diff --git a/providers/src/databricks_account/v00.00.00000/provider.yaml b/providers/src/databricks_account/v00.00.00000/provider.yaml index e703b8af..c3cc9a4b 100644 --- a/providers/src/databricks_account/v00.00.00000/provider.yaml +++ b/providers/src/databricks_account/v00.00.00000/provider.yaml @@ -24,6 +24,17 @@ providerServices: description: >- OpenAPI specification for the Databricks catalog service (account-level APIs), generated from the Databricks Python SDK. + disasterrecovery: + id: disasterrecovery:v00.00.00000 + name: disasterrecovery + preferred: true + service: + $ref: databricks_account/v00.00.00000/services/disasterrecovery.yaml + title: Databricks Disasterrecovery API (account) + version: v00.00.00000 + description: >- + OpenAPI specification for the Databricks disasterrecovery service + (account-level APIs), generated from the Databricks Python SDK. iam: id: iam:v00.00.00000 name: iam diff --git a/providers/src/databricks_account/v00.00.00000/services/billing.yaml b/providers/src/databricks_account/v00.00.00000/services/billing.yaml index cbc3a783..81fb7b58 100644 --- a/providers/src/databricks_account/v00.00.00000/services/billing.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/billing.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks billing service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.billing servers: - url: https://accounts.cloud.databricks.com @@ -21,7 +21,10 @@ paths: Returns billable usage logs in CSV format for the specified account and date range. For the data schema, see: - - AWS: [CSV file schema]. - GCP: [CSV file schema]. + - AWS: `CSV file schema + `__. + - GCP: `CSV file schema + `__. Note that this method might take multiple minutes to complete. @@ -29,14 +32,12 @@ paths: the response and the internet speed of the caller, this API may hit a timeout after a few minutes. If you experience this, try to mitigate by calling the API with narrower date ranges. - [CSV file schema]: https://docs.gcp.databricks.com/administration-guide/account-settings/usage-analysis.html#csv-file-schema - :param start_month: str - Format specification for month in the format `YYYY-MM`. This is used to specify billable usage - `start_month` and `end_month` properties. **Note**: Billable usage logs are unavailable before March - 2019 (`2019-03`). + Format specification for month in the format ``YYYY-MM``. This is used to specify billable usage + ``start_month`` and ``end_month`` properties. **Note**: Billable usage logs are unavailable before + March 2019 (``2019-03``). :param end_month: str - Format: `YYYY-MM`. Last month to return billable usage logs for. This field is required. + Format: ``YYYY-MM``. Last month to return billable usage logs for. This field is required. :param personal_data: bool (optional) Specify whether to include personally identifiable information in the billable usage logs, for example the email addresses of cluster creators. Handle this information with care. Defaults to @@ -54,14 +55,14 @@ paths: required: true schema: type: string - description: 'Format specification for month in the format `YYYY-MM`. This is used to specify billable usage `start_month` - and `end_month` properties. **Note**: Billable usage logs are unavailable before March 2019 (`2019-03`).' + description: 'Format specification for month in the format ``YYYY-MM``. This is used to specify billable usage ``start_month`` + and ``end_month`` properties. **Note**: Billable usage logs are unavailable before March 2019 (``2019-03``).' - name: end_month in: query required: true schema: type: string - description: 'Format: `YYYY-MM`. Last month to return billable usage logs for. This field is required.' + description: 'Format: ``YYYY-MM``. Last month to return billable usage logs for. This field is required.' - name: personal_data in: query required: false @@ -102,12 +103,12 @@ paths: Creates a new policy. :param policy: :class:`BudgetPolicy` (optional) - The policy to create. `policy_id` needs to be empty as it will be generated `policy_name` must be - provided, custom_tags may need to be provided depending on the cloud provider. All other fields are - optional. + The policy to create. ``policy_id`` needs to be empty as it will be generated ``policy_name`` must + be provided, custom_tags may need to be provided depending on the cloud provider. All other fields + are optional. :param request_id: str (optional) A unique identifier for this request. Restricted to 36 ASCII characters. A random UUID is - recommended. This request is only idempotent if a `request_id` is provided. + recommended. This request is only idempotent if a ``request_id`` is provided. :returns: :class:`BudgetPolicy` parameters: @@ -140,14 +141,14 @@ paths: policy_name: type: string description: The name of the policy. - Must be unique among active policies. - Can contain only characters - from the ISO 8859-1 (latin1) set. - Can't start with reserved keywords such as `databricks:default-policy`. - description: The policy to create. `policy_id` needs to be empty as it will be generated `policy_name` must - be provided, custom_tags may need to be provided depending on the cloud provider. All other fields are - optional. + from the ISO 8859-1 (latin1) set. - Can't start with reserved keywords such as ``databricks:default-policy``. + description: The policy to create. ``policy_id`` needs to be empty as it will be generated ``policy_name`` + must be provided, custom_tags may need to be provided depending on the cloud provider. All other fields + are optional. request_id: type: string description: A unique identifier for this request. Restricted to 36 ASCII characters. A random UUID is recommended. - This request is only idempotent if a `request_id` is provided. + This request is only idempotent if a ``request_id`` is provided. responses: '200': description: Success @@ -182,10 +183,10 @@ paths: The maximum number of budget policies to return. If unspecified, at most 100 budget policies will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. :param page_token: str (optional) - A page token, received from a previous `ListServerlessPolicies` call. Provide this to retrieve the + A page token, received from a previous ``ListServerlessPolicies`` call. Provide this to retrieve the subsequent page. If unspecified, the first page will be returned. - When paginating, all other parameters provided to `ListServerlessPoliciesRequest` must match the + When paginating, all other parameters provided to ``ListServerlessPoliciesRequest`` must match the call that provided the page token. :param sort_spec: :class:`SortSpec` (optional) The sort specification. @@ -215,9 +216,9 @@ paths: required: false schema: type: string - description: A page token, received from a previous `ListServerlessPolicies` call. Provide this to retrieve the subsequent - page. If unspecified, the first page will be returned. When paginating, all other parameters provided to `ListServerlessPoliciesRequest` - must match the call that provided the page token. + description: A page token, received from a previous ``ListServerlessPolicies`` call. Provide this to retrieve the + subsequent page. If unspecified, the first page will be returned. When paginating, all other parameters provided + to ``ListServerlessPoliciesRequest`` must match the call that provided the page token. - name: sort_spec in: query required: false @@ -338,10 +339,12 @@ paths: :param policy_id: str The Id of the policy. This field is generated by Databricks and globally unique. :param policy: :class:`BudgetPolicy` - The policy to update. `creator_user_id` cannot be specified in the request. All other fields must be - specified even if not changed. The `policy_id` is used to identify the policy to update. + The policy to update. ``creator_user_id`` cannot be specified in the request. All other fields must + be specified even if not changed. The ``policy_id`` is used to identify the policy to update. :param limit_config: :class:`LimitConfig` (optional) DEPRECATED. This is redundant field as LimitConfig is part of the BudgetPolicy + :param update_mask: str (optional) + Field mask specifying which fields to update. When not provided, all fields are updated. :returns: :class:`BudgetPolicy` parameters: @@ -362,6 +365,12 @@ paths: schema: $ref: '#/components/schemas/LimitConfig' description: DEPRECATED. This is redundant field as LimitConfig is part of the BudgetPolicy + - name: update_mask + in: query + required: false + schema: + type: string + description: Field mask specifying which fields to update. When not provided, all fields are updated. requestBody: content: application/json: @@ -386,9 +395,9 @@ paths: policy_name: type: string description: The name of the policy. - Must be unique among active policies. - Can contain only characters - from the ISO 8859-1 (latin1) set. - Can't start with reserved keywords such as `databricks:default-policy`. - description: The policy to update. `creator_user_id` cannot be specified in the request. All other fields - must be specified even if not changed. The `policy_id` is used to identify the policy to update. + from the ISO 8859-1 (latin1) set. - Can't start with reserved keywords such as ``databricks:default-policy``. + description: The policy to update. ``creator_user_id`` cannot be specified in the request. All other fields + must be specified even if not changed. The ``policy_id`` is used to identify the policy to update. required: - policy responses: @@ -456,55 +465,19 @@ paths: type: string x-enum: - EMAIL_NOTIFICATION - description: |- - Create a collection of name/value pairs. - - Example enumeration: - - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 - - Access them by: - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many members they have: - - >>> len(Color) - 3 - - >>> list(Color) - [, , ] - - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. + description: Type of action that a budget alert executes when its threshold is crossed. target: type: string description: Target for the action. For example, an email address. quantity_threshold: type: string description: The threshold for the budget alert to determine if it is in a triggered state. The - number is evaluated based on `quantity_type`. + number is evaluated based on ``quantity_type``. quantity_type: type: string x-enum: - LIST_PRICE_DOLLARS_USD - description: The way to calculate cost for this budget alert. This is what `quantity_threshold` + description: The way to calculate cost for this budget alert. This is what ``quantity_threshold`` is measured in. time_period: type: string @@ -848,12 +821,12 @@ paths: quantity_threshold: type: string description: The threshold for the budget alert to determine if it is in a triggered state. The - number is evaluated based on `quantity_type`. + number is evaluated based on ``quantity_type``. quantity_type: type: string x-enum: - LIST_PRICE_DOLLARS_USD - description: The way to calculate cost for this budget alert. This is what `quantity_threshold` + description: The way to calculate cost for this budget alert. This is what ``quantity_threshold`` is measured in. time_period: type: string @@ -1018,8 +991,11 @@ paths: object](:method:Credentials/Create) (which encapsulates a cross-account service IAM role) and a [storage configuration object](:method:Storage/Create) (which encapsulates an S3 bucket). - For full details, including the required IAM role policies and bucket policies, see [Deliver and - access billable usage logs] or [Configure audit logging]. + For full details, including the required IAM role policies and bucket policies, see `Deliver and + access billable usage logs + `__ or + `Configure audit logging + `__. **Note**: There is a limit on the number of log delivery configurations available per account (each limit applies separately to each log type including billable usage and audit logs). You can create a @@ -1031,9 +1007,6 @@ paths: You cannot delete a log delivery configuration, but you can disable it (see [Enable or disable log delivery configuration](:method:LogDelivery/PatchStatus)). - [Configure audit logging]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [Deliver and access billable usage logs]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html - :param log_delivery_configuration: :class:`CreateLogDeliveryConfigurationParams` :returns: :class:`WrappedLogDeliveryConfiguration` @@ -1057,33 +1030,30 @@ paths: x-enum: - AUDIT_LOGS - BILLABLE_USAGE - description: 'Log delivery type. Supported values are: * `BILLABLE_USAGE` — Configure [billable usage - log delivery]. For the CSV schema, see the [View billable usage]. * `AUDIT_LOGS` — Configure [audit - log delivery]. For the JSON schema, see [Configure audit logging] [Configure audit logging]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [View billable usage]: https://docs.databricks.com/administration-guide/account-settings/usage.html - [audit log delivery]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [billable usage log delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: 'Log delivery type. Supported values are: - ``BILLABLE_USAGE`` — Configure `billable usage + log delivery `__. + For the CSV schema, see the `View billable usage `__. + - ``AUDIT_LOGS`` — Configure `audit log delivery `__. + For the JSON schema, see `Configure audit logging `__' output_format: type: string x-enum: - CSV - JSON - description: 'The file type of log delivery. * If `log_type` is `BILLABLE_USAGE`, this value must be - `CSV`. Only the CSV (comma-separated values) format is supported. For the schema, see the [View billable - usage] * If `log_type` is `AUDIT_LOGS`, this value must be `JSON`. Only the JSON (JavaScript Object - Notation) format is supported. For the schema, see the [Configuring audit logs]. [Configuring audit - logs]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html [View billable - usage]: https://docs.databricks.com/administration-guide/account-settings/usage.html' + description: The file type of log delivery. - If ``log_type`` is ``BILLABLE_USAGE``, this value must + be ``CSV``. Only the CSV (comma-separated values) format is supported. For the schema, see the `View + billable usage `__ - + If ``log_type`` is ``AUDIT_LOGS``, this value must be ``JSON``. Only the JSON (JavaScript Object Notation) + format is supported. For the schema, see the `Configuring audit logs `__. credentials_id: type: string - description: 'The ID for a method:credentials/create that represents the AWS IAM role with policy and - trust relationship as described in the main billable usage documentation page. See [Configure billable - usage delivery]. [Configure billable usage delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: The ID for a method:credentials/create that represents the AWS IAM role with policy and + trust relationship as described in the main billable usage documentation page. See `Configure billable + usage delivery `__. storage_configuration_id: type: string - description: 'The ID for a method:storage/create that represents the S3 bucket with bucket policy as - described in the main billable usage documentation page. See [Configure billable usage delivery]. - [Configure billable usage delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: The ID for a method:storage/create that represents the S3 bucket with bucket policy as + described in the main billable usage documentation page. See `Configure billable usage delivery `__. config_name: type: string description: The optional human-readable name of the log delivery configuration. Defaults to empty. @@ -1102,8 +1072,8 @@ paths: x-enum: - DISABLED - ENABLED - description: Status of log delivery configuration. Set to `ENABLED` (enabled) or `DISABLED` (disabled). - Defaults to `ENABLED`. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) + description: Status of log delivery configuration. Set to ``ENABLED`` (enabled) or ``DISABLED`` (disabled). + Defaults to ``ENABLED``. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. Deletion of a configuration is not supported, so disable a log delivery configuration that is no longer needed. workspace_ids_filter: @@ -1113,18 +1083,18 @@ paths: description: Optional filter that specifies workspace IDs to deliver logs for. By default the workspace filter is empty and log delivery applies at the account level, delivering workspace-level logs for all workspaces in your account, plus account level logs. You can optionally set this field to an array - of workspace IDs (each one is an `int64`) to which log delivery should apply, in which case only workspace-level - logs relating to the specified workspaces are delivered. If you plan to use different log delivery - configurations for different workspaces, set this field explicitly. Be aware that delivery configurations - mentioning specific workspaces won't apply to new workspaces created in the future, and delivery won't - include account level logs. For some types of Databricks deployments there is only one workspace per - account ID, so this field is unnecessary. + of workspace IDs (each one is an ``int64``) to which log delivery should apply, in which case only + workspace-level logs relating to the specified workspaces are delivered. If you plan to use different + log delivery configurations for different workspaces, set this field explicitly. Be aware that delivery + configurations mentioning specific workspaces won't apply to new workspaces created in the future, + and delivery won't include account level logs. For some types of Databricks deployments there is only + one workspace per account ID, so this field is unnecessary. required: - log_type - output_format - credentials_id - storage_configuration_id - description: '* Log Delivery Configuration' + description: Log Delivery Configuration required: - log_delivery_configuration responses: @@ -1276,8 +1246,8 @@ paths: :param log_delivery_configuration_id: str The log delivery configuration id of customer :param status: :class:`LogDeliveryConfigStatus` - Status of log delivery configuration. Set to `ENABLED` (enabled) or `DISABLED` (disabled). Defaults - to `ENABLED`. You can [enable or disable the + Status of log delivery configuration. Set to ``ENABLED`` (enabled) or ``DISABLED`` (disabled). + Defaults to ``ENABLED``. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. Deletion of a configuration is not supported, so disable a log delivery configuration that is no longer needed. parameters: @@ -1303,8 +1273,8 @@ paths: x-enum: - DISABLED - ENABLED - description: Status of log delivery configuration. Set to `ENABLED` (enabled) or `DISABLED` (disabled). - Defaults to `ENABLED`. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) + description: Status of log delivery configuration. Set to ``ENABLED`` (enabled) or ``DISABLED`` (disabled). + Defaults to ``ENABLED``. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. Deletion of a configuration is not supported, so disable a log delivery configuration that is no longer needed. required: @@ -1444,6 +1414,305 @@ paths: message: type: string x-stackql-sdk-source: UsageDashboardsAPI + /api/2.1/accounts/{account_id}/usage-policies: + post: + operationId: usage_policy_create + summary: Creates a new usage policy. + tags: + - billing + - usage_policy + description: |- + Creates a new usage policy. + + :param policy: :class:`UsagePolicy` (optional) + The policy to create. ``policy_id`` needs to be empty as it will be generated + :param request_id: str (optional) + A unique identifier for this request. Restricted to 36 ASCII characters. + + :returns: :class:`UsagePolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + policy: + type: object + properties: + binding_workspace_ids: + type: array + items: + type: integer + description: List of workspaces that this usage policy will be exclusively bound to. + custom_tags: + type: string + description: A list of tags defined by the customer. At most 20 entries are allowed per policy. + policy_id: + type: string + description: The Id of the policy. This field is generated by Databricks and globally unique. + policy_name: + type: string + description: The name of the policy. + description: The policy to create. ``policy_id`` needs to be empty as it will be generated + request_id: + type: string + description: A unique identifier for this request. Restricted to 36 ASCII characters. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UsagePolicy' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: UsagePolicyAPI + get: + operationId: usage_policy_list + summary: Lists all usage policies. Policies are returned in the alphabetically ascending order of their names. + tags: + - billing + - usage_policy + description: |- + Lists all usage policies. Policies are returned in the alphabetically ascending order of their names. + + :param filter_by: :class:`Filter` (optional) + A filter to apply to the list of policies. + :param page_size: int (optional) + The maximum number of usage policies to return. + :param page_token: str (optional) + A page token, received from a previous ``ListUsagePolicies`` call. + :param sort_spec: :class:`SortSpec` (optional) + The sort specification. + + :returns: Iterator over :class:`UsagePolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: filter_by + in: query + required: false + schema: + $ref: '#/components/schemas/Filter' + description: A filter to apply to the list of policies. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of usage policies to return. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ``ListUsagePolicies`` call. + - name: sort_spec + in: query + required: false + schema: + $ref: '#/components/schemas/SortSpec' + description: The sort specification. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListUsagePoliciesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: UsagePolicyAPI + /api/2.1/accounts/{account_id}/usage-policies/{policy_id}: + delete: + operationId: usage_policy_delete + summary: Deletes a usage policy + tags: + - billing + - usage_policy + description: |- + Deletes a usage policy + + :param policy_id: str + The Id of the policy. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: policy_id + in: path + required: true + schema: + type: string + description: The Id of the policy. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: UsagePolicyAPI + get: + operationId: usage_policy_get + summary: Retrieves a usage policy by it's ID. + tags: + - billing + - usage_policy + description: |- + Retrieves a usage policy by it's ID. + + :param policy_id: str + The Id of the policy. + + :returns: :class:`UsagePolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: policy_id + in: path + required: true + schema: + type: string + description: The Id of the policy. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UsagePolicy' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: UsagePolicyAPI + patch: + operationId: usage_policy_update + summary: Updates a usage policy + tags: + - billing + - usage_policy + description: |- + Updates a usage policy + + :param policy_id: str + The Id of the policy. This field is generated by Databricks and globally unique. + :param policy: :class:`UsagePolicy` + The policy to update. ``creator_user_id`` cannot be specified in the request. + :param limit_config: :class:`LimitConfig` (optional) + DEPRECATED. This is redundant field as LimitConfig is part of the UsagePolicy + + :returns: :class:`UsagePolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: policy_id + in: path + required: true + schema: + type: string + description: The Id of the policy. This field is generated by Databricks and globally unique. + - name: limit_config + in: query + required: false + schema: + $ref: '#/components/schemas/LimitConfig' + description: DEPRECATED. This is redundant field as LimitConfig is part of the UsagePolicy + requestBody: + content: + application/json: + schema: + type: object + properties: + policy: + type: object + properties: + binding_workspace_ids: + type: array + items: + type: integer + description: List of workspaces that this usage policy will be exclusively bound to. + custom_tags: + type: string + description: A list of tags defined by the customer. At most 20 entries are allowed per policy. + policy_id: + type: string + description: The Id of the policy. This field is generated by Databricks and globally unique. + policy_name: + type: string + description: The name of the policy. + description: The policy to update. ``creator_user_id`` cannot be specified in the request. + required: + - policy + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UsagePolicy' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: UsagePolicyAPI components: schemas: ActionConfiguration: @@ -1470,10 +1739,10 @@ components: quantity_threshold: type: string description: The threshold for the budget alert to determine if it is in a triggered state. The number is evaluated - based on `quantity_type`. + based on ``quantity_type``. quantity_type: $ref: '#/components/schemas/AlertConfigurationQuantityType' - description: The way to calculate cost for this budget alert. This is what `quantity_threshold` is measured in. + description: The way to calculate cost for this budget alert. This is what ``quantity_threshold`` is measured in. time_period: $ref: '#/components/schemas/AlertConfigurationTimePeriod' description: The time window of usage data for the budget. @@ -1560,7 +1829,7 @@ components: policy_name: type: string description: The name of the policy. - Must be unique among active policies. - Can contain only characters from - the ISO 8859-1 (latin1) set. - Can't start with reserved keywords such as `databricks:default-policy`. + the ISO 8859-1 (latin1) set. - Can't start with reserved keywords such as ``databricks:default-policy``. description: Contains the BudgetPolicy details. CreateBillingUsageDashboardResponse: type: object @@ -1603,10 +1872,10 @@ components: quantity_threshold: type: string description: The threshold for the budget alert to determine if it is in a triggered state. The number is evaluated - based on `quantity_type`. + based on ``quantity_type``. quantity_type: $ref: '#/components/schemas/AlertConfigurationQuantityType' - description: The way to calculate cost for this budget alert. This is what `quantity_threshold` is measured in. + description: The way to calculate cost for this budget alert. This is what ``quantity_threshold`` is measured in. time_period: $ref: '#/components/schemas/AlertConfigurationTimePeriod' description: The time window of usage data for the budget. @@ -1623,29 +1892,25 @@ components: properties: log_type: $ref: '#/components/schemas/LogType' - description: 'Log delivery type. Supported values are: * `BILLABLE_USAGE` — Configure [billable usage log delivery]. - For the CSV schema, see the [View billable usage]. * `AUDIT_LOGS` — Configure [audit log delivery]. For the JSON - schema, see [Configure audit logging] [Configure audit logging]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [View billable usage]: https://docs.databricks.com/administration-guide/account-settings/usage.html [audit log - delivery]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html [billable usage log - delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: 'Log delivery type. Supported values are: - ``BILLABLE_USAGE`` — Configure `billable usage log delivery + `__. For the CSV + schema, see the `View billable usage `__. + - ``AUDIT_LOGS`` — Configure `audit log delivery `__. + For the JSON schema, see `Configure audit logging `__' output_format: $ref: '#/components/schemas/OutputFormat' - description: 'The file type of log delivery. * If `log_type` is `BILLABLE_USAGE`, this value must be `CSV`. Only - the CSV (comma-separated values) format is supported. For the schema, see the [View billable usage] * If `log_type` - is `AUDIT_LOGS`, this value must be `JSON`. Only the JSON (JavaScript Object Notation) format is supported. For - the schema, see the [Configuring audit logs]. [Configuring audit logs]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [View billable usage]: https://docs.databricks.com/administration-guide/account-settings/usage.html' + description: The file type of log delivery. - If ``log_type`` is ``BILLABLE_USAGE``, this value must be ``CSV``. + Only the CSV (comma-separated values) format is supported. For the schema, see the `View billable usage `__ + - If ``log_type`` is ``AUDIT_LOGS``, this value must be ``JSON``. Only the JSON (JavaScript Object Notation) format + is supported. For the schema, see the `Configuring audit logs `__. credentials_id: type: string - description: 'The ID for a method:credentials/create that represents the AWS IAM role with policy and trust relationship - as described in the main billable usage documentation page. See [Configure billable usage delivery]. [Configure - billable usage delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: The ID for a method:credentials/create that represents the AWS IAM role with policy and trust relationship + as described in the main billable usage documentation page. See `Configure billable usage delivery `__. storage_configuration_id: type: string - description: 'The ID for a method:storage/create that represents the S3 bucket with bucket policy as described in - the main billable usage documentation page. See [Configure billable usage delivery]. [Configure billable usage - delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: The ID for a method:storage/create that represents the S3 bucket with bucket policy as described in + the main billable usage documentation page. See `Configure billable usage delivery `__. config_name: type: string description: The optional human-readable name of the log delivery configuration. Defaults to empty. @@ -1661,8 +1926,8 @@ components: for usage before March 2019 (2019-03). status: $ref: '#/components/schemas/LogDeliveryConfigStatus' - description: Status of log delivery configuration. Set to `ENABLED` (enabled) or `DISABLED` (disabled). Defaults - to `ENABLED`. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. + description: Status of log delivery configuration. Set to ``ENABLED`` (enabled) or ``DISABLED`` (disabled). Defaults + to ``ENABLED``. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. Deletion of a configuration is not supported, so disable a log delivery configuration that is no longer needed. workspace_ids_filter: type: array @@ -1671,7 +1936,7 @@ components: description: Optional filter that specifies workspace IDs to deliver logs for. By default the workspace filter is empty and log delivery applies at the account level, delivering workspace-level logs for all workspaces in your account, plus account level logs. You can optionally set this field to an array of workspace IDs (each one is - an `int64`) to which log delivery should apply, in which case only workspace-level logs relating to the specified + an ``int64``) to which log delivery should apply, in which case only workspace-level logs relating to the specified workspaces are delivered. If you plan to use different log delivery configurations for different workspaces, set this field explicitly. Be aware that delivery configurations mentioning specific workspaces won't apply to new workspaces created in the future, and delivery won't include account level logs. For some types of Databricks @@ -1681,7 +1946,7 @@ components: - output_format - credentials_id - storage_configuration_id - description: '* Log Delivery Configuration' + description: Log Delivery Configuration DeleteBudgetConfigurationResponse: type: object properties: {} @@ -1699,7 +1964,8 @@ components: description: The policy creator user id to be filtered on. If unspecified, all policies will be returned. creator_user_name: type: string - description: The policy creator user name to be filtered on. If unspecified, all policies will be returned. + description: 'Deprecated: Do not use this field in new integrations. Creator filtering will be removed in a future + version. The policy creator user name to be filtered on. If unspecified, all policies will be returned.' policy_name: type: string description: The partial name of policies to be filtered on. If unspecified, all policies will be returned. @@ -1739,14 +2005,14 @@ components: $ref: '#/components/schemas/BudgetConfiguration' next_page_token: type: string - description: Token which can be sent as `page_token` to retrieve the next page of results. If this field is omitted, + description: Token which can be sent as ``page_token`` to retrieve the next page of results. If this field is omitted, there are no subsequent budgets. ListBudgetPoliciesResponse: type: object properties: next_page_token: type: string - description: A token that can be sent as `page_token` to retrieve the next page. If this field is omitted, there + description: A token that can be sent as ``page_token`` to retrieve the next page. If this field is omitted, there are no subsequent pages. policies: type: array @@ -1754,37 +2020,47 @@ components: $ref: '#/components/schemas/BudgetPolicy' previous_page_token: type: string - description: A token that can be sent as `page_token` to retrieve the previous page. In this field is omitted, there - are no previous pages. + description: A token that can be sent as ``page_token`` to retrieve the previous page. In this field is omitted, + there are no previous pages. description: A list of policies. + ListUsagePoliciesResponse: + type: object + properties: + next_page_token: + type: string + description: A token that can be sent as ``page_token`` to retrieve the next page. + policies: + type: array + items: + $ref: '#/components/schemas/UsagePolicy' + previous_page_token: + type: string + description: A token that can be sent as ``page_token`` to retrieve the previous page. + description: A list of usage policies. LogDeliveryConfiguration: type: object properties: log_type: $ref: '#/components/schemas/LogType' - description: 'Log delivery type. Supported values are: * `BILLABLE_USAGE` — Configure [billable usage log delivery]. - For the CSV schema, see the [View billable usage]. * `AUDIT_LOGS` — Configure [audit log delivery]. For the JSON - schema, see [Configure audit logging] [Configure audit logging]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [View billable usage]: https://docs.databricks.com/administration-guide/account-settings/usage.html [audit log - delivery]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html [billable usage log - delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: 'Log delivery type. Supported values are: - ``BILLABLE_USAGE`` — Configure `billable usage log delivery + `__. For the CSV + schema, see the `View billable usage `__. + - ``AUDIT_LOGS`` — Configure `audit log delivery `__. + For the JSON schema, see `Configure audit logging `__' output_format: $ref: '#/components/schemas/OutputFormat' - description: 'The file type of log delivery. * If `log_type` is `BILLABLE_USAGE`, this value must be `CSV`. Only - the CSV (comma-separated values) format is supported. For the schema, see the [View billable usage] * If `log_type` - is `AUDIT_LOGS`, this value must be `JSON`. Only the JSON (JavaScript Object Notation) format is supported. For - the schema, see the [Configuring audit logs]. [Configuring audit logs]: https://docs.databricks.com/administration-guide/account-settings/audit-logs.html - [View billable usage]: https://docs.databricks.com/administration-guide/account-settings/usage.html' + description: The file type of log delivery. - If ``log_type`` is ``BILLABLE_USAGE``, this value must be ``CSV``. + Only the CSV (comma-separated values) format is supported. For the schema, see the `View billable usage `__ + - If ``log_type`` is ``AUDIT_LOGS``, this value must be ``JSON``. Only the JSON (JavaScript Object Notation) format + is supported. For the schema, see the `Configuring audit logs `__. credentials_id: type: string - description: 'The ID for a method:credentials/create that represents the AWS IAM role with policy and trust relationship - as described in the main billable usage documentation page. See [Configure billable usage delivery]. [Configure - billable usage delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: The ID for a method:credentials/create that represents the AWS IAM role with policy and trust relationship + as described in the main billable usage documentation page. See `Configure billable usage delivery `__. storage_configuration_id: type: string - description: 'The ID for a method:storage/create that represents the S3 bucket with bucket policy as described in - the main billable usage documentation page. See [Configure billable usage delivery]. [Configure billable usage - delivery]: https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html' + description: The ID for a method:storage/create that represents the S3 bucket with bucket policy as described in + the main billable usage documentation page. See `Configure billable usage delivery `__. account_id: type: string description: Databricks account ID. @@ -1812,8 +2088,8 @@ components: description: The LogDeliveryStatus of this log delivery configuration status: $ref: '#/components/schemas/LogDeliveryConfigStatus' - description: Status of log delivery configuration. Set to `ENABLED` (enabled) or `DISABLED` (disabled). Defaults - to `ENABLED`. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. + description: Status of log delivery configuration. Set to ``ENABLED`` (enabled) or ``DISABLED`` (disabled). Defaults + to ``ENABLED``. You can [enable or disable the configuration](#operation/patch-log-delivery-config-status) later. Deletion of a configuration is not supported, so disable a log delivery configuration that is no longer needed. update_time: type: integer @@ -1825,7 +2101,7 @@ components: description: Optional filter that specifies workspace IDs to deliver logs for. By default the workspace filter is empty and log delivery applies at the account level, delivering workspace-level logs for all workspaces in your account, plus account level logs. You can optionally set this field to an array of workspace IDs (each one is - an `int64`) to which log delivery should apply, in which case only workspace-level logs relating to the specified + an ``int64``) to which log delivery should apply, in which case only workspace-level logs relating to the specified workspaces are delivered. If you plan to use different log delivery configurations for different workspaces, set this field explicitly. Be aware that delivery configurations mentioning specific workspaces won't apply to new workspaces created in the future, and delivery won't include account level logs. For some types of Databricks @@ -1835,7 +2111,7 @@ components: - output_format - credentials_id - storage_configuration_id - description: '* Log Delivery Configuration' + description: Log Delivery Configuration LogDeliveryStatus: type: object properties: @@ -1891,6 +2167,24 @@ components: properties: budget: $ref: '#/components/schemas/BudgetConfiguration' + UsagePolicy: + type: object + properties: + binding_workspace_ids: + type: array + items: + type: integer + description: List of workspaces that this usage policy will be exclusively bound to. + custom_tags: + type: string + description: A list of tags defined by the customer. At most 20 entries are allowed per policy. + policy_id: + type: string + description: The Id of the policy. This field is generated by Databricks and globally unique. + policy_name: + type: string + description: The name of the policy. + description: Contains the UsagePolicy details (same structure as BudgetPolicy) WrappedLogDeliveryConfiguration: type: object properties: @@ -1905,49 +2199,13 @@ components: $ref: '#/components/schemas/LogDeliveryConfiguration' next_page_token: type: string - description: Token which can be sent as `page_token` to retrieve the next page of results. If this field is omitted, + description: Token which can be sent as ``page_token`` to retrieve the next page of results. If this field is omitted, there are no subsequent budgets. ActionConfigurationType: type: string x-enum: - EMAIL_NOTIFICATION - description: |- - Create a collection of name/value pairs. - - Example enumeration: - - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 - - Access them by: - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many members they have: - - >>> len(Color) - 3 - - >>> list(Color) - [, , ] - - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. + description: Type of action that a budget alert executes when its threshold is crossed. AlertConfigurationQuantityType: type: string x-enum: @@ -2121,12 +2379,12 @@ components: - SYSTEM_FAILURE - USER_FAILURE description: |- - * The status string for log delivery. Possible values are: `CREATED`: There were no log delivery - attempts since the config was created. `SUCCEEDED`: The latest attempt of log delivery has - succeeded completely. `USER_FAILURE`: The latest attempt of log delivery failed because of - misconfiguration of customer provided permissions on role or storage. `SYSTEM_FAILURE`: The + The status string for log delivery. Possible values are: ``CREATED``: There were no log delivery + attempts since the config was created. ``SUCCEEDED``: The latest attempt of log delivery has + succeeded completely. ``USER_FAILURE``: The latest attempt of log delivery failed because of + misconfiguration of customer provided permissions on role or storage. ``SYSTEM_FAILURE``: The latest attempt of log delivery failed because of an Databricks internal error. Contact support - if it doesn't go away soon. `NOT_FOUND`: The log delivery status as the configuration has been + if it doesn't go away soon. ``NOT_FOUND``: The log delivery status as the configuration has been disabled since the release of this feature or there are no workspaces in the account. LogDeliveryConfigStatus: type: string @@ -2134,22 +2392,22 @@ components: - DISABLED - ENABLED description: |- - * Log Delivery Status + Log Delivery Status - `ENABLED`: All dependencies have executed and succeeded `DISABLED`: At least one dependency has - succeeded + ``ENABLED``: All dependencies have executed and succeeded ``DISABLED``: At least one dependency + has succeeded LogType: type: string x-enum: - AUDIT_LOGS - BILLABLE_USAGE - description: '* Log Delivery Type' + description: Log Delivery Type OutputFormat: type: string x-enum: - CSV - JSON - description: '* Log Delivery Output Format' + description: Log Delivery Output Format SortSpecField: type: string x-enum: @@ -2480,6 +2738,59 @@ components: update: [] delete: [] replace: [] + usage_policy: + id: databricks_account.billing.usage_policy + name: usage_policy + title: Usage Policy + methods: + usage_policy_create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1accounts~1{account_id}~1usage-policies/post' + response: + mediaType: application/json + openAPIDocKey: '200' + usage_policy_list: + operation: + $ref: '#/paths/~1api~12.1~1accounts~1{account_id}~1usage-policies/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.policies + usage_policy_delete: + operation: + $ref: '#/paths/~1api~12.1~1accounts~1{account_id}~1usage-policies~1{policy_id}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + usage_policy_get: + operation: + $ref: '#/paths/~1api~12.1~1accounts~1{account_id}~1usage-policies~1{policy_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + usage_policy_update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1accounts~1{account_id}~1usage-policies~1{policy_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/usage_policy/methods/usage_policy_get' + - $ref: '#/components/x-stackQL-resources/usage_policy/methods/usage_policy_list' + insert: + - $ref: '#/components/x-stackQL-resources/usage_policy/methods/usage_policy_create' + update: + - $ref: '#/components/x-stackQL-resources/usage_policy/methods/usage_policy_update' + delete: + - $ref: '#/components/x-stackQL-resources/usage_policy/methods/usage_policy_delete' + replace: [] vw_budget_policies: name: vw_budget_policies id: databricks_account.billing.vw_budget_policies diff --git a/providers/src/databricks_account/v00.00.00000/services/catalog.yaml b/providers/src/databricks_account/v00.00.00000/services/catalog.yaml index e4b2dd23..e16bd1c1 100644 --- a/providers/src/databricks_account/v00.00.00000/services/catalog.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/catalog.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks catalog service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.catalog servers: - url: https://accounts.cloud.databricks.com @@ -339,7 +339,7 @@ paths: description: Whether to allow non-DBR clients to directly access entities under the metastore. region: type: string - description: Cloud region which the metastore serves (e.g., `us-west-2`, `westus`). + description: Cloud region which the metastore serves (e.g., ``us-west-2``, ``westus``). storage_root: type: string description: The storage root URL for metastore @@ -552,7 +552,7 @@ paths: description: The owner of the metastore. privilege_model_version: type: string - description: Privilege model version of the metastore, of the form `major.minor` (e.g., `1.0`). + description: Privilege model version of the metastore, of the form ``major.minor`` (e.g., ``1.0``). storage_root_credential_id: type: string description: UUID of storage credential to access the metastore storage_root. @@ -579,16 +579,18 @@ paths: /api/2.0/accounts/{account_id}/metastores/{metastore_id}/storage-credentials: post: operationId: account_storage_credentials_create - summary: 'Creates a new storage credential. The request object is specific to the cloud: - **AwsIamRole** for' + summary: 'Creates a new storage credential. The request object is specific to the cloud:' tags: - catalog - account_storage_credentials description: |- - Creates a new storage credential. The request object is specific to the cloud: - **AwsIamRole** for - AWS credentials - **AzureServicePrincipal** for Azure credentials - **GcpServiceAccountKey** for GCP - credentials + Creates a new storage credential. The request object is specific to the cloud: - The caller must be a metastore admin and have the `CREATE_STORAGE_CREDENTIAL` privilege on the + - **AwsIamRole** for AWS credentials + - **AzureServicePrincipal** for Azure credentials + - **GcpServiceAccountKey** for GCP credentials + + The caller must be a metastore admin and have the ``CREATE_STORAGE_CREDENTIAL`` privilege on the metastore. :param metastore_id: str @@ -636,10 +638,10 @@ paths: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -934,13 +936,13 @@ paths: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -992,6 +994,7 @@ paths: x-enum: - ISOLATION_MODE_ISOLATED - ISOLATION_MODE_OPEN + - ISOLATION_MODE_OPEN_IN_ACCOUNT description: Whether the current securable is accessible from all workspaces or a specific set of workspaces. owner: type: string @@ -1087,13 +1090,13 @@ paths: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -1169,12 +1172,12 @@ paths: x-stackql-sdk-source: CredentialsAPI get: operationId: credentials_list_credentials - summary: Gets an array of credentials (as __CredentialInfo__ objects). + summary: Gets an array of credentials (as **CredentialInfo** objects). tags: - catalog - credentials description: |- - Gets an array of credentials (as __CredentialInfo__ objects). + Gets an array of credentials (as **CredentialInfo** objects). The array is limited to only the credentials that the caller has permission to access. If the caller is a metastore admin, retrieval of credentials is unrestricted. There is no guarantee of a specific @@ -1188,10 +1191,13 @@ paths: Whether to include credentials not bound to the workspace. Effective only if the user has permission to update the credential–workspace binding. :param max_results: int (optional) - Maximum number of credentials to return. - If not set, the default max page size is used. - When set - to a value greater than 0, the page length is the minimum of this value and a server-configured - value. - When set to 0, the page length is set to a server-configured value (recommended). - When - set to a value less than 0, an invalid parameter error is returned. + Maximum number of credentials to return. + + - If not set, the default max page size is used. + - When set to a value greater than 0, the page length is the minimum of this value and a + server-configured value. + - When set to 0, the page length is set to a server-configured value (recommended). + - When set to a value less than 0, an invalid parameter error is returned. :param page_token: str (optional) Opaque token to retrieve the next page of results. :param purpose: :class:`CredentialPurpose` (optional) @@ -1340,8 +1346,8 @@ paths: description: |- Updates a service or storage credential on the metastore. - The caller must be the owner of the credential or a metastore admin or have the `MANAGE` permission. - If the caller is a metastore admin, only the __owner__ field can be changed. + The caller must be the owner of the credential or a metastore admin or have the ``MANAGE`` permission. + If the caller is a metastore admin, only the **owner** field can be changed. :param name_arg: str Name of the credential. @@ -1403,13 +1409,13 @@ paths: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -1458,6 +1464,7 @@ paths: x-enum: - ISOLATION_MODE_ISOLATED - ISOLATION_MODE_OPEN + - ISOLATION_MODE_OPEN_IN_ACCOUNT description: Whether the current securable is accessible from all workspaces or a specific set of workspaces. new_name: type: string @@ -1568,13 +1575,13 @@ paths: description: |- Validates a credential. - For service credentials (purpose is **SERVICE**), either the __credential_name__ or the cloud-specific + For service credentials (purpose is **SERVICE**), either the **credential_name** or the cloud-specific credential must be provided. - For storage credentials (purpose is **STORAGE**), at least one of __external_location_name__ and - __url__ need to be provided. If only one of them is provided, it will be used for validation. And if - both are provided, the __url__ will be used for validation, and __external_location_name__ will be - ignored when checking overlapping urls. Either the __credential_name__ or the cloud-specific + For storage credentials (purpose is **STORAGE**), at least one of **external_location_name** and + **url** need to be provided. If only one of them is provided, it will be used for validation. And if + both are provided, the **url** will be used for validation, and **external_location_name** will be + ignored when checking overlapping urls. Either the **credential_name** or the cloud-specific credential must be provided. The caller must be a metastore admin or the credential owner or have the required permission on the @@ -1622,13 +1629,13 @@ paths: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -1786,6 +1793,109 @@ components: credential_info: $ref: '#/components/schemas/StorageCredentialInfo' description: The storage credential was successfully updated. + AgentService: + type: object + properties: + agent_service_type: + $ref: '#/components/schemas/AgentServiceAgentServiceType' + description: Type of agent service. Required on create. Immutable after creation. For future internal / genie / + app kinds, determines how the backing agent surface is provisioned. + browse_only: + type: boolean + description: Whether the caller sees only metadata available through the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + $ref: '#/components/schemas/AgentServiceConfig' + description: 'Operational configuration: connection, base_path, system_prompt, model_service, optional inference + table, optional rate limits. Required on CreateAgentService; on UpdateAgentService it is required only when ``config`` + (or a ``config.*`` subpath) appears in ``update_mask``.' + create_time: + type: string + format: date-time + description: When the agent service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: The resolved owner of the agent service. Falls back to the caller's identity when ``owner`` is not + explicitly set on creation. + etag: + type: string + description: Optimistic concurrency control token. Server-generated from the entity's state and returned on every + read. To use it as an if-match precondition on a mutation, echo the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the agent service. + name: + type: string + description: 'Resource name of the agent service. Format: ``agent-services/{catalog}.{schema}.{agent_service}``. + Each ``{...}`` component is capped at 255 characters individually. Server-derived on Create from ``parent`` + + ``agent_service_id``; required and immutable on Update/Get/Delete.' + owner: + type: string + description: The owner of the agent service. Write-only; read owner via effective_owner. + update_time: + type: string + format: date-time + description: When the agent service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: |- + An agent service represents a governed registration of an external AI agent (Bedrock Agents, + Snowflake Cortex Agents, Foundry, etc) in Unity Catalog. It acts as a schema-scoped securable + that references a UC Connection holding the agent platform's auth + endpoint, and exposes the + agent for discovery, authorization, and audit logging. Initial kind: AGENT_SERVICE_STANDARD. + Future kinds (INTERNAL, GENIE, APP) can be added as separate enum values. + AgentServiceConfig: + type: object + properties: + base_path: + type: string + description: 'Vendor base path appended to the connection url at invocation time. Bedrock example: ``/agents/AGENT123/agentAliases/ALIAS1/sessions``. + Empty when the agent platform does not require a per-agent suffix.' + inference_table: + $ref: '#/components/schemas/InferenceTableConfig' + description: Inference table config for invocation payload logging. Reuses the shared InferenceTableConfig (same + as ModelServiceConfig.inference_table). + model_service: + type: string + description: 'Resource name of the model service backing this agent. Empty when the agent platform manages its own + model selection (e.g. Bedrock Agents). Format: ``model-services/{catalog}.{schema}.{model_service}``.' + rate_limits: + type: array + items: + $ref: '#/components/schemas/RateLimit' + description: Per-principal rate limits applied to invocations routed through this agent service. Repeated to support + per-USER / USER_GROUP / SERVICE_PRINCIPAL / SERVICE / USER_DEFAULT scopes simultaneously, mirroring McpServiceConfig.rate_limits. + Empty when no rate limit is configured. + source_connection: + $ref: '#/components/schemas/AgentServiceConfigSourceConnection' + description: External agent referenced via a UC Connection. + system_prompt: + type: string + description: System prompt prepended to invocations. Max 8 KB. + description: |- + Operational configuration for an agent service. Groups the source connection, base path, system + prompt, model service reference, optional inference table, and optional rate limits. The + ``source_connection.name`` is supplied on create. + AgentServiceConfigSourceConnection: + type: object + properties: + name: + type: string + is_deleted: + type: boolean + required: + - name + description: |- + UC Connection for the agent platform, such as Bedrock Agents, Snowflake Cortex Agents, or + Foundry. On create, provide ``name`` in the schema-scoped form + ``connections/{catalog}.{schema}.{connection}``. On read, the service returns ``name``. ArtifactAllowlistInfo: type: object properties: @@ -1893,18 +2003,29 @@ components: Azure Active Directory token, essentially the Oauth token for Azure Service Principal or Managed Identity. Read more at https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token + AzureEncryptionSettings: + type: object + properties: + azure_tenant_id: + type: string + azure_cmk_access_connector_id: + type: string + azure_cmk_managed_identity_id: + type: string + required: + - azure_tenant_id AzureManagedIdentity: type: object properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -1916,10 +2037,10 @@ components: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -1931,13 +2052,13 @@ components: properties: access_connector_id: type: string - description: The Azure resource ID of the Azure Databricks Access Connector. Use the format `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + description: The Azure resource ID of the Azure Databricks Access Connector. Use the format ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. managed_identity_id: type: string - description: The Azure resource ID of the managed identity. Use the format, `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + description: The Azure resource ID of the managed identity. Use the format, ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume the AzureManagedIdentity is using the system-assigned identity. @@ -1997,11 +2118,21 @@ components: CancelRefreshResponse: type: object properties: {} + CatalogFederationConfig: + type: object + properties: + schema_filter_type: + $ref: '#/components/schemas/SchemaFilterType' + description: Controls how the catalog's schema filter items are treated. + description: Federation-only configuration for a FOREIGN catalog. CatalogInfo: type: object properties: browse_only: type: boolean + catalog_federation_config: + $ref: '#/components/schemas/CatalogFederationConfig' + description: Federation-only configuration, present only for FOREIGN catalogs. catalog_type: $ref: '#/components/schemas/CatalogType' comment: @@ -2010,12 +2141,21 @@ components: connection_name: type: string description: The name of the connection to an external data source. + conversion_info: + $ref: '#/components/schemas/ConversionInfo' + description: Status of conversion of FOREIGN catalog to UC Native catalog. created_at: type: integer description: Time at which this catalog was created, in epoch milliseconds. created_by: type: string description: Username of catalog creator. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the catalog + dr_replication_info: + $ref: '#/components/schemas/DrReplicationInfo' + description: Disaster Recovery replication state snapshot. effective_predictive_optimization_flag: $ref: '#/components/schemas/EffectivePredictiveOptimizationFlag' enable_predictive_optimization: @@ -2027,6 +2167,9 @@ components: isolation_mode: $ref: '#/components/schemas/CatalogIsolationMode' description: Whether the current securable is accessible from all workspaces or a specific set of workspaces. + managed_encryption_settings: + $ref: '#/components/schemas/EncryptionSettings' + description: Control CMK encryption for managed catalog data metastore_id: type: string description: Unique identifier of parent metastore. @@ -2163,19 +2306,53 @@ components: type: string target: type: string + ColumnTagValueExtraction: + type: object + properties: + column_alias: + type: string + description: The alias from MATCH COLUMNS that identifies the column. + tag_key: + type: string + description: 1024 matches the max_length on FunctionArgument.constant above. + required: + - column_alias + - tag_key + description: 'Extracts the value of a column-level tag: get_column_tag_value(col, "tagKey").' + ConditionalDisplay: + type: object + properties: + depends_on_option: + type: string + description: The name of the option whose value determines visibility of this option. + hidden_when_values: + type: array + items: + type: string + description: The values of the depends_on_option that will hide this option. If empty or not set, this option follows + default visibility (shown unless is_hidden is true). If depends_on_option has any of these values, this option + is hidden. + description: |- + Defines when an option should be hidden based on another option's value. For example, for + pre-created OAuth connections, some options are conditionally hidden. This field works in + conjunction with OptionSpec.is_hidden: + + - If OptionSpec.is_hidden is true, the option is always hidden regardless of ConditionalDisplay. + - If OptionSpec.is_hidden is false (or unset), ConditionalDisplay determines visibility: + - If depends_on_option matches any value in hidden_when_values, hide this option. + - Otherwise, show this option. ConnectionDependency: type: object properties: connection_name: type: string - description: Full name of the dependent connection, in the form of __connection_name__. + description: Full name of the dependent connection, in the form of **connection_name**. description: A connection that is dependent on a SQL object. ConnectionInfo: type: object properties: comment: type: string - description: User-provided free-form text description. connection_id: type: string description: Unique identifier of the Connection. @@ -2191,6 +2368,9 @@ components: credential_type: $ref: '#/components/schemas/CredentialType' description: The type of credential. + environment_settings: + $ref: '#/components/schemas/EnvironmentSettings' + description: '[Create,Update:OPT] Connection environment settings as EnvironmentSettings object.' full_name: type: string description: Full name of connection. @@ -2214,6 +2394,10 @@ components: read_only: type: boolean description: If the connection is read only. + secrets: + type: object + description: A map of option names to UC Secret references. Keys are connection option names (same as in OptionsKVPairs) + and values are UC Secret fully qualified names. securable_type: $ref: '#/components/schemas/SecurableType' updated_at: @@ -2225,7 +2409,6 @@ components: url: type: string description: URL of the remote data source, extracted from options. - description: 'Next ID: 24' ContinuousUpdateStatus: type: object properties: @@ -2242,6 +2425,13 @@ components: description: |- Detailed status of an online table. Shown if the online table is in the ONLINE_CONTINUOUS_UPDATE or the ONLINE_UPDATING_PIPELINE_RESOURCES state. + ConversionInfo: + type: object + properties: + state: + $ref: '#/components/schemas/ConversionInfoState' + description: The conversion state of the resource. + description: Status of conversion of FOREIGN entity into UC Native entity. CreateAccessRequest: type: object properties: @@ -2277,7 +2467,7 @@ components: description: Whether to allow non-DBR clients to directly access entities under the metastore. region: type: string - description: Cloud region which the metastore serves (e.g., `us-west-2`, `westus`). + description: Cloud region which the metastore serves (e.g., ``us-west-2``, ``westus``). storage_root: type: string description: The storage root URL for metastore @@ -2436,7 +2626,7 @@ components: properties: credential_name: type: string - description: Full name of the dependent credential, in the form of __credential_name__. + description: Full name of the dependent credential, in the form of **credential_name**. description: A credential that is dependent on a SQL object. CredentialInfo: type: object @@ -2569,7 +2759,16 @@ components: - delta_runtime_properties description: |- Properties pertaining to the current state of the delta table as given by the commit server. - This does not contain **delta.*** (input) properties in __TableInfo.properties__. + This does not contain **delta.*** (input) properties in **TableInfo.properties**. + DenyOptions: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges Dependency: type: object properties: @@ -2579,11 +2778,21 @@ components: $ref: '#/components/schemas/CredentialDependency' function: $ref: '#/components/schemas/FunctionDependency' + metastore_name: + type: string + description: Metastore context for the dependency. Global UC server responses currently populate this with the metastore + UUID string so clients can disambiguate dependencies with identical names across metastores. + secret: + $ref: '#/components/schemas/SecretDependency' + description: A dependency on a Unity Catalog secret. table: $ref: '#/components/schemas/TableDependency' + volume: + $ref: '#/components/schemas/VolumeDependency' + description: A dependency on a Unity Catalog volume. description: |- - A dependency of a SQL object. One of the following fields must be defined: __table__, - __function__, __connection__, or __credential__. + A dependency of a SQL object. One of the following fields must be defined: **table**, + **function**, **connection**, **credential**, **volume**, or **secret**. DependencyList: type: object properties: @@ -2596,6 +2805,19 @@ components: DisableResponse: type: object properties: {} + DrReplicationInfo: + type: object + properties: + last_failover_time_ms: + type: integer + description: Wall-clock epoch milliseconds when this catalog was last promoted to primary via failover or failback. + Set by DR Manager. Used by Predictive Optimization to suppress operations until sufficient workload history accumulates. + replicated_entities: + type: string + description: See https://docs.google.com/document/d/1X0A_3hMhzuS2V1E3zB0x5wxPsFx70bVYK5rHep2AjW8. + status: + $ref: '#/components/schemas/DrReplicationStatus' + description: Metadata related to Disaster Recovery. EffectivePermissionsList: type: object properties: @@ -2653,6 +2875,21 @@ components: $ref: '#/components/schemas/SseEncryptionDetails' description: Server-Side Encryption properties for clients communicating with AWS s3. description: Encryption options that apply to clients connecting to cloud storage. + EncryptionSettings: + type: object + properties: + azure_encryption_settings: + $ref: '#/components/schemas/AzureEncryptionSettings' + description: optional Azure settings - only required if an Azure CMK is used. + azure_key_vault_key_id: + type: string + description: the AKV URL in Azure, null otherwise. + customer_managed_key_id: + type: string + description: the CMK uuid in AWS and GCP, null otherwise. + description: |- + Encryption Settings are used to carry metadata for securable encryption at rest. Currently used + for catalogs, we can use the information supplied here to interact with a CMK. EntityTagAssignment: type: object properties: @@ -2664,8 +2901,10 @@ components: description: The key of the tag entity_type: type: string - description: 'The type of the entity to which the tag is assigned. Allowed values are: catalogs, schemas, tables, - columns, volumes.' + description: The type of the entity to which the tag is assigned. + inherited: + type: boolean + description: Boolean which indicates whether this tag is inherited. source_type: $ref: '#/components/schemas/TagAssignmentSourceType' description: The source type of the tag assignment, e.g., user-assigned or system-assigned @@ -2684,6 +2923,15 @@ components: - tag_key - entity_type description: Represents a tag assignment to an entity + EnvironmentSettings: + type: object + properties: + environment_version: + type: string + java_dependencies: + type: array + items: + type: string ExternalLineageExternalMetadata: type: object properties: @@ -2742,6 +2990,9 @@ components: table_info: $ref: '#/components/schemas/ExternalLineageTableInfo' description: Information about the table involved in the lineage relationship. + volume_info: + $ref: '#/components/schemas/ExternalLineageVolumeInfo' + description: Information about the volume involved in the lineage relationship. description: Lineage response containing lineage information of a data asset. ExternalLineageModelVersion: type: object @@ -2774,6 +3025,8 @@ components: $ref: '#/components/schemas/ExternalLineagePath' table: $ref: '#/components/schemas/ExternalLineageTable' + volume: + $ref: '#/components/schemas/ExternalLineageVolume' ExternalLineagePath: type: object properties: @@ -2844,6 +3097,26 @@ components: type: string description: Name of Schema. description: Represents the table information in the lineage event. + ExternalLineageVolume: + type: object + properties: + name: + type: string + ExternalLineageVolumeInfo: + type: object + properties: + catalog_name: + type: string + event_time: + type: string + format: date-time + description: Timestamp of the lineage event. + name: + type: string + description: Name of the volume. + schema_name: + type: string + description: Name of the schema. ExternalLocationInfo: type: object properties: @@ -2866,11 +3139,16 @@ components: description: Name of the storage credential used with this location. effective_enable_file_events: type: boolean - description: The effective value of `enable_file_events` after applying server-side defaults. + description: The effective value of ``enable_file_events`` after applying server-side defaults. + effective_file_event_queue: + $ref: '#/components/schemas/FileEventQueue' + description: The effective file event queue configuration after applying server-side defaults. Always populated + when a queue is provisioned, regardless of whether the user explicitly set ``enable_file_events``. Use this field + instead of ``file_event_queue`` for reading the actual queue state. enable_file_events: type: boolean - description: Whether to enable file events on this external location. Default to `true`. Set to `false` to disable - file events. The actual applied value may differ due to server-side defaults; check `effective_enable_file_events` + description: Whether to enable file events on this external location. Default to ``true``. Set to ``false`` to disable + file events. The actual applied value may differ due to server-side defaults; check ``effective_enable_file_events`` for the effective state. encryption_details: $ref: '#/components/schemas/EncryptionDetails' @@ -2880,7 +3158,7 @@ components: the access to the location falls back to cluster credentials if UC credentials are not sufficient. file_event_queue: $ref: '#/components/schemas/FileEventQueue' - description: File event queue settings. If `enable_file_events` is not `false`, must be defined and have exactly + description: File event queue settings. If ``enable_file_events`` is not ``false``, must be defined and have exactly one of the documented properties. isolation_mode: $ref: '#/components/schemas/IsolationMode' @@ -2975,12 +3253,16 @@ components: properties: managed_aqs: $ref: '#/components/schemas/AzureQueueStorage' + managed_onelake: + $ref: '#/components/schemas/OneLakeEventQueue' managed_pubsub: $ref: '#/components/schemas/GcpPubsub' managed_sqs: $ref: '#/components/schemas/AwsSqsQueue' provided_aqs: $ref: '#/components/schemas/AzureQueueStorage' + provided_onelake: + $ref: '#/components/schemas/OneLakeEventQueue' provided_pubsub: $ref: '#/components/schemas/GcpPubsub' provided_sqs: @@ -3011,6 +3293,15 @@ components: - child_columns - parent_table - parent_columns + FunctionArgExpression: + type: object + properties: + tag_introspection: + $ref: '#/components/schemas/TagIntrospectionExpression' + description: An expression that introspects tags at query time. + description: |- + An expression that is evaluated at query time against per-request context. New variants (e.g., + identity attributes) are added as additional oneof cases. FunctionArgument: type: object properties: @@ -3019,12 +3310,16 @@ components: constant: type: string description: A constant literal. + function_arg_expression: + $ref: '#/components/schemas/FunctionArgExpression' + description: An expression evaluated at query time. Wraps per-request expression variants (e.g., tag introspection) + so new variants can be added without extending the FunctionArgument oneof. FunctionDependency: type: object properties: function_full_name: type: string - description: Full name of the dependent function, in the form of __catalog_name__.__schema_name__.__function_name__. + description: Full name of the dependent function, in the form of **catalog_name**.**schema_name**.**function_name**. required: - function_full_name description: A function that is dependent on a SQL object. @@ -3206,6 +3501,8 @@ components: $ref: '#/components/schemas/GcpOauthToken' r2_temp_credentials: $ref: '#/components/schemas/R2Credentials' + uc_encrypted_token: + $ref: '#/components/schemas/UcEncryptedToken' url: type: string description: The URL of the storage path accessible by the temporary credential. @@ -3246,6 +3543,30 @@ components: $ref: '#/components/schemas/GcpOauthToken' r2_temp_credentials: $ref: '#/components/schemas/R2Credentials' + uc_encrypted_token: + $ref: '#/components/schemas/UcEncryptedToken' + url: + type: string + description: The URL of the storage path accessible by the temporary credential. + GenerateTemporaryVolumeCredentialResponse: + type: object + properties: + aws_temp_credentials: + $ref: '#/components/schemas/AwsCredentials' + azure_aad: + $ref: '#/components/schemas/AzureActiveDirectoryToken' + azure_user_delegation_sas: + $ref: '#/components/schemas/AzureUserDelegationSas' + expiration_time: + type: integer + description: Server time when the credential will expire, in epoch milliseconds. The API client is advised to cache + the credential given this expiration time. + gcp_oauth_token: + $ref: '#/components/schemas/GcpOauthToken' + r2_temp_credentials: + $ref: '#/components/schemas/R2Credentials' + uc_encrypted_token: + $ref: '#/components/schemas/UcEncryptedToken' url: type: string description: The URL of the storage path accessible by the temporary credential. @@ -3285,7 +3606,7 @@ components: description: Whether to allow non-DBR clients to directly access entities under the metastore. global_metastore_id: type: string - description: Globally unique metastore ID across clouds and regions, of the form `cloud:region:metastore_id`. + description: Globally unique metastore ID across clouds and regions, of the form ``cloud:region:metastore_id``. metastore_id: type: string description: Unique identifier of metastore. @@ -3297,10 +3618,10 @@ components: description: The owner of the metastore. privilege_model_version: type: string - description: Privilege model version of the metastore, of the form `major.minor` (e.g., `1.0`). + description: Privilege model version of the metastore, of the form ``major.minor`` (e.g., ``1.0``). region: type: string - description: Cloud region which the metastore serves (e.g., `us-west-2`, `westus`). + description: Cloud region which the metastore serves (e.g., ``us-west-2``, ``westus``). storage_root: type: string description: The storage root URL for metastore @@ -3340,8 +3661,54 @@ components: $ref: '#/components/schemas/WorkspaceBinding' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). + GrantOptions: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges + InferenceTableConfig: + type: object + properties: + parent: + type: string + description: 'Parent UC schema where the inference table is created. Format: ``schemas/{catalog}.{schema}``. Set + at create time and immutable thereafter; changing it on an existing service is rejected.' + disabled: + type: boolean + description: Indicates whether payload logging is disabled (opt-out). Unset means that payload logging is active + (the on-by-default state coincides with the proto zero-value, so the server never fills this field for a client + that leaves it unset). Set ``disabled = true`` to pause runtime logging while keeping the sub-message attached + (preserving ``parent`` and ``table_name_prefix`` for a later flip back to active). ``parent`` remains required + either way. + is_deleted: + type: boolean + description: True when the bound inference TABLE has been deleted but the parent service still references it. The + dangling reference is surfaced (not silently dropped) so callers can see the broken dependency. AI Gateway payload + logging fails closed in this state. + table: + type: string + description: 'Resolved UC table for payload logs. Format: ``tables/{catalog}.{schema}.{table}``.' + table_name_prefix: + type: string + description: Prefix for the inference-table's UC-registered name. The actual leaf name UC stores is ``_payload``; + the ``_payload`` suffix is appended automatically. To find the actual UC table after Create, read the ``table`` + field on the response. Defaults to ``_payload`` when unset. Set at create time and immutable + thereafter; changing it on an existing service is rejected. + required: + - parent + description: |- + Inference table configuration for payload logging on a model service. + + ``parent`` is always REQUIRED when the sub-message is set; the destination UC schema is needed + to construct or rebind the payload TABLE regardless of whether payload logging is currently + active. Payload logging is active by default; set ``disabled = true`` to pause runtime logging + without dropping the table or the binding. ListAccountMetastoreAssignmentsResponse: type: object properties: @@ -3359,6 +3726,18 @@ components: $ref: '#/components/schemas/StorageCredentialInfo' description: An array of metastore storage credentials. description: The metastore storage credentials were successfully returned. + ListAgentServicesResponse: + type: object + properties: + agent_services: + type: array + items: + $ref: '#/components/schemas/AgentService' + description: The list of agent services. + next_page_token: + type: string + description: Pagination token for retrieving the next page of results. + description: Response for listing agent services. ListCatalogsResponse: type: object properties: @@ -3368,7 +3747,7 @@ components: $ref: '#/components/schemas/CatalogInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). ListConnectionsResponse: type: object @@ -3379,7 +3758,7 @@ components: $ref: '#/components/schemas/ConnectionInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). ListCredentialsResponse: type: object @@ -3390,7 +3769,18 @@ components: $ref: '#/components/schemas/CredentialInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** + should be set to this value for the next request (for the next page of results). + ListEffectivePrivilegeAssignmentsResponse: + type: object + properties: + effective_privilege_assignments: + type: array + items: + $ref: '#/components/schemas/EffectivePrivilegeAssignment' + next_page_token: + type: string + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). ListEntityTagAssignmentsResponse: type: object @@ -3420,7 +3810,7 @@ components: $ref: '#/components/schemas/ExternalLocationInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). ListExternalMetadataResponse: type: object @@ -3440,8 +3830,20 @@ components: $ref: '#/components/schemas/FunctionInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). + ListMcpServicesResponse: + type: object + properties: + mcp_services: + type: array + items: + $ref: '#/components/schemas/McpService' + description: The list of MCP services. + next_page_token: + type: string + description: Pagination token for retrieving the next page of results. + description: Response for listing MCP services. ListMetastoresResponse: type: object properties: @@ -3451,8 +3853,32 @@ components: $ref: '#/components/schemas/MetastoreInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). + ListModelProviderServicesResponse: + type: object + properties: + model_provider_services: + type: array + items: + $ref: '#/components/schemas/ModelProviderService' + description: The list of model provider services. + next_page_token: + type: string + description: Pagination token for retrieving the next page of results. + description: Response for listing model provider services. + ListModelServicesResponse: + type: object + properties: + model_services: + type: array + items: + $ref: '#/components/schemas/ModelService' + description: The list of model services. + next_page_token: + type: string + description: Pagination token for retrieving the next page of results. + description: Response for listing model services. ListModelVersionsResponse: type: object properties: @@ -3462,7 +3888,7 @@ components: $ref: '#/components/schemas/ModelVersionInfo' next_page_token: type: string - description: Opaque token to retrieve the next page of results. Absent if there are no more pages. __page_token__ + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** should be set to this value for the next request (for the next page of results). ListPoliciesResponse: type: object @@ -3474,6 +3900,15 @@ components: items: $ref: '#/components/schemas/PolicyInfo' description: The list of retrieved policies. + ListPrivilegeAssignmentsResponse: + type: object + properties: + next_page_token: + type: string + privilege_assignments: + type: array + items: + $ref: '#/components/schemas/PrivilegeAssignment' ListQuotasResponse: type: object properties: @@ -3503,6 +3938,19 @@ components: items: $ref: '#/components/schemas/SchemaInfo' description: An array of schema information objects. + ListSecretsResponse: + type: object + properties: + next_page_token: + type: string + description: Opaque token to retrieve the next page of results. Absent if there are no more pages. **page_token** + should be set to this value for the next request. + secrets: + type: array + items: + $ref: '#/components/schemas/Secret' + description: An array of secret objects. + description: Response message for ListSecrets. ListStorageCredentialsResponse: type: object properties: @@ -3559,6 +4007,93 @@ components: condition: type: string description: The condition expression used to match a table column. + McpService: + type: object + properties: + browse_only: + type: boolean + description: Whether the caller sees only metadata available through the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + $ref: '#/components/schemas/McpServiceConfig' + description: 'Operational configuration: connection, tool selectors, rate limit. Required on CreateMcpService; on + UpdateMcpService it is required only when ``config`` (or a ``config.*`` subpath) appears in ``update_mask``.' + create_time: + type: string + format: date-time + description: When the MCP service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: The resolved owner of the MCP service. Falls back to the caller's identity when ``owner`` is not explicitly + set on creation. + etag: + type: string + description: Optimistic concurrency control token. Server-generated from the entity's state and returned on every + read. To use it as an if-match precondition on a mutation, echo the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the MCP service. + name: + type: string + description: 'Resource name of the MCP service. Format: ``mcp-services/{catalog}.{schema}.{mcp_service}``. Each + ``{...}`` component is capped at 255 characters individually. Server-derived on Create from ``parent`` + ``mcp_service_id``; + required and immutable on Update/Get/Delete.' + owner: + type: string + description: The owner of the MCP service. Write-only; read owner via effective_owner. + update_time: + type: string + format: date-time + description: When the MCP service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: |- + A governed MCP server registration in Unity Catalog. Acts as a container securable that + references an MCP server -- customer-external via a UC Connection, or Databricks-hosted via an + internal server -- and exposes its tools for discovery, authorization, and invocation. + McpServiceConfig: + type: object + properties: + include_tool_selectors: + type: array + items: + type: string + description: Glob or exact-match patterns selecting which tools from the MCP server to expose. Prefix match for + patterns with ``*``, exact match otherwise. An empty list means all tools are included. Per-element max 256 chars. + rate_limits: + type: array + items: + $ref: '#/components/schemas/RateLimit' + description: Per-principal rate limits applied to tool invocations routed through this MCP service. Repeated to + support per-USER / USER_GROUP / SERVICE_PRINCIPAL / SERVICE / USER_DEFAULT scopes simultaneously, mirroring the + ``ModelServiceConfig.rate_limits`` shape. Empty when no rate limit is configured. + source_connection: + $ref: '#/components/schemas/McpServiceConfigSourceConnection' + description: UC Connection referencing the MCP server. + description: |- + Operational configuration for an MCP service. Groups the source reference, tool selectors, and + rate limit -- the fields that configure how the MCP service behaves at invocation time. + McpServiceConfigSourceConnection: + type: object + properties: + name: + type: string + is_deleted: + type: boolean + required: + - name + description: |- + UC Connection that hosts the MCP server. On create, provide ``name`` in the schema-scoped form + ``connections/{catalog}.{schema}.{connection}``. On read, the service populates the resolved + connection metadata and preserves a dangling source so callers can diagnose a deleted backing + connection. MetastoreAssignment: type: object properties: @@ -3603,7 +4138,7 @@ components: description: Whether to allow non-DBR clients to directly access entities under the metastore. global_metastore_id: type: string - description: Globally unique metastore ID across clouds and regions, of the form `cloud:region:metastore_id`. + description: Globally unique metastore ID across clouds and regions, of the form ``cloud:region:metastore_id``. metastore_id: type: string description: Unique identifier of metastore. @@ -3615,10 +4150,10 @@ components: description: The owner of the metastore. privilege_model_version: type: string - description: Privilege model version of the metastore, of the form `major.minor` (e.g., `1.0`). + description: Privilege model version of the metastore, of the form ``major.minor`` (e.g., ``1.0``). region: type: string - description: Cloud region which the metastore serves (e.g., `us-west-2`, `westus`). + description: Cloud region which the metastore serves (e.g., ``us-west-2``, ``westus``). storage_root: type: string description: The storage root URL for metastore @@ -3634,30 +4169,669 @@ components: updated_by: type: string description: Username of user who last modified the metastore. - ModelVersionInfo: + ModelProviderService: type: object properties: - aliases: - type: array - items: - $ref: '#/components/schemas/RegisteredModelAlias' - catalog_name: - type: string - description: The name of the catalog containing the model version + browse_only: + type: boolean + description: Whether the caller sees only metadata available through the BROWSE privilege. comment: type: string - description: The comment attached to the model version - created_at: - type: integer + description: User-provided description. + config: + $ref: '#/components/schemas/ModelProviderServiceConfig' + description: 'Behavioral configuration: provider connection, model catalog, and passthrough policy. See ``ModelProviderServiceConfig`` + for the per-field contract. Required on CreateModelProviderService; on Update it is required only when ``config`` + (or a ``config.*`` subpath) appears in ``update_mask``.' + create_time: + type: string + format: date-time + description: When the provider service was created. created_by: type: string - description: The identifier of the user who created the model version - id: + description: Creator identity. + effective_owner: type: string - description: The unique identifier of the model version + description: The resolved owner of the model provider service. Falls back to the caller's identity when ``owner`` + is not explicitly set on creation. + etag: + type: string + description: Optimistic concurrency control token. Server-generated from the entity's state and returned on every + read. To use it as an if-match precondition on a mutation, echo the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects the mutation if the stored etag differs. metastore_id: type: string - description: The unique identifier of the metastore containing the model version + description: Metastore hosting the provider service. + name: + type: string + description: 'Resource name of the provider service. Format: ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at 255 characters individually. Server-derived on Create from ``parent`` + + ``model_provider_service_id``; required and immutable on Update/Get/Delete.' + owner: + type: string + description: The owner of the model provider service. Write-only; read owner via effective_owner. + update_time: + type: string + format: date-time + description: When the provider service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: |- + A governed external model-provider connection stored in Unity Catalog (e.g. an OpenAI API + account, an Azure OpenAI deployment, an Amazon Bedrock account). Owns the provider type and the + auth/configuration the platform needs to invoke that provider, and is referenced from + ``ExternalModelConfig.model_provider_service`` on a ModelService. + + One ModelProviderService can back many ModelServices (e.g. an ``openai_prod`` provider serving + multiple models); a single ModelService can fan out across multiple ModelProviderServices for + traffic split or failover. + ModelProviderServiceConfig: + type: object + properties: + allow_all_targets: + type: boolean + description: When true, accepts any model exposed by the upstream provider; ``targets`` is not required and does + not restrict routability. When false, only models listed in ``targets`` are routable. + amazon_bedrock: + $ref: '#/components/schemas/ModelProviderServiceConfigAmazonBedrockProviderConfig' + anthropic: + $ref: '#/components/schemas/ModelProviderServiceConfigAnthropicProviderConfig' + azure_openai: + $ref: '#/components/schemas/ModelProviderServiceConfigAzureOpenAiProviderConfig' + custom: + $ref: '#/components/schemas/ModelProviderServiceConfigCustomProviderConfig' + forward_headers: + type: boolean + description: Whether to forward incoming request headers to the upstream provider. Applies to managed (multi-model) + requests as well as passthrough requests served by this provider service. Governance-level decision by the provider + service owner; not selectable per inference call. + forward_query_parameters: + type: boolean + description: Whether to forward incoming request query parameters to the upstream provider. Same trust-boundary + semantics as ``forward_headers``. + forward_unmanaged_paths: + type: boolean + description: Whether to forward request paths that fall outside this service's managed API set to the upstream provider + as opaque passthrough. When true, requests addressed to subpaths not recognized by the managed API surface are + proxied to the upstream provider over the same provider connection. When false, only managed-API paths are served. + Governance-level decision by the provider service owner; expanding this expands the trust boundary that the ModelProviderService + exposes. + gemini_enterprise: + $ref: '#/components/schemas/ModelProviderServiceConfigGeminiEnterpriseProviderConfig' + inference_table: + $ref: '#/components/schemas/InferenceTableConfig' + description: Inference table configuration for payload logging when this provider service is invoked directly. When + it is invoked through a model service, the model service's own inference table captures the invocation instead. + Mirrors ``ModelServiceConfig.inference_table`` / ``AgentServiceConfig.inference_table``. + microsoft_foundry: + $ref: '#/components/schemas/ModelProviderServiceConfigMicrosoftFoundryProviderConfig' + openai: + $ref: '#/components/schemas/ModelProviderServiceConfigOpenAiProviderConfig' + provider_type: + $ref: '#/components/schemas/ModelProviderServiceConfigExternalModelProviderType' + description: Provider type discriminator. Required at create time; immutable after. Determines which variant of + the ``provider`` oneof must be set. May not be changed via Update; attempts to include ``config.provider_type`` + in ``UpdateModelProviderServiceRequest.update_mask`` are rejected. Required on CreateModelProviderService and + immutable thereafter. + rate_limits: + type: array + items: + $ref: '#/components/schemas/RateLimit' + description: Rate limits applied when this provider service is invoked directly. When it is invoked through a model + service, the model service's own ``rate_limits`` apply instead. Mirrors ``ModelServiceConfig.rate_limits`` / ``McpServiceConfig.rate_limits``. + targets: + type: array + items: + $ref: '#/components/schemas/ModelProviderServiceConfigModelTargetConfig' + description: Routing targets this provider service exposes (provider-side model identifier + unified API types per + entry). Required (>=1) when ``allow_all_targets = false``; optional and additive when ``allow_all_targets = true``. + References from ``ExternalModelConfig.target`` must match an entry here unless ``allow_all_targets = true``. + description: |- + Behavioral configuration for a ModelProviderService: provider connection (auth + + provider-specific fields), the catalog of models this provider service can route to, and the + passthrough policy that governs how request headers, query parameters, and unmanaged subpaths + cross the trust boundary to the upstream provider. + ModelProviderServiceConfigAmazonBedrockProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigAmazonBedrockProviderDirectConfig' + description: Amazon Bedrock provider configuration. + ModelProviderServiceConfigAmazonBedrockProviderDirectConfig: + type: object + properties: + aws_access_key: + $ref: '#/components/schemas/ModelProviderServiceConfigAwsAccessKey' + description: AWS access-key-pair auth. Mutually exclusive with ``service_credential``. Supersedes the flat ``aws_access_key_id`` + / ``aws_secret_access_key`` fields. + aws_access_key_id: + type: string + description: 'Deprecated flat AWS access key ID. Superseded by ``aws_access_key.access_key_id``. Kept for one migration + cycle; the handler mirrors it to/from ``aws_access_key``. Treated as username-equivalent (not a secret value): + round-trips on reads and is scrubbed from audit logs.' + aws_secret_access_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Deprecated flat AWS secret access key. Superseded by ``aws_access_key.secret_access_key``. Kept for + one migration cycle; the handler mirrors it to/from ``aws_access_key``. Supplied as inline plaintext via ``ProviderSecret.plaintext``. + region: + type: string + description: AWS region where the Bedrock endpoint is hosted (e.g., ``us-east-1``). Required on Create. + service_credential: + $ref: '#/components/schemas/ModelProviderServiceConfigServiceCredential' + description: Reference to a UC service credential authorizing Bedrock requests. On Create the caller supplies ``service_credential.name`` + in the AIP-122 resource-name form ``credentials/{name}``. Required on Create when using UC-service-credential + auth; mutually exclusive with ``aws_access_key``. The credential is referenced by name; its value is not carried + here. On read the resolved ``id`` and ``is_deleted`` are also populated. Only supported on AWS-hosted workspaces; + Create requests from other clouds are rejected with INVALID_PARAMETER_VALUE. + description: |- + Direct form of Amazon Bedrock provider config. + + Authentication is one of two mutually exclusive modes, exactly one of which must be supplied on + Create: + + - Access keys: set ``aws_access_key``, leave ``service_credential`` unset. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``aws_access_key`` unset. The credential value lives in UC and + is referenced by name, not held on this message. Setting more than one mode is rejected. + ModelProviderServiceConfigAnthropicProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigAnthropicProviderDirectConfig' + description: 'Direct (inline-credentials) form: caller supplies the API key in the request body. Required on Create + unless ``relayed`` is set.' + relayed: + $ref: '#/components/schemas/ModelProviderServiceConfigAnthropicProviderRelayedConfig' + description: 'Relayed (credential-less) form: no Anthropic credential is stored. Each inference request instead + carries the caller''s own OAuth token, which the platform forwards to Anthropic on outbound requests. Mutually + exclusive with ``direct``; no ``api_key`` is required or persisted.' + description: |- + Anthropic provider configuration. Exactly one of ``direct`` or ``relayed`` must be set on + Create; the two are mutually exclusive. + ModelProviderServiceConfigAnthropicProviderDirectConfig: + type: object + properties: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Anthropic API key. Required on Create. Sent as the ``x-api-key`` header on outbound requests. Supplied + as inline plaintext via ``ProviderSecret.plaintext``. + description: Direct form of Anthropic provider config. + ModelProviderServiceConfigAnthropicProviderRelayedConfig: + type: object + properties: + plan_type: + $ref: '#/components/schemas/ModelProviderServiceConfigAnthropicProviderRelayedConfigAnthropicRelayedPlanType' + description: Which Anthropic subscription tier the relayed token belongs to. Optional; when unset the MPS gets the + full governance surface (see TEAM_ENTERPRISE). Immutable after Create, so the tier cannot be flipped in place. + description: |- + Relayed form of Anthropic provider config: no credential is stored. Authentication is the + caller's own OAuth token, forwarded to Anthropic on outbound requests, so there is no persisted + secret. Presence of this variant is the signal that the provider service uses relayed auth; + ``plan_type`` further distinguishes which Anthropic subscription tier the token belongs to. + ModelProviderServiceConfigAwsAccessKey: + type: object + properties: + access_key_id: + type: string + description: 'AWS access key ID. Required on Create when using access-key auth. Treated as username-equivalent (not + a secret value): round-trips on reads and is scrubbed from audit logs.' + secret_access_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: AWS secret access key paired with ``access_key_id``. Required on Create when using access-key auth. + Supplied as inline plaintext via ``ProviderSecret.plaintext``. + description: 'AWS access-key-pair auth for Amazon Bedrock: a SigV4-signing key pair.' + ModelProviderServiceConfigAzureOpenAiProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigAzureOpenAiProviderDirectConfig' + description: Azure OpenAI provider configuration. + ModelProviderServiceConfigAzureOpenAiProviderDirectConfig: + type: object + properties: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Azure OpenAI API key. Mutually exclusive with the Entra and service-credential modes. Supplied as inline + plaintext via ``ProviderSecret.plaintext``. + base_url: + type: string + description: Full Azure OpenAI endpoint base URL, e.g. ``https://myresource.openai.azure.com``. Required on Create. + client_id: + type: string + description: Deprecated flat Entra client ID. Superseded by ``entra_service_principal.client_id``. Kept for one + migration cycle; the handler mirrors it to/from ``entra_service_principal``. + client_secret: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Deprecated flat Entra client secret. Superseded by ``entra_service_principal.client_secret``. Kept + for one migration cycle; the handler mirrors it to/from ``entra_service_principal``. Supplied as inline plaintext + via ``ProviderSecret.plaintext``. + entra_service_principal: + $ref: '#/components/schemas/ModelProviderServiceConfigEntraServicePrincipal' + description: Entra ID (service principal) auth. Mutually exclusive with ``api_key`` and ``service_credential``. + Supersedes the flat ``tenant_id`` / ``client_id`` / ``client_secret`` fields. + service_credential: + $ref: '#/components/schemas/ModelProviderServiceConfigServiceCredential' + description: Reference to a UC service credential authorizing Azure OpenAI requests. On Create the caller supplies + ``service_credential.name`` in the AIP-122 resource-name form ``credentials/{name}``. Required on Create when + using UC-service-credential auth; mutually exclusive with ``api_key`` and ``entra_service_principal``. The credential + is referenced by name; its value is not carried here. On read the resolved ``id`` and ``is_deleted`` are also + populated. Only supported on Azure-hosted workspaces; Create requests from other clouds are rejected with INVALID_PARAMETER_VALUE. + tenant_id: + type: string + description: Deprecated flat Entra tenant ID. Superseded by ``entra_service_principal.tenant_id``. Kept for one + migration cycle; the handler mirrors it to/from ``entra_service_principal``. + description: |- + Direct form of Azure OpenAI provider config. Exactly one of three mutually-exclusive auth modes + must be supplied on Create: + + - API key: set ``api_key``, leave ``entra_service_principal`` and ``service_credential`` unset. + - Entra ID (service principal): set ``entra_service_principal``, leave ``api_key`` and + ``service_credential`` unset. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``api_key`` and ``entra_service_principal`` unset. The + credential value lives in UC and is referenced by name, not held on this message. Only + supported on Azure-hosted workspaces. Setting more than one mode is rejected. + ModelProviderServiceConfigCustomProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigCustomProviderDirectConfig' + description: 'Custom provider configuration: arbitrary HTTP endpoint with bearer-token auth.' + ModelProviderServiceConfigCustomProviderDirectConfig: + type: object + properties: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: 'Bearer token forwarded as the ``Authorization: Bearer ...`` header on outbound requests. Supplied + as inline plaintext via ``ProviderSecret.plaintext``. Set this for bearer-token auth.' + base_url: + type: string + description: Endpoint URL of the OpenAI-compatible service (e.g., ``https://api.example.com/v1``). Required on Create. + description: |- + Direct form of custom provider config. + + Authentication is one of two mutually exclusive modes, exactly one of which must be supplied on + Create: + + - Bearer: set ``api_key``, leave ``header_auth`` unset. The secret is forwarded as + ``Authorization: Bearer ``. + - Header: set ``header_auth``, leave ``api_key`` unset. The secret is forwarded as + ``: ``. Setting both modes or neither mode is rejected. + ModelProviderServiceConfigEntraServicePrincipal: + type: object + properties: + client_id: + type: string + description: Entra ID client (application) ID. Required on Create. + client_secret: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Entra ID client secret. Supplied as inline plaintext via ``ProviderSecret.plaintext``. + tenant_id: + type: string + description: Entra ID (Azure AD) tenant ID. Required on Create. + description: |- + Entra ID (Azure AD) service-principal auth: AI Gateway exchanges the ``tenant_id`` + + ``client_id`` identify the service principal, and the ``credential`` oneof proves that identity, + exchanged for an Entra bearer token on outbound requests via the OAuth2 client-credentials + grant. Shared by the Azure OpenAI and Microsoft Foundry provider configs. + ModelProviderServiceConfigGeminiEnterpriseProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigGeminiEnterpriseProviderDirectConfig' + description: Gemini Enterprise provider configuration. + ModelProviderServiceConfigGeminiEnterpriseProviderDirectConfig: + type: object + properties: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Google Gemini Enterprise API key. Required on Create. Supplied as inline plaintext via ``ProviderSecret.plaintext``. + project_id: + type: string + description: GCP project ID hosting the Gemini Enterprise endpoint. Required on Create. + region: + type: string + description: GCP region of the Gemini Enterprise endpoint (e.g., ``us-central1``). Required on Create. + description: Direct form of Gemini Enterprise provider config. + ModelProviderServiceConfigMicrosoftFoundryProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigMicrosoftFoundryProviderDirectConfig' + description: Microsoft Foundry provider configuration. + ModelProviderServiceConfigMicrosoftFoundryProviderDirectConfig: + type: object + properties: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Microsoft AI Foundry API key. Mutually exclusive with the Entra and service-credential modes. Supplied + as inline plaintext via ``ProviderSecret.plaintext``. + base_url: + type: string + description: Microsoft AI Foundry endpoint URL. Required on Create. + client_id: + type: string + description: Deprecated flat Entra client ID. Superseded by ``entra_service_principal.client_id``. Kept for one + migration cycle; the handler mirrors it to/from ``entra_service_principal``. + client_secret: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: Deprecated flat Entra client secret. Superseded by ``entra_service_principal.client_secret``. Kept + for one migration cycle; the handler mirrors it to/from ``entra_service_principal``. Supplied as inline plaintext + via ``ProviderSecret.plaintext``. + entra_service_principal: + $ref: '#/components/schemas/ModelProviderServiceConfigEntraServicePrincipal' + description: Entra ID (service principal) auth. Mutually exclusive with ``api_key`` and ``service_credential``. + Supersedes the flat ``tenant_id`` / ``client_id`` / ``client_secret`` fields. + service_credential: + $ref: '#/components/schemas/ModelProviderServiceConfigServiceCredential' + description: Reference to a UC service credential authorizing Microsoft Foundry requests. On Create the caller supplies + ``service_credential.name`` in the AIP-122 resource-name form ``credentials/{name}``. Required on Create when + using UC-service-credential auth; mutually exclusive with ``api_key`` and ``entra_service_principal``. The credential + is referenced by name; its value is not carried here. On read the resolved ``id`` and ``is_deleted`` are also + populated. Only supported on Azure-hosted workspaces; Create requests from other clouds are rejected with INVALID_PARAMETER_VALUE. + tenant_id: + type: string + description: Deprecated flat Entra tenant ID. Superseded by ``entra_service_principal.tenant_id``. Kept for one + migration cycle; the handler mirrors it to/from ``entra_service_principal``. + description: |- + Direct form of Microsoft Foundry provider config. + + Authentication is one of three mutually exclusive modes, exactly one of which must be supplied + on Create: + + - API key: set ``api_key``, leave ``entra_service_principal`` and ``service_credential`` unset. + - Entra ID (service principal): set ``entra_service_principal``, leave ``api_key`` and + ``service_credential`` unset. AI Gateway exchanges these for an Entra bearer token on outbound + requests via the OAuth2 client-credentials grant. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``api_key`` and ``entra_service_principal`` unset. The + credential value lives in UC and is referenced by name, not held on this message. Only + supported on Azure-hosted workspaces. Setting more than one mode is rejected. + ModelProviderServiceConfigModelTargetConfig: + type: object + properties: + model: + type: string + description: Provider-side model identifier (e.g. "gpt-5", "claude-opus-4-7"). This is a string on the LLM provider's + side, not a UC entity. The UC governance hook for external destinations is the ModelProviderService referenced + by ``ExternalModelConfig.model_provider_service``, not the model itself. + native_api_types: + type: array + items: + type: string + description: Provider-native API types the model supports (e.g. "openai/v1/chat/completions"). Used by the platform + for request/response translation from the unified API type. At most 64 entries of at most 256 characters each; + the list is persisted into the destination binding's bounded storage envelope. + required: + - model + description: Model target configuration for an external model destination. + ModelProviderServiceConfigOpenAiProviderConfig: + type: object + properties: + direct: + $ref: '#/components/schemas/ModelProviderServiceConfigOpenAiProviderDirectConfig' + description: OpenAI provider configuration. + ModelProviderServiceConfigOpenAiProviderDirectConfig: + type: object + properties: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: OpenAI API key. Required on Create. Supplied as inline plaintext via ``ProviderSecret.plaintext``. + base_url: + type: string + description: Optional custom base URL. Defaults to ``https://api.openai.com/v1``. Use for OpenAI-API-compatible + third-party endpoints or in-network proxies. + organization: + type: string + description: Optional OpenAI organization ID. When set, the platform forwards it as the ``OpenAI-Organization`` + header. + description: Direct (inline-credentials) form of the OpenAI provider config. + ModelProviderServiceConfigProviderSecret: + type: object + properties: + plaintext: + type: string + description: 'Inline plaintext credential. INPUT_ONLY: the value never round-trips on reads. Get and List responses + omit ``plaintext``; the field''s presence in the read shape only indicates that a secret is configured.' + description: |- + A secret value supplied as part of an inline provider config. The caller supplies the value as + inline ``plaintext`` on writes; the platform stores it encrypted. The ``plaintext`` field is + ``INPUT_ONLY`` and never round-trips on reads. + ModelProviderServiceConfigServiceCredential: + type: object + properties: + name: + type: string + description: Resource name of the bound UC service credential, in the AIP-122 form ``credentials/{name}`` (a metastore-level + single-part credential name). On create the caller supplies the name here. On read it reflects the credential's + current name at read time. + required: + - name + description: |- + ---- Provider configuration (nested; see the ``provider`` oneof below) ---- The customer-owned + UC service credential a ModelProviderService uses to authenticate to its provider, referenced by + name. + ModelService: + type: object + properties: + browse_only: + type: boolean + description: Whether the caller sees only metadata available through the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + $ref: '#/components/schemas/ModelServiceConfig' + description: 'Operational configuration: destinations, routing, rate limits, inference table. Required on CreateModelService; + on UpdateModelService it is required only when ``config`` (or a ``config.*`` subpath) appears in ``update_mask``.' + create_time: + type: string + format: date-time + description: When the model service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: The resolved owner of the ModelService. Falls back to the caller's identity when ``owner`` is not explicitly + set on creation. + etag: + type: string + description: Optimistic concurrency control token. Server-generated from the entity's state and returned on every + read. To use it as an if-match precondition on a mutation, echo the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the model service. + name: + type: string + description: 'Resource name of the model service. Format: ``model-services/{catalog}.{schema}.{model_service}``. + Each ``{...}`` component is capped at 255 characters individually. Server-derived on Create from ``parent`` + + ``model_service_id``; required and immutable on Update/Get/Delete.' + owner: + type: string + description: The owner of the model service. Write-only; read owner via effective_owner. + supported_api_types: + type: array + items: + type: string + description: Unified API types this endpoint supports (e.g. "chat", "embeddings", "completions"). Derived from the + destinations' backing models / providers at read time. + update_time: + type: string + format: date-time + description: When the model service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: |- + A governed AI Gateway endpoint in Unity Catalog that routes inference requests to one or more + model destinations (for example a foundation model or an external LLM reached through a + ModelProviderService). Applies centralized access control, rate limits, guardrails, and auditing + to the traffic it serves. + ModelServiceConfig: + type: object + properties: + inference_table: + $ref: '#/components/schemas/InferenceTableConfig' + description: Inference table config for payload logging. + rate_limits: + type: array + items: + $ref: '#/components/schemas/RateLimit' + description: Rate limits applied to requests routed through this model service. + routing: + $ref: '#/components/schemas/ModelServiceConfigRoutingConfig' + description: 'Routing configuration: destinations, routing strategy, and fallback.' + description: Operational configuration wrapped around the ModelService resource. + ModelServiceConfigDestinationConfig: + type: object + properties: + name: + type: string + description: User-facing label for this destination, used in routing references. + destination_type: + $ref: '#/components/schemas/ModelServiceConfigDestinationConfigDestinationType' + description: Backing-model category. Determines which oneof variant is populated. + external_model_config: + $ref: '#/components/schemas/ModelServiceConfigExternalModelConfig' + is_deleted: + type: boolean + description: True when the destination's backing UC entity (MODEL for foundation-model destinations, MODEL_PROVIDER_SERVICE + for external destinations) has been deleted but the destination row still references it. The dangling destination + is surfaced (not silently dropped) so callers can see the broken routing. Inference traffic through this destination + fails closed (BAD_REQUEST / FAILED_PRECONDITION). + pay_per_token_config: + $ref: '#/components/schemas/ModelServiceConfigPayPerTokenConfig' + provisioned_throughput_config: + $ref: '#/components/schemas/ModelServiceConfigProvisionedThroughputConfig' + traffic_percentage: + type: integer + description: Share of traffic sent to this destination, 0-100. Optional on fallback destinations; see FallbackConfig. + required: + - name + - destination_type + description: |- + A destination the model service can route traffic to. Exactly one of the per-type configs inside + ``type_config`` must be set, and it must match ``destination_type``. + ModelServiceConfigExternalModelConfig: + type: object + properties: + model_provider_service: + type: string + description: 'Resource name of the governed ModelProviderService that owns provider auth and provider-specific configuration. + The referenced ModelProviderService also carries the provider type, so this message does not surface it directly. + Format: ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. Each ``{...}`` component is capped + at 255 characters individually.' + target: + $ref: '#/components/schemas/ModelProviderServiceConfigModelTargetConfig' + description: 'Routing target for the destination: the provider-side model selected from the referenced ModelProviderService''s + ``targets`` catalog, plus the unified API types the platform should translate to/from at request time.' + required: + - model_provider_service + - target + description: |- + Configuration for an external-foundation-model destination. Provider auth and provider-specific + cloud configuration are owned by a separate, governed ModelProviderService entity referenced via + ``model_provider_service``; the platform resolves the provider at invocation time. + ModelServiceConfigFallbackConfig: + type: object + properties: + destinations: + type: array + items: + $ref: '#/components/schemas/ModelServiceConfigDestinationConfig' + description: Ordered list of fallback destinations. Traversal is in list order; the attempt count is the length + of the list. At most 5 are allowed. + description: |- + Fallback routing, applied after the primary destination returns a retryable error. Traversal is + in list order; the attempt count is the length of the list. + ModelServiceConfigPayPerTokenConfig: + type: object + properties: + model: + type: string + description: 'Resource name of the UC model. Format: ``models/{catalog}.{schema}.{model}``.' + required: + - model + description: |- + Configuration for a pay-per-token foundation-model destination. Identifies the foundation model + by its UC resource name; the platform resolves it to a Model Serving endpoint at request time. + ModelServiceConfigProvisionedThroughputConfig: + type: object + properties: + model_serving_endpoint: + type: string + description: Name of the backing Model Serving endpoint serving the provisioned- throughput foundation model, as + the AIP-122 typed resource name ``serving-endpoints/{name}``. The same UC model can be served on multiple Model + Serving endpoints (different throughput / region / config); the caller picks which one this destination routes + to. The endpoint must exist at create time. + model: + type: string + description: UC model FQN of the model served by the backing endpoint (e.g., ``system.ai.databricks-claude-opus-4-6``). + Resolved from Model Serving at Create/Update time. + required: + - model_serving_endpoint + description: |- + Configuration for a provisioned-throughput foundation-model destination. References a + pre-existing Model Serving endpoint that serves the model; sizing (provisioned throughput, burst + scaling, model version) is owned by the Model Serving endpoint itself, not by this message. + ModelServiceConfigRoutingConfig: + type: object + properties: + destinations: + type: array + items: + $ref: '#/components/schemas/ModelServiceConfigDestinationConfig' + description: Primary routing destinations. At most 10 are allowed. At least one is required on CreateModelService; + on UpdateModelService it is required only when ``config.routing`` (or a ``config.routing.*`` subpath) appears + in ``update_mask``. + fallback: + $ref: '#/components/schemas/ModelServiceConfigFallbackConfig' + description: Fallback routing config, applied after primary destinations fail. + first_token_timeout: + type: string + description: Timeout for the first token of a streaming response. If a destination does not return its first token + within this duration, AI Gateway aborts the attempt and fails over to the next destination. Applies to streaming + requests only. Leave unset for no first-token timeout. + traffic_splitting: + $ref: '#/components/schemas/ModelServiceConfigRoutingConfigTrafficSplitting' + description: Marker message selecting request-based traffic splitting. Traffic is distributed according to each + destination's traffic_percentage value; no configuration lives on this message itself. + description: |- + Routing configuration for a model service, nesting destinations, routing strategy, and fallback + under a single sub-message. + ModelServiceConfigRoutingConfigTrafficSplitting: + type: object + properties: {} + description: |- + Marker message selecting request-based traffic splitting across primary destinations. Split + weights are read from each DestinationConfig.traffic_percentage. + ModelVersionInfo: + type: object + properties: + aliases: + type: array + items: + $ref: '#/components/schemas/RegisteredModelAlias' + catalog_name: + type: string + description: The name of the catalog containing the model version + comment: + type: string + description: The comment attached to the model version + created_at: + type: integer + created_by: + type: string + description: The identifier of the user who created the model version + id: + type: string + description: The unique identifier of the model version + metastore_id: + type: string + description: The unique identifier of the metastore containing the model version model_name: type: string description: The name of the parent registered model of the model version, relative to parent schema @@ -3759,16 +4933,16 @@ components: type: string table_name: type: string - description: '[Create:ERR Update:IGN] UC table to monitor. Format: `catalog.schema.table_name`' + description: '[Create:ERR Update:IGN] UC table to monitor. Format: ``catalog.schema.table_name``' status: $ref: '#/components/schemas/MonitorInfoStatus' description: '[Create:ERR Update:IGN] The monitor status.' profile_metrics_table_name: type: string - description: '[Create:ERR Update:IGN] Table that stores profile metrics data. Format: `catalog.schema.table_name`.' + description: '[Create:ERR Update:IGN] Table that stores profile metrics data. Format: ``catalog.schema.table_name``.' drift_metrics_table_name: type: string - description: '[Create:ERR Update:IGN] Table that stores drift metrics data. Format: `catalog.schema.table_name`.' + description: '[Create:ERR Update:IGN] Table that stores drift metrics data. Format: ``catalog.schema.table_name``.' monitor_version: type: integer description: '[Create:ERR Update:IGN] Represents the current monitor configuration version in use. The version will @@ -3781,7 +4955,7 @@ components: baseline_table_name: type: string description: '[Create:OPT Update:OPT] Baseline table name. Baseline data is used to compute drift from the data - in the monitored `table_name`. The baseline table and the monitored table shall have the same schema.' + in the monitored ``table_name``. The baseline table and the monitored table shall have the same schema.' custom_metrics: type: array items: @@ -3811,9 +4985,9 @@ components: type: string description: '[Create:OPT Update:OPT] List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. - For example `slicing_exprs=[“col_1”, “col_2 > 10”]` will generate the following slices: two slices for `col_2 - > 10` (True and False), and one slice per unique value in `col1`. For high-cardinality columns, only the top 100 - unique values by frequency will generate slices.' + For example ``slicing_exprs=[“col_1”, “col_2 > 10”]`` will generate the following slices: two slices for ``col_2 + > 10`` (True and False), and one slice per unique value in ``col1``. For high-cardinality columns, only the top + 100 unique values by frequency will generate slices.' snapshot: $ref: '#/components/schemas/MonitorSnapshot' description: Configuration for monitoring snapshot tables. @@ -3835,8 +5009,8 @@ components: description: Name of the metric in the output tables. definition: type: string - description: 'Jinja template for a SQL expression that specifies how to compute the metric. See [create metric definition]. - [create metric definition]: https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition' + description: Jinja template for a SQL expression that specifies how to compute the metric. See `create metric definition + `__. input_columns: type: array items: @@ -3942,6 +5116,32 @@ components: The special destination cannot be assigned to a securable and only represents the default destination of the securable. The securable types that support default special destinations are: "catalog", "external_location", "connection", "credential", and "metastore". The **destination_type** of a **special_destination** is always EMAIL.' + OneLakeEventQueue: + type: object + properties: + consumer_group: + type: string + description: 'Event Hubs consumer group used to consume file events. Defaults to "$Default" when unset. Recommended + for provided_onelake: create a dedicated consumer group on the Event Hub for file events to avoid contending with + the customer''s other consumers.' + event_hub_name: + type: string + description: Event Hub entity name within the namespace. Only required for provided_onelake. + event_hub_url: + type: string + description: 'The Event Hub URL in the format https://{namespace}.servicebus.windows.net/{event_hub_name}. Deprecated: + use fully_qualified_namespace + event_hub_name instead.' + fully_qualified_namespace: + type: string + description: The fully qualified domain name of the Event Hubs namespace, e.g. {yournamespace}.servicebus.windows.net. + Only required for provided_onelake. + managed_resource_id: + type: string + description: Unique identifier included in the name of the file events managed resources. + description: |- + File event queue for OneLake (Microsoft Fabric) locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). OnlineTable: type: object properties: @@ -4028,6 +5228,10 @@ components: type: string description: For drop down / radio button selections, UI will want to know the possible input values, it can also be used by other option types to limit input selections. + conditional_display: + $ref: '#/components/schemas/ConditionalDisplay' + description: Conditional display configuration. Specifies when this option should be hidden based on another option's + value. default_value: type: string description: The default value of the option, for example, value '443' for 'port' option. @@ -4083,6 +5287,12 @@ components: type: string description: The principal whose privileges we are changing. Only one of principal or principal_id should be specified, never both at the same time. + principal_id: + type: integer + description: An opaque internal ID that identifies the principal whose privileges should be removed. This field + is intended for removing privileges associated with a deleted user. When set, only the entries specified in the + remove field are processed; any entries in the add field will be rejected. Only one of principal or principal_id + should be specified, never both at the same time. remove: type: array items: @@ -4129,14 +5339,14 @@ components: type: string for_securable_type: $ref: '#/components/schemas/SecurableType' - description: Type of securables that the policy should take effect on. Only `TABLE` is supported at this moment. + description: Type of securables that the policy should take effect on. Only ``TABLE`` is supported at this moment. Required on create and optional on update. policy_type: $ref: '#/components/schemas/PolicyType' description: Type of the policy. Required on create. column_mask: $ref: '#/components/schemas/ColumnMaskOptions' - description: Options for column mask policies. Valid only if `policy_type` is `POLICY_TYPE_COLUMN_MASK`. Required + description: Options for column mask policies. Valid only if ``policy_type`` is ``POLICY_TYPE_COLUMN_MASK``. Required on create and optional on update. When specified on update, the new options will replace the existing options as a whole. comment: @@ -4148,11 +5358,19 @@ components: created_by: type: string description: Username of the user who created the policy. Output only. + deny: + $ref: '#/components/schemas/DenyOptions' + description: Options for deny policies. Valid only if ``policy_type`` is ``POLICY_TYPE_DENY``. Required on create + and optional on update. When specified on update, the new options will replace the existing options as a whole. except_principals: type: array items: type: string description: Optional list of user or group names that should be excluded from the policy. + grant: + $ref: '#/components/schemas/GrantOptions' + description: Options for grant policies. Valid only if ``policy_type`` is ``POLICY_TYPE_GRANT``. Required on create + and optional on update. When specified on update, the new options will replace the existing options as a whole. id: type: string description: Unique identifier of the policy. This field is output only and is generated by the system. @@ -4160,22 +5378,22 @@ components: type: array items: $ref: '#/components/schemas/MatchColumn' - description: Optional list of condition expressions used to match table columns. Only valid when `for_securable_type` - is `TABLE`. When specified, the policy only applies to tables whose columns satisfy all match conditions. + description: Optional list of condition expressions used to match table columns. Only valid when ``for_securable_type`` + is ``TABLE``. When specified, the policy only applies to tables whose columns satisfy all match conditions. name: type: string - description: Name of the policy. Required on create and optional on update. To rename the policy, set `name` to + description: Name of the policy. Required on create and optional on update. To rename the policy, set ``name`` to a different value on update. on_securable_fullname: type: string description: Full name of the securable on which the policy is defined. Required on create. on_securable_type: $ref: '#/components/schemas/SecurableType' - description: Type of the securable on which the policy is defined. Only `CATALOG`, `SCHEMA` and `TABLE` are supported - at this moment. Required on create. + description: Type of the securable on which the policy is defined. Only ``CATALOG``, ``SCHEMA`` and ``TABLE`` are + supported at this moment. Required on create. row_filter: $ref: '#/components/schemas/RowFilterOptions' - description: Options for row filter policies. Valid only if `policy_type` is `POLICY_TYPE_ROW_FILTER`. Required + description: Options for row filter policies. Valid only if ``policy_type`` is ``POLICY_TYPE_ROW_FILTER``. Required on create and optional on update. When specified on update, the new options will replace the existing options as a whole. updated_at: @@ -4184,6 +5402,13 @@ components: updated_by: type: string description: Username of the user who last modified the policy. Output only. + use_session_identity: + type: boolean + description: Temporary for migrating customers to session identity. Customers not currently using ABAC will not + be able to set this field to false and all new policies will have this field default to true. Existing customers + will have this field default to false, but can set it to true to opt in to session identity. after a grace period, + this field will be removed and all policies will use session identity. Only for row filter and column mask policies. + Not applicable to deny policies. when_condition: type: string description: Optional condition when the policy should take effect. @@ -4224,6 +5449,10 @@ components: properties: principal: type: string + principal_id: + type: integer + description: Unique identifier of the principal. For active principals, both ``principal`` and ``principal_id`` + are present. privileges: type: array items: @@ -4281,6 +5510,41 @@ components: description: |- R2 temporary credentials for API authentication. Read more at https://developers.cloudflare.com/r2/api/s3/tokens/. + RateLimit: + type: object + properties: + key: + $ref: '#/components/schemas/RateLimitRateLimitKey' + description: Scope key. Determines whether ``principal`` is required. + renewal_period: + $ref: '#/components/schemas/RateLimitRateLimitRenewalPeriod' + description: Renewal period. + principal: + type: string + description: 'Principal this limit applies to: user email, group name, or service principal application ID. Required + unless ``key`` is ``RATE_LIMIT_KEY_SERVICE``, ``RATE_LIMIT_KEY_USER_DEFAULT``, or ``RATE_LIMIT_KEY_REQUEST_TAG`` + (which must not set a principal).' + request_tag_key: + type: string + description: Request tag key this limit applies to. Required when ``key`` is ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden + otherwise. + request_tag_value: + type: string + description: Request tag value this limit applies to. Only valid when ``key`` is ``RATE_LIMIT_KEY_REQUEST_TAG``. + Leave unset to apply the limit to every value of ``request_tag_key`` (an any-value default); a set value is a + specific override for that value. + requests: + type: integer + description: Max requests allowed within a renewal period. Leave unset for no request limit. + tokens: + type: integer + description: Max tokens allowed within a renewal period. Leave unset for no token limit. + required: + - key + - renewal_period + description: |- + A rate limit applied to service requests. Leave ``requests`` or ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. RegenerateDashboardResponse: type: object properties: @@ -4374,8 +5638,6 @@ components: properties: browse_only: type: boolean - description: Indicates whether the principal is limited to retrieving metadata for the associated object through - the BROWSE privilege when include_browse is enabled in the request. catalog_name: type: string description: Name of parent catalog. @@ -4391,6 +5653,9 @@ components: created_by: type: string description: Username of schema creator. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the schema. effective_predictive_optimization_flag: $ref: '#/components/schemas/EffectivePredictiveOptimizationFlag' enable_predictive_optimization: @@ -4398,7 +5663,7 @@ components: description: Whether predictive optimization should be enabled for this object and objects under it. full_name: type: string - description: Full name of schema, in form of __catalog_name__.__schema_name__. + description: Full name of schema, in form of **catalog_name**.**schema_name**. metastore_id: type: string description: Unique identifier of parent metastore. @@ -4426,7 +5691,81 @@ components: updated_by: type: string description: Username of user who last modified schema. - description: 'Next ID: 45' + Secret: + type: object + properties: + name: + type: string + description: The name of the secret, relative to its parent schema. + catalog_name: + type: string + description: The name of the catalog where the schema and the secret reside. + schema_name: + type: string + description: The name of the schema where the secret resides. + value: + type: string + description: The secret value to store. This field is input-only and is not returned in responses — use the **effective_value** + field (via GetSecret with **include_value** set to true) to read the secret value. The maximum size is 60 KiB + (pre-encryption). Accepted content includes passwords, tokens, keys, and other sensitive credential data. + comment: + type: string + description: User-provided free-form text description of the secret. + create_time: + type: string + format: date-time + description: The time at which this secret was created. + created_by: + type: string + description: The principal that created the secret. + effective_owner: + type: string + description: The effective owner of the secret, which may differ from the directly-set **owner** due to inheritance. + effective_value: + type: string + description: The secret value. Only populated in responses when you have the **READ_SECRET** privilege and **include_value** + is set to true in the request. The maximum size is 60 KiB. + expire_time: + type: string + format: date-time + description: User-provided expiration time of the secret. This field indicates when the secret should no longer + be used and may be displayed as a warning in the UI. It is purely informational and does not trigger any automatic + actions or affect the secret's lifecycle. + full_name: + type: string + description: The three-level (fully qualified) name of the secret, in the form of **catalog_name.schema_name.secret_name**. + metastore_id: + type: string + description: Unique identifier of the metastore hosting the secret. + owner: + type: string + description: The owner of the secret. Defaults to the creating principal on creation. Can be updated to transfer + ownership of the secret to another principal. + update_time: + type: string + format: date-time + description: The time at which this secret was last updated. + updated_by: + type: string + description: The principal that last updated the secret. + required: + - name + - catalog_name + - schema_name + - value + description: |- + A secret stored in Unity Catalog. Secrets are three-level namespace objects + (catalog.schema.secret) that securely store sensitive credential data such as passwords, tokens, + and keys. + SecretDependency: + type: object + properties: + secret_full_name: + type: string + description: Full name of the dependent secret, in the form of **catalog_name**.**schema_name**.**secret_name**. + required: + - secret_full_name + description: A secret that is dependent on a SQL object. Securable: type: object properties: @@ -4570,13 +5909,13 @@ components: $ref: '#/components/schemas/PrimaryKeyConstraint' description: |- A table constraint, as defined by *one* of the following fields being set: - __primary_key_constraint__, __foreign_key_constraint__, __named_table_constraint__. + **primary_key_constraint**, **foreign_key_constraint**, **named_table_constraint**. TableDependency: type: object properties: table_full_name: type: string - description: Full name of the dependent table, in the form of __catalog_name__.__schema_name__.__table_name__. + description: Full name of the dependent table, in the form of **catalog_name**.**schema_name**.**table_name**. required: - table_full_name description: A table that is dependent on a SQL object. @@ -4601,7 +5940,7 @@ components: type: array items: $ref: '#/components/schemas/ColumnInfo' - description: The array of __ColumnInfo__ definitions of the table's columns. + description: The array of **ColumnInfo** definitions of the table's columns. comment: type: string description: User-provided free-form text description. @@ -4630,7 +5969,7 @@ components: $ref: '#/components/schemas/EncryptionDetails' full_name: type: string - description: Full name of table, in form of __catalog_name__.__schema_name__.__table_name__ + description: Full name of table, in form of **catalog_name**.**schema_name**.**table_name** metastore_id: type: string description: Unique identifier of parent metastore. @@ -4668,7 +6007,7 @@ components: type: array items: $ref: '#/components/schemas/TableConstraint' - description: 'List of table constraints. Note: this field is not set in the output of the __listTables__ API.' + description: 'List of table constraints. Note: this field is not set in the output of the **listTables** API.' table_id: type: string description: The unique identifier of the table. @@ -4682,13 +6021,13 @@ components: description: Username of user who last modified the table. view_definition: type: string - description: View definition SQL (when __table_type__ is **VIEW**, **MATERIALIZED_VIEW**, or **STREAMING_TABLE**) + description: View definition SQL (when **table_type** is **VIEW**, **MATERIALIZED_VIEW**, or **STREAMING_TABLE**) view_dependencies: $ref: '#/components/schemas/DependencyList' description: 'View dependencies (when table_type == **VIEW** or **MATERIALIZED_VIEW**, **STREAMING_TABLE**) - when DependencyList is None, the dependency is not provided; - when DependencyList is an empty list, the dependency is provided but is empty; - when DependencyList is not an empty list, dependencies are provided and recorded. - Note: this field is not set in the output of the __listTables__ API.' + Note: this field is not set in the output of the **listTables** API.' TableRowFilter: type: object properties: @@ -4720,6 +6059,16 @@ components: description: SecurableKindManifest of table, including capabilities the table has. table_type: $ref: '#/components/schemas/TableType' + TagIntrospectionExpression: + type: object + properties: + column_tag_value: + $ref: '#/components/schemas/ColumnTagValueExtraction' + description: Extracts the value of a column-level tag. + tag_value: + $ref: '#/components/schemas/TagValueExtraction' + description: Extracts the value of a securable-level tag. + description: An expression that introspects tags at query time. TagKeyValue: type: object properties: @@ -4728,6 +6077,15 @@ components: value: type: string description: value of the tag associated with the key, could be optional + TagValueExtraction: + type: object + properties: + tag_key: + type: string + description: 1024 matches the max_length on FunctionArgument.constant above. + required: + - tag_key + description: 'Extracts the value of a securable-level tag: get_tag_value("tagKey").' TemporaryCredentials: type: object properties: @@ -4741,6 +6099,8 @@ components: the credential given this expiration time. gcp_oauth_token: $ref: '#/components/schemas/GcpOauthToken' + r2_temp_credentials: + $ref: '#/components/schemas/R2Credentials' TriggeredUpdateStatus: type: object properties: @@ -4757,6 +6117,15 @@ components: description: |- Detailed status of an online table. Shown if the online table is in the ONLINE_TRIGGERED_UPDATE or the ONLINE_NO_PENDING_UPDATE state. + UcEncryptedToken: + type: object + properties: + encrypted_payload: + type: string + description: Stores encrypted ScopedCloudToken as a base64-encoded string + description: |- + Encrypted token used when we cannot downscope the cloud provider token appropriately See: + https://docs.google.com/document/d/1hEKDnSckuU5PIS798CtfqBElrMR6OJuR2wgz_BjhMSY UnassignResponse: type: object properties: {} @@ -4779,7 +6148,7 @@ components: description: The owner of the metastore. privilege_model_version: type: string - description: Privilege model version of the metastore, of the form `major.minor` (e.g., `1.0`). + description: Privilege model version of the metastore, of the form ``major.minor`` (e.g., ``1.0``). storage_root_credential_id: type: string description: UUID of storage credential to access the metastore storage_root. @@ -4908,6 +6277,15 @@ components: result: $ref: '#/components/schemas/ValidationResultResult' description: The results of the tested operation. + VolumeDependency: + type: object + properties: + volume_full_name: + type: string + description: Full name of the dependent volume, in the form of **catalog_name**.**schema_name**.**volume_name**. + required: + - volume_full_name + description: A volume that is dependent on a SQL object. VolumeInfo: type: object properties: @@ -4958,9 +6336,9 @@ components: description: The unique identifier of the volume volume_type: $ref: '#/components/schemas/VolumeType' - description: 'The type of the volume. An external volume is located in the specified external location. A managed + description: The type of the volume. An external volume is located in the specified external location. A managed volume is located in the default location which is specified by the parent schema, or the parent catalog, or the - Metastore. [Learn more] [Learn more]: https://docs.databricks.com/aws/en/volumes/managed-vs-external' + Metastore. `Learn more `__ WorkspaceBinding: type: object properties: @@ -4971,6 +6349,14 @@ components: description: One of READ_WRITE/READ_ONLY. Default is READ_WRITE. required: - workspace_id + AgentServiceAgentServiceType: + type: string + x-enum: + - AGENT_SERVICE_TYPE_EXTERNAL + description: |- + Type of agent service. Determines how the backing agent surface is provisioned and how + invocation routes to the runtime. Nested to scope enum values (guardrail: + avoid-generic-enum-names). ArtifactType: type: string x-enum: @@ -4983,6 +6369,7 @@ components: x-enum: - ISOLATED - OPEN + - OPEN_IN_ACCOUNT description: |- Create a collection of name/value pairs. @@ -5041,6 +6428,7 @@ components: - DATE - DECIMAL - DOUBLE + - FILE - FLOAT - GEOGRAPHY - GEOMETRY @@ -5052,7 +6440,9 @@ components: - SHORT - STRING - STRUCT + - TABLEREF_TYPE - TABLE_TYPE + - TIME - TIMESTAMP - TIMESTAMP_NTZ - USER_DEFINED_TYPE @@ -5097,27 +6487,173 @@ components: ConnectionType: type: string x-enum: + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AMPLITUDE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS + - BIGLAKE - BIGQUERY + - CERIDIAN_DAYFORCE + - CONFLUENCE - DATABRICKS + - DELIGHTED + - DYNAMICS365 + - EPIC_CLARITY + - FRESHSERVICE + - FRONT - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB - GLUE + - GMAIL + - GONG + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_CLOUD_LAKEHOUSE + - GOOGLE_WORKSPACE + - GURU + - HIBOB - HIVE_METASTORE - HTTP + - HUBSPOT + - ICEBERG_REST + - IRONCLAD + - JDBC + - KINESIS + - LINEAR + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MONDAY_COM - MYSQL + - NETSKOPE_LOGS + - NOTION - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PALANTIR + - PARTNERSTACK + - PENDO - POSTGRESQL - POWER_BI + - PUBSUB + - QUICKBOOKS - REDSHIFT - SALESFORCE - SALESFORCE_DATA_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID - SERVICENOW + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SMARTSHEET + - SNAPCHAT_ADS - SNOWFLAKE + - SPLUNK - SQLDW - SQLSERVER + - SQUARE - TERADATA - UNKNOWN_CONNECTION_TYPE + - VERKADA - WORKDAY_RAAS - description: 'Next Id: 72' + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + ConversionInfoState: + type: string + x-enum: + - COMPLETED + - IN_PROGRESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. CreateFunctionParameterStyle: type: string x-enum: @@ -5333,7 +6869,11 @@ components: - ANY_STATIC_CREDENTIAL - BEARER_TOKEN - EDGEGRID_AKAMAI + - GENERIC_TOKEN_EXCHANGE + - INLINE_YAML - OAUTH_ACCESS_TOKEN + - OAUTH_DCR + - OAUTH_GOOGLE_SERVICE_ACCOUNT - OAUTH_M2M - OAUTH_MTLS - OAUTH_REFRESH_TOKEN @@ -5346,7 +6886,43 @@ components: - SSWS_TOKEN - UNKNOWN_CREDENTIAL_TYPE - USERNAME_PASSWORD - description: 'Next Id: 17' + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. DataSourceFormat: type: string x-enum: @@ -5466,6 +7042,48 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + DrReplicationStatus: + type: string + x-enum: + - DR_REPLICATION_STATUS_PRIMARY + - DR_REPLICATION_STATUS_SECONDARY + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. EffectivePredictiveOptimizationFlagInheritedFromType: @@ -5808,6 +7426,7 @@ components: x-enum: - ISOLATION_MODE_ISOLATED - ISOLATION_MODE_OPEN + - ISOLATION_MODE_OPEN_IN_ACCOUNT description: |- Create a collection of name/value pairs. @@ -5887,11 +7506,66 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + ListMcpServicesRequestView: + type: string + x-enum: + - BASIC + - FULL + description: |- + Controls which fields are populated on each McpService in the response. The server treats unset + / VIEW_UNSPECIFIED as BASIC. Callers needing the full configuration must request it explicitly + with ``view = FULL``. + ListModelProviderServicesRequestView: + type: string + x-enum: + - BASIC + - FULL + description: |- + Controls which fields are populated on each ModelProviderService in the response. The server + treats unset / VIEW_UNSPECIFIED as BASIC. Callers needing the full configuration must request it + explicitly with ``view = FULL``. + ListModelServicesRequestView: + type: string + x-enum: + - BASIC + - FULL + description: |- + Controls which fields are populated on each ModelService in the response. The server treats + unset / VIEW_UNSPECIFIED as BASIC. Callers needing the full configuration must request it + explicitly with ``view = FULL``. MatchType: type: string x-enum: - PREFIX_MATCH description: The artifact pattern matching type + ModelProviderServiceConfigAnthropicProviderRelayedConfigAnthropicRelayedPlanType: + type: string + x-enum: + - ANTHROPIC_RELAYED_PLAN_TYPE_MAX + - ANTHROPIC_RELAYED_PLAN_TYPE_TEAM_ENTERPRISE + description: |- + Which Anthropic subscription tier the relayed OAuth token belongs to. Immutable after Create + (switching tiers changes which governance controls the platform enforces). Only MAX and + TEAM_ENTERPRISE differ in the governance surface the gateway can enforce, not in how the token + is relayed. + ModelProviderServiceConfigExternalModelProviderType: + type: string + x-enum: + - EXTERNAL_MODEL_PROVIDER_TYPE_AMAZON_BEDROCK + - EXTERNAL_MODEL_PROVIDER_TYPE_ANTHROPIC + - EXTERNAL_MODEL_PROVIDER_TYPE_AZURE_OPENAI + - EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM + - EXTERNAL_MODEL_PROVIDER_TYPE_GEMINI_ENTERPRISE + - EXTERNAL_MODEL_PROVIDER_TYPE_MICROSOFT_FOUNDRY + - EXTERNAL_MODEL_PROVIDER_TYPE_OPENAI + description: External LLM provider for an EXTERNAL_FOUNDATION_MODEL destination. + ModelServiceConfigDestinationConfigDestinationType: + type: string + x-enum: + - DESTINATION_TYPE_EXTERNAL_FOUNDATION_MODEL + - DESTINATION_TYPE_PAY_PER_TOKEN_FOUNDATION_MODEL + - DESTINATION_TYPE_PROVISIONED_THROUGHPUT_FOUNDATION_MODEL + description: Backing-model category for a model service destination. ModelVersionInfoStatus: type: string x-enum: @@ -6044,9 +7718,11 @@ components: ``"CUSTOM_METRIC_TYPE_DRIFT"``. The ``"CUSTOM_METRIC_TYPE_AGGREGATE"`` and ``"CUSTOM_METRIC_TYPE_DERIVED"`` metrics are computed on a single table, whereas the ``"CUSTOM_METRIC_TYPE_DRIFT"`` compare metrics across baseline and input table, or across the - two consecutive time windows. - CUSTOM_METRIC_TYPE_AGGREGATE: only depend on the existing - columns in your table - CUSTOM_METRIC_TYPE_DERIVED: depend on previously computed aggregate - metrics - CUSTOM_METRIC_TYPE_DRIFT: depend on previously computed aggregate or derived metrics + two consecutive time windows. + + - CUSTOM_METRIC_TYPE_AGGREGATE: only depend on the existing columns in your table + - CUSTOM_METRIC_TYPE_DERIVED: depend on previously computed aggregate metrics + - CUSTOM_METRIC_TYPE_DRIFT: depend on previously computed aggregate or derived metrics MonitorRefreshInfoState: type: string x-enum: @@ -6135,6 +7811,7 @@ components: - OPTION_MULTILINE_STRING - OPTION_NUMBER - OPTION_SERVICE_CREDENTIAL + - OPTION_STORAGE_CREDENTIAL - OPTION_STRING description: |- Type of the option, we purposely follow JavaScript types so that the UI can map the options to @@ -6187,6 +7864,8 @@ components: type: string x-enum: - POLICY_TYPE_COLUMN_MASK + - POLICY_TYPE_DENY + - POLICY_TYPE_GRANT - POLICY_TYPE_ROW_FILTER description: |- Create a collection of name/value pairs. @@ -6276,40 +7955,70 @@ components: - APPLY_TAG - BROWSE - CREATE + - CREATE_CASE_COLLECTION - CREATE_CATALOG - CREATE_CLEAN_ROOM - CREATE_CONNECTION + - CREATE_DATASOURCE - CREATE_EXTERNAL_LOCATION - CREATE_EXTERNAL_TABLE - CREATE_EXTERNAL_VOLUME + - CREATE_FEATURE + - CREATE_FLOW - CREATE_FOREIGN_CATALOG - CREATE_FOREIGN_SECURABLE - CREATE_FUNCTION - CREATE_MANAGED_STORAGE - CREATE_MATERIALIZED_VIEW + - CREATE_MEMORY_STORE - CREATE_MODEL - CREATE_PROVIDER - CREATE_RECIPIENT + - CREATE_RULE - CREATE_SCHEMA + - CREATE_SEMANTIC_GRAPH + - CREATE_SERVICE - CREATE_SERVICE_CREDENTIAL - CREATE_SHARE + - CREATE_SKILL - CREATE_STORAGE_CREDENTIAL + - CREATE_STREAM - CREATE_TABLE - CREATE_VIEW - CREATE_VOLUME + - DELETE + - DELETE_EVENTS + - DELETE_SECURITY_DATA - EXECUTE - EXECUTE_CLEAN_ROOM_TASK - EXTERNAL_USE_SCHEMA + - INSERT + - INSERT_SECURITY_DATA - MANAGE + - MANAGE_ACCESS + - MANAGE_ACCESS_CONTROL - MANAGE_ALLOWLIST + - MANAGE_GRANTS - MODIFY - MODIFY_CLEAN_ROOM + - READ_EVENTS + - READ_FEATURE - READ_FILES + - READ_FLOW + - READ_MEMORY_STORE + - READ_METADATA - READ_PRIVATE_FILES + - READ_SECURITY_DATA + - READ_SEMANTIC_GRAPH + - READ_SKILL + - READ_STREAM - READ_VOLUME - REFRESH - SELECT - SET_SHARE_PERMISSION + - UPDATE + - UPDATE_EVENTS + - UPDATE_SECURITY_DATA - USAGE - USE_CATALOG - USE_CONNECTION @@ -6318,8 +8027,16 @@ components: - USE_RECIPIENT - USE_SCHEMA - USE_SHARE + - USE_VOLUME + - VIEW_ADMIN_METADATA + - VIEW_METADATA + - VIEW_OBJECT - WRITE_FILES + - WRITE_FLOW + - WRITE_MEMORY_STORE - WRITE_PRIVATE_FILES + - WRITE_SEMANTIC_GRAPH + - WRITE_SKILL - WRITE_VOLUME description: |- Create a collection of name/value pairs. @@ -6404,9 +8121,49 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + RateLimitRateLimitKey: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: Scope key for a rate limit. + RateLimitRateLimitRenewalPeriod: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period for a rate limit. + SchemaFilterType: + type: string + x-enum: + - SCHEMA_FILTER_TYPE_EXCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE_ALL + description: Determines how a FOREIGN catalog's schema filter items are interpreted. SecurableKind: type: string x-enum: + - CATALOG_FOREIGN_BIGLAKE + - CATALOG_FOREIGN_GOOGLE_CLOUD_LAKEHOUSE + - CATALOG_FOREIGN_ICEBERG_REST + - CONNECTION_BIGLAKE_SERVICE_ACCOUNT + - CONNECTION_COMMUNITY_OAUTH_M2M + - CONNECTION_COMMUNITY_OAUTH_U2M + - CONNECTION_COMMUNITY_OAUTH_U2M_MAPPING + - CONNECTION_GOOGLE_CLOUD_LAKEHOUSE_OAUTH_U2M + - CONNECTION_GOOGLE_CLOUD_LAKEHOUSE_SERVICE_ACCOUNT + - CONNECTION_ICEBERG_REST_BEARER_TOKEN + - CONNECTION_ICEBERG_REST_OAUTH_M2M + - RECIPIENT_EMAIL + - RECIPIENT_EMAIL_DATABRICKS + - RECIPIENT_EMAIL_TOKEN + - SCHEMA_FOREIGN_BIGLAKE + - SCHEMA_FOREIGN_GOOGLE_CLOUD_LAKEHOUSE + - SCHEMA_FOREIGN_ICEBERG_REST - TABLE_DB_STORAGE - TABLE_DELTA - TABLE_DELTASHARING @@ -6417,6 +8174,7 @@ components: - TABLE_DELTA_ICEBERG_MANAGED - TABLE_DELTA_UNIFORM_HUDI_EXTERNAL - TABLE_DELTA_UNIFORM_ICEBERG_EXTERNAL + - TABLE_DELTA_UNIFORM_ICEBERG_EXTERNAL_DELTASHARING - TABLE_DELTA_UNIFORM_ICEBERG_FOREIGN_DELTASHARING - TABLE_DELTA_UNIFORM_ICEBERG_FOREIGN_HIVE_METASTORE_EXTERNAL - TABLE_DELTA_UNIFORM_ICEBERG_FOREIGN_HIVE_METASTORE_MANAGED @@ -6424,9 +8182,12 @@ components: - TABLE_EXTERNAL - TABLE_FEATURE_STORE - TABLE_FEATURE_STORE_EXTERNAL + - TABLE_FOREIGN_BIGLAKE - TABLE_FOREIGN_BIGQUERY - TABLE_FOREIGN_DATABRICKS - TABLE_FOREIGN_DELTASHARING + - TABLE_FOREIGN_DELTA_DELTASHARING + - TABLE_FOREIGN_GOOGLE_CLOUD_LAKEHOUSE - TABLE_FOREIGN_HIVE_METASTORE - TABLE_FOREIGN_HIVE_METASTORE_DBFS_EXTERNAL - TABLE_FOREIGN_HIVE_METASTORE_DBFS_MANAGED @@ -6438,10 +8199,12 @@ components: - TABLE_FOREIGN_HIVE_METASTORE_SHALLOW_CLONE_EXTERNAL - TABLE_FOREIGN_HIVE_METASTORE_SHALLOW_CLONE_MANAGED - TABLE_FOREIGN_HIVE_METASTORE_VIEW + - TABLE_FOREIGN_ICEBERG_REST - TABLE_FOREIGN_MONGODB - TABLE_FOREIGN_MYSQL - TABLE_FOREIGN_NETSUITE - TABLE_FOREIGN_ORACLE + - TABLE_FOREIGN_PALANTIR - TABLE_FOREIGN_POSTGRESQL - TABLE_FOREIGN_REDSHIFT - TABLE_FOREIGN_SALESFORCE @@ -6470,7 +8233,43 @@ components: - TABLE_SYSTEM_DELTASHARING - TABLE_VIEW - TABLE_VIEW_DELTASHARING - description: 'Latest kind: CONNECTION_GOOGLE_DRIVE_SERVICE_ACCOUNT = 301; Next id: 302' + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. SecurableType: type: string x-enum: @@ -6499,7 +8298,12 @@ components: - SPECIAL_DESTINATION_CONNECTION_OWNER - SPECIAL_DESTINATION_CREDENTIAL_OWNER - SPECIAL_DESTINATION_EXTERNAL_LOCATION_OWNER + - SPECIAL_DESTINATION_FUNCTION_OWNER - SPECIAL_DESTINATION_METASTORE_OWNER + - SPECIAL_DESTINATION_REGISTERED_MODEL_OWNER + - SPECIAL_DESTINATION_SCHEMA_OWNER + - SPECIAL_DESTINATION_TABLE_OWNER + - SPECIAL_DESTINATION_VOLUME_OWNER description: |- Create a collection of name/value pairs. @@ -6763,6 +8567,48 @@ components: - PASS - SKIP description: A enum represents the result of the file operation + VolumeOperation: + type: string + x-enum: + - READ_VOLUME + - WRITE_VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. VolumeType: type: string x-enum: @@ -6811,8 +8657,8 @@ components: - BINDING_TYPE_READ_ONLY - BINDING_TYPE_READ_WRITE description: |- - Using `BINDING_TYPE_` prefix here to avoid conflict with `TableOperation` enum in - `credentials_common.proto`. + Using ``BINDING_TYPE_`` prefix here to avoid conflict with ``TableOperation`` enum in + ``credentials_common.proto``. x-stackQL-resources: account_metastore_assignments: id: databricks_account.catalog.account_metastore_assignments @@ -7017,7 +8863,7 @@ components: response: mediaType: application/json openAPIDocKey: '200' - credentials_generate_temporary_service_credential: + generate_temporary_service_credential: config: requestBodyTranslate: algorithm: naive @@ -7026,7 +8872,7 @@ components: response: mediaType: application/json openAPIDocKey: '200' - credentials_validate_credential: + validate: config: requestBodyTranslate: algorithm: naive @@ -7041,8 +8887,6 @@ components: - $ref: '#/components/x-stackQL-resources/credentials/methods/list_credentials' insert: - $ref: '#/components/x-stackQL-resources/credentials/methods/create_credential' - - $ref: '#/components/x-stackQL-resources/credentials/methods/credentials_generate_temporary_service_credential' - - $ref: '#/components/x-stackQL-resources/credentials/methods/credentials_validate_credential' update: - $ref: '#/components/x-stackQL-resources/credentials/methods/update_credential' delete: diff --git a/providers/src/databricks_account/v00.00.00000/services/disasterrecovery.yaml b/providers/src/databricks_account/v00.00.00000/services/disasterrecovery.yaml new file mode 100644 index 00000000..8159f22f --- /dev/null +++ b/providers/src/databricks_account/v00.00.00000/services/disasterrecovery.yaml @@ -0,0 +1,1433 @@ +openapi: 3.0.0 +info: + title: Databricks Disasterrecovery API (account) + description: >- + OpenAPI specification for the Databricks disasterrecovery service + (account-level APIs), generated from the Databricks Python SDK. + version: 0.1.0 + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' + x-stackql-sdk-namespace: databricks.sdk.service.disasterrecovery +servers: + - url: https://accounts.cloud.databricks.com +paths: + /api/disaster-recovery/v1/{parent}/failover-groups: + post: + operationId: disaster_recovery_create_failover_group + summary: Create a new failover group. + tags: + - disasterrecovery + - disaster_recovery + description: |- + Create a new failover group. + + :param parent: str + The parent resource. Format: accounts/{account_id}. + :param failover_group: :class:`FailoverGroup` + The failover group to create. + :param failover_group_id: str + Client-provided identifier for the failover group. Used to construct the resource name as + {parent}/failover-groups/{failover_group_id}. + :param validate_only: bool (optional) + When true, validates the request without creating the failover group. + + :returns: :class:`FailoverGroup` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent resource. Format: accounts/{account_id}.' + - name: failover_group_id + in: query + required: true + schema: + type: string + description: >- + Client-provided identifier for the failover group. Used to construct + the resource name as {parent}/failover-groups/{failover_group_id}. + - name: validate_only + in: query + required: false + schema: + type: boolean + description: >- + When true, validates the request without creating the failover + group. + requestBody: + content: + application/json: + schema: + type: object + properties: + failover_group: + type: object + properties: + regions: + type: array + items: + type: string + description: >- + List of all regions participating in this failover + group. + workspace_sets: + type: array + items: + type: object + properties: + name: + type: string + description: Resource name for this workspace set. + workspace_ids: + type: array + items: + type: string + description: >- + Workspace IDs in this set. The system derives and + validates regions. All workspaces must be in the + Mission Critical tier. + asset_replication_config: + type: object + properties: + enable_jobs: + type: boolean + description: Whether to replicate jobs. + enable_clusters: + type: boolean + description: >- + Whether to replicate clusters (including + cluster policies, instance profiles, global + init scripts, and worker environment + overlays). + enable_warehouse: + type: boolean + description: Whether to replicate SQL warehouses. + enable_sql_workspace: + type: boolean + description: Whether to replicate the SQL workspace. + enable_libraries: + type: boolean + description: Whether to replicate libraries. + enable_notebooks: + type: boolean + description: Whether to replicate notebooks. + enable_files: + type: boolean + description: Whether to replicate workspace files. + enable_queries: + type: boolean + description: Whether to replicate SQL queries. + enable_dashboards: + type: boolean + description: Whether to replicate dashboards. + required: + - enable_jobs + - enable_clusters + - enable_warehouse + - enable_sql_workspace + - enable_libraries + - enable_notebooks + - enable_files + - enable_queries + - enable_dashboards + description: >- + Per-asset-type control over which workspace assets + are replicated. Applies only when + replicate_workspace_assets is true. When omitted + while control plane DR is enabled, every asset + type is replicated. + replicate_workspace_assets: + type: boolean + description: >- + Whether to enable control plane DR (notebooks, + jobs, clusters, etc.) for this set. Defaults to + false. + stable_url_names: + type: array + items: + type: string + description: >- + Resource names of stable URLs associated with this + workspace set. Format: + accounts/{account_id}/stable-urls/{stable_url_id}. + The referenced stable URLs must already exist (via + CreateStableUrl). + required: + - name + - workspace_ids + description: >- + A set of workspaces that replicate to each other + across regions. + description: >- + Workspace sets, each containing workspaces that + replicate to each other. + initial_primary_region: + type: string + description: >- + Initial primary region. Used only in Create requests to + set the starting primary region. Not returned in + responses. + create_time: + type: string + format: date-time + description: Time at which this failover group was created. + effective_primary_region: + type: string + description: >- + Current effective primary region. Replication flows FROM + workspaces in this region. Changes after a successful + failover. + etag: + type: string + description: >- + Opaque version string for optimistic locking. + Server-generated and returned in responses. + name: + type: string + description: >- + Fully qualified resource name in the format + accounts/{account_id}/failover-groups/{failover_group_id}. + replication_point: + type: string + format: date-time + description: >- + The latest point in time to which data has been + replicated. + state: + type: string + x-enum: + - ACTIVE + - CREATING + - CREATION_FAILED + - DELETING + - DELETION_FAILED + - FAILING_OVER + - FAILOVER_FAILED + - INITIAL_REPLICATION + description: Aggregate state of the failover group. + unity_catalog_assets: + type: object + properties: + catalogs: + type: array + items: + type: object + properties: + name: + type: string + description: The name of the UC catalog to replicate. + required: + - name + description: A Unity Catalog catalog to replicate. + description: UC catalogs to replicate. + data_replication_workspace_set: + type: string + description: >- + The workspace set whose workspaces will be used for + data replication of all UC catalogs' underlying + storage. + location_mappings: + type: array + items: + type: object + properties: + name: + type: string + description: Resource name for this location. + uri_by_region: + type: array + items: + type: object + properties: + region: + type: string + description: The region name. + uri: + type: string + description: The storage URI for this region. + required: + - region + - uri + description: >- + A single entry in a location mapping, + mapping a region to a storage URI. Used + instead of + map for proto2 compatibility. + description: >- + URI for each region. Each entry maps a region + name to a storage URI. + required: + - name + - uri_by_region + description: >- + A location mapping identified by a name, with URIs + per region. The system derives replication + direction from effective_primary_region. + description: >- + Location mappings - storage URI per region for each + location. + required: + - catalogs + - data_replication_workspace_set + description: Unity Catalog replication configuration. + update_time: + type: string + format: date-time + description: Time at which this failover group was last modified. + required: + - regions + - workspace_sets + - initial_primary_region + description: The failover group to create. + required: + - failover_group + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FailoverGroup' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + get: + operationId: disaster_recovery_list_failover_groups + summary: List failover groups. + tags: + - disasterrecovery + - disaster_recovery + description: >- + List failover groups. + + + List entries are abbreviated: ``state`` and ``replication_point`` are + not populated. Call + + GetFailoverGroup to retrieve the full resource. + + + :param parent: str + The parent resource. Format: accounts/{account_id}. + :param page_size: int (optional) + Maximum number of failover groups to return per page: + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0 or unset, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned. + :param page_token: str (optional) + Page token received from a previous ListFailoverGroups call. Provide this to retrieve the subsequent + page. + + :returns: Iterator over :class:`FailoverGroup` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent resource. Format: accounts/{account_id}.' + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Maximum number of failover groups to return per page: - when set to + a value greater than 0, the page length is the minimum of this value + and a server configured value; - when set to 0 or unset, the page + length is set to a server configured value (recommended); - when set + to a value less than 0, an invalid parameter error is returned. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + Page token received from a previous ListFailoverGroups call. Provide + this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListFailoverGroupsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + /api/disaster-recovery/v1/{parent}/stable-urls: + post: + operationId: disaster_recovery_create_stable_url + summary: Create a new stable URL. + tags: + - disasterrecovery + - disaster_recovery + description: |- + Create a new stable URL. + + :param parent: str + The parent resource. Format: accounts/{account_id}. + :param stable_url: :class:`StableUrl` + The stable URL to create. + :param stable_url_id: str + Client-provided identifier for the stable URL. Used to construct the resource name as + {parent}/stable-urls/{stable_url_id}. + :param validate_only: bool (optional) + When true, validates the request without creating the stable URL. + + :returns: :class:`StableUrl` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent resource. Format: accounts/{account_id}.' + - name: stable_url_id + in: query + required: true + schema: + type: string + description: >- + Client-provided identifier for the stable URL. Used to construct the + resource name as {parent}/stable-urls/{stable_url_id}. + - name: validate_only + in: query + required: false + schema: + type: boolean + description: When true, validates the request without creating the stable URL. + requestBody: + content: + application/json: + schema: + type: object + properties: + stable_url: + type: object + properties: + initial_workspace_id: + type: string + description: >- + The workspace this stable URL is initially bound to. + Used only in Create requests to associate the stable URL + with a workspace. Not returned in responses. + effective_workspace_id: + type: string + description: >- + The workspace this stable URL currently routes to. Set + to ``initial_workspace_id`` at creation, advanced to the + failover group's primary while attached (including + across a failover), and preserved when the stable URL is + detached from its failover group. Read this to see where + an unattached stable URL points: after a failover + followed by a detach it reflects the post-failover + primary, not ``initial_workspace_id``. + failover_group_name: + type: string + description: >- + Fully qualified resource name of the FailoverGroup this + stable URL is currently linked to, in the format + ``accounts/{account_id}/failover-groups/{failover_group_id}``. + Empty when the stable URL is not attached to any + failover group. + name: + type: string + description: >- + Fully qualified resource name. Format: + accounts/{account_id}/stable-urls/{stable_url_id}. + stable_workspace_id: + type: string + description: >- + The stable workspace ID for this stable URL. Generated + on creation and immutable thereafter; identifies the URL + across failovers and is the same value embedded in the + ``url`` (as the ``w=`` query parameter for SPOG URLs, or + in the ``conn-`` hostname for Private-Link URLs). + url: + type: string + description: >- + The stable URL endpoint. Generated on creation and + immutable thereafter. For non-Private-Link workspaces + this is ``https:///?w=``. For + Private-Link workspaces this is the per-connection + hostname. + required: + - initial_workspace_id + description: The stable URL to create. + required: + - stable_url + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/StableUrl' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + get: + operationId: disaster_recovery_list_stable_urls + summary: List stable URLs for an account. + tags: + - disasterrecovery + - disaster_recovery + description: |- + List stable URLs for an account. + + :param parent: str + The parent resource. Format: accounts/{account_id}. + :param page_size: int (optional) + Maximum number of stable URLs to return per page: + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0 or unset, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned. + :param page_token: str (optional) + Page token received from a previous ListStableUrls call. Provide this to retrieve the subsequent + page. + + :returns: Iterator over :class:`StableUrl` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent resource. Format: accounts/{account_id}.' + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Maximum number of stable URLs to return per page: - when set to a + value greater than 0, the page length is the minimum of this value + and a server configured value; - when set to 0 or unset, the page + length is set to a server configured value (recommended); - when set + to a value less than 0, an invalid parameter error is returned. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + Page token received from a previous ListStableUrls call. Provide + this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListStableUrlsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + /api/disaster-recovery/v1/{name}: + delete: + operationId: disaster_recovery_delete_stable_url + summary: Delete a stable URL. + tags: + - disasterrecovery + - disaster_recovery + description: |- + Delete a stable URL. + + :param name: str + The fully qualified resource name. Format: accounts/{account_id}/stable-urls/{stable_url_id}. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The fully qualified resource name. Format: + accounts/{account_id}/stable-urls/{stable_url_id}. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + get: + operationId: disaster_recovery_get_stable_url + summary: Get a stable URL. + tags: + - disasterrecovery + - disaster_recovery + description: |- + Get a stable URL. + + :param name: str + The fully qualified resource name. Format: accounts/{account_id}/stable-urls/{stable_url_id}. + + :returns: :class:`StableUrl` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The fully qualified resource name. Format: + accounts/{account_id}/stable-urls/{stable_url_id}. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/StableUrl' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + patch: + operationId: disaster_recovery_update_failover_group + summary: Update a failover group. + tags: + - disasterrecovery + - disaster_recovery + description: |- + Update a failover group. + + :param name: str + Fully qualified resource name in the format + accounts/{account_id}/failover-groups/{failover_group_id}. + :param failover_group: :class:`FailoverGroup` + The failover group with updated fields. The name field identifies the resource and is populated from + the URL path. + :param update_mask: FieldMask + Comma-separated list of fields to update. + :param etag: str (optional) + Optional opaque version string for optimistic locking, obtained from a prior read of the failover + group. If provided, the update is rejected unless it matches the failover group's current etag. If + omitted, the update proceeds without an optimistic-lock check. + + :returns: :class:`FailoverGroup` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Fully qualified resource name in the format + accounts/{account_id}/failover-groups/{failover_group_id}. + - name: update_mask + in: query + required: true + schema: + type: object + description: Comma-separated list of fields to update. + - name: etag + in: query + required: false + schema: + type: string + description: >- + Optional opaque version string for optimistic locking, obtained from + a prior read of the failover group. If provided, the update is + rejected unless it matches the failover group's current etag. If + omitted, the update proceeds without an optimistic-lock check. + requestBody: + content: + application/json: + schema: + type: object + properties: + failover_group: + type: object + properties: + regions: + type: array + items: + type: string + description: >- + List of all regions participating in this failover + group. + workspace_sets: + type: array + items: + type: object + properties: + name: + type: string + description: Resource name for this workspace set. + workspace_ids: + type: array + items: + type: string + description: >- + Workspace IDs in this set. The system derives and + validates regions. All workspaces must be in the + Mission Critical tier. + asset_replication_config: + type: object + properties: + enable_jobs: + type: boolean + description: Whether to replicate jobs. + enable_clusters: + type: boolean + description: >- + Whether to replicate clusters (including + cluster policies, instance profiles, global + init scripts, and worker environment + overlays). + enable_warehouse: + type: boolean + description: Whether to replicate SQL warehouses. + enable_sql_workspace: + type: boolean + description: Whether to replicate the SQL workspace. + enable_libraries: + type: boolean + description: Whether to replicate libraries. + enable_notebooks: + type: boolean + description: Whether to replicate notebooks. + enable_files: + type: boolean + description: Whether to replicate workspace files. + enable_queries: + type: boolean + description: Whether to replicate SQL queries. + enable_dashboards: + type: boolean + description: Whether to replicate dashboards. + required: + - enable_jobs + - enable_clusters + - enable_warehouse + - enable_sql_workspace + - enable_libraries + - enable_notebooks + - enable_files + - enable_queries + - enable_dashboards + description: >- + Per-asset-type control over which workspace assets + are replicated. Applies only when + replicate_workspace_assets is true. When omitted + while control plane DR is enabled, every asset + type is replicated. + replicate_workspace_assets: + type: boolean + description: >- + Whether to enable control plane DR (notebooks, + jobs, clusters, etc.) for this set. Defaults to + false. + stable_url_names: + type: array + items: + type: string + description: >- + Resource names of stable URLs associated with this + workspace set. Format: + accounts/{account_id}/stable-urls/{stable_url_id}. + The referenced stable URLs must already exist (via + CreateStableUrl). + required: + - name + - workspace_ids + description: >- + A set of workspaces that replicate to each other + across regions. + description: >- + Workspace sets, each containing workspaces that + replicate to each other. + initial_primary_region: + type: string + description: >- + Initial primary region. Used only in Create requests to + set the starting primary region. Not returned in + responses. + create_time: + type: string + format: date-time + description: Time at which this failover group was created. + effective_primary_region: + type: string + description: >- + Current effective primary region. Replication flows FROM + workspaces in this region. Changes after a successful + failover. + etag: + type: string + description: >- + Opaque version string for optimistic locking. + Server-generated and returned in responses. + name: + type: string + description: >- + Fully qualified resource name in the format + accounts/{account_id}/failover-groups/{failover_group_id}. + replication_point: + type: string + format: date-time + description: >- + The latest point in time to which data has been + replicated. + state: + type: string + x-enum: + - ACTIVE + - CREATING + - CREATION_FAILED + - DELETING + - DELETION_FAILED + - FAILING_OVER + - FAILOVER_FAILED + - INITIAL_REPLICATION + description: Aggregate state of the failover group. + unity_catalog_assets: + type: object + properties: + catalogs: + type: array + items: + type: object + properties: + name: + type: string + description: The name of the UC catalog to replicate. + required: + - name + description: A Unity Catalog catalog to replicate. + description: UC catalogs to replicate. + data_replication_workspace_set: + type: string + description: >- + The workspace set whose workspaces will be used for + data replication of all UC catalogs' underlying + storage. + location_mappings: + type: array + items: + type: object + properties: + name: + type: string + description: Resource name for this location. + uri_by_region: + type: array + items: + type: object + properties: + region: + type: string + description: The region name. + uri: + type: string + description: The storage URI for this region. + required: + - region + - uri + description: >- + A single entry in a location mapping, + mapping a region to a storage URI. Used + instead of + map for proto2 compatibility. + description: >- + URI for each region. Each entry maps a region + name to a storage URI. + required: + - name + - uri_by_region + description: >- + A location mapping identified by a name, with URIs + per region. The system derives replication + direction from effective_primary_region. + description: >- + Location mappings - storage URI per region for each + location. + required: + - catalogs + - data_replication_workspace_set + description: Unity Catalog replication configuration. + update_time: + type: string + format: date-time + description: Time at which this failover group was last modified. + required: + - regions + - workspace_sets + - initial_primary_region + description: >- + The failover group with updated fields. The name field + identifies the resource and is populated from the URL path. + required: + - failover_group + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FailoverGroup' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI + /api/disaster-recovery/v1/{name}/failover: + post: + operationId: disaster_recovery_failover_failover_group + summary: Initiate a failover to a new primary region. + tags: + - disasterrecovery + - disaster_recovery + description: |- + Initiate a failover to a new primary region. + + :param name: str + The fully qualified resource name of the failover group to failover. Format: + accounts/{account_id}/failover-groups/{failover_group_id}. + :param target_primary_region: str + The target primary region. Must be one of the participating regions and different from the current + effective_primary_region. Serves as an idempotency check. + :param failover_type: :class:`FailoverFailoverGroupRequestFailoverType` + The type of failover to perform. + :param etag: str (optional) + Opaque version string for optimistic locking. If provided, must match the current etag. If omitted, + the failover proceeds regardless of current state. + + :returns: :class:`FailoverGroup` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The fully qualified resource name of the failover group to failover. + Format: accounts/{account_id}/failover-groups/{failover_group_id}. + requestBody: + content: + application/json: + schema: + type: object + properties: + target_primary_region: + type: string + description: >- + The target primary region. Must be one of the participating + regions and different from the current + effective_primary_region. Serves as an idempotency check. + failover_type: + type: string + x-enum: + - FORCED + description: The type of failover to perform. + etag: + type: string + description: >- + Opaque version string for optimistic locking. If provided, + must match the current etag. If omitted, the failover + proceeds regardless of current state. + required: + - target_primary_region + - failover_type + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FailoverGroup' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DisasterRecoveryAPI +components: + schemas: + AssetReplicationConfig: + type: object + properties: + enable_jobs: + type: boolean + description: Whether to replicate jobs. + enable_clusters: + type: boolean + description: >- + Whether to replicate clusters (including cluster policies, instance + profiles, global init scripts, and worker environment overlays). + enable_warehouse: + type: boolean + description: Whether to replicate SQL warehouses. + enable_sql_workspace: + type: boolean + description: Whether to replicate the SQL workspace. + enable_libraries: + type: boolean + description: Whether to replicate libraries. + enable_notebooks: + type: boolean + description: Whether to replicate notebooks. + enable_files: + type: boolean + description: Whether to replicate workspace files. + enable_queries: + type: boolean + description: Whether to replicate SQL queries. + enable_dashboards: + type: boolean + description: Whether to replicate dashboards. + required: + - enable_jobs + - enable_clusters + - enable_warehouse + - enable_sql_workspace + - enable_libraries + - enable_notebooks + - enable_files + - enable_queries + - enable_dashboards + description: >- + Per-asset-type control over which workspace assets are replicated for a + workspace set. Applies + only when replicate_workspace_assets is true. Each toggle enables replication of one asset type + (or, for the workspace tree, one sub-type). Folders and permissions are always replicated for a + coherent recovery copy and therefore have no toggle. + + This message is all-or-nothing: if it is provided, every toggle must be set (a partial config is + rejected). Omitting the containing asset_replication_config field entirely selects the default + of replicating every asset type; sending the message means the caller opts into explicit + per-type control and must therefore state each type. + FailoverGroup: + type: object + properties: + regions: + type: array + items: + type: string + description: List of all regions participating in this failover group. + workspace_sets: + type: array + items: + $ref: '#/components/schemas/WorkspaceSet' + description: >- + Workspace sets, each containing workspaces that replicate to each + other. + initial_primary_region: + type: string + description: >- + Initial primary region. Used only in Create requests to set the + starting primary region. Not returned in responses. + create_time: + type: string + format: date-time + description: Time at which this failover group was created. + effective_primary_region: + type: string + description: >- + Current effective primary region. Replication flows FROM workspaces + in this region. Changes after a successful failover. + etag: + type: string + description: >- + Opaque version string for optimistic locking. Server-generated and + returned in responses. + name: + type: string + description: >- + Fully qualified resource name in the format + accounts/{account_id}/failover-groups/{failover_group_id}. + replication_point: + type: string + format: date-time + description: The latest point in time to which data has been replicated. + state: + $ref: '#/components/schemas/FailoverGroupState' + description: Aggregate state of the failover group. + unity_catalog_assets: + $ref: '#/components/schemas/UcReplicationConfig' + description: Unity Catalog replication configuration. + update_time: + type: string + format: date-time + description: Time at which this failover group was last modified. + required: + - regions + - workspace_sets + - initial_primary_region + description: >- + A failover group manages disaster recovery across workspace sets, + coordinating Unity Catalog and + workspace assets replication. + ListFailoverGroupsResponse: + type: object + properties: + failover_groups: + type: array + items: + $ref: '#/components/schemas/FailoverGroup' + description: The failover groups for this account. + next_page_token: + type: string + description: >- + A token that can be sent as page_token to retrieve the next page. If + omitted, there are no subsequent pages. + description: Response for listing failover groups. + ListStableUrlsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token that can be sent as page_token to retrieve the next page. If + omitted, there are no subsequent pages. + stable_urls: + type: array + items: + $ref: '#/components/schemas/StableUrl' + description: The stable URLs for this account. + description: Response for listing stable URLs. + LocationMapping: + type: object + properties: + name: + type: string + description: Resource name for this location. + uri_by_region: + type: array + items: + $ref: '#/components/schemas/LocationMappingEntry' + description: URI for each region. Each entry maps a region name to a storage URI. + required: + - name + - uri_by_region + description: >- + A location mapping identified by a name, with URIs per region. The + system derives replication + direction from effective_primary_region. + LocationMappingEntry: + type: object + properties: + region: + type: string + description: The region name. + uri: + type: string + description: The storage URI for this region. + required: + - region + - uri + description: >- + A single entry in a location mapping, mapping a region to a storage URI. + Used instead of + map for proto2 compatibility. + StableUrl: + type: object + properties: + initial_workspace_id: + type: string + description: >- + The workspace this stable URL is initially bound to. Used only in + Create requests to associate the stable URL with a workspace. Not + returned in responses. + effective_workspace_id: + type: string + description: >- + The workspace this stable URL currently routes to. Set to + ``initial_workspace_id`` at creation, advanced to the failover + group's primary while attached (including across a failover), and + preserved when the stable URL is detached from its failover group. + Read this to see where an unattached stable URL points: after a + failover followed by a detach it reflects the post-failover primary, + not ``initial_workspace_id``. + failover_group_name: + type: string + description: >- + Fully qualified resource name of the FailoverGroup this stable URL + is currently linked to, in the format + ``accounts/{account_id}/failover-groups/{failover_group_id}``. Empty + when the stable URL is not attached to any failover group. + name: + type: string + description: >- + Fully qualified resource name. Format: + accounts/{account_id}/stable-urls/{stable_url_id}. + stable_workspace_id: + type: string + description: >- + The stable workspace ID for this stable URL. Generated on creation + and immutable thereafter; identifies the URL across failovers and is + the same value embedded in the ``url`` (as the ``w=`` query + parameter for SPOG URLs, or in the ``conn-`` hostname for + Private-Link URLs). + url: + type: string + description: >- + The stable URL endpoint. Generated on creation and immutable + thereafter. For non-Private-Link workspaces this is + ``https:///?w=``. For Private-Link + workspaces this is the per-connection hostname. + required: + - initial_workspace_id + description: >- + A stable URL provides a failover-aware endpoint for accessing a + workspace. Its lifecycle is + independent of any failover group. + UcCatalog: + type: object + properties: + name: + type: string + description: The name of the UC catalog to replicate. + required: + - name + description: A Unity Catalog catalog to replicate. + UcReplicationConfig: + type: object + properties: + catalogs: + type: array + items: + $ref: '#/components/schemas/UcCatalog' + description: UC catalogs to replicate. + data_replication_workspace_set: + type: string + description: >- + The workspace set whose workspaces will be used for data replication + of all UC catalogs' underlying storage. + location_mappings: + type: array + items: + $ref: '#/components/schemas/LocationMapping' + description: Location mappings - storage URI per region for each location. + required: + - catalogs + - data_replication_workspace_set + description: Unity Catalog replication configuration (top-level, not per-set). + WorkspaceSet: + type: object + properties: + name: + type: string + description: Resource name for this workspace set. + workspace_ids: + type: array + items: + type: string + description: >- + Workspace IDs in this set. The system derives and validates regions. + All workspaces must be in the Mission Critical tier. + asset_replication_config: + $ref: '#/components/schemas/AssetReplicationConfig' + description: >- + Per-asset-type control over which workspace assets are replicated. + Applies only when replicate_workspace_assets is true. When omitted + while control plane DR is enabled, every asset type is replicated. + replicate_workspace_assets: + type: boolean + description: >- + Whether to enable control plane DR (notebooks, jobs, clusters, etc.) + for this set. Defaults to false. + stable_url_names: + type: array + items: + type: string + description: >- + Resource names of stable URLs associated with this workspace set. + Format: accounts/{account_id}/stable-urls/{stable_url_id}. The + referenced stable URLs must already exist (via CreateStableUrl). + required: + - name + - workspace_ids + description: A set of workspaces that replicate to each other across regions. + FailoverFailoverGroupRequestFailoverType: + type: string + x-enum: + - FORCED + description: The type of failover to perform. + FailoverGroupState: + type: string + x-enum: + - ACTIVE + - CREATING + - CREATION_FAILED + - DELETING + - DELETION_FAILED + - FAILING_OVER + - FAILOVER_FAILED + - INITIAL_REPLICATION + description: The aggregate state of a FailoverGroup. + x-stackQL-resources: + failover_groups: + id: databricks_account.disasterrecovery.failover_groups + name: failover_groups + title: Failover Groups + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~1disaster-recovery~1v1~1{parent}~1failover-groups/post + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: >- + #/paths/~1api~1disaster-recovery~1v1~1{parent}~1failover-groups/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.failover_groups + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~1disaster-recovery~1v1~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + failover: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~1disaster-recovery~1v1~1{name}~1failover/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/failover_groups/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/failover_groups/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/failover_groups/methods/update' + delete: [] + replace: [] + stable_urls: + id: databricks_account.disasterrecovery.stable_urls + name: stable_urls + title: Stable Urls + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~1disaster-recovery~1v1~1{parent}~1stable-urls/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~1disaster-recovery~1v1~1{parent}~1stable-urls/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.stable_urls + delete: + operation: + $ref: '#/paths/~1api~1disaster-recovery~1v1~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~1disaster-recovery~1v1~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/stable_urls/methods/list' + - $ref: '#/components/x-stackQL-resources/stable_urls/methods/get' + insert: + - $ref: '#/components/x-stackQL-resources/stable_urls/methods/create' + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/stable_urls/methods/delete' + replace: [] +x-stackQL-config: + pagination: + requestToken: + key: page_token + location: query + responseToken: + key: next_page_token + location: body diff --git a/providers/src/databricks_account/v00.00.00000/services/iam.yaml b/providers/src/databricks_account/v00.00.00000/services/iam.yaml index 75839fe0..0a0b375c 100644 --- a/providers/src/databricks_account/v00.00.00000/services/iam.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/iam.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks iam service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.iam servers: - url: https://accounts.cloud.databricks.com @@ -24,10 +24,10 @@ paths: :param resource: str The resource name for which assignable roles will be listed. - Examples | Summary :--- | :--- `resource=accounts/` | A resource name for the account. - `resource=accounts//groups/` | A resource name for the group. - `resource=accounts//servicePrincipals/` | A resource name for the service - principal. `resource=accounts//tagPolicies/` | A resource name for the + Examples | Summary :--- | :--- ``resource=accounts/`` | A resource name for the account. + ``resource=accounts//groups/`` | A resource name for the group. + ``resource=accounts//servicePrincipals/`` | A resource name for the service + principal. ``resource=accounts//tagPolicies/`` | A resource name for the tag policy. :returns: :class:`GetAssignableRolesForResourceResponse` @@ -42,10 +42,10 @@ paths: required: true schema: type: string - description: The resource name for which assignable roles will be listed. Examples | Summary :--- | :--- `resource=accounts/` - | A resource name for the account. `resource=accounts//groups/` | A resource name for the - group. `resource=accounts//servicePrincipals/` | A resource name for the service principal. `resource=accounts//tagPolicies/` - | A resource name for the tag policy. + description: The resource name for which assignable roles will be listed. Examples | Summary :--- | :--- ``resource=accounts/`` + | A resource name for the account. ``resource=accounts//groups/`` | A resource name for the + group. ``resource=accounts//servicePrincipals/`` | A resource name for the service principal. + ``resource=accounts//tagPolicies/`` | A resource name for the tag policy. responses: '200': description: Success @@ -79,13 +79,13 @@ paths: :param name: str The ruleset name associated with the request. - Examples | Summary :--- | :--- `name=accounts//ruleSets/default` | A name for a rule set - on the account. `name=accounts//groups//ruleSets/default` | A name for a rule - set on the group. - `name=accounts//servicePrincipals//ruleSets/default` | - A name for a rule set on the service principal. - `name=accounts//tagPolicies//ruleSets/default` | A name for a rule set on - the tag policy. + Examples | Summary :--- | :--- ``name=accounts//ruleSets/default`` | A name for a rule + set on the account. ``name=accounts//groups//ruleSets/default`` | A name for a + rule set on the group. + ``name=accounts//servicePrincipals//ruleSets/default`` + | A name for a rule set on the service principal. + ``name=accounts//tagPolicies//ruleSets/default`` | A name for a rule set + on the tag policy. :param etag: str Etag used for versioning. The response is at least as fresh as the eTag provided. Etag is used for optimistic concurrency control as a way to help prevent simultaneous updates of a rule set from @@ -94,8 +94,8 @@ paths: etag from a GET rule set request, and pass it with the PUT update request to identify the rule set version you are updating. - Examples | Summary :--- | :--- `etag=` | An empty etag can only be used in GET to indicate no - freshness requirements. `etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==` | An + Examples | Summary :--- | :--- ``etag=`` | An empty etag can only be used in GET to indicate no + freshness requirements. ``etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==`` | An etag encoded a specific version of the rule set to get or to be updated. :returns: :class:`RuleSetResponse` @@ -110,10 +110,10 @@ paths: required: true schema: type: string - description: The ruleset name associated with the request. Examples | Summary :--- | :--- `name=accounts//ruleSets/default` - | A name for a rule set on the account. `name=accounts//groups//ruleSets/default` | A name - for a rule set on the group. `name=accounts//servicePrincipals//ruleSets/default` - | A name for a rule set on the service principal. `name=accounts//tagPolicies//ruleSets/default` + description: The ruleset name associated with the request. Examples | Summary :--- | :--- ``name=accounts//ruleSets/default`` + | A name for a rule set on the account. ``name=accounts//groups//ruleSets/default`` | A name + for a rule set on the group. ``name=accounts//servicePrincipals//ruleSets/default`` + | A name for a rule set on the service principal. ``name=accounts//tagPolicies//ruleSets/default`` | A name for a rule set on the tag policy. - name: etag in: query @@ -124,8 +124,8 @@ paths: concurrency control as a way to help prevent simultaneous updates of a rule set from overwriting each other. It is strongly suggested that systems make use of the etag in the read -> modify -> write pattern to perform rule set updates in order to avoid race conditions that is get an etag from a GET rule set request, and pass it with the - PUT update request to identify the rule set version you are updating. Examples | Summary :--- | :--- `etag=` | An - empty etag can only be used in GET to indicate no freshness requirements. `etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==` + PUT update request to identify the rule set version you are updating. Examples | Summary :--- | :--- ``etag=`` | + An empty etag can only be used in GET to indicate no freshness requirements. ``etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==`` | An etag encoded a specific version of the rule set to get or to be updated. responses: '200': @@ -202,7 +202,7 @@ paths: type: string description: 'Principals this grant rule applies to. A principal can be a user (for end users), a service principal (for applications and compute workloads), or an account group. Each principal - has its own identifier format: * users/ * groups/ * servicePrincipals/' + has its own identifier format: - users/ - groups/ - servicePrincipals/' required: - role required: @@ -336,8 +336,8 @@ paths: - account_groups_v2 description: |- Gets all details of the groups associated with the Databricks account. As of 08/22/2025, this endpoint - will no longer return members. Instead, members should be retrieved by iterating through `Get group - details`. Existing accounts that rely on this attribute will not be impacted and will continue + will no longer return members. Instead, members should be retrieved by iterating through ``Get group + details``. Existing accounts that rely on this attribute will not be impacted and will continue receiving member data as before. :param attributes: str (optional) @@ -347,12 +347,11 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) Attribute to sort the results. :param sort_order: :class:`ListSortOrder` (optional) @@ -390,9 +389,10 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false @@ -816,12 +816,11 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) Attribute to sort the results. :param sort_order: :class:`ListSortOrder` (optional) @@ -859,9 +858,10 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false @@ -1187,7 +1187,7 @@ paths: :param active: bool (optional) If this user is active :param display_name: str (optional) - String that represents a concatenation of given and family names. For example `John Smith`. + String that represents a concatenation of given and family names. For example ``John Smith``. :param emails: List[:class:`ComplexValue`] (optional) All the emails associated with the Databricks user. :param external_id: str (optional) @@ -1218,7 +1218,7 @@ paths: description: If this user is active displayName: type: string - description: String that represents a concatenation of given and family names. For example `John Smith`. + description: String that represents a concatenation of given and family names. For example ``John Smith``. emails: type: array items: @@ -1303,15 +1303,14 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) - Attribute to sort the results. Multi-part paths are supported. For example, `userName`, - `name.givenName`, and `emails`. + Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, + ``name.givenName``, and ``emails``. :param sort_order: :class:`ListSortOrder` (optional) The order to sort the results. :param start_index: int (optional) @@ -1347,16 +1346,17 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false schema: type: string - description: Attribute to sort the results. Multi-part paths are supported. For example, `userName`, `name.givenName`, - and `emails`. + description: Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, ``name.givenName``, + and ``emails``. - name: sort_order in: query required: false @@ -1446,15 +1446,14 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) - Attribute to sort the results. Multi-part paths are supported. For example, `userName`, - `name.givenName`, and `emails`. + Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, + ``name.givenName``, and ``emails``. :param sort_order: :class:`GetSortOrder` (optional) The order to sort the results. :param start_index: int (optional) @@ -1496,16 +1495,17 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false schema: type: string - description: Attribute to sort the results. Multi-part paths are supported. For example, `userName`, `name.givenName`, - and `emails`. + description: Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, ``name.givenName``, + and ``emails``. - name: sort_order in: query required: false @@ -1539,12 +1539,14 @@ paths: x-stackql-sdk-source: AccountUsersV2API patch: operationId: account_users_v2_patch - summary: Partially updates a user resource by applying the supplied operations on specific user attributes. + summary: Partially updates a user resource by applying the supplied operations on specific user attributes. The tags: - iam - account_users_v2 description: |- - Partially updates a user resource by applying the supplied operations on specific user attributes. + Partially updates a user resource by applying the supplied operations on specific user attributes. The + ``userName`` and ``emails`` attributes cannot be updated through this API; any supplied changes to + them are ignored (no-op). :param id: str Unique ID in the Databricks workspace. @@ -1660,7 +1662,7 @@ paths: :param active: bool (optional) If this user is active :param display_name: str (optional) - String that represents a concatenation of given and family names. For example `John Smith`. + String that represents a concatenation of given and family names. For example ``John Smith``. :param emails: List[:class:`ComplexValue`] (optional) All the emails associated with the Databricks user. :param external_id: str (optional) @@ -1693,7 +1695,7 @@ paths: description: If this user is active displayName: type: string - description: String that represents a concatenation of given and family names. For example `John Smith`. + description: String that represents a concatenation of given and family names. For example ``John Smith``. emails: type: array items: @@ -1974,6 +1976,12 @@ paths: :param workspace_id: int The workspace ID for the account. + :param filter: str (optional) + Filter string to search principals. + :param max_results: int (optional) + Maximum number of permission assignments to return. + :param page_token: str (optional) + Page token returned by previous call to retrieve the next page of results. :returns: Iterator over :class:`PermissionAssignment` parameters: @@ -1988,6 +1996,24 @@ paths: schema: type: integer description: The workspace ID for the account. + - name: filter + in: query + required: false + schema: + type: string + description: Filter string to search principals. + - name: max_results + in: query + required: false + schema: + type: integer + description: Maximum number of permission assignments to return. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token returned by previous call to retrieve the next page of results. responses: '200': description: Success @@ -2101,12 +2127,13 @@ components: description: If this user is active displayName: type: string - description: String that represents a concatenation of given and family names. For example `John Smith`. + description: String that represents a concatenation of given and family names. For example ``John Smith``. emails: type: array items: $ref: '#/components/schemas/ComplexValue' - description: All the emails associated with the Databricks user. + description: All the emails associated with the Databricks user. This attribute cannot be updated through the SCIM + PATCH or PUT APIs; any supplied change is ignored. externalId: type: string description: External ID is not currently supported. It is reserved for future use. @@ -2122,7 +2149,8 @@ components: description: Indicates if the group has the admin role. userName: type: string - description: Email address of the Databricks user. + description: Email address of the Databricks user. This attribute cannot be updated through the SCIM PATCH or PUT + APIs; any supplied change is ignored. Actor: type: object properties: @@ -2194,8 +2222,8 @@ components: items: type: string description: 'Principals this grant rule applies to. A principal can be a user (for end users), a service principal - (for applications and compute workloads), or an account group. Each principal has its own identifier format: * - users/ * groups/ * servicePrincipals/' + (for applications and compute workloads), or an account group. Each principal has its own identifier format: - + users/ - groups/ - servicePrincipals/' required: - role Group: @@ -2207,8 +2235,8 @@ components: type: array items: $ref: '#/components/schemas/ComplexValue' - description: 'Entitlements assigned to the group. See [assigning entitlements] for a full list of supported values. - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the group. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string description: external_id should be unique for identifying groups @@ -2477,10 +2505,16 @@ components: PermissionAssignments: type: object properties: + next_page_token: + type: string permission_assignments: type: array items: $ref: '#/components/schemas/PermissionAssignment' + description: Array of permissions assignments defined for a workspace. + prev_page_token: + type: string + description: Token to retrieve the previous page of results. PermissionOutput: type: object properties: @@ -2594,8 +2628,8 @@ components: type: array items: $ref: '#/components/schemas/ComplexValue' - description: 'Entitlements assigned to the service principal. See [assigning entitlements] for a full list of supported - values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the service principal. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string groups: @@ -2622,20 +2656,21 @@ components: type: boolean displayName: type: string - description: 'String that represents a concatenation of given and family names. For example `John Smith`. This field - cannot be updated through the Workspace SCIM APIs when [identity federation is enabled]. Use Account SCIM APIs - to update `displayName`. [identity federation is enabled]: https://docs.databricks.com/administration-guide/users-groups/best-practices.html#enable-identity-federation' + description: String that represents a concatenation of given and family names. For example ``John Smith``. This + field cannot be updated through the Workspace SCIM APIs when `identity federation is enabled `__. + Use Account SCIM APIs to update ``displayName``. emails: type: array items: $ref: '#/components/schemas/ComplexValue' - description: All the emails associated with the Databricks user. + description: All the emails associated with the Databricks user. This attribute cannot be updated through the SCIM + PATCH or PUT APIs; any supplied change is ignored. entitlements: type: array items: $ref: '#/components/schemas/ComplexValue' - description: 'Entitlements assigned to the user. See [assigning entitlements] for a full list of supported values. - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the user. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string description: External ID is not currently supported. It is reserved for future use. @@ -2660,7 +2695,8 @@ components: description: The schema of the user. userName: type: string - description: Email address of the Databricks user. + description: Email address of the Databricks user. This attribute cannot be updated through the SCIM PATCH or PUT + APIs; any supplied change is ignored. WorkspacePermissions: type: object properties: @@ -2668,6 +2704,21 @@ components: type: array items: $ref: '#/components/schemas/PermissionOutput' + AutoscopeState: + type: string + x-enum: + - AUTOSCOPE_STATE_API_NOT_COVERED + - AUTOSCOPE_STATE_BACKFILLED + - AUTOSCOPE_STATE_COMPLETED + - AUTOSCOPE_STATE_DISABLED + - AUTOSCOPE_STATE_RUNNING + - AUTOSCOPE_STATE_USER_SELECTED + description: |- + State of inferred scope collection (autoscope) for an external PAT. Mirrored in + databricks.identity.AutoscopeState in common/principal-context/api/proto/tokendetails.proto. + Token store and token management proto can depend on this. Principal context proto should NOT + depend on this proto definitions because too many services depend on the principal context + proto. GetSortOrder: type: string x-enum: @@ -2910,6 +2961,7 @@ components: - CAN_VIEW - CAN_VIEW_METADATA - IS_OWNER + - UNSPECIFIED description: Permission level RequestAuthzIdentity: type: string diff --git a/providers/src/databricks_account/v00.00.00000/services/iamv2.yaml b/providers/src/databricks_account/v00.00.00000/services/iamv2.yaml index 115edd14..1a8d7307 100644 --- a/providers/src/databricks_account/v00.00.00000/services/iamv2.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/iamv2.yaml @@ -5,81 +5,178 @@ info: OpenAPI specification for the Databricks iamv2 service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.iamv2 servers: - url: https://accounts.cloud.databricks.com paths: - /api/2.0/identity/accounts/{account_id}/workspaces/{workspace_id}/workspaceAccessDetails/{principal_id}: - get: - operationId: account_iam_v2_get_workspace_access_detail + /api/2.0/{parent}/account-access-identity-rules: + post: + operationId: account_iam_v2_create_account_access_identity_rule summary: >- - Returns the access details for a principal in a workspace. Allows for - checking access details for any + Creates a new account access identity rule for a given account. This + allows administrators to tags: - iamv2 - account_iam_v2 description: >- - Returns the access details for a principal in a workspace. Allows for - checking access details for any + Creates a new account access identity rule for a given account. This + allows administrators to - provisioned principal (user, service principal, or group) in a - workspace. * Provisioned principal here + explicitly allow or deny specific principals from accessing the account. - refers to one that has been synced into Databricks from the customer's - IdP or added explicitly to - Databricks via SCIM/UI. Allows for passing in a "view" parameter to - control what fields are returned + :param parent: str + Required. The account under which to create the rule. Format: accounts/{account_id} + :param account_access_identity_rule: :class:`AccountAccessIdentityRule` + Required. The rule to create. + :param external_principal_id: str + Required. The external ID of the principal in the customer's IdP. + + :returns: :class:`AccountAccessIdentityRule` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Required. The account under which to create the rule. Format: + accounts/{account_id} + - name: external_principal_id + in: query + required: true + schema: + type: string + description: Required. The external ID of the principal in the customer's IdP. + requestBody: + content: + application/json: + schema: + type: object + properties: + account_access_identity_rule: + type: object + properties: + action: + type: string + x-enum: + - DENY + description: Currently, only DENY action is supported. + display_name: + type: string + description: Display name of the principal. + external_principal_id: + type: string + description: External ID of the principal in the customer's IdP. + name: + type: string + description: >- + Fully qualified name for the rule. Format: + accounts/{account_id}/account-access-identity-rules/{external_principal_id} + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: >- + The type of the principal (user/service + principal/group). This field is populated by the server + based on the external_principal_id. + required: + - action + description: Required. The rule to create. + required: + - account_access_identity_rule + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AccountAccessIdentityRule' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_list_account_access_identity_rules + summary: >- + Lists all account access identity rules for a given account. These rules + control which principals + tags: + - iamv2 + - account_iam_v2 + description: >- + Lists all account access identity rules for a given account. These rules + control which principals + + (users, service principals, groups) from the customer's IdP are allowed + or denied access to the - (BASIC by default or FULL). + Databricks account. - :param workspace_id: int - Required. The workspace ID for which the access details are being requested. - :param principal_id: int - Required. The internal ID of the principal (user/sp/group) for which the access details are being - requested. - :param view: :class:`WorkspaceAccessDetailView` (optional) - Controls what fields are returned. + :param parent: str + Required. The account for which to list the rules. Format: accounts/{account_id} + :param filter: str (optional) + Optional. Filter to apply to the list. Supports filtering by displayName. + :param page_size: int (optional) + Optional. The maximum number of rules to return. The service may return fewer than this value. + :param page_token: str (optional) + Optional. A page token, received from a previous call. Provide this to retrieve the subsequent page. - :returns: :class:`WorkspaceAccessDetail` + :returns: :class:`ListAccountAccessIdentityRulesResponse` parameters: - - name: account_id + - name: parent in: path required: true schema: type: string - - name: workspace_id - in: path - required: true + description: >- + Required. The account for which to list the rules. Format: + accounts/{account_id} + - name: filter + in: query + required: false schema: - type: integer + type: string description: >- - Required. The workspace ID for which the access details are being - requested. - - name: principal_id - in: path - required: true + Optional. Filter to apply to the list. Supports filtering by + displayName. + - name: page_size + in: query + required: false schema: type: integer description: >- - Required. The internal ID of the principal (user/sp/group) for which - the access details are being requested. - - name: view + Optional. The maximum number of rules to return. The service may + return fewer than this value. + - name: page_token in: query required: false schema: - $ref: '#/components/schemas/WorkspaceAccessDetailView' - description: Controls what fields are returned. + type: string + description: >- + Optional. A page token, received from a previous call. Provide this + to retrieve the subsequent page. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/WorkspaceAccessDetail' + $ref: '#/components/schemas/ListAccountAccessIdentityRulesResponse' default: description: Error response content: @@ -92,55 +189,78 @@ paths: message: type: string x-stackql-sdk-source: AccountIamV2API - /api/2.0/identity/accounts/{account_id}/groups/resolveByExternalId: + /api/2.0/{parent}/attribute-control-entries: post: - operationId: account_iam_v2_resolve_group + operationId: account_iam_v2_create_attribute_control_entry summary: >- - Resolves a group with the given external ID from the customer's IdP. If - the group does not exist, it + Creates (allows) an identity attribute control-list entry for an + account. tags: - iamv2 - account_iam_v2 description: >- - Resolves a group with the given external ID from the customer's IdP. If - the group does not exist, it + Creates (allows) an identity attribute control-list entry for an + account. - will be created in the account. If the customer is not onboarded onto - Automatic Identity Management - - (AIM), this will return an error. + :param parent: str + Required. The account under which to create the entry. Format: accounts/{account_id} + :param attribute_control_entry: :class:`AttributeControlEntry` + Required. The entry to create. + :param attribute_control_entry_id: str (optional) + Optional. The ID to use for the entry, which becomes the last segment of its resource name: the IdP + attribute being governed (e.g. "department"). - :param external_id: str - Required. The external ID of the group in the customer's IdP. - - :returns: :class:`ResolveGroupResponse` + :returns: :class:`AttributeControlEntry` parameters: - - name: account_id + - name: parent in: path required: true schema: type: string + description: >- + Required. The account under which to create the entry. Format: + accounts/{account_id} + - name: attribute_control_entry_id + in: query + required: false + schema: + type: string + description: >- + Optional. The ID to use for the entry, which becomes the last + segment of its resource name: the IdP attribute being governed (e.g. + "department"). requestBody: content: application/json: schema: type: object properties: - external_id: - type: string - description: >- - Required. The external ID of the group in the customer's - IdP. + attribute_control_entry: + type: object + properties: + is_allowed: + type: boolean + description: >- + Whether the attribute is permitted. Always true in the + Identity Attributes PrPr. + name: + type: string + description: >- + The resource name of the entry. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + where {attribute_name} is the IdP attribute being + governed (e.g. "department"). + description: Required. The entry to create. required: - - external_id + - attribute_control_entry responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ResolveGroupResponse' + $ref: '#/components/schemas/AttributeControlEntry' default: description: Error response content: @@ -153,55 +273,140 @@ paths: message: type: string x-stackql-sdk-source: AccountIamV2API - /api/2.0/identity/accounts/{account_id}/servicePrincipals/resolveByExternalId: - post: - operationId: account_iam_v2_resolve_service_principal - summary: >- - Resolves an SP with the given external ID from the customer's IdP. If - the SP does not exist, it will + get: + operationId: account_iam_v2_list_attribute_control_entries + summary: Lists the identity attribute control-list entries for an account. tags: - iamv2 - account_iam_v2 - description: >- - Resolves an SP with the given external ID from the customer's IdP. If - the SP does not exist, it will - - be created. If the customer is not onboarded onto Automatic Identity - Management (AIM), this will + description: |- + Lists the identity attribute control-list entries for an account. - return an error. + :param parent: str + Required. The account for which to list entries. Format: accounts/{account_id} + :param page_size: int (optional) + Optional. The maximum number of entries to return. + :param page_token: str (optional) + Optional. A page token from a previous call, to retrieve the next page. + :returns: :class:`ListAttributeControlEntriesResponse` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Required. The account for which to list entries. Format: + accounts/{account_id} + - name: page_size + in: query + required: false + schema: + type: integer + description: Optional. The maximum number of entries to return. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + Optional. A page token from a previous call, to retrieve the next + page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListAttributeControlEntriesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/groups/{group_id}/direct-members: + post: + operationId: account_iam_v2_create_direct_group_member + summary: Creates a group membership (assigns a principal to a group). + tags: + - iamv2 + - account_iam_v2 + description: |- + Creates a group membership (assigns a principal to a group). - :param external_id: str - Required. The external ID of the service principal in the customer's IdP. + :param group_id: int + Required. Internal ID of the group in Databricks. + :param direct_group_member: :class:`DirectGroupMember` + Required. The direct group member to be added to the group. - :returns: :class:`ResolveServicePrincipalResponse` + :returns: :class:`DirectGroupMember` parameters: - name: account_id in: path required: true schema: type: string + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks. requestBody: content: application/json: schema: type: object properties: - external_id: - type: string - description: >- - Required. The external ID of the service principal in the - customer's IdP. + direct_group_member: + type: object + properties: + principal_id: + type: integer + description: Internal ID of the principal in Databricks. + display_name: + type: string + description: Display name of the principal. + external_id: + type: string + description: The external ID of the principal in Databricks. + membership_source: + type: string + x-enum: + - IDENTITY_PROVIDER + - INTERNAL + description: >- + The source of group membership (internal or from + identity provider). + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: >- + The type of the principal (user/service + principal/group). + required: + - principal_id + description: Required. The direct group member to be added to the group. required: - - external_id + - direct_group_member responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ResolveServicePrincipalResponse' + $ref: '#/components/schemas/DirectGroupMember' default: description: Error response content: @@ -214,29 +419,99 @@ paths: message: type: string x-stackql-sdk-source: AccountIamV2API - /api/2.0/identity/accounts/{account_id}/users/resolveByExternalId: - post: - operationId: account_iam_v2_resolve_user + get: + operationId: account_iam_v2_list_direct_group_members summary: >- - Resolves a user with the given external ID from the customer's IdP. If - the user does not exist, it + Lists provisioned direct members of a group with their membership source + (internal or from identity tags: - iamv2 - account_iam_v2 description: >- - Resolves a user with the given external ID from the customer's IdP. If - the user does not exist, it + Lists provisioned direct members of a group with their membership source + (internal or from identity - will be created. If the customer is not onboarded onto Automatic - Identity Management (AIM), this will + provider). - return an error. + :param group_id: int + Required. Internal ID of the group in Databricks whose direct members are being listed. + :param page_size: int (optional) + The maximum number of members to return. The service may return fewer than this value. If not + provided, defaults to 1000 (also the maximum allowed). + :param page_token: str (optional) + A page token, received from a previous ListDirectGroupMembers call. Provide this to retrieve the + subsequent page. - :param external_id: str - Required. The external ID of the user in the customer's IdP. + :returns: :class:`ListDirectGroupMembersResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: group_id + in: path + required: true + schema: + type: integer + description: >- + Required. Internal ID of the group in Databricks whose direct + members are being listed. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of members to return. The service may return + fewer than this value. If not provided, defaults to 1000 (also the + maximum allowed). + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ListDirectGroupMembers call. + Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListDirectGroupMembersResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/groups: + post: + operationId: account_iam_v2_create_group + summary: >- + Creates a group in the Databricks account and returns the resulting + Group resource. + tags: + - iamv2 + - account_iam_v2 + description: >- + Creates a group in the Databricks account and returns the resulting + Group resource. - :returns: :class:`ResolveUserResponse` + + :param group: :class:`Group` + Required. Group to be created in + + :returns: :class:`Group` parameters: - name: account_id in: path @@ -249,18 +524,31 @@ paths: schema: type: object properties: - external_id: - type: string - description: Required. The external ID of the user in the customer's IdP. + group: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + group_name: + type: string + description: Display name of the group. + internal_id: + type: string + description: Internal group ID of the group in Databricks. + description: Required. Group to be created in required: - - external_id + - group responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ResolveUserResponse' + $ref: '#/components/schemas/Group' default: description: Error response content: @@ -273,24 +561,2759 @@ paths: message: type: string x-stackql-sdk-source: AccountIamV2API -components: - schemas: - Group: - type: object - properties: - account_id: - type: string - description: The parent account ID for group in Databricks. - external_id: - type: string - description: ExternalId of the group in the customer's IdP. - group_name: - type: string - description: Display name of the group. - internal_id: - type: integer - description: Internal group ID of the group in Databricks. + get: + operationId: account_iam_v2_list_groups + summary: >- + Lists the groups in the Databricks account, returning one page per call. + Supports filtering by group + tags: + - iamv2 + - account_iam_v2 + description: >- + Lists the groups in the Databricks account, returning one page per call. + Supports filtering by group + + name or external ID. + + + :param filter: str (optional) + Optional. Allows filtering groups by group name or external id. + :param page_size: int (optional) + The maximum number of groups to return. The service may return fewer than this value. + :param page_token: str (optional) + A page token, received from a previous ListGroups call. Provide this to retrieve the subsequent + page. + + :returns: :class:`ListGroupsResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: filter + in: query + required: false + schema: + type: string + description: Optional. Allows filtering groups by group name or external id. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of groups to return. The service may return fewer + than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ListGroups call. Provide this + to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListGroupsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/service-principals: + post: + operationId: account_iam_v2_create_service_principal + summary: >- + Creates a service principal in the Databricks account and returns the + resulting ServicePrincipal + tags: + - iamv2 + - account_iam_v2 + description: >- + Creates a service principal in the Databricks account and returns the + resulting ServicePrincipal + + resource. + + + :param service_principal: :class:`ServicePrincipal` + Required. Service principal to be created in + + :returns: :class:`ServicePrincipal` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + service_principal: + type: object + properties: + account_id: + type: string + description: >- + The parent account ID for the service principal in + Databricks. + account_sp_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: >- + The activity status of a service principal in a + Databricks account. + application_id: + type: string + description: Application ID of the service principal. + display_name: + type: string + description: Display name of the service principal. + external_id: + type: string + description: >- + ExternalId of the service principal in the customer's + IdP. + internal_id: + type: string + description: >- + Internal service principal ID of the service principal + in Databricks. + description: Required. Service principal to be created in + required: + - service_principal + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServicePrincipal' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_list_service_principals + summary: >- + Lists the service principals in the Databricks account, returning one + page per call. Supports + tags: + - iamv2 + - account_iam_v2 + description: >- + Lists the service principals in the Databricks account, returning one + page per call. Supports + + filtering by application ID or external ID. + + + :param filter: str (optional) + Optional. Allows filtering service principals by application id or external id. + :param page_size: int (optional) + The maximum number of service principals to return. The service may return fewer than this value. + :param page_token: str (optional) + A page token, received from a previous ListServicePrincipals call. Provide this to retrieve the + subsequent page. + + :returns: :class:`ListServicePrincipalsResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: filter + in: query + required: false + schema: + type: string + description: >- + Optional. Allows filtering service principals by application id or + external id. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of service principals to return. The service may + return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ListServicePrincipals call. + Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListServicePrincipalsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/users: + post: + operationId: account_iam_v2_create_user + summary: >- + Creates a user in the Databricks account and returns the resulting User + resource. Behavior depends on + tags: + - iamv2 + - account_iam_v2 + description: >- + Creates a user in the Databricks account and returns the resulting User + resource. Behavior depends on + + whether Account Identity Management (AIM) is enabled: + + + - When AIM is enabled: the user is provisioned with an internalId. If an + externalId is provided, the + identity provider is treated as the source of truth for user metadata, and customer-supplied field + values may be overridden. + - When AIM is disabled: the user is provisioned with an internalId only, + and customer-supplied + metadata is used as-is. + + :param user: :class:`User` + Required. User to be created in + + :returns: :class:`User` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + user: + type: object + properties: + account_id: + type: string + description: The accountId parent of the user in Databricks. + account_user_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of a user in a Databricks account. + external_id: + type: string + description: ExternalId of the user in the customer's IdP. + full_name: + type: object + properties: + family_name: + type: string + given_name: + type: string + internal_id: + type: string + description: Internal userId of the user in Databricks. + username: + type: string + description: Username/email of the user. + description: Required. User to be created in + required: + - user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/User' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_list_users + summary: >- + Lists the users in the Databricks account, returning one page per call. + Supports filtering by username + tags: + - iamv2 + - account_iam_v2 + description: >- + Lists the users in the Databricks account, returning one page per call. + Supports filtering by username + + or external ID. + + + :param filter: str (optional) + Optional. Allows filtering users by username or external id. + :param page_size: int (optional) + The maximum number of users to return. The service may return fewer than this value. + :param page_token: str (optional) + A page token, received from a previous ListUsers call. Provide this to retrieve the subsequent page. + + :returns: :class:`ListUsersResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: filter + in: query + required: false + schema: + type: string + description: Optional. Allows filtering users by username or external id. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of users to return. The service may return fewer + than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ListUsers call. Provide this + to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListUsersResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/workspaces/{workspace_id}/workspace-assignment-details: + post: + operationId: account_iam_v2_create_workspace_assignment_detail + summary: >- + Creates a workspace assignment detail for a principal. Entitlement + grants are applied individually and + tags: + - iamv2 + - account_iam_v2 + description: >- + Creates a workspace assignment detail for a principal. Entitlement + grants are applied individually and + + non-atomically — if a failure occurs partway through, the principal will + be assigned to the + + workspace but with only a subset of the requested entitlements. Use + GetWorkspaceAssignmentDetail to + + confirm which entitlements were successfully granted. + + + :param workspace_id: int + Required. The workspace ID for which the workspace assignment detail is being created. + :param workspace_assignment_detail: :class:`WorkspaceAssignmentDetail` + Required. Workspace assignment detail to be created in . + + :returns: :class:`WorkspaceAssignmentDetail` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: >- + Required. The workspace ID for which the workspace assignment detail + is being created. + requestBody: + content: + application/json: + schema: + type: object + properties: + workspace_assignment_detail: + type: object + properties: + principal_id: + type: integer + description: >- + The internal ID of the principal (user/sp/group) in + Databricks. + account_id: + type: string + description: >- + The account ID parent of the workspace where the + principal is assigned + effective_entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + description: >- + The principal's full effective entitlements granted in + this workspace: every entitlement it holds whether + granted directly or via group membership. Populated on + Get; empty on List. + entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + description: >- + Entitlements granted directly to the principal on this + workspace. The only client-settable field: create and + update manage exactly this set (including entitlements + the principal also holds via a group). Not populated by + ListWorkspaceAssignmentDetails (omitted for + scalability); call GetWorkspaceAssignmentDetail to read + the entitlements for a single principal. + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the principal (user/sp/group). + workspace_id: + type: integer + description: The workspace ID where the principal is assigned + required: + - principal_id + description: >- + Required. Workspace assignment detail to be created in + . + required: + - workspace_assignment_detail + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_list_workspace_assignment_details + summary: >- + Lists workspace assignment details for a workspace. For scalability, the + response omits the + tags: + - iamv2 + - account_iam_v2 + description: >- + Lists workspace assignment details for a workspace. For scalability, the + response omits the + + per-principal entitlement fields (``entitlements`` and + ``effective_entitlements``); call + + GetWorkspaceAssignmentDetail to read entitlements for a single + principal. + + + :param workspace_id: int + Required. The workspace ID for which the workspace assignment details are being fetched. + :param page_size: int (optional) + The maximum number of workspace assignment details to return. The service may return fewer than this + value. + :param page_token: str (optional) + A page token, received from a previous ListWorkspaceAssignmentDetails call. Provide this to retrieve + the subsequent page. + + :returns: :class:`ListWorkspaceAssignmentDetailsResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: >- + Required. The workspace ID for which the workspace assignment + details are being fetched. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of workspace assignment details to return. The + service may return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous + ListWorkspaceAssignmentDetails call. Provide this to retrieve the + subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListWorkspaceAssignmentDetailsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/{parent}/account-access-identity-rules/{external_principal_id}: + delete: + operationId: account_iam_v2_delete_account_access_identity_rule + summary: Deletes an account access identity rule for a given principal. + tags: + - iamv2 + - account_iam_v2 + description: |- + Deletes an account access identity rule for a given principal. + + :param parent: str + Required. The account for which to delete the rule. Format: accounts/{account_id} + :param external_principal_id: str + Required. The external ID of the principal whose rule should be deleted. + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Required. The account for which to delete the rule. Format: + accounts/{account_id} + - name: external_principal_id + in: path + required: true + schema: + type: string + description: >- + Required. The external ID of the principal whose rule should be + deleted. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_account_access_identity_rule + summary: Gets an account access identity rule for a given principal. + tags: + - iamv2 + - account_iam_v2 + description: |- + Gets an account access identity rule for a given principal. + + :param parent: str + Required. The account for which to get the rule. Format: accounts/{account_id} + :param external_principal_id: str + Required. The external ID of the principal whose rule should be retrieved. + + :returns: :class:`AccountAccessIdentityRule` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Required. The account for which to get the rule. Format: + accounts/{account_id} + - name: external_principal_id + in: path + required: true + schema: + type: string + description: >- + Required. The external ID of the principal whose rule should be + retrieved. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AccountAccessIdentityRule' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/{name}: + delete: + operationId: account_iam_v2_delete_attribute_control_entry + summary: Deletes an identity attribute control-list entry. + tags: + - iamv2 + - account_iam_v2 + description: |- + Deletes an identity attribute control-list entry. + + :param name: str + Required. The resource name of the entry to delete. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Required. The resource name of the entry to delete. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_attribute_control_entry + summary: Gets an identity attribute control-list entry. + tags: + - iamv2 + - account_iam_v2 + description: |- + Gets an identity attribute control-list entry. + + :param name: str + Required. The resource name of the entry to get. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + + :returns: :class:`AttributeControlEntry` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Required. The resource name of the entry to get. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AttributeControlEntry' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + patch: + operationId: account_iam_v2_update_attribute_control_entry + summary: Updates an identity attribute control-list entry for an account. + tags: + - iamv2 + - account_iam_v2 + description: |- + Updates an identity attribute control-list entry for an account. + + :param name: str + The resource name of the entry. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} where {attribute_name} is the IdP + attribute being governed (e.g. "department"). + :param attribute_control_entry: :class:`AttributeControlEntry` + Required. The entry to update; its ``name`` identifies the existing entry. + :param update_mask: FieldMask + Required. The fields to update. For the PrPr only is_allowed is mutable, so the backend currently + applies the full entry regardless of the mask. + + :returns: :class:`AttributeControlEntry` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The resource name of the entry. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + where {attribute_name} is the IdP attribute being governed (e.g. + "department"). + - name: update_mask + in: query + required: true + schema: + type: object + description: >- + Required. The fields to update. For the PrPr only is_allowed is + mutable, so the backend currently applies the full entry regardless + of the mask. + requestBody: + content: + application/json: + schema: + type: object + properties: + attribute_control_entry: + type: object + properties: + is_allowed: + type: boolean + description: >- + Whether the attribute is permitted. Always true in the + Identity Attributes PrPr. + name: + type: string + description: >- + The resource name of the entry. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + where {attribute_name} is the IdP attribute being + governed (e.g. "department"). + description: >- + Required. The entry to update; its ``name`` identifies the + existing entry. + required: + - attribute_control_entry + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AttributeControlEntry' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/groups/{group_id}/direct-members/{principal_id}: + delete: + operationId: account_iam_v2_delete_direct_group_member + summary: Deletes a group membership (unassigns a principal from a group). + tags: + - iamv2 + - account_iam_v2 + description: |- + Deletes a group membership (unassigns a principal from a group). + + :param group_id: int + Required. Internal ID of the group in Databricks. + :param principal_id: int + Required. Internal ID of the principal to be unassigned from the group. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks. + - name: principal_id + in: path + required: true + schema: + type: integer + description: >- + Required. Internal ID of the principal to be unassigned from the + group. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_direct_group_member + summary: Gets a provisioned direct member of a group. + tags: + - iamv2 + - account_iam_v2 + description: |- + Gets a provisioned direct member of a group. + + :param group_id: int + Required. Internal ID of the group in Databricks. + :param principal_id: int + Required. Internal ID of the principal belonging to the group in Databricks. + + :returns: :class:`DirectGroupMember` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks. + - name: principal_id + in: path + required: true + schema: + type: integer + description: >- + Required. Internal ID of the principal belonging to the group in + Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DirectGroupMember' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/groups/{internal_id}: + delete: + operationId: account_iam_v2_delete_group + summary: Deletes a group from the Databricks account by its internal ID. + tags: + - iamv2 + - account_iam_v2 + description: |- + Deletes a group from the Databricks account by its internal ID. + + :param internal_id: str + Required. Internal ID of the group in Databricks. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the group in Databricks. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_group + summary: Fetches a group from the Databricks account by its internal ID. + tags: + - iamv2 + - account_iam_v2 + description: |- + Fetches a group from the Databricks account by its internal ID. + + :param internal_id: str + Required. Internal ID of the group in Databricks. + + :returns: :class:`Group` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the group in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Group' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + patch: + operationId: account_iam_v2_update_group + summary: >- + Updates an existing group in the Databricks account. Only the fields + named in the update mask are + tags: + - iamv2 + - account_iam_v2 + description: >- + Updates an existing group in the Databricks account. Only the fields + named in the update mask are + + modified. Returns the updated Group resource. + + + :param internal_id: str + Required. Internal ID of the group in Databricks. + :param group: :class:`Group` + Required. Group to be updated in + :param update_mask: str + Optional. The list of fields to update. + + :returns: :class:`Group` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the group in Databricks. + - name: update_mask + in: query + required: true + schema: + type: string + description: Optional. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + group: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + group_name: + type: string + description: Display name of the group. + internal_id: + type: string + description: Internal group ID of the group in Databricks. + description: Required. Group to be updated in + required: + - group + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Group' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/service-principals/{internal_id}: + delete: + operationId: account_iam_v2_delete_service_principal + summary: >- + Deletes a service principal from the Databricks account by its internal + ID. + tags: + - iamv2 + - account_iam_v2 + description: >- + Deletes a service principal from the Databricks account by its internal + ID. + + + :param internal_id: str + Required. Internal ID of the service principal in Databricks. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the service principal in Databricks. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_service_principal + summary: >- + Fetches a service principal from the Databricks account by its internal + ID. + tags: + - iamv2 + - account_iam_v2 + description: >- + Fetches a service principal from the Databricks account by its internal + ID. + + + :param internal_id: str + Required. Internal ID of the service principal in Databricks. + + :returns: :class:`ServicePrincipal` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the service principal in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServicePrincipal' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + patch: + operationId: account_iam_v2_update_service_principal + summary: >- + Updates an existing service principal in the Databricks account. Only + the fields named in the update + tags: + - iamv2 + - account_iam_v2 + description: >- + Updates an existing service principal in the Databricks account. Only + the fields named in the update + + mask are modified. Returns the updated ServicePrincipal resource. + + + :param internal_id: str + Required. Internal ID of the service principal in Databricks. + :param service_principal: :class:`ServicePrincipal` + Required. Service Principal to be updated in + :param update_mask: str + Optional. The list of fields to update. + + :returns: :class:`ServicePrincipal` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the service principal in Databricks. + - name: update_mask + in: query + required: true + schema: + type: string + description: Optional. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + service_principal: + type: object + properties: + account_id: + type: string + description: >- + The parent account ID for the service principal in + Databricks. + account_sp_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: >- + The activity status of a service principal in a + Databricks account. + application_id: + type: string + description: Application ID of the service principal. + display_name: + type: string + description: Display name of the service principal. + external_id: + type: string + description: >- + ExternalId of the service principal in the customer's + IdP. + internal_id: + type: string + description: >- + Internal service principal ID of the service principal + in Databricks. + description: Required. Service Principal to be updated in + required: + - service_principal + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServicePrincipal' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/users/{internal_id}: + delete: + operationId: account_iam_v2_delete_user + summary: Deletes a user from the Databricks account by its internal ID. + tags: + - iamv2 + - account_iam_v2 + description: |- + Deletes a user from the Databricks account by its internal ID. + + :param internal_id: str + Required. Internal ID of the user in Databricks. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the user in Databricks. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_user + summary: Fetches a user from the Databricks account by its internal ID. + tags: + - iamv2 + - account_iam_v2 + description: |- + Fetches a user from the Databricks account by its internal ID. + + :param internal_id: str + Required. Internal ID of the user in Databricks. + + :returns: :class:`User` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the user in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/User' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + patch: + operationId: account_iam_v2_update_user + summary: >- + Updates an existing user in the Databricks account, returning the + updated User resource. The behavior + tags: + - iamv2 + - account_iam_v2 + description: >- + Updates an existing user in the Databricks account, returning the + updated User resource. The behavior + + is the same whether or not Account Identity Management (AIM) is enabled. + Only the fields named in the + + update mask are modified; the updatable fields are fullName.givenName, + fullName.familyName, status, + + and externalId. + + + :param internal_id: str + Required. Internal ID of the user in Databricks. + :param user: :class:`User` + Required. User to be updated in + :param update_mask: str + Optional. The list of fields to update. + + :returns: :class:`User` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the user in Databricks. + - name: update_mask + in: query + required: true + schema: + type: string + description: Optional. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + user: + type: object + properties: + account_id: + type: string + description: The accountId parent of the user in Databricks. + account_user_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of a user in a Databricks account. + external_id: + type: string + description: ExternalId of the user in the customer's IdP. + full_name: + type: object + properties: + family_name: + type: string + given_name: + type: string + internal_id: + type: string + description: Internal userId of the user in Databricks. + username: + type: string + description: Username/email of the user. + description: Required. User to be updated in + required: + - user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/User' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/workspaces/{workspace_id}/workspace-assignment-details/{principal_id}: + delete: + operationId: account_iam_v2_delete_workspace_assignment_detail + summary: >- + Deletes a workspace assignment detail for a principal, revoking all + associated entitlements. + tags: + - iamv2 + - account_iam_v2 + description: >- + Deletes a workspace assignment detail for a principal, revoking all + associated entitlements. + + Entitlement revocations are applied individually and non-atomically — if + a failure occurs partway + + through, the principal remains assigned with a subset of its original + entitlements, and the operation + + is safe to retry. + + + :param workspace_id: int + The workspace ID where the principal has access. + :param principal_id: int + Required. ID of the principal in Databricks to delete workspace assignment for. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: The workspace ID where the principal has access. + - name: principal_id + in: path + required: true + schema: + type: integer + description: >- + Required. ID of the principal in Databricks to delete workspace + assignment for. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + get: + operationId: account_iam_v2_get_workspace_assignment_detail + summary: Returns the assignment details for a principal in a workspace. + tags: + - iamv2 + - account_iam_v2 + description: |- + Returns the assignment details for a principal in a workspace. + + :param workspace_id: int + Required. The workspace ID for which the assignment details are being requested. + :param principal_id: int + Required. The internal ID of the principal (user/sp/group) for which the assignment details are + being requested. + + :returns: :class:`WorkspaceAssignmentDetail` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: >- + Required. The workspace ID for which the assignment details are + being requested. + - name: principal_id + in: path + required: true + schema: + type: integer + description: >- + Required. The internal ID of the principal (user/sp/group) for which + the assignment details are being requested. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + patch: + operationId: account_iam_v2_update_workspace_assignment_detail + summary: >- + Updates the entitlements of a directly assigned principal in a + workspace. Entitlement changes are + tags: + - iamv2 + - account_iam_v2 + description: >- + Updates the entitlements of a directly assigned principal in a + workspace. Entitlement changes are + + applied individually and non-atomically — if a failure occurs partway + through, only a subset of the + + requested changes may have been applied. Use + GetWorkspaceAssignmentDetail to confirm the final state. + + + :param workspace_id: int + Required. The workspace ID for which the workspace assignment detail is being updated. + :param principal_id: int + Required. ID of the principal in Databricks. + :param workspace_assignment_detail: :class:`WorkspaceAssignmentDetail` + Required. Workspace assignment detail to be updated in . + :param update_mask: FieldMask + Required. The list of fields to update. + + :returns: :class:`WorkspaceAssignmentDetail` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: >- + Required. The workspace ID for which the workspace assignment detail + is being updated. + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. ID of the principal in Databricks. + - name: update_mask + in: query + required: true + schema: + type: object + description: Required. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + workspace_assignment_detail: + type: object + properties: + principal_id: + type: integer + description: >- + The internal ID of the principal (user/sp/group) in + Databricks. + account_id: + type: string + description: >- + The account ID parent of the workspace where the + principal is assigned + effective_entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + description: >- + The principal's full effective entitlements granted in + this workspace: every entitlement it holds whether + granted directly or via group membership. Populated on + Get; empty on List. + entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + description: >- + Entitlements granted directly to the principal on this + workspace. The only client-settable field: create and + update manage exactly this set (including entitlements + the principal also holds via a group). Not populated by + ListWorkspaceAssignmentDetails (omitted for + scalability); call GetWorkspaceAssignmentDetail to read + the entitlements for a single principal. + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the principal (user/sp/group). + workspace_id: + type: integer + description: The workspace ID where the principal is assigned + required: + - principal_id + description: >- + Required. Workspace assignment detail to be updated in + . + required: + - workspace_assignment_detail + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/{name}: + get: + operationId: account_iam_v2_get_external_user + summary: >- + Retrieves an external user with the given external ID from the + customer's IdP. If the user does not + tags: + - iamv2 + - account_iam_v2 + description: >- + Retrieves an external user with the given external ID from the + customer's IdP. If the user does not + + exist, it will be created. If the customer is not onboarded onto + Automatic Identity Management (AIM), + + this will return an error. + + + :param name: str + Required. The resource name of the external user. Format: + accounts/{account_id}/external-users/{external_user_id} + + :returns: :class:`ExternalUser` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Required. The resource name of the external user. Format: + accounts/{account_id}/external-users/{external_user_id} + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalUser' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/workspaces/{workspace_id}/workspace-access-details/{principal_id}: + get: + operationId: account_iam_v2_get_workspace_access_detail + summary: >- + Returns the access details for a principal in a workspace. Allows for + checking access details for any + tags: + - iamv2 + - account_iam_v2 + description: >- + Returns the access details for a principal in a workspace. Allows for + checking access details for any + + provisioned principal (user, service principal, or group) in a + workspace. + + + - Provisioned principal here refers to one that has been synced into + Databricks from the customer's + IdP or added explicitly to Databricks via SCIM/UI. Allows for passing in a "view" parameter to + control what fields are returned (BASIC by default or FULL). + + :param workspace_id: int + Required. The workspace ID for which the access details are being requested. + :param principal_id: int + Required. The internal ID of the principal (user/sp/group) for which the access details are being + requested. + :param view: :class:`WorkspaceAccessDetailView` (optional) + Controls what fields are returned. + + :returns: :class:`WorkspaceAccessDetail` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: >- + Required. The workspace ID for which the access details are being + requested. + - name: principal_id + in: path + required: true + schema: + type: integer + description: >- + Required. The internal ID of the principal (user/sp/group) for which + the access details are being requested. + - name: view + in: query + required: false + schema: + $ref: '#/components/schemas/WorkspaceAccessDetailView' + description: Controls what fields are returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAccessDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/principals/{principal_id}/transitive-parent-groups: + get: + operationId: account_iam_v2_list_transitive_parent_groups + summary: Lists all transitive parent groups of a principal. + tags: + - iamv2 + - account_iam_v2 + description: |- + Lists all transitive parent groups of a principal. + + :param principal_id: int + Required. Internal ID of the principal in Databricks whose transitive parent groups are being + listed. + :param page_size: int (optional) + The maximum number of parent groups to return. The service may return fewer than this value. If not + provided, defaults to 1000 (also the maximum allowed). + :param page_token: str (optional) + A page token, received from a previous ListTransitiveParentGroups call. Provide this to retrieve the + subsequent page. + + :returns: :class:`ListTransitiveParentGroupsResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: principal_id + in: path + required: true + schema: + type: integer + description: >- + Required. Internal ID of the principal in Databricks whose + transitive parent groups are being listed. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of parent groups to return. The service may + return fewer than this value. If not provided, defaults to 1000 + (also the maximum allowed). + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ListTransitiveParentGroups + call. Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListTransitiveParentGroupsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/workspaces/{workspace_id}/workspace-access-details: + get: + operationId: account_iam_v2_list_workspace_access_details + summary: >- + Lists the access details of every provisioned principal (user, service + principal, or group) with + tags: + - iamv2 + - account_iam_v2 + description: >- + Lists the access details of every provisioned principal (user, service + principal, or group) with + + access to the given workspace, returning one page per call. + + + - Provisioned principal here refers to one that has been synced into + Databricks from the customer's + IdP or added explicitly to Databricks via SCIM/UI. + + :param workspace_id: int + The workspace ID for which the workspace access details are being fetched. + :param page_size: int (optional) + The maximum number of workspace access details to return. The service may return fewer than this + value. + :param page_token: str (optional) + A page token, received from a previous ListWorkspaceAccessDetails call. Provide this to retrieve the + subsequent page. + + :returns: :class:`ListWorkspaceAccessDetailsResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: workspace_id + in: path + required: true + schema: + type: integer + description: >- + The workspace ID for which the workspace access details are being + fetched. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of workspace access details to return. The + service may return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ListWorkspaceAccessDetails + call. Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListWorkspaceAccessDetailsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/groups/resolve-by-external-id: + post: + operationId: account_iam_v2_resolve_group + summary: >- + Resolves a group with the given external ID from the customer's IdP. If + the group does not exist, it + tags: + - iamv2 + - account_iam_v2 + description: >- + Resolves a group with the given external ID from the customer's IdP. If + the group does not exist, it + + will be created in the account. If the customer is not onboarded onto + Automatic Identity Management + + (AIM), this will return an error. + + + :param external_id: str + Required. The external ID of the group in the customer's IdP. + + :returns: :class:`ResolveGroupResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + external_id: + type: string + description: >- + Required. The external ID of the group in the customer's + IdP. + required: + - external_id + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ResolveGroupResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/service-principals/resolve-by-external-id: + post: + operationId: account_iam_v2_resolve_service_principal + summary: >- + Resolves an SP with the given external ID from the customer's IdP. If + the SP does not exist, it will + tags: + - iamv2 + - account_iam_v2 + description: >- + Resolves an SP with the given external ID from the customer's IdP. If + the SP does not exist, it will + + be created. If the customer is not onboarded onto Automatic Identity + Management (AIM), this will + + return an error. + + + :param external_id: str + Required. The external ID of the service principal in the customer's IdP. + + :returns: :class:`ResolveServicePrincipalResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + external_id: + type: string + description: >- + Required. The external ID of the service principal in the + customer's IdP. + required: + - external_id + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ResolveServicePrincipalResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API + /api/2.0/identity/accounts/{account_id}/users/resolve-by-external-id: + post: + operationId: account_iam_v2_resolve_user + summary: >- + Resolves a user with the given external ID from the customer's IdP. If + the user does not exist, it + tags: + - iamv2 + - account_iam_v2 + description: >- + Resolves a user with the given external ID from the customer's IdP. If + the user does not exist, it + + will be created. If the customer is not onboarded onto Automatic + Identity Management (AIM), this will + + return an error. + + + :param external_id: str + Required. The external ID of the user in the customer's IdP. + + :returns: :class:`ResolveUserResponse` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + external_id: + type: string + description: Required. The external ID of the user in the customer's IdP. + required: + - external_id + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ResolveUserResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AccountIamV2API +components: + schemas: + AccountAccessIdentityRule: + type: object + properties: + action: + $ref: '#/components/schemas/AccountAccessRuleAction' + description: Currently, only DENY action is supported. + display_name: + type: string + description: Display name of the principal. + external_principal_id: + type: string + description: External ID of the principal in the customer's IdP. + name: + type: string + description: >- + Fully qualified name for the rule. Format: + accounts/{account_id}/account-access-identity-rules/{external_principal_id} + principal_type: + $ref: '#/components/schemas/PrincipalType' + description: >- + The type of the principal (user/service principal/group). This field + is populated by the server based on the external_principal_id. + required: + - action + description: An identity rule that controls which principals can access an account. + AttributeControlEntry: + type: object + properties: + is_allowed: + type: boolean + description: >- + Whether the attribute is permitted. Always true in the Identity + Attributes PrPr. + name: + type: string + description: >- + The resource name of the entry. Format: + accounts/{account_id}/attribute-control-entries/{attribute_name} + where {attribute_name} is the IdP attribute being governed (e.g. + "department"). + description: >- + A single entry in an account's identity attribute control list. The + presence of an entry means + the governed attribute (the last segment of ``name``) is controlled across all write sources + (SCIM, Identity APIs, AIM); is_allowed records whether the attribute is permitted (always true + in the Identity Attributes PrPr). For the Identity Attributes PrPr the control list is an + explicit allowlist: an attribute that is NOT present in the control list is treated as not + allowed. This may change post-PrPr. + DirectGroupMember: + type: object + properties: + principal_id: + type: integer + description: Internal ID of the principal in Databricks. + display_name: + type: string + description: Display name of the principal. + external_id: + type: string + description: The external ID of the principal in Databricks. + membership_source: + $ref: '#/components/schemas/GroupMembershipSource' + description: The source of group membership (internal or from identity provider). + principal_type: + $ref: '#/components/schemas/PrincipalType' + description: The type of the principal (user/service principal/group). + required: + - principal_id + description: >- + Represents a principal that is a direct member of a group, with its + source of membership. + ExternalGroup: + type: object + properties: + account_id: + type: string + description: The parent account ID, from Databricks. + display_name: + type: string + description: Display name of the group from the customer's IdP. + external_group_id: + type: string + description: The external ID of the group in the customer's IdP. + internal_id: + type: string + description: Internal groupId of the group in Databricks. + name: + type: string + description: >- + The resource name of the external group. Format: + accounts/{account_id}/external-groups/{external_group_id} + description: >- + An external group from the customer's Identity Provider, resolved into + Databricks. This is a + read-only resource keyed by the IdP external ID. The Get method may trigger an idempotent sync + from the customer's IdP to provision or refresh the group's data in Databricks. + ExternalServicePrincipal: + type: object + properties: + account_id: + type: string + description: The parent account ID, from Databricks. + account_sp_status: + $ref: '#/components/schemas/State' + description: >- + The activity status of the service principal in the Databricks + account. + application_id: + type: string + description: Application ID of the service principal, from the customer's IdP. + display_name: + type: string + description: Display name of the service principal, from the customer's IdP. + external_service_principal_id: + type: string + description: The external ID of the service principal in the customer's IdP. + internal_id: + type: string + description: Internal servicePrincipalId of the service principal in Databricks. + name: + type: string + description: >- + The resource name of the external service principal. Format: + accounts/{account_id}/external-service-principals/{external_service_principal_id} + description: >- + An external service principal from the customer's Identity Provider, + resolved into Databricks. + This is a read-only resource keyed by the IdP external ID. The Get method may trigger an + idempotent sync from the customer's IdP to provision or refresh the service principal's data in + Databricks. + ExternalUser: + type: object + properties: + account_id: + type: string + description: The parent account ID, from Databricks. + account_user_status: + $ref: '#/components/schemas/State' + description: The activity status of the user in the Databricks account. + display_name: + type: string + description: Display name of the user from the customer's IdP. + external_user_id: + type: string + description: The external ID of the user in the customer's IdP. + full_name: + $ref: '#/components/schemas/FullName' + description: The full name of the user, from the customer's IdP. + internal_id: + type: string + description: Internal userId of the user in Databricks. + name: + type: string + description: >- + The resource name of the external user. Format: + accounts/{account_id}/external-users/{external_user_id} + username: + type: string + description: Username/email of the user, from Databricks. + description: >- + An external user from the customer's Identity Provider, resolved into + Databricks. This is a + read-only resource that allows customers to look up external user identities by their IdP + external ID and retrieve the corresponding Databricks internal ID and metadata. The Get method + may trigger an idempotent sync from the customer's IdP to provision or refresh the user's data + in Databricks. + FullName: + type: object + properties: + family_name: + type: string + description: The family (last) name of the user, from the customer's IdP. + given_name: + type: string + description: The given (first) name of the user, from the customer's IdP. + description: The full name of a user. + Group: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + group_name: + type: string + description: Display name of the group. + internal_id: + type: string + description: Internal group ID of the group in Databricks. description: The details of a Group resource. + ListAccountAccessIdentityRulesResponse: + type: object + properties: + account_access_identity_rules: + type: array + items: + $ref: '#/components/schemas/AccountAccessIdentityRule' + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + description: Response message for listing account access identity rules. + ListAttributeControlEntriesResponse: + type: object + properties: + attribute_control_entries: + type: array + items: + $ref: '#/components/schemas/AttributeControlEntry' + next_page_token: + type: string + description: >- + A token to retrieve the next page; empty when there are no more + results. + description: Response message for listing identity attribute control-list entries. + ListDirectGroupMembersResponse: + type: object + properties: + direct_group_members: + type: array + items: + $ref: '#/components/schemas/DirectGroupMember' + description: The list of direct group members with their membership source type. + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + description: Response message for listing direct group members. + ListGroupsResponse: + type: object + properties: + groups: + type: array + items: + $ref: '#/components/schemas/Group' + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + description: Response message containing a page of groups in the account. + ListServicePrincipalsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + service_principals: + type: array + items: + $ref: '#/components/schemas/ServicePrincipal' + description: Response message containing a page of service principals in the account. + ListTransitiveParentGroupsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + transitive_parent_groups: + type: array + items: + $ref: '#/components/schemas/TransitiveParentGroup' + description: The list of transitive parent groups. + description: >- + Response message for listing all transitive parent groups of a + principal. + ListUsersResponse: + type: object + properties: + next_page_token: + type: string + users: + type: array + items: + $ref: '#/components/schemas/User' + ListWorkspaceAccessDetailsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + workspace_access_details: + type: array + items: + $ref: '#/components/schemas/WorkspaceAccessDetail' + description: Response message containing a page of workspace access details. + ListWorkspaceAssignmentDetailsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as page_token to retrieve the next page. + If this field is omitted, there are no subsequent pages. + workspace_assignment_details: + type: array + items: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + description: Response message for listing workspace assignment details. ResolveGroupResponse: type: object properties: @@ -325,11 +3348,24 @@ components: type: string description: ExternalId of the service principal in the customer's IdP. internal_id: - type: integer + type: string description: >- Internal service principal ID of the service principal in Databricks. description: The details of a ServicePrincipal resource. + TransitiveParentGroup: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + internal_id: + type: string + description: Internal group ID of the group in Databricks. + description: Represents a group that is a transitive parent of a principal. User: type: object properties: @@ -342,16 +3378,16 @@ components: external_id: type: string description: ExternalId of the user in the customer's IdP. + full_name: + $ref: '#/components/schemas/UserFullName' internal_id: - type: integer + type: string description: Internal userId of the user in Databricks. - name: - $ref: '#/components/schemas/UserName' username: type: string description: Username/email of the user. description: The details of a User resource. - UserName: + UserFullName: type: object properties: family_name: @@ -387,6 +3423,142 @@ components: type: integer description: The workspace ID where the principal has access. description: The details of a principal's access to a workspace. + WorkspaceAssignmentDetail: + type: object + properties: + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + account_id: + type: string + description: >- + The account ID parent of the workspace where the principal is + assigned + effective_entitlements: + type: array + items: + $ref: '#/components/schemas/Entitlement' + description: >- + The principal's full effective entitlements granted in this + workspace: every entitlement it holds whether granted directly or + via group membership. Populated on Get; empty on List. + entitlements: + type: array + items: + $ref: '#/components/schemas/Entitlement' + description: >- + Entitlements granted directly to the principal on this workspace. + The only client-settable field: create and update manage exactly + this set (including entitlements the principal also holds via a + group). Not populated by ListWorkspaceAssignmentDetails (omitted for + scalability); call GetWorkspaceAssignmentDetail to read the + entitlements for a single principal. + principal_type: + $ref: '#/components/schemas/PrincipalType' + workspace_id: + type: integer + description: The workspace ID where the principal is assigned + required: + - principal_id + description: >- + The direct assignment of a provisioned account-level principal (user, + service principal, or + group) to a workspace, together with the entitlements that assignment grants in the workspace. + + A WorkspaceAssignmentDetail exists only for principals that are directly assigned to the + workspace; principals that merely inherit workspace access through a group are not represented + here (see WorkspaceAccessDetail / WorkspaceIdentityDetail for the effective, direct-or-indirect + view). Creating the resource assigns the principal to the workspace; deleting it removes the + assignment. The ``entitlements`` field is the only client-settable field and defines the + entitlements granted directly on this assignment; ``effective_entitlements`` is the read-only + union of those plus any granted via group membership. + + A direct assignment always carries at least one directly-assigned entitlement: the assignment is + what grants the entitlement, so a WorkspaceAssignmentDetail with an empty ``entitlements`` set + is not a valid state. Both create and update require a non-empty ``entitlements`` set (an empty + set is rejected); to remove a principal's assignment entirely, delete the resource. + + This resource replaces workspace assignment previously managed through the workspace SCIM and + permission-assignment APIs, and is intended for account and workspace admins. + WorkspaceIdentityDetail: + type: object + properties: + assignment_type: + $ref: '#/components/schemas/WorkspaceIdentityDetailAssignmentType' + description: >- + The type of assignment the principal has to the workspace (direct or + indirect). + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + principal_type: + $ref: '#/components/schemas/PrincipalType' + description: The type of the principal (user/service principal/group). + workspace_identity_status: + $ref: '#/components/schemas/State' + description: The activity status of an identity in a Databricks workspace. + description: >- + The details of a directly or indirectly assigned principal's details in + a workspace. + AccountAccessRuleAction: + type: string + x-enum: + - DENY + description: >- + The action type for an account access identity rule (currently DENY + only). + Entitlement: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + GroupMembershipSource: + type: string + x-enum: + - IDENTITY_PROVIDER + - INTERNAL + description: The source of the group membership (internal or from identity provider). PrincipalType: type: string x-enum: @@ -416,6 +3588,12 @@ components: description: >- Controls what fields are returned in the GetWorkspaceAccessDetail response. + WorkspaceIdentityDetailAssignmentType: + type: string + x-enum: + - DIRECT + - INDIRECT + description: The type of assignment the principal has to the workspace. WorkspacePermission: type: string x-enum: @@ -423,52 +3601,490 @@ components: - USER_PERMISSION description: The type of permission a principal has to a workspace (admin/user). x-stackQL-resources: - account_iam: - id: databricks_account.iamv2.account_iam - name: account_iam - title: Account Iam + account_access_identity_rules: + id: databricks_account.iamv2.account_access_identity_rules + name: account_access_identity_rules + title: Account Access Identity Rules + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1{parent}~1account-access-identity-rules/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1{parent}~1account-access-identity-rules/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.account_access_identity_rules + delete: + operation: + $ref: >- + #/paths/~1api~12.0~1{parent}~1account-access-identity-rules~1{external_principal_id}/delete + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: >- + #/paths/~1api~12.0~1{parent}~1account-access-identity-rules~1{external_principal_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/account_access_identity_rules/methods/get + - $ref: >- + #/components/x-stackQL-resources/account_access_identity_rules/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/account_access_identity_rules/methods/create + update: [] + delete: + - $ref: >- + #/components/x-stackQL-resources/account_access_identity_rules/methods/delete + replace: [] + attribute_control_entries: + id: databricks_account.iamv2.attribute_control_entries + name: attribute_control_entries + title: Attribute Control Entries + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1{parent}~1attribute-control-entries/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1{parent}~1attribute-control-entries/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.attribute_control_entries + delete: + operation: + $ref: '#/paths/~1api~12.0~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/attribute_control_entries/methods/list + - $ref: >- + #/components/x-stackQL-resources/attribute_control_entries/methods/get + insert: + - $ref: >- + #/components/x-stackQL-resources/attribute_control_entries/methods/create + update: + - $ref: >- + #/components/x-stackQL-resources/attribute_control_entries/methods/update + delete: + - $ref: >- + #/components/x-stackQL-resources/attribute_control_entries/methods/delete + replace: [] + direct_group_members: + id: databricks_account.iamv2.direct_group_members + name: direct_group_members + title: Direct Group Members + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{group_id}~1direct-members/post + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{group_id}~1direct-members/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.direct_group_members + delete: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{group_id}~1direct-members~1{principal_id}/delete + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{group_id}~1direct-members~1{principal_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/direct_group_members/methods/get' + - $ref: '#/components/x-stackQL-resources/direct_group_members/methods/list' + insert: + - $ref: >- + #/components/x-stackQL-resources/direct_group_members/methods/create + update: [] + delete: + - $ref: >- + #/components/x-stackQL-resources/direct_group_members/methods/delete + replace: [] + groups: + id: databricks_account.iamv2.groups + name: groups + title: Groups + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.groups + delete: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{internal_id}/delete + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{internal_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1{internal_id}/patch + response: + mediaType: application/json + openAPIDocKey: '200' + resolve: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1resolve-by-external-id/post + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/groups/methods/get' + - $ref: '#/components/x-stackQL-resources/groups/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/groups/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/groups/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/groups/methods/delete' + replace: [] + service_principals: + id: databricks_account.iamv2.service_principals + name: service_principals + title: Service Principals + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1service-principals/post + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1service-principals/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.service_principals + delete: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1service-principals~1{internal_id}/delete + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1service-principals~1{internal_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1service-principals~1{internal_id}/patch + response: + mediaType: application/json + openAPIDocKey: '200' + resolve: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1service-principals~1resolve-by-external-id/post + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/get' + - $ref: '#/components/x-stackQL-resources/service_principals/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/delete' + replace: [] + users: + id: databricks_account.iamv2.users + name: users + title: Users methods: - get_workspace_access_detail: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.users + delete: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users~1{internal_id}/delete + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users~1{internal_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive operation: $ref: >- - #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspaceAccessDetails~1{principal_id}/get + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users~1{internal_id}/patch response: mediaType: application/json openAPIDocKey: '200' - resolve_group: + resolve: config: requestBodyTranslate: algorithm: naive operation: $ref: >- - #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1groups~1resolveByExternalId/post + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users~1resolve-by-external-id/post response: mediaType: application/json openAPIDocKey: '200' - resolve_service_principal: + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/users/methods/get' + - $ref: '#/components/x-stackQL-resources/users/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/users/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/users/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/users/methods/delete' + replace: [] + workspace_assignment_details: + id: databricks_account.iamv2.workspace_assignment_details + name: workspace_assignment_details + title: Workspace Assignment Details + methods: + create: config: requestBodyTranslate: algorithm: naive operation: $ref: >- - #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1servicePrincipals~1resolveByExternalId/post + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-assignment-details/post + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-assignment-details/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.workspace_assignment_details + delete: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-assignment-details~1{principal_id}/delete + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-assignment-details~1{principal_id}/get response: mediaType: application/json openAPIDocKey: '200' - resolve_user: + update: config: requestBodyTranslate: algorithm: naive operation: $ref: >- - #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1users~1resolveByExternalId/post + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-assignment-details~1{principal_id}/patch + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/workspace_assignment_details/methods/get + - $ref: >- + #/components/x-stackQL-resources/workspace_assignment_details/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/workspace_assignment_details/methods/create + update: + - $ref: >- + #/components/x-stackQL-resources/workspace_assignment_details/methods/update + delete: + - $ref: >- + #/components/x-stackQL-resources/workspace_assignment_details/methods/delete + replace: [] + external_users: + id: databricks_account.iamv2.external_users + name: external_users + title: External Users + methods: + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/external_users/methods/get' + insert: [] + update: [] + delete: [] + replace: [] + workspace_access_details: + id: databricks_account.iamv2.workspace_access_details + name: workspace_access_details + title: Workspace Access Details + methods: + get: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-access-details~1{principal_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1workspaces~1{workspace_id}~1workspace-access-details/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.workspace_access_details + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/workspace_access_details/methods/get + - $ref: >- + #/components/x-stackQL-resources/workspace_access_details/methods/list + insert: [] + update: [] + delete: [] + replace: [] + transitive_parent_groups: + id: databricks_account.iamv2.transitive_parent_groups + name: transitive_parent_groups + title: Transitive Parent Groups + methods: + list: + operation: + $ref: >- + #/paths/~1api~12.0~1identity~1accounts~1{account_id}~1principals~1{principal_id}~1transitive-parent-groups/get response: mediaType: application/json openAPIDocKey: '200' + objectKey: $.transitive_parent_groups sqlVerbs: select: - $ref: >- - #/components/x-stackQL-resources/account_iam/methods/get_workspace_access_detail + #/components/x-stackQL-resources/transitive_parent_groups/methods/list insert: [] update: [] delete: [] diff --git a/providers/src/databricks_account/v00.00.00000/services/oauth2.yaml b/providers/src/databricks_account/v00.00.00000/services/oauth2.yaml index b8ea02ba..418c3921 100644 --- a/providers/src/databricks_account/v00.00.00000/services/oauth2.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/oauth2.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks oauth2 service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.oauth2 servers: - url: https://accounts.cloud.databricks.com @@ -55,8 +55,8 @@ paths: description: Description of the federation policy. name: type: string - description: Resource name for the federation policy. Example values include `accounts//federationPolicies/my-federation-policy` - for Account Federation Policies, and `accounts//servicePrincipals//federationPolicies/my-federation-policy` + description: Resource name for the federation policy. Example values include ``accounts//federationPolicies/my-federation-policy`` + for Account Federation Policies, and ``accounts//servicePrincipals//federationPolicies/my-federation-policy`` for Service Principal Federation Policies. Typically an output parameter, which does not need to be specified in create or update requests. If specified in a request, must match the value in the request URL. @@ -321,8 +321,8 @@ paths: description: Description of the federation policy. name: type: string - description: Resource name for the federation policy. Example values include `accounts//federationPolicies/my-federation-policy` - for Account Federation Policies, and `accounts//servicePrincipals//federationPolicies/my-federation-policy` + description: Resource name for the federation policy. Example values include ``accounts//federationPolicies/my-federation-policy`` + for Account Federation Policies, and ``accounts//servicePrincipals//federationPolicies/my-federation-policy`` for Service Principal Federation Policies. Typically an output parameter, which does not need to be specified in create or update requests. If specified in a request, must match the value in the request URL. @@ -409,10 +409,15 @@ paths: description: |- Create Custom OAuth App Integration. - You can retrieve the custom OAuth app integration via :method:CustomAppIntegration/get. + You can retrieve the custom OAuth app integration via `CustomAppIntegration/get + `__. :param confidential: bool (optional) This field indicates whether an OAuth client secret is required to authenticate this client. + :param is_agent: bool (optional) + When true, the custom OAuth app integration is backed by an agent service principal instead of a + standard application service principal. The public surface stays a simple boolean; the server maps + it to the internal OAuthClientApp.agent message. :param name: str (optional) Name of the custom OAuth app :param redirect_urls: List[str] (optional) @@ -442,6 +447,11 @@ paths: confidential: type: boolean description: This field indicates whether an OAuth client secret is required to authenticate this client. + is_agent: + type: boolean + description: When true, the custom OAuth app integration is backed by an agent service principal instead + of a standard application service principal. The public surface stays a simple boolean; the server maps + it to the internal OAuthClientApp.agent message. name: type: string description: Name of the custom OAuth app @@ -513,6 +523,10 @@ paths: :param include_creator_username: bool (optional) :param page_size: int (optional) :param page_token: str (optional) + :param search_keyword: str (optional) + Search term to filter app integrations (case-insensitive substring match). Matches if the display + name, the OAuth client_id (a UUID for custom apps or a stable string like "databricks-cli" for + published apps), or the integration id contains the term. :returns: Iterator over :class:`GetCustomAppIntegrationOutput` parameters: @@ -536,6 +550,14 @@ paths: required: false schema: type: string + - name: search_keyword + in: query + required: false + schema: + type: string + description: Search term to filter app integrations (case-insensitive substring match). Matches if the display name, + the OAuth client_id (a UUID for custom apps or a stable string like "databricks-cli" for published apps), or the + integration id contains the term. responses: '200': description: Success @@ -564,7 +586,7 @@ paths: - custom_app_integration description: |- Delete an existing Custom OAuth App Integration. You can retrieve the custom OAuth app integration via - :method:CustomAppIntegration/get. + `CustomAppIntegration/get `__. :param integration_id: str parameters: @@ -645,7 +667,7 @@ paths: - custom_app_integration description: |- Updates an existing custom OAuth App Integration. You can retrieve the custom OAuth app integration - via :method:CustomAppIntegration/get. + via `CustomAppIntegration/get `__. :param integration_id: str :param redirect_urls: List[str] (optional) @@ -790,7 +812,8 @@ paths: description: |- Create Published OAuth App Integration. - You can retrieve the published OAuth app integration via :method:PublishedAppIntegration/get. + You can retrieve the published OAuth app integration via `PublishedAppIntegration/get + `__. :param app_id: str (optional) App id of the OAuth published app integration. For example power-bi, tableau-deskop @@ -863,6 +886,9 @@ paths: :param page_size: int (optional) :param page_token: str (optional) + :param search_keyword: str (optional) + Search term to filter app integrations (case-insensitive substring match). Matches if the display + name, the OAuth client_id, or the integration id contains the term. :returns: Iterator over :class:`GetPublishedAppIntegrationOutput` parameters: @@ -881,6 +907,13 @@ paths: required: false schema: type: string + - name: search_keyword + in: query + required: false + schema: + type: string + description: Search term to filter app integrations (case-insensitive substring match). Matches if the display name, + the OAuth client_id, or the integration id contains the term. responses: '200': description: Success @@ -909,7 +942,8 @@ paths: - published_app_integration description: |- Delete an existing Published OAuth App Integration. You can retrieve the published OAuth app - integration via :method:PublishedAppIntegration/get. + integration via `PublishedAppIntegration/get + `__. :param integration_id: str parameters: @@ -988,7 +1022,8 @@ paths: - published_app_integration description: |- Updates an existing published OAuth App Integration. You can retrieve the published OAuth app - integration via :method:PublishedAppIntegration/get. + integration via `PublishedAppIntegration/get + `__. :param integration_id: str :param token_access_policy: :class:`TokenAccessPolicy` (optional) @@ -1098,8 +1133,8 @@ paths: description: Description of the federation policy. name: type: string - description: Resource name for the federation policy. Example values include `accounts//federationPolicies/my-federation-policy` - for Account Federation Policies, and `accounts//servicePrincipals//federationPolicies/my-federation-policy` + description: Resource name for the federation policy. Example values include ``accounts//federationPolicies/my-federation-policy`` + for Account Federation Policies, and ``accounts//servicePrincipals//federationPolicies/my-federation-policy`` for Service Principal Federation Policies. Typically an output parameter, which does not need to be specified in create or update requests. If specified in a request, must match the value in the request URL. @@ -1396,8 +1431,8 @@ paths: description: Description of the federation policy. name: type: string - description: Resource name for the federation policy. Example values include `accounts//federationPolicies/my-federation-policy` - for Account Federation Policies, and `accounts//servicePrincipals//federationPolicies/my-federation-policy` + description: Resource name for the federation policy. Example values include ``accounts//federationPolicies/my-federation-policy`` + for Account Federation Policies, and ``accounts//servicePrincipals//federationPolicies/my-federation-policy`` for Service Principal Federation Policies. Typically an output parameter, which does not need to be specified in create or update requests. If specified in a request, must match the value in the request URL. @@ -1489,6 +1524,9 @@ paths: :param lifetime: str (optional) The lifetime of the secret in seconds. If this parameter is not provided, the secret will have a default lifetime of 730 days (63072000s). + :param scopes: List[str] (optional) + OAuth API scopes to bind to the secret; a minted token cannot exceed them. Empty = unrestricted + (all-apis). E.g. all-apis, sql, iam.groups:read. :returns: :class:`CreateServicePrincipalSecretResponse` parameters: @@ -1513,6 +1551,12 @@ paths: type: string description: The lifetime of the secret in seconds. If this parameter is not provided, the secret will have a default lifetime of 730 days (63072000s). + scopes: + type: array + items: + type: string + description: OAuth API scopes to bind to the secret; a minted token cannot exceed them. Empty = unrestricted + (all-apis). E.g. all-apis, sql, iam.groups:read. responses: '200': description: Success @@ -1546,12 +1590,12 @@ paths: The service principal ID. :param page_size: int (optional) :param page_token: str (optional) - An opaque page token which was the `next_page_token` in the response of the previous request to list - the secrets for this service principal. Provide this token to retrieve the next page of secret - entries. When providing a `page_token`, all other parameters provided to the request must match the - previous request. To list all of the secrets for a service principal, it is necessary to continue - requesting pages of entries until the response contains no `next_page_token`. Note that the number - of entries returned must not be used to determine when the listing is complete. + An opaque page token which was the ``next_page_token`` in the response of the previous request to + list the secrets for this service principal. Provide this token to retrieve the next page of secret + entries. When providing a ``page_token``, all other parameters provided to the request must match + the previous request. To list all of the secrets for a service principal, it is necessary to + continue requesting pages of entries until the response contains no ``next_page_token``. Note that + the number of entries returned must not be used to determine when the listing is complete. :returns: Iterator over :class:`SecretInfo` parameters: @@ -1576,12 +1620,12 @@ paths: required: false schema: type: string - description: An opaque page token which was the `next_page_token` in the response of the previous request to list + description: An opaque page token which was the ``next_page_token`` in the response of the previous request to list the secrets for this service principal. Provide this token to retrieve the next page of secret entries. When providing - a `page_token`, all other parameters provided to the request must match the previous request. To list all of the + a ``page_token``, all other parameters provided to the request must match the previous request. To list all of the secrets for a service principal, it is necessary to continue requesting pages of entries until the response contains - no `next_page_token`. Note that the number of entries returned must not be used to determine when the listing is - complete. + no ``next_page_token``. Note that the number of entries returned must not be used to determine when the listing + is complete. responses: '200': description: Success @@ -1659,9 +1703,16 @@ components: type: string description: OAuth client-secret generated by the Databricks. If this is a confidential OAuth app client-secret will be generated. + client_secret_expire_time: + type: string + format: date-time + description: The expiration time of the client secret. If not present, the secret does not expire. integration_id: type: string description: Unique integration id for the custom OAuth app + principal_id: + type: integer + description: The principal id of the service principal associated with this OAuth app. CreatePublishedAppIntegrationOutput: type: object properties: @@ -1678,6 +1729,11 @@ components: id: type: string description: ID of the secret + scopes: + type: array + items: + type: string + description: OAuth API scopes bound to this secret. Empty = unrestricted (all-apis). secret: type: string description: Secret Value @@ -1706,8 +1762,8 @@ components: description: Description of the federation policy. name: type: string - description: Resource name for the federation policy. Example values include `accounts//federationPolicies/my-federation-policy` - for Account Federation Policies, and `accounts//servicePrincipals//federationPolicies/my-federation-policy` + description: Resource name for the federation policy. Example values include ``accounts//federationPolicies/my-federation-policy`` + for Account Federation Policies, and ``accounts//servicePrincipals//federationPolicies/my-federation-policy`` for Service Principal Federation Policies. Typically an output parameter, which does not need to be specified in create or update requests. If specified in a request, must match the value in the request URL. oidc_policy: @@ -1745,6 +1801,9 @@ components: name: type: string description: The display name of the custom OAuth app + principal_id: + type: integer + description: The principal id of the service principal associated with this OAuth app. redirect_urls: type: array items: @@ -1901,6 +1960,11 @@ components: id: type: string description: ID of the secret + scopes: + type: array + items: + type: string + description: OAuth API scopes bound to this secret. Empty = unrestricted (all-apis). secret_hash: type: string description: Secret Hash diff --git a/providers/src/databricks_account/v00.00.00000/services/provisioning.yaml b/providers/src/databricks_account/v00.00.00000/services/provisioning.yaml index 4a34af70..d5d62e91 100644 --- a/providers/src/databricks_account/v00.00.00000/services/provisioning.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/provisioning.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks provisioning service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.provisioning servers: - url: https://accounts.cloud.databricks.com @@ -23,13 +23,11 @@ paths: clusters. For your AWS IAM role, you need to trust the External ID (the Databricks Account API account ID) in the returned credential object, and configure the required access policy. - Save the response's `credentials_id` field, which is the ID for your new credential configuration + Save the response's ``credentials_id`` field, which is the ID for your new credential configuration object. - For information about how to create a new workspace with this API, see [Create a new workspace using - the Account API] - - [Create a new workspace using the Account API]: http://docs.databricks.com/administration-guide/account-api/new-workspace.html + For information about how to create a new workspace with this API, see `Create a new workspace using + the Account API `__ :param credentials_name: str The human-readable name of the credential configuration object. @@ -229,9 +227,17 @@ paths: This operation is available only if your account is on the E2 version of the platform or on a select custom plan that allows multiple workspaces per account. + **GCP only**: To create a customer-managed key on GCP, you must include the + ``X-Databricks-GCP-SA-Access-Token`` HTTP header in your request. This header must contain a Google + Cloud OAuth access token with the ``cloud-platform`` scope. The Google identity associated with the + token must also have the ``setIamPermissions`` and ``getIamPermissions`` IAM permissions on the key + resource. For details on obtaining this token, see `Authenticate with Google ID tokens + `__. + :param use_cases: List[:class:`KeyUseCase`] The cases that the key can be used for. :param aws_key_info: :class:`CreateAwsKeyInfo` (optional) + :param azure_key_info: :class:`CreateAzureKeyInfo` (optional) :param gcp_key_info: :class:`CreateGcpKeyInfo` (optional) :returns: :class:`CustomerManagedKey` @@ -305,11 +311,38 @@ paths: description: The AWS KMS key region. reuse_key_for_cluster_volumes: type: boolean - description: This field applies only if the `use_cases` property includes `STORAGE`. If this is set - to true or omitted, the key is also used to encrypt cluster EBS volumes. If you do not want to use - this key for encrypting EBS volumes, set to false. + description: This field applies only if the ``use_cases`` property includes ``STORAGE``. If this is + set to true or omitted, the key is also used to encrypt cluster EBS volumes. If you do not want to + use this key for encrypting EBS volumes, set to false. required: - key_arn + azure_key_info: + type: object + properties: + disk_encryption_set_id: + type: string + key_access_configuration: + type: object + properties: + credential_id: + type: string + description: The structure to store key access credential This is set if the Managed Identity is being + used to access the Azure Key Vault key. + key_name: + type: string + description: The name of the key in KeyVault. + key_vault_uri: + type: string + description: The base URI of the KeyVault. + subscription_id: + type: string + description: The Azure subscription ID to use for Key Vault access validation during CMK creation. + tenant_id: + type: string + description: The tenant id where the KeyVault lives. + version: + type: string + description: The current key version. gcp_key_info: type: object properties: @@ -322,6 +355,10 @@ paths: type: string description: Globally unique service account email that has access to the KMS key. The service account exists within the Databricks CP project. + manual: + type: boolean + description: When true, Databricks will not use OAuth to grant the service account access to the KMS + key. The customer is responsible for granting access manually. required: - kms_key_id required: @@ -748,10 +785,10 @@ paths: internet, see IP access lists. :param private_access_level: :class:`PrivateAccessLevel` (optional) The private access level controls which VPC endpoints can connect to the UI or API of any workspace - that attaches this private access settings object. `ACCOUNT` level access (the default) allows only - VPC endpoints that are registered in your Databricks account connect to your workspace. `ENDPOINT` - level access allows only specified VPC endpoints connect to your workspace. For details, see - allowed_vpc_endpoint_ids. + that attaches this private access settings object. ``ACCOUNT`` level access (the default) allows + only VPC endpoints that are registered in your Databricks account connect to your workspace. + ``ENDPOINT`` level access allows only specified VPC endpoints connect to your workspace. For + details, see allowed_vpc_endpoint_ids. :param private_access_settings_name: str (optional) The human-readable name of the private access settings object. :param public_access_enabled: bool (optional) @@ -790,9 +827,9 @@ paths: - ACCOUNT - ENDPOINT description: The private access level controls which VPC endpoints can connect to the UI or API of any workspace - that attaches this private access settings object. `ACCOUNT` level access (the default) allows only VPC - endpoints that are registered in your Databricks account connect to your workspace. `ENDPOINT` level access - allows only specified VPC endpoints connect to your workspace. For details, see allowed_vpc_endpoint_ids. + that attaches this private access settings object. ``ACCOUNT`` level access (the default) allows only + VPC endpoints that are registered in your Databricks account connect to your workspace. ``ENDPOINT`` level + access allows only specified VPC endpoints connect to your workspace. For details, see allowed_vpc_endpoint_ids. private_access_settings_name: type: string description: The human-readable name of the private access settings object. @@ -1009,9 +1046,9 @@ paths: - ACCOUNT - ENDPOINT description: The private access level controls which VPC endpoints can connect to the UI or API of any - workspace that attaches this private access settings object. `ACCOUNT` level access (the default) + workspace that attaches this private access settings object. ``ACCOUNT`` level access (the default) allows only VPC endpoints that are registered in your Databricks account connect to your workspace. - `ENDPOINT` level access allows only specified VPC endpoints connect to your workspace. For details, + ``ENDPOINT`` level access allows only specified VPC endpoints connect to your workspace. For details, see allowed_vpc_endpoint_ids. private_access_settings_id: type: string @@ -1245,23 +1282,21 @@ paths: /api/2.0/accounts/{account_id}/vpc-endpoints: post: operationId: vpc_endpoints_create - summary: Creates a VPC endpoint configuration, which represents a [VPC endpoint] object in AWS used to + summary: Creates a VPC endpoint configuration, which represents a `VPC endpoint tags: - provisioning - vpc_endpoints description: |- - Creates a VPC endpoint configuration, which represents a [VPC endpoint] object in AWS used to - communicate privately with Databricks over [AWS PrivateLink]. - - After you create the VPC endpoint configuration, the Databricks [endpoint service] automatically - accepts the VPC endpoint. + Creates a VPC endpoint configuration, which represents a `VPC endpoint + `__ object in AWS used to + communicate privately with Databricks over `AWS PrivateLink `__. - Before configuring PrivateLink, read the [Databricks article about PrivateLink]. + After you create the VPC endpoint configuration, the Databricks `endpoint service + `__ + automatically accepts the VPC endpoint. - [AWS PrivateLink]: https://aws.amazon.com/privatelink - [Databricks article about PrivateLink]: https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html - [VPC endpoint]: https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html - [endpoint service]: https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html + Before configuring PrivateLink, read the `Databricks article about PrivateLink + `__. :param aws_vpc_endpoint_id: str (optional) The ID of the VPC endpoint object in AWS. @@ -1413,16 +1448,14 @@ paths: x-stackql-sdk-source: VpcEndpointsAPI get: operationId: vpc_endpoints_get - summary: Gets a VPC endpoint configuration, which represents a [VPC endpoint] object in AWS used to communicate + summary: Gets a VPC endpoint configuration, which represents a `VPC endpoint tags: - provisioning - vpc_endpoints description: |- - Gets a VPC endpoint configuration, which represents a [VPC endpoint] object in AWS used to communicate - privately with Databricks over [AWS PrivateLink]. - - [AWS PrivateLink]: https://aws.amazon.com/privatelink - [VPC endpoint]: https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html + Gets a VPC endpoint configuration, which represents a `VPC endpoint + `__ object in AWS used to + communicate privately with Databricks over `AWS PrivateLink `__. :param vpc_endpoint_id: str Databricks VPC endpoint ID. @@ -1486,7 +1519,8 @@ paths: information, you cannot reuse a Databricks Account API network configuration across workspaces. For information about how to create a new workspace with this API including error handling, see - [Create a new workspace using the Account API]. + `Create a new workspace using the Account API + `__. Important: Customer-managed VPCs, PrivateLink, and customer-managed keys are supported on a limited set of deployment and subscription types. If you have questions about availability, contact your @@ -1495,18 +1529,17 @@ paths: This operation is available only if your account is on the E2 version of the platform or on a select custom plan that allows multiple workspaces per account. - [Create a new workspace using the Account API]: http://docs.databricks.com/administration-guide/account-api/new-workspace.html - :param aws_region: str (optional) + :param azure_workspace_info: :class:`AzureWorkspaceInfo` (optional) :param cloud: str (optional) DEPRECATED: This field is being ignored by the server and will be removed in the future. The cloud - name. This field always has the value `gcp`. + name. This field always has the value ``gcp``. :param cloud_resource_container: :class:`CloudResourceContainer` (optional) :param compute_mode: :class:`CustomerFacingComputeMode` (optional) - If the compute mode is `SERVERLESS`, a serverless workspace is created that comes pre-configured + If the compute mode is ``SERVERLESS``, a serverless workspace is created that comes pre-configured with serverless compute and default storage, providing a fully-managed, enterprise-ready SaaS experience. This means you don't need to provide any resources managed by you, such as credentials, - storage, or network. If the compute mode is `HYBRID` (which is the default option), a classic + storage, or network. If the compute mode is ``HYBRID`` (which is the default option), a classic workspace is created that uses customer-managed resources. :param credentials_id: str (optional) ID of the workspace's credential configuration object. @@ -1534,7 +1567,7 @@ paths: :param gke_config: :class:`GkeConfig` (optional) :param location: str (optional) The Google Cloud region of the workspace data plane in your Google account (for example, - `us-east4`). + ``us-east4``). :param managed_services_customer_managed_key_id: str (optional) The ID of the workspace's managed services encryption key configuration object. This is used to help protect and control access to the workspace's notebooks, secrets, Databricks SQL queries, and query @@ -1549,12 +1582,10 @@ paths: :param pricing_tier: :class:`PricingTier` (optional) :param private_access_settings_id: str (optional) ID of the workspace's private access settings object. Only used for PrivateLink. You must specify - this ID if you are using [AWS PrivateLink] for either front-end (user-to-workspace connection), - back-end (data plane to control plane connection), or both connection types. Before configuring - PrivateLink, read the [Databricks article about PrivateLink].", - - [AWS PrivateLink]: https://aws.amazon.com/privatelink/ - [Databricks article about PrivateLink]: https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html + this ID if you are using `AWS PrivateLink `__ for either + front-end (user-to-workspace connection), back-end (data plane to control plane connection), or both + connection types. Before configuring PrivateLink, read the `Databricks article about PrivateLink + `__.", :param storage_configuration_id: str (optional) ID of the workspace's storage configuration object. :param storage_customer_managed_key_id: str (optional) @@ -1581,10 +1612,18 @@ paths: properties: aws_region: type: string + azure_workspace_info: + type: object + properties: + resource_group: + type: string + subscription_id: + type: string + description: Azure Subscription ID cloud: type: string description: 'DEPRECATED: This field is being ignored by the server and will be removed in the future. The - cloud name. This field always has the value `gcp`.' + cloud name. This field always has the value ``gcp``.' cloud_resource_container: type: object properties: @@ -1598,10 +1637,10 @@ paths: x-enum: - HYBRID - SERVERLESS - description: If the compute mode is `SERVERLESS`, a serverless workspace is created that comes pre-configured + description: If the compute mode is ``SERVERLESS``, a serverless workspace is created that comes pre-configured with serverless compute and default storage, providing a fully-managed, enterprise-ready SaaS experience. This means you don't need to provide any resources managed by you, such as credentials, storage, or network. - If the compute mode is `HYBRID` (which is the default option), a classic workspace is created that uses + If the compute mode is ``HYBRID`` (which is the default option), a classic workspace is created that uses customer-managed resources. credentials_id: type: string @@ -1656,7 +1695,7 @@ paths: description: The configurations of the GKE cluster used by the GCP workspace. location: type: string - description: The Google Cloud region of the workspace data plane in your Google account (for example, `us-east4`). + description: The Google Cloud region of the workspace data plane in your Google account (for example, ``us-east4``). managed_services_customer_managed_key_id: type: string description: The ID of the workspace's managed services encryption key configuration object. This is used @@ -1718,11 +1757,10 @@ paths: attributes -- see the documentation for details. private_access_settings_id: type: string - description: 'ID of the workspace''s private access settings object. Only used for PrivateLink. You must - specify this ID if you are using [AWS PrivateLink] for either front-end (user-to-workspace connection), - back-end (data plane to control plane connection), or both connection types. Before configuring PrivateLink, - read the [Databricks article about PrivateLink].", [AWS PrivateLink]: https://aws.amazon.com/privatelink/ - [Databricks article about PrivateLink]: https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html' + description: ID of the workspace's private access settings object. Only used for PrivateLink. You must specify + this ID if you are using `AWS PrivateLink `__ for either front-end + (user-to-workspace connection), back-end (data plane to control plane connection), or both connection + types. Before configuring PrivateLink, read the `Databricks article about PrivateLink `__.", storage_configuration_id: type: string description: ID of the workspace's storage configuration object. @@ -1840,13 +1878,11 @@ paths: - workspaces description: |- Gets information including status for a Databricks workspace, specified by ID. In the response, the - `workspace_status` field indicates the current status. After initial workspace creation (which is - asynchronous), make repeated `GET` requests with the workspace ID and check its status. The workspace - becomes available when the status changes to `RUNNING`. For information about how to create a new - workspace with this API **including error handling**, see [Create a new workspace using the Account - API]. - - [Create a new workspace using the Account API]: http://docs.databricks.com/administration-guide/account-api/new-workspace.html + ``workspace_status`` field indicates the current status. After initial workspace creation (which is + asynchronous), make repeated ``GET`` requests with the workspace ID and check its status. The + workspace becomes available when the status changes to ``RUNNING``. For information about how to + create a new workspace with this API **including error handling**, see `Create a new workspace using + the Account API `__. :param workspace_id: int @@ -1895,14 +1931,14 @@ paths: :param customer_facing_workspace: :class:`Workspace` :param update_mask: str (optional) The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: Long-running operation waiter for :class:`Workspace`. @@ -1925,11 +1961,11 @@ paths: schema: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). The field - path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). Specification - of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field - names must exactly match the resource field names. A field mask of `*` indicates full replacement. It’s recommended - to always explicitly list the fields being updated and avoid using `*` wildcards, as it can lead to unintended results - if the API changes in the future. + path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). + Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. + Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s + recommended to always explicitly list the fields being updated and avoid using ``*`` wildcards, as it can lead to + unintended results if the API changes in the future. requestBody: content: application/json: @@ -1953,7 +1989,7 @@ paths: description: Azure Subscription ID cloud: type: string - description: The cloud name. This field can have values like `azure`, `gcp`. + description: The cloud name. This field can have values like ``azure``, ``gcp``. cloud_resource_container: type: object properties: @@ -2023,7 +2059,7 @@ paths: location: type: string description: The Google Cloud region of the workspace data plane in your Google account (for example, - `us-east4`). + ``us-east4``). managed_services_customer_managed_key_id: type: string description: ID of the key configuration for encrypting managed services. @@ -2060,14 +2096,13 @@ paths: services and master IP range must be mutually exclusive.' description: The mutually exclusive network deployment modes. The option decides which network mode the workspace will use. The network config for GCP workspace with Databricks managed network. - This object is input-only and will not be provided when listing workspaces. See go/gcp-byovpc-alpha-design - for interface decisions. + This object is input-only and will not be provided when listing workspaces. network_id: type: string description: The ID of the network object, if the workspace is a BYOVPC workspace. This should apply to workspaces on all clouds in internal services. In accounts-rest-api, user will use workspace.network_id for input and output instead. Currently (2021-06-19) the network ID is only used by GCP. - description: The network configuration for the workspace. DEPRECATED. Use `network_id` instead. + description: The network configuration for the workspace. DEPRECATED. Use ``network_id`` instead. network_connectivity_config_id: type: string description: The object ID of network connectivity config. @@ -2123,11 +2158,10 @@ paths: attributes -- see the documentation for details. private_access_settings_id: type: string - description: 'ID of the workspace''s private access settings object. Only used for PrivateLink. You - must specify this ID if you are using [AWS PrivateLink] for either front-end (user-to-workspace connection), - back-end (data plane to control plane connection), or both connection types. Before configuring PrivateLink, - read the [Databricks article about PrivateLink].", [AWS PrivateLink]: https://aws.amazon.com/privatelink/ - [Databricks article about PrivateLink]: https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html' + description: ID of the workspace's private access settings object. Only used for PrivateLink. You must + specify this ID if you are using `AWS PrivateLink `__ for either + front-end (user-to-workspace connection), back-end (data plane to control plane connection), or both + connection types. Before configuring PrivateLink, read the `Databricks article about PrivateLink `__.", storage_configuration_id: type: string description: ID of the workspace's storage configuration object. @@ -2200,9 +2234,9 @@ components: description: The AWS KMS key alias. reuse_key_for_cluster_volumes: type: boolean - description: This field applies only if the `use_cases` property includes `STORAGE`. If this is set to true or omitted, - the key is also used to encrypt cluster EBS volumes. If you do not want to use this key for encrypting EBS volumes, - set to false. + description: This field applies only if the ``use_cases`` property includes ``STORAGE``. If this is set to true + or omitted, the key is also used to encrypt cluster EBS volumes. If you do not want to use this key for encrypting + EBS volumes, set to false. required: - key_arn - key_region @@ -2221,6 +2255,9 @@ components: key_vault_uri: type: string description: The base URI of the KeyVault. + subscription_id: + type: string + description: The Azure subscription ID to use for Key Vault access validation during CMK creation. tenant_id: type: string description: The tenant id where the KeyVault lives. @@ -2253,11 +2290,35 @@ components: description: The AWS KMS key region. reuse_key_for_cluster_volumes: type: boolean - description: This field applies only if the `use_cases` property includes `STORAGE`. If this is set to true or omitted, - the key is also used to encrypt cluster EBS volumes. If you do not want to use this key for encrypting EBS volumes, - set to false. + description: This field applies only if the ``use_cases`` property includes ``STORAGE``. If this is set to true + or omitted, the key is also used to encrypt cluster EBS volumes. If you do not want to use this key for encrypting + EBS volumes, set to false. required: - key_arn + CreateAzureKeyInfo: + type: object + properties: + disk_encryption_set_id: + type: string + key_access_configuration: + $ref: '#/components/schemas/KeyAccessConfiguration' + description: The structure to store key access credential This is set if the Managed Identity is being used to access + the Azure Key Vault key. + key_name: + type: string + description: The name of the key in KeyVault. + key_vault_uri: + type: string + description: The base URI of the KeyVault. + subscription_id: + type: string + description: The Azure subscription ID to use for Key Vault access validation during CMK creation. + tenant_id: + type: string + description: The tenant id where the KeyVault lives. + version: + type: string + description: The current key version. CreateCredentialAwsCredentials: type: object properties: @@ -2277,6 +2338,10 @@ components: $ref: '#/components/schemas/GcpServiceAccount' description: Globally unique service account email that has access to the KMS key. The service account exists within the Databricks CP project. + manual: + type: boolean + description: When true, Databricks will not use OAuth to grant the service account access to the KMS key. The customer + is responsible for granting access manually. required: - kms_key_id Credential: @@ -2344,6 +2409,10 @@ components: $ref: '#/components/schemas/GcpServiceAccount' description: Globally unique service account email that has access to the KMS key. The service account exists within the Databricks CP project. + manual: + type: boolean + description: When true, Databricks will not use OAuth to grant the service account access to the KMS key. The customer + is responsible for granting access manually. required: - kms_key_id GcpManagedNetworkConfig: @@ -2519,8 +2588,8 @@ components: private_access_level: $ref: '#/components/schemas/PrivateAccessLevel' description: The private access level controls which VPC endpoints can connect to the UI or API of any workspace - that attaches this private access settings object. `ACCOUNT` level access (the default) allows only VPC endpoints - that are registered in your Databricks account connect to your workspace. `ENDPOINT` level access allows only + that attaches this private access settings object. ``ACCOUNT`` level access (the default) allows only VPC endpoints + that are registered in your Databricks account connect to your workspace. ``ENDPOINT`` level access allows only specified VPC endpoints connect to your workspace. For details, see allowed_vpc_endpoint_ids. private_access_settings_id: type: string @@ -2536,7 +2605,6 @@ components: region: type: string description: The AWS region for workspaces attached to this private access settings object. - description: '*' RootBucketInfo: type: object properties: @@ -2575,17 +2643,15 @@ components: properties: account_id: type: string - description: The Databricks account ID that hosts the VPC endpoint configuration. TODO - This may signal an OpenAPI - diff; it does not show up in the generated spec + description: The Databricks account ID that hosts the VPC endpoint configuration. aws_account_id: type: string description: The AWS Account in which the VPC endpoint object exists. aws_endpoint_service_id: type: string - description: 'The ID of the Databricks [endpoint service] that this VPC endpoint is connected to. For a list of - endpoint service IDs for each supported AWS region, see the [Databricks PrivateLink documentation]. [Databricks - PrivateLink documentation]: https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html - [endpoint service]: https://docs.aws.amazon.com/vpc/latest/privatelink/endpoint-service.html' + description: The ID of the Databricks `endpoint service `__ + that this VPC endpoint is connected to. For a list of endpoint service IDs for each supported AWS region, see + the `Databricks PrivateLink documentation `__. aws_vpc_endpoint_id: type: string description: The ID of the VPC endpoint object in AWS. @@ -2597,8 +2663,8 @@ components: description: The AWS region in which this VPC endpoint object exists. state: type: string - description: 'The current state (such as `available` or `rejected`) of the VPC endpoint. Derived from AWS. For the - full set of values, see [AWS DescribeVpcEndpoint documentation]. [AWS DescribeVpcEndpoint documentation]: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-vpc-endpoints.html' + description: The current state (such as ``available`` or ``rejected``) of the VPC endpoint. Derived from AWS. For + the full set of values, see `AWS DescribeVpcEndpoint documentation `__. use_case: $ref: '#/components/schemas/EndpointUseCase' description: This enumeration represents the type of Databricks VPC endpoint service that was used when creating @@ -2608,11 +2674,10 @@ components: vpc_endpoint_id: type: string description: Databricks VPC endpoint ID. This is the Databricks-specific name of the VPC endpoint. Do not confuse - this with the `aws_vpc_endpoint_id`, which is the ID within AWS of the VPC endpoint. + this with the ``aws_vpc_endpoint_id``, which is the ID within AWS of the VPC endpoint. vpc_endpoint_name: type: string description: The human-readable name of the storage configuration. - description: '*' Workspace: type: object properties: @@ -2624,7 +2689,7 @@ components: $ref: '#/components/schemas/AzureWorkspaceInfo' cloud: type: string - description: The cloud name. This field can have values like `azure`, `gcp`. + description: The cloud name. This field can have values like ``azure``, ``gcp``. cloud_resource_container: $ref: '#/components/schemas/CloudResourceContainer' compute_mode: @@ -2653,13 +2718,13 @@ components: $ref: '#/components/schemas/GkeConfig' location: type: string - description: The Google Cloud region of the workspace data plane in your Google account (for example, `us-east4`). + description: The Google Cloud region of the workspace data plane in your Google account (for example, ``us-east4``). managed_services_customer_managed_key_id: type: string description: ID of the key configuration for encrypting managed services. network: $ref: '#/components/schemas/WorkspaceNetwork' - description: The network configuration for the workspace. DEPRECATED. Use `network_id` instead. + description: The network configuration for the workspace. DEPRECATED. Use ``network_id`` instead. network_connectivity_config_id: type: string description: The object ID of network connectivity config. @@ -2671,11 +2736,10 @@ components: $ref: '#/components/schemas/PricingTier' private_access_settings_id: type: string - description: 'ID of the workspace''s private access settings object. Only used for PrivateLink. You must specify - this ID if you are using [AWS PrivateLink] for either front-end (user-to-workspace connection), back-end (data - plane to control plane connection), or both connection types. Before configuring PrivateLink, read the [Databricks - article about PrivateLink].", [AWS PrivateLink]: https://aws.amazon.com/privatelink/ [Databricks article about - PrivateLink]: https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html' + description: ID of the workspace's private access settings object. Only used for PrivateLink. You must specify this + ID if you are using `AWS PrivateLink `__ for either front-end (user-to-workspace + connection), back-end (data plane to control plane connection), or both connection types. Before configuring PrivateLink, + read the `Databricks article about PrivateLink `__.", storage_configuration_id: type: string description: ID of the workspace's storage configuration object. @@ -2708,7 +2772,7 @@ components: $ref: '#/components/schemas/GcpManagedNetworkConfig' description: The mutually exclusive network deployment modes. The option decides which network mode the workspace will use. The network config for GCP workspace with Databricks managed network. This object is input-only and - will not be provided when listing workspaces. See go/gcp-byovpc-alpha-design for interface decisions. + will not be provided when listing workspaces. network_id: type: string description: The ID of the network object, if the workspace is a BYOVPC workspace. This should apply to workspaces @@ -2827,10 +2891,10 @@ components: description: |- Specifies the network connectivity types for the GKE nodes and the GKE master network. - Set to `PRIVATE_NODE_PUBLIC_MASTER` for a private GKE cluster for the workspace. The GKE nodes + Set to ``PRIVATE_NODE_PUBLIC_MASTER`` for a private GKE cluster for the workspace. The GKE nodes will not have public IPs. - Set to `PUBLIC_NODE_PUBLIC_MASTER` for a public GKE cluster. The nodes of a public GKE cluster + Set to ``PUBLIC_NODE_PUBLIC_MASTER`` for a public GKE cluster. The nodes of a public GKE cluster have public IP addresses. KeyUseCase: type: string @@ -3063,7 +3127,6 @@ components: RUNNING -> FAILED -> CANCELLED (note that this transition is disallowed in the MultiWorkspace Project) RUNNING -> PROVISIONING -> BANNED -> CANCELLED FAILED -> PROVISIONING -> CANCELLED BANNED -> RUNNING -> CANCELLED Note that a transition from any state to itself is also valid. - TODO(PLAT-5867): add a transition from CANCELLED to some other value (e.g. RECOVERING) x-stackQL-resources: credentials: id: databricks_account.provisioning.credentials diff --git a/providers/src/databricks_account/v00.00.00000/services/settings.yaml b/providers/src/databricks_account/v00.00.00000/services/settings.yaml index ca22b922..c75d0aa9 100644 --- a/providers/src/databricks_account/v00.00.00000/services/settings.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/settings.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks settings service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.settings servers: - url: https://accounts.cloud.databricks.com @@ -25,10 +25,11 @@ paths: When creating or updating an IP access list: - * For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, - where one CIDR counts as a single value. Attempts to exceed that number return error 400 with - `error_code` value `QUOTA_EXCEEDED`. * If the new list would block the calling user's current IP, - error 400 is returned with `error_code` value `INVALID_STATE`. + - For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, + where one CIDR counts as a single value. Attempts to exceed that number return error 400 with + ``error_code`` value ``QUOTA_EXCEEDED``. + - If the new list would block the calling user's current IP, error 400 is returned with ``error_code`` + value ``INVALID_STATE``. It can take a few minutes for the changes to take effect. @@ -61,8 +62,9 @@ paths: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. ip_addresses: type: array items: @@ -218,12 +220,13 @@ paths: Replaces an IP access list, specified by its ID. A list can include allow lists and block lists. See the top of this file for a description of how the - server treats allow lists and block lists at run time. When replacing an IP access list: * For all - allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, where one - CIDR counts as a single value. Attempts to exceed that number return error 400 with `error_code` value - `QUOTA_EXCEEDED`. * If the resulting list would block the calling user's current IP, error 400 is - returned with `error_code` value `INVALID_STATE`. It can take a few minutes for the changes to take - effect. + server treats allow lists and block lists at run time. When replacing an IP access list: + + - For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, + where one CIDR counts as a single value. Attempts to exceed that number return error 400 with + ``error_code`` value ``QUOTA_EXCEEDED``. + - If the resulting list would block the calling user's current IP, error 400 is returned with + ``error_code`` value ``INVALID_STATE``. It can take a few minutes for the changes to take effect. :param ip_access_list_id: str The ID for the corresponding IP access list @@ -262,8 +265,9 @@ paths: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. enabled: type: boolean description: Specifies whether this IP access list is enabled. @@ -304,10 +308,11 @@ paths: When updating an IP access list: - * For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, - where one CIDR counts as a single value. Attempts to exceed that number return error 400 with - `error_code` value `QUOTA_EXCEEDED`. * If the updated list would block the calling user's current IP, - error 400 is returned with `error_code` value `INVALID_STATE`. + - For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, + where one CIDR counts as a single value. Attempts to exceed that number return error 400 with + ``error_code`` value ``QUOTA_EXCEEDED``. + - If the updated list would block the calling user's current IP, error 400 is returned with + ``error_code`` value ``INVALID_STATE``. It can take a few minutes for the changes to take effect. @@ -355,8 +360,9 @@ paths: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. responses: '200': description: Success @@ -388,9 +394,8 @@ paths: **IMPORTANT**: After you create the network connectivity configuration, you must assign one or more workspaces to the new network connectivity configuration. You can share one network connectivity configuration with multiple workspaces from the same Azure region within the same Databricks account. - See [configure serverless secure connectivity]. - - [configure serverless secure connectivity]: https://learn.microsoft.com/azure/databricks/security/network/serverless-network-security + See `configure serverless secure connectivity + `__. :param network_connectivity_config: :class:`CreateNetworkConnectivityConfiguration` @@ -501,10 +506,9 @@ paths: resource. **IMPORTANT**: You must use Azure portal or other Azure tools to approve the private endpoint to - complete the connection. To get the information of the private endpoint created, make a `GET` request - on the new private endpoint rule. See [serverless private link]. - - [serverless private link]: https://learn.microsoft.com/azure/databricks/security/network/serverless-network-security/serverless-private-link + complete the connection. To get the information of the private endpoint created, make a ``GET`` + request on the new private endpoint rule. See `serverless private link + `__. :param network_connectivity_config_id: str Your Network Connectivity Configuration ID. @@ -545,6 +549,27 @@ paths: the private endpoint. error_message: type: string + gcp_endpoint: + type: object + properties: + all_vpc_sc_services: + type: boolean + google_api_endpoints: + type: object + properties: + endpoints: + type: array + items: + type: string + description: Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". + Use "googleapis.com" to cover all Google APIs. + description: Selected Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". + psc_endpoint_uri: + type: string + description: Output only. The URI of the created PSC endpoint. + service_attachment: + type: string + description: 'The full url of the target service attachment. Example: projects/my-gcp-project/regions/us-east4/serviceAttachments/my-service-attachment' group_id: type: string description: Not used by customer-managed private endpoint services. The sub-resource type (group ID) @@ -596,6 +621,7 @@ paths: :param network_connectivity_config_id: str Your Network Connectvity Configuration ID. + :param page_size: int (optional) :param page_token: str (optional) Pagination token to go to next page based on previous query. @@ -612,6 +638,11 @@ paths: schema: type: string description: Your Network Connectvity Configuration ID. + - name: page_size + in: query + required: false + schema: + type: integer - name: page_token in: query required: false @@ -730,8 +761,8 @@ paths: description: |- Initiates deleting a private endpoint rule. If the connection state is PENDING or EXPIRED, the private endpoint is immediately deleted. Otherwise, the private endpoint is deactivated and will be deleted - after seven days of deactivation. When a private endpoint is deactivated, the `deactivated` field is - set to `true` and the private endpoint is not available to your serverless compute resources. + after one day of deactivation. When a private endpoint is deactivated, the ``deactivated`` field is + set to ``true`` and the private endpoint is not available to your serverless compute resources. :param network_connectivity_config_id: str Your Network Connectvity Configuration ID. @@ -846,8 +877,8 @@ paths: :param private_endpoint_rule: :class:`UpdatePrivateEndpointRule` :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. @@ -877,9 +908,9 @@ paths: schema: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). The field - path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). Specification - of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field - names must exactly match the resource field names. + path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). + Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. + Field names must exactly match the resource field names. requestBody: content: application/json: @@ -898,10 +929,31 @@ paths: be specified. enabled: type: boolean - description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate - this private endpoint to allow egress access from serverless compute resources. + description: Update this field to activate/deactivate this private endpoint to allow egress access from + serverless compute resources. Only honored for first-party services on each cloud (e.g. AWS S3). error_message: type: string + gcp_endpoint: + type: object + properties: + all_vpc_sc_services: + type: boolean + google_api_endpoints: + type: object + properties: + endpoints: + type: array + items: + type: string + description: Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". + Use "googleapis.com" to cover all Google APIs. + description: Selected Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". + psc_endpoint_uri: + type: string + description: Output only. The URI of the created PSC endpoint. + service_attachment: + type: string + description: 'The full url of the target service attachment. Example: projects/my-gcp-project/regions/us-east4/serviceAttachments/my-service-attachment' resource_names: type: array items: @@ -982,6 +1034,17 @@ paths: Databricks can access Internet. No blocking rules will apply. RESTRICTED_ACCESS: Databricks can only access explicitly allowed internet and storage destinations, as well as UC connections and external locations. + allowed_databricks_destinations: + type: array + items: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: List of Databricks workspace destinations that serverless workloads are allowed + to access when in RESTRICTED_ACCESS mode. allowed_internet_destinations: type: array items: @@ -1099,236 +1162,7 @@ paths: description: Users can specify accessible storage destinations. description: List of storage destinations that serverless workloads are allowed to access when in RESTRICTED_ACCESS mode. - policy_enforcement: - type: object - properties: - dry_run_mode_product_filter: - type: array - items: - type: string - x-enum: - - DBSQL - - ML_SERVING - description: The values should match the list of workloads used in networkconfig.proto - enforcement_mode: - type: string - x-enum: - - DRY_RUN - - ENFORCED - description: The mode of policy enforcement. ENFORCED blocks traffic that violates policy, - while DRY_RUN only logs violations without blocking. When not specified, defaults to ENFORCED. - description: Optional. When policy_enforcement is not provided, we default to ENFORCE_MODE_ALL_SERVICES - required: - - restriction_mode - description: The access policy enforced for egress traffic to the internet. - description: The network policies applying for egress traffic. - network_policy_id: - type: string - description: The unique identifier for the network policy. - description: Network policy configuration details. - required: - - network_policy - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/AccountNetworkPolicy' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: NetworkPoliciesAPI - get: - operationId: network_policies_list_network_policies_rpc - summary: Gets an array of network policies. - tags: - - settings - - network_policies - description: |- - Gets an array of network policies. - - :param page_token: str (optional) - Pagination token to go to next page based on previous query. - - :returns: Iterator over :class:`AccountNetworkPolicy` - parameters: - - name: account_id - in: path - required: true - schema: - type: string - - name: page_token - in: query - required: false - schema: - type: string - description: Pagination token to go to next page based on previous query. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListNetworkPoliciesResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: NetworkPoliciesAPI - /api/2.0/accounts/{account_id}/network-policies/{network_policy_id}: - delete: - operationId: network_policies_delete_network_policy_rpc - summary: Deletes a network policy. Cannot be called on 'default-policy'. - tags: - - settings - - network_policies - description: |- - Deletes a network policy. Cannot be called on 'default-policy'. - - :param network_policy_id: str - The unique identifier of the network policy to delete. - parameters: - - name: account_id - in: path - required: true - schema: - type: string - - name: network_policy_id - in: path - required: true - schema: - type: string - description: The unique identifier of the network policy to delete. - responses: - '200': - description: Success - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: NetworkPoliciesAPI - get: - operationId: network_policies_get_network_policy_rpc - summary: Gets a network policy. - tags: - - settings - - network_policies - description: |- - Gets a network policy. - - :param network_policy_id: str - The unique identifier of the network policy to retrieve. - - :returns: :class:`AccountNetworkPolicy` - parameters: - - name: account_id - in: path - required: true - schema: - type: string - - name: network_policy_id - in: path - required: true - schema: - type: string - description: The unique identifier of the network policy to retrieve. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/AccountNetworkPolicy' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: NetworkPoliciesAPI - put: - operationId: network_policies_update_network_policy_rpc - summary: Updates a network policy. This allows you to modify the configuration of a network policy. - tags: - - settings - - network_policies - description: |- - Updates a network policy. This allows you to modify the configuration of a network policy. - - :param network_policy_id: str - The unique identifier for the network policy. - :param network_policy: :class:`AccountNetworkPolicy` - Updated network policy configuration details. - - :returns: :class:`AccountNetworkPolicy` - parameters: - - name: account_id - in: path - required: true - schema: - type: string - - name: network_policy_id - in: path - required: true - schema: - type: string - description: The unique identifier for the network policy. - requestBody: - content: - application/json: - schema: - type: object - properties: - network_policy: - type: object - properties: - account_id: - type: string - egress: - type: object - properties: - network_access: - type: object - properties: - restriction_mode: - type: string - x-enum: - - FULL_ACCESS - - RESTRICTED_ACCESS - description: |- - At which level can Databricks and Databricks managed compute access Internet. FULL_ACCESS: - Databricks can access Internet. No blocking rules will apply. RESTRICTED_ACCESS: Databricks can - only access explicitly allowed internet and storage destinations, as well as UC connections and - external locations. - allowed_internet_destinations: + blocked_internet_destinations: type: array items: type: object @@ -1382,92 +1216,6402 @@ paths: Users can specify accessible internet destinations when outbound access is restricted. We only support DNS_NAME (FQDN format) destinations for the time being. Going forward we may extend support to host names and IP addresses. - description: List of internet destinations that serverless workloads are allowed to access when - in RESTRICTED_ACCESS mode. - allowed_storage_destinations: + description: List of internet destinations that serverless workloads are blocked from accessing. + These destinations are enforced when restriction mode is RESTRICTED_ACCESS or DRY_RUN. Currently + supports DNS_NAME type only; IP_RANGE support is planned. + policy_enforcement: + type: object + properties: + dry_run_mode_product_filter: + type: array + items: + type: string + x-enum: + - DBSQL + - ML_SERVING + description: The values should match the list of workloads used in networkconfig.proto + enforcement_mode: + type: string + x-enum: + - DRY_RUN + - ENFORCED + description: The mode of policy enforcement. ENFORCED blocks traffic that violates policy, + while DRY_RUN only logs violations without blocking. When not specified, defaults to ENFORCED. + description: Optional. When policy_enforcement is not provided, we default to ENFORCE_MODE_ALL_SERVICES + required: + - restriction_mode + description: The access policy enforced for egress traffic to the internet. + description: The network policies applying for egress traffic. + ingress: + type: object + properties: + cross_workspace_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + required: + - restriction_mode + private_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - ALLOW_ALL_REGISTERED_ENDPOINTS + - RESTRICTED_ACCESS + description: The restriction mode for private access. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Allow rules are evaluated after deny rules. A request matching any allow rule is + allowed; a request matching no rule is denied by default. Only applies when restriction_mode + is RESTRICTED_ACCESS. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Deny rules are evaluated first. A request matching any deny rule is denied, regardless + of allow rules. Only applies when restriction_mode is RESTRICTED_ACCESS. + required: + - restriction_mode + description: The network policy restrictions for private access. Configures how requests arriving + over private connectivity are governed. + public_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + required: + - restriction_mode + description: The network policy restrictions for public access to the workspace. Configures how + public internet traffic is allowed or denied access. + description: The network policies applying for ingress traffic. + ingress_dry_run: + type: object + properties: + cross_workspace_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + required: + - restriction_mode + private_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - ALLOW_ALL_REGISTERED_ENDPOINTS + - RESTRICTED_ACCESS + description: The restriction mode for private access. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Allow rules are evaluated after deny rules. A request matching any allow rule is + allowed; a request matching no rule is denied by default. Only applies when restriction_mode + is RESTRICTED_ACCESS. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Deny rules are evaluated first. A request matching any deny rule is denied, regardless + of allow rules. Only applies when restriction_mode is RESTRICTED_ACCESS. + required: + - restriction_mode + description: The network policy restrictions for private access. Configures how requests arriving + over private connectivity are governed. + public_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + required: + - restriction_mode + description: The network policy restrictions for public access to the workspace. Configures how + public internet traffic is allowed or denied access. + description: The ingress policy for dry run mode. Dry run will always run even if the request is allowed + by the ingress policy. When this field is set, the policy will be evaluated and emit logs only without + blocking requests. + network_policy_id: + type: string + description: The unique identifier for the network policy. + description: Network policy configuration details. + required: + - network_policy + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AccountNetworkPolicy' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: NetworkPoliciesAPI + get: + operationId: network_policies_list_network_policies_rpc + summary: Gets an array of network policies. + tags: + - settings + - network_policies + description: |- + Gets an array of network policies. + + :param page_token: str (optional) + Pagination token to go to next page based on previous query. + + :returns: Iterator over :class:`AccountNetworkPolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token to go to next page based on previous query. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListNetworkPoliciesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: NetworkPoliciesAPI + /api/2.0/accounts/{account_id}/network-policies/{network_policy_id}: + delete: + operationId: network_policies_delete_network_policy_rpc + summary: Deletes a network policy. Cannot be called on 'default-policy'. + tags: + - settings + - network_policies + description: |- + Deletes a network policy. Cannot be called on 'default-policy'. + + :param network_policy_id: str + The unique identifier of the network policy to delete. + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: network_policy_id + in: path + required: true + schema: + type: string + description: The unique identifier of the network policy to delete. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: NetworkPoliciesAPI + get: + operationId: network_policies_get_network_policy_rpc + summary: Gets a network policy. + tags: + - settings + - network_policies + description: |- + Gets a network policy. + + :param network_policy_id: str + The unique identifier of the network policy to retrieve. + + :returns: :class:`AccountNetworkPolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: network_policy_id + in: path + required: true + schema: + type: string + description: The unique identifier of the network policy to retrieve. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AccountNetworkPolicy' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: NetworkPoliciesAPI + put: + operationId: network_policies_update_network_policy_rpc + summary: Updates a network policy. This allows you to modify the configuration of a network policy. + tags: + - settings + - network_policies + description: |- + Updates a network policy. This allows you to modify the configuration of a network policy. + + :param network_policy_id: str + The unique identifier for the network policy. + :param network_policy: :class:`AccountNetworkPolicy` + Updated network policy configuration details. + + :returns: :class:`AccountNetworkPolicy` + parameters: + - name: account_id + in: path + required: true + schema: + type: string + - name: network_policy_id + in: path + required: true + schema: + type: string + description: The unique identifier for the network policy. + requestBody: + content: + application/json: + schema: + type: object + properties: + network_policy: + type: object + properties: + account_id: + type: string + egress: + type: object + properties: + network_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + At which level can Databricks and Databricks managed compute access Internet. FULL_ACCESS: + Databricks can access Internet. No blocking rules will apply. RESTRICTED_ACCESS: Databricks can + only access explicitly allowed internet and storage destinations, as well as UC connections and + external locations. + allowed_databricks_destinations: + type: array + items: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: List of Databricks workspace destinations that serverless workloads are allowed + to access when in RESTRICTED_ACCESS mode. + allowed_internet_destinations: + type: array + items: + type: object + properties: + destination: + type: string + description: The internet destination to which access will be allowed. Format dependent + on the destination type. + internet_destination_type: + type: string + x-enum: + - DNS_NAME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: |- + Users can specify accessible internet destinations when outbound access is restricted. We only + support DNS_NAME (FQDN format) destinations for the time being. Going forward we may extend + support to host names and IP addresses. + description: List of internet destinations that serverless workloads are allowed to access when + in RESTRICTED_ACCESS mode. + allowed_storage_destinations: + type: array + items: + type: object + properties: + azure_storage_account: + type: string + description: The Azure storage account name. + azure_storage_service: + type: string + description: The Azure storage service type (blob, dfs, etc.). + bucket_name: + type: string + region: + type: string + storage_destination_type: + type: string + x-enum: + - AWS_S3 + - AZURE_STORAGE + - GOOGLE_CLOUD_STORAGE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: Users can specify accessible storage destinations. + description: List of storage destinations that serverless workloads are allowed to access when + in RESTRICTED_ACCESS mode. + blocked_internet_destinations: + type: array + items: + type: object + properties: + destination: + type: string + description: The internet destination to which access will be allowed. Format dependent + on the destination type. + internet_destination_type: + type: string + x-enum: + - DNS_NAME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: |- + Users can specify accessible internet destinations when outbound access is restricted. We only + support DNS_NAME (FQDN format) destinations for the time being. Going forward we may extend + support to host names and IP addresses. + description: List of internet destinations that serverless workloads are blocked from accessing. + These destinations are enforced when restriction mode is RESTRICTED_ACCESS or DRY_RUN. Currently + supports DNS_NAME type only; IP_RANGE support is planned. + policy_enforcement: + type: object + properties: + dry_run_mode_product_filter: + type: array + items: + type: string + x-enum: + - DBSQL + - ML_SERVING + description: The values should match the list of workloads used in networkconfig.proto + enforcement_mode: + type: string + x-enum: + - DRY_RUN + - ENFORCED + description: The mode of policy enforcement. ENFORCED blocks traffic that violates policy, + while DRY_RUN only logs violations without blocking. When not specified, defaults to ENFORCED. + description: Optional. When policy_enforcement is not provided, we default to ENFORCE_MODE_ALL_SERVICES + required: + - restriction_mode + description: The access policy enforced for egress traffic to the internet. + description: The network policies applying for egress traffic. + ingress: + type: object + properties: + cross_workspace_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + required: + - restriction_mode + private_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - ALLOW_ALL_REGISTERED_ENDPOINTS + - RESTRICTED_ACCESS + description: The restriction mode for private access. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Allow rules are evaluated after deny rules. A request matching any allow rule is + allowed; a request matching no rule is denied by default. Only applies when restriction_mode + is RESTRICTED_ACCESS. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Deny rules are evaluated first. A request matching any deny rule is denied, regardless + of allow rules. Only applies when restriction_mode is RESTRICTED_ACCESS. + required: + - restriction_mode + description: The network policy restrictions for private access. Configures how requests arriving + over private connectivity are governed. + public_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + required: + - restriction_mode + description: The network policy restrictions for public access to the workspace. Configures how + public internet traffic is allowed or denied access. + description: The network policies applying for ingress traffic. + ingress_dry_run: + type: object + properties: + cross_workspace_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: Specific source workspace IDs to match. + required: + - restriction_mode + private_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - ALLOW_ALL_REGISTERED_ENDPOINTS + - RESTRICTED_ACCESS + description: The restriction mode for private access. + allow_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Allow rules are evaluated after deny rules. A request matching any allow rule is + allowed; a request matching no rule is denied by default. Only applies when restriction_mode + is RESTRICTED_ACCESS. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: The authenticated identity the request must match. When unset, the rule matches + all users and service principals. On the account-level network policy, scoping to specific + identities is not currently supported, so this field must be unset (the rule matches + all users and service principals). + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + description: The destination the request must match — the resource being accessed, for + example the workspace UI, workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered + endpoints and the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when + specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. + Must be set to true when specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link + (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. + Must be set to true when specified. + endpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint + ID. + description: Matches requests arriving through any of the specified registered endpoints. + description: The origin the request must match — the private connectivity the request + arrives through, for example a specific set of registered endpoints or any endpoint + registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + description: Deny rules are evaluated first. A request matching any deny rule is denied, regardless + of allow rules. Only applies when restriction_mode is RESTRICTED_ACCESS. + required: + - restriction_mode + description: The network policy restrictions for private access. Configures how requests arriving + over private connectivity are governed. + public_access: + type: object + properties: + restriction_mode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + allow_rules: type: array items: type: object properties: - azure_storage_account: - type: string - description: The Azure storage account name. - azure_storage_service: - type: string - description: The Azure storage service type (blob, dfs, etc.). - bucket_name: - type: string - region: - type: string - storage_destination_type: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: type: string - x-enum: - - AWS_S3 - - AZURE_STORAGE - - GOOGLE_CLOUD_STORAGE - description: |- - Create a collection of name/value pairs. + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + deny_rules: + type: array + items: + type: object + properties: + authentication: + type: object + properties: + identities: + type: array + items: + type: object + properties: + principal_id: + type: integer + principal_type: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. - Example enumeration: + Example enumeration: - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 - Access them by: + Access them by: - - attribute access: + - attribute access: - >>> Color.RED - + >>> Color.RED + - - value lookup: + - value lookup: - >>> Color(1) - + >>> Color(1) + - - name lookup: + - name lookup: - >>> Color['RED'] - + >>> Color['RED'] + - Enumerations can be iterated over, and know how many members they have: + Enumerations can be iterated over, and know how many members they have: - >>> len(Color) - 3 + >>> len(Color) + 3 - >>> list(Color) - [, , ] + >>> list(Color) + [, , ] - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. - description: Users can specify accessible storage destinations. - description: List of storage destinations that serverless workloads are allowed to access when - in RESTRICTED_ACCESS mode. - policy_enforcement: - type: object - properties: - dry_run_mode_product_filter: - type: array - items: + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + identity_type: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + destination: + type: object + properties: + account_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level + API. + description: Matches account-level Databricks API endpoints for an ingress network + policy rule. + account_databricks_one: + type: object + properties: + all_destinations: + type: boolean + description: Account DatabricksOne destination is not supported. + account_ui: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: Matches requests to the account console UI. Can only be used in the account-level + network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be + set. When not set or false, at least one specific destination must be provided. + apps_runtime: + type: object + properties: + all_destinations: + type: boolean + lakebase_runtime: + type: object + properties: + all_destinations: + type: boolean + workspace_api: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed + here ("allow all except"). Mutually exclusive with ``scopes`` — a single destination + may set at most one of the two.' + scope_constraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network + policy rule. + workspace_ui: + type: object + properties: + all_destinations: + type: boolean + label: type: string - x-enum: - - DBSQL - - ML_SERVING - description: The values should match the list of workloads used in networkconfig.proto - enforcement_mode: - type: string - x-enum: - - DRY_RUN - - ENFORCED - description: The mode of policy enforcement. ENFORCED blocks traffic that violates policy, - while DRY_RUN only logs violations without blocking. When not specified, defaults to ENFORCED. - description: Optional. When policy_enforcement is not provided, we default to ENFORCE_MODE_ALL_SERVICES + description: The label for this ingress rule. + origin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + description: Will not allow IP ranges with private IPs. + managed_ip_range: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region + must belong to the partner named in ``name``. Empty list means all regions of + the partner. Region casing matches the partner's source (e.g., Microsoft uses + "EastUS"), so values are stored verbatim, not lowercased. + description: A single Databricks-maintained IP range for a well-known partner (e.g., + Power BI, dbt), optionally scoped to specific partner regions. Customers create + one rule per partner. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. required: - restriction_mode - description: The access policy enforced for egress traffic to the internet. - description: The network policies applying for egress traffic. + description: The network policy restrictions for public access to the workspace. Configures how + public internet traffic is allowed or denied access. + description: The ingress policy for dry run mode. Dry run will always run even if the request is allowed + by the ingress policy. When this field is set, the policy will be evaluated and emit logs only without + blocking requests. network_policy_id: type: string description: The unique identifier for the network policy. @@ -1632,6 +7776,13 @@ components: egress: $ref: '#/components/schemas/NetworkPolicyEgress' description: The network policies applying for egress traffic. + ingress: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicy' + description: The network policies applying for ingress traffic. + ingress_dry_run: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicy' + description: The ingress policy for dry run mode. Dry run will always run even if the request is allowed by the + ingress policy. When this field is set, the policy will be evaluated and emit logs only without blocking requests. network_policy_id: type: string description: The unique identifier for the network policy. @@ -1770,10 +7921,13 @@ components: type: array items: $ref: '#/components/schemas/ComplianceStandard' - description: Set by customers when they request Compliance Security Profile (CSP) + description: Compliance standards selected by the customer for this Compliance Security Profile. is_enabled: type: boolean - description: 'SHIELD feature: CSP' + description: Whether Compliance Security Profile (CSP) is enabled on the workspace. + description: |- + SHIELD feature: CSP Compliance Security Profile (CSP) enables enhanced compliance controls on + the workspace. ComplianceSecurityProfileSetting: type: object properties: @@ -1851,6 +8005,8 @@ components: endpoint. error_message: type: string + gcp_endpoint: + $ref: '#/components/schemas/GcpEndpoint' group_id: type: string description: Not used by customer-managed private endpoint services. The sub-resource type (group ID) of the target @@ -1910,6 +8066,314 @@ components: name is required to be 'default' if the setting only has one instance per workspace. required: - csp_enablement_account + CustomerFacingIngressNetworkPolicy: + type: object + properties: + cross_workspace_access: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceAccess' + private_access: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateAccess' + description: The network policy restrictions for private access. Configures how requests arriving over private connectivity + are governed. + public_access: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicAccess' + description: The network policy restrictions for public access to the workspace. Configures how public internet + traffic is allowed or denied access. + description: The network policies applying for ingress traffic. + CustomerFacingIngressNetworkPolicyAccountApiDestination: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed here ("allow all except"). Mutually + exclusive with ``scopes`` — a single destination may set at most one of the two.' + scope_constraint: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeConstraint' + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeQualifier' + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: The API scopes to match. Use "all-apis" to match any account-level API. + description: Matches account-level Databricks API endpoints for an ingress network policy rule. + CustomerFacingIngressNetworkPolicyAccountDatabricksOneDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingIngressNetworkPolicyAccountUiDestination: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: The account console UI destination. + CustomerFacingIngressNetworkPolicyAppsRuntimeDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingIngressNetworkPolicyAuthentication: + type: object + properties: + identities: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthenticationIdentity' + identity_type: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthenticationIdentityType' + CustomerFacingIngressNetworkPolicyAuthenticationIdentity: + type: object + properties: + principal_id: + type: integer + principal_type: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthenticationIdentityPrincipalType' + CustomerFacingIngressNetworkPolicyCrossWorkspaceAccess: + type: object + properties: + restriction_mode: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceAccessRestrictionMode' + allow_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceIngressRule' + deny_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceIngressRule' + required: + - restriction_mode + CustomerFacingIngressNetworkPolicyCrossWorkspaceIngressRule: + type: object + properties: + authentication: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthentication' + destination: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyRequestDestination' + label: + type: string + description: The label for this ingress rule. + origin: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceRequestOrigin' + CustomerFacingIngressNetworkPolicyCrossWorkspaceRequestOrigin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyWorkspaceIdList' + description: Specific source workspace IDs to match. + CustomerFacingIngressNetworkPolicyEndpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint ID. + description: A set of registered endpoints, identified by their endpoint IDs. + CustomerFacingIngressNetworkPolicyIpRanges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + CustomerFacingIngressNetworkPolicyLakebaseRuntimeDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingIngressNetworkPolicyManagedIpRange: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region must belong to the partner named + in ``name``. Empty list means all regions of the partner. Region casing matches the partner's source (e.g., Microsoft + uses "EastUS"), so values are stored verbatim, not lowercased. + description: |- + A Databricks-managed IP range identified by name (e.g., "powerbi", "dbt"), optionally scoped to + specific partner regions. Customers reference a name instead of raw CIDRs; Databricks keeps the + underlying IP list current. + CustomerFacingIngressNetworkPolicyPrivateAccess: + type: object + properties: + restriction_mode: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateAccessRestrictionMode' + description: The restriction mode for private access. + allow_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateIngressRule' + description: Allow rules are evaluated after deny rules. A request matching any allow rule is allowed; a request + matching no rule is denied by default. Only applies when restriction_mode is RESTRICTED_ACCESS. + deny_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateIngressRule' + description: Deny rules are evaluated first. A request matching any deny rule is denied, regardless of allow rules. + Only applies when restriction_mode is RESTRICTED_ACCESS. + required: + - restriction_mode + description: |- + Configures how requests arriving over private connectivity, such as registered endpoints, are + allowed or denied access. + CustomerFacingIngressNetworkPolicyPrivateIngressRule: + type: object + properties: + authentication: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthentication' + description: The authenticated identity the request must match. When unset, the rule matches all users and service + principals. On the account-level network policy, scoping to specific identities is not currently supported, so + this field must be unset (the rule matches all users and service principals). + destination: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyRequestDestination' + description: The destination the request must match — the resource being accessed, for example the workspace UI, + workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateRequestOrigin' + description: The origin the request must match — the private connectivity the request arrives through, for example + a specific set of registered endpoints or any endpoint registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + CustomerFacingIngressNetworkPolicyPrivateRequestOrigin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered endpoints and the workspace's + Azure Private Link (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. Must + be set to true when specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. Must be set to true when + specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when specified. + endpoints: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyEndpoints' + description: Matches requests arriving through any of the specified registered endpoints. + description: |- + The origin of a private access request, identified by the endpoint through which the request + arrives. + CustomerFacingIngressNetworkPolicyPublicAccess: + type: object + properties: + restriction_mode: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicAccessRestrictionMode' + allow_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicIngressRule' + deny_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicIngressRule' + required: + - restriction_mode + CustomerFacingIngressNetworkPolicyPublicIngressRule: + type: object + properties: + authentication: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthentication' + destination: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyRequestDestination' + label: + type: string + description: The label for this ingress rule. + origin: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicRequestOrigin' + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + CustomerFacingIngressNetworkPolicyPublicRequestOrigin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyIpRanges' + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyIpRanges' + description: Will not allow IP ranges with private IPs. + managed_ip_range: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyManagedIpRange' + description: A single Databricks-maintained IP range for a well-known partner (e.g., Power BI, dbt), optionally + scoped to specific partner regions. Customers create one rule per partner. + CustomerFacingIngressNetworkPolicyRequestDestination: + type: object + properties: + account_api: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAccountApiDestination' + account_databricks_one: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAccountDatabricksOneDestination' + description: Account DatabricksOne destination is not supported. + account_ui: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAccountUiDestination' + description: Matches requests to the account console UI. Can only be used in the account-level network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be set. When not set or false, at + least one specific destination must be provided. + apps_runtime: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAppsRuntimeDestination' + lakebase_runtime: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyLakebaseRuntimeDestination' + workspace_api: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyWorkspaceApiDestination' + workspace_ui: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyWorkspaceUiDestination' + CustomerFacingIngressNetworkPolicyWorkspaceApiDestination: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed here ("allow all except"). Mutually + exclusive with ``scopes`` — a single destination may set at most one of the two.' + scope_constraint: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeConstraint' + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeQualifier' + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network policy rule. + CustomerFacingIngressNetworkPolicyWorkspaceIdList: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + CustomerFacingIngressNetworkPolicyWorkspaceUiDestination: + type: object + properties: + all_destinations: + type: boolean CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRule: type: object properties: @@ -2335,6 +8799,12 @@ components: properties: restriction_mode: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyRestrictionMode' + allowed_databricks_destinations: + type: array + items: + $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyDatabricksDestination' + description: List of Databricks workspace destinations that serverless workloads are allowed to access when in RESTRICTED_ACCESS + mode. allowed_internet_destinations: type: array items: @@ -2347,11 +8817,25 @@ components: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyStorageDestination' description: List of storage destinations that serverless workloads are allowed to access when in RESTRICTED_ACCESS mode. + blocked_internet_destinations: + type: array + items: + $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyInternetDestination' + description: List of internet destinations that serverless workloads are blocked from accessing. These destinations + are enforced when restriction mode is RESTRICTED_ACCESS or DRY_RUN. Currently supports DNS_NAME type only; IP_RANGE + support is planned. policy_enforcement: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyPolicyEnforcement' description: Optional. When policy_enforcement is not provided, we default to ENFORCE_MODE_ALL_SERVICES required: - restriction_mode + EgressNetworkPolicyNetworkAccessPolicyDatabricksDestination: + type: object + properties: + workspace_ids: + type: array + items: + type: integer EgressNetworkPolicyNetworkAccessPolicyInternetDestination: type: object properties: @@ -2436,7 +8920,10 @@ components: properties: is_enabled: type: boolean - description: 'SHIELD feature: ESM' + description: Whether Enhanced Security Monitoring (ESM) is enabled on the workspace. + description: |- + SHIELD feature: ESM Enhanced Security Monitoring (ESM) enables additional security monitoring on + the workspace. EnhancedSecurityMonitoringSetting: type: object properties: @@ -2516,11 +9003,57 @@ components: ip_access_list: $ref: '#/components/schemas/IpAccessListInfo' description: An IP access list was successfully returned. + GcpEndpoint: + type: object + properties: + all_vpc_sc_services: + type: boolean + google_api_endpoints: + $ref: '#/components/schemas/GoogleApiEndpoints' + description: Selected Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". + psc_endpoint_uri: + type: string + description: Output only. The URI of the created PSC endpoint. + service_attachment: + type: string + description: 'The full url of the target service attachment. Example: projects/my-gcp-project/regions/us-east4/serviceAttachments/my-service-attachment' GenericWebhookConfig: type: object properties: + oauth_audience: + type: string + oauth_audience_set: + type: boolean + description: '[Output-Only] Whether OAuth2 audience is set.' + oauth_client_id: + type: string + description: '[Input-Only][Optional] OAuth2 client ID for the client_credentials grant.' + oauth_client_id_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client ID is set.' + oauth_client_secret: + type: string + description: '[Input-Only][Optional] OAuth2 client secret for the client_credentials grant.' + oauth_client_secret_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client secret is set.' + oauth_scopes: + type: array + items: + type: string + description: '[Input-Only][Optional] OAuth2 scopes requested at token mint (required by some IdPs).' + oauth_scopes_set: + type: boolean + description: '[Output-Only] Whether OAuth2 scopes are set.' + oauth_token_url: + type: string + description: '[Input-Only][Optional] OAuth2 token endpoint URL (customer IdP) where access tokens are minted.' + oauth_token_url_set: + type: boolean + description: '[Output-Only] Whether OAuth2 token URL is set.' password: type: string + description: '[Input-Only][Optional] Password for webhook.' password_set: type: boolean description: '[Output-Only] Whether password is set.' @@ -2562,6 +9095,18 @@ components: token_info: $ref: '#/components/schemas/TokenInfo' description: Token with specified Token ID was successfully returned. + GoogleApiEndpoints: + type: object + properties: + endpoints: + type: array + items: + type: string + description: Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". Use "googleapis.com" + to cover all Google APIs. + description: |- + Wrapper for a list of Google API hostnames. Wrapped in a message because proto3 oneof does not + support repeated fields directly. IpAccessListInfo: type: object properties: @@ -2862,6 +9407,8 @@ components: $ref: '#/components/schemas/NccAwsStableIpRule' azure_service_endpoint_rule: $ref: '#/components/schemas/NccAzureServiceEndpointRule' + gcp_project_id_rule: + $ref: '#/components/schemas/NetworkConnectivityConfigEgressConfigDefaultRuleGcpProjectIdRule' description: Default rules don't have specific targets. NccEgressTargetRules: type: object @@ -2910,8 +9457,8 @@ components: private link service. When updating this field, the full list of target domain_names must be specified. enabled: type: boolean - description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate - this private endpoint to allow egress access from serverless compute resources. + description: Update this field to activate/deactivate this private endpoint to allow egress access from serverless + compute resources. Only honored for first-party services on each cloud (e.g. AWS S3). endpoint_name: type: string description: The name of the Azure private endpoint resource. @@ -2921,6 +9468,8 @@ components: endpoint. error_message: type: string + gcp_endpoint: + $ref: '#/components/schemas/GcpEndpoint' group_id: type: string description: Not used by customer-managed private endpoint services. The sub-resource type (group ID) of the target @@ -2953,6 +9502,13 @@ components: description: |- Properties of the new private endpoint rule. Note that you must approve the endpoint in Azure portal after initialization. + NetworkConnectivityConfigEgressConfigDefaultRuleGcpProjectIdRule: + type: object + properties: + project_ids: + type: array + items: + type: string NetworkConnectivityConfiguration: type: object properties: @@ -2986,12 +9542,7 @@ components: network_access: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicy' description: The access policy enforced for egress traffic to the internet. - description: |- - The network policies applying for egress traffic. This message is used by the UI/REST API. We - translate this message to the format expected by the dataplane in Lakehouse Network Manager (for - the format expected by the dataplane, see networkconfig.textproto). This policy should be - consistent with [[com.databricks.api.proto.settingspolicy.EgressNetworkPolicy]]. Details see - API-design: https://docs.google.com/document/d/1DKWO_FpZMCY4cF2O62LpwII1lx8gsnDGG-qgE3t3TOA/ + description: The network policies applying for egress traffic. NotificationDestination: type: object properties: @@ -3050,14 +9601,35 @@ components: PublicTokenInfo: type: object properties: + autoscope_state: + type: string + backfill_scopes: + type: array + items: + type: string + description: Output only. Scopes inferred from offline backfill processing. comment: type: string + description: Comment the token was created with, if applicable. creation_time: type: integer description: Server time (in epoch milliseconds) when the token was created. expiry_time: type: integer description: Server time (in epoch milliseconds) when the token will expire, or -1 if not applicable. + inferred_scopes: + type: array + items: + type: string + description: Output only. Inferred API path scopes collected for this token when autoscope is enabled. + last_accessed_time: + type: integer + description: Server time (in epoch milliseconds) when the token was accessed most recently. + scopes: + type: array + items: + type: string + description: Scope of the token was created with, if applicable. token_id: type: string description: The ID of this token. @@ -3173,8 +9745,16 @@ components: TokenInfo: type: object properties: + autoscope_state: + type: string + backfill_scopes: + type: array + items: + type: string + description: Output only. Scopes inferred from offline backfill processing. comment: type: string + description: Comment that describes the purpose of the token, specified by the token creator. created_by_id: type: integer description: User ID of the user that created the token. @@ -3187,12 +9767,22 @@ components: expiry_time: type: integer description: Timestamp when the token expires. + inferred_scopes: + type: array + items: + type: string + description: Output only. Inferred API path scopes collected for this token when autoscope is enabled. last_used_day: type: integer description: Approximate timestamp for the day the token was last used. Accurate up to 1 day. owner_id: type: integer description: User ID of the user that owns the token. + scopes: + type: array + items: + type: string + description: Scope of the token was created with, if applicable. token_id: type: string description: ID of the token. @@ -3239,10 +9829,12 @@ components: private link service. When updating this field, the full list of target domain_names must be specified. enabled: type: boolean - description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate - this private endpoint to allow egress access from serverless compute resources. + description: Update this field to activate/deactivate this private endpoint to allow egress access from serverless + compute resources. Only honored for first-party services on each cloud (e.g. AWS S3). error_message: type: string + gcp_endpoint: + $ref: '#/components/schemas/GcpEndpoint' resource_names: type: array items: @@ -3254,6 +9846,9 @@ components: description: |- Properties of the new private endpoint rule. Note that you must approve the endpoint in Azure portal after initialization. + UpdateTokenResponse: + type: object + properties: {} WorkspaceNetworkOption: type: object properties: @@ -3402,6 +9997,7 @@ components: ComplianceStandard: type: string x-enum: + - ARC_AMPE - CANADA_PROTECTED_B - CYBER_ESSENTIAL_PLUS - FEDRAMP_HIGH @@ -3414,12 +10010,207 @@ components: - IRAP_PROTECTED - ISMAP - ITAR_EAR + - KSA_ECC_CCC_DCC - K_FSI - NONE - PCI_DSS description: |- Compliance standard for SHIELD customers. See README.md for how instructions of how to add new standards. + CustomerFacingIngressNetworkPolicyApiScopeConstraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use ApiScopeQualifier instead.' + CustomerFacingIngressNetworkPolicyApiScopeQualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: |- + Qualifies the breadth of API access permitted by an ingress network policy rule. + API_SCOPE_QUALIFIER_READ narrows matching to read-only variants of the listed scopes; + API_SCOPE_QUALIFIER_ALL matches any scope. When unset, scopes match exactly as listed. + CustomerFacingIngressNetworkPolicyAuthenticationIdentityPrincipalType: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CustomerFacingIngressNetworkPolicyAuthenticationIdentityType: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CustomerFacingIngressNetworkPolicyCrossWorkspaceAccessRestrictionMode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CustomerFacingIngressNetworkPolicyPrivateAccessRestrictionMode: + type: string + x-enum: + - ALLOW_ALL_REGISTERED_ENDPOINTS + - RESTRICTED_ACCESS + description: |- + The restriction mode for private access. In ALLOW_ALL_REGISTERED_ENDPOINTS mode, requests + arriving through any endpoint registered to the account are allowed, and deny rules and allow + rules cannot be set. In RESTRICTED_ACCESS mode, access is restricted based on deny rules and + allow rules; requests that do not match any allow rule are denied. + CustomerFacingIngressNetworkPolicyPublicAccessRestrictionMode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRulePrivateLinkConnectionState: type: string x-enum: @@ -3824,8 +10615,9 @@ components: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. NccAzurePrivateEndpointRuleConnectionState: type: string x-enum: @@ -3987,7 +10779,7 @@ components: - ARCLIGHT_MULTI_TENANT_AZURE_EXCHANGE_TOKEN - ARCLIGHT_MULTI_TENANT_AZURE_EXCHANGE_TOKEN_WITH_USER_DELEGATION_KEY - AZURE_ACTIVE_DIRECTORY_TOKEN - description: The type of token request. As of now, only `AZURE_ACTIVE_DIRECTORY_TOKEN` is supported. + description: The type of token request. As of now, only ``AZURE_ACTIVE_DIRECTORY_TOKEN`` is supported. x-stackQL-resources: account_ip_access_lists: id: databricks_account.settings.account_ip_access_lists diff --git a/providers/src/databricks_account/v00.00.00000/services/settingsv2.yaml b/providers/src/databricks_account/v00.00.00000/services/settingsv2.yaml index 93bc4fab..e0eaaeec 100644 --- a/providers/src/databricks_account/v00.00.00000/services/settingsv2.yaml +++ b/providers/src/databricks_account/v00.00.00000/services/settingsv2.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks settingsv2 service (account-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.settingsv2 servers: - url: https://accounts.cloud.databricks.com @@ -186,6 +186,18 @@ paths: is the setting value set by consumers, check effective_aibi_dashboard_embedding_approved_domains for final setting value. + allowed_apps_user_api_scopes: + type: object + properties: + allowed_scopes: + type: array + items: + type: string + description: >- + Setting value for allowed_apps_user_api_scopes setting. + This is the setting value set by consumers, check + effective_allowed_apps_user_api_scopes for final setting + value. automatic_cluster_update_workspace: type: object properties: @@ -376,6 +388,75 @@ paths: Setting value for boolean type setting. This is the setting value set by consumers, check effective_boolean_val for final setting value. + collaboration_platform_connectivity: + type: object + properties: + connectivity: + type: string + x-enum: + - ALLOW_ALL + - ALLOW_SLACK + - ALLOW_TEAMS + - DENY_ALL + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + required: + - connectivity + description: >- + Setting value for collaboration_platform_connectivity + setting. This is the setting value set by consumers, + check effective_collaboration_platform_connectivity for + final setting value. effective_aibi_dashboard_embedding_access_policy: type: object properties: @@ -456,6 +537,17 @@ paths: aibi_dashboard_embedding_approved_domains setting. This is the final effective value of setting. To set a value use aibi_dashboard_embedding_approved_domains. + effective_allowed_apps_user_api_scopes: + type: object + properties: + allowed_scopes: + type: array + items: + type: string + description: >- + Effective setting value for allowed_apps_user_api_scopes + setting. This is the final effective value of setting. + To set a value use allowed_apps_user_api_scopes. effective_automatic_cluster_update_workspace: type: object properties: @@ -646,6 +738,86 @@ paths: Effective setting value for boolean type setting. This is the final effective value of setting. To set a value use boolean_val. + effective_collaboration_platform_connectivity: + type: object + properties: + connectivity: + type: string + x-enum: + - ALLOW_ALL + - ALLOW_SLACK + - ALLOW_TEAMS + - DENY_ALL + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + required: + - connectivity + description: >- + Effective setting value for + collaboration_platform_connectivity setting. This is the + final effective value of setting. To set a value use + collaboration_platform_connectivity. + effective_genie_chat_share_policy: + type: object + properties: + disable_make_chat_private: + type: boolean + disable_share_with_account_users: + type: boolean + description: >- + Effective setting value for genie_chat_share_policy + setting. This is the final effective value of setting. + To set a value use genie_chat_share_policy. effective_integer_val: type: object properties: @@ -655,6 +827,16 @@ paths: Effective setting value for integer type setting. This is the final effective value of setting. To set a value use integer_val. + effective_operational_email_custom_recipient: + type: object + properties: + email: + type: string + description: >- + Effective setting value for + operational_email_custom_recipient setting. This is the + final effective value of setting. To set a value use + operational_email_custom_recipient. effective_personal_compute: type: object properties: @@ -763,6 +945,18 @@ paths: Effective setting value for string type setting. This is the final effective value of setting. To set a value use string_val. + genie_chat_share_policy: + type: object + properties: + disable_make_chat_private: + type: boolean + disable_share_with_account_users: + type: boolean + description: >- + Setting value for genie_chat_share_policy setting. This + is the setting value set by consumers, check + effective_genie_chat_share_policy for final setting + value. integer_val: type: object properties: @@ -775,6 +969,16 @@ paths: name: type: string description: Name of the setting. + operational_email_custom_recipient: + type: object + properties: + email: + type: string + description: >- + Setting value for operational_email_custom_recipient + setting. This is the setting value set by consumers, + check effective_operational_email_custom_recipient for + final setting value. personal_compute: type: object properties: @@ -1100,10 +1304,10 @@ paths: unspecified, at most 200 settings will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. :param page_token: str (optional) - A page token, received from a previous `ListAccountSettingsMetadataRequest` call. Provide this to + A page token, received from a previous ``ListAccountSettingsMetadataRequest`` call. Provide this to retrieve the subsequent page. - When paginating, all other parameters provided to `ListAccountSettingsMetadataRequest` must match + When paginating, all other parameters provided to ``ListAccountSettingsMetadataRequest`` must match the call that provided the page token. :returns: Iterator over :class:`SettingsMetadata` @@ -1130,10 +1334,10 @@ paths: type: string description: >- A page token, received from a previous - `ListAccountSettingsMetadataRequest` call. Provide this to retrieve - the subsequent page. When paginating, all other parameters provided - to `ListAccountSettingsMetadataRequest` must match the call that - provided the page token. + ``ListAccountSettingsMetadataRequest`` call. Provide this to + retrieve the subsequent page. When paginating, all other parameters + provided to ``ListAccountSettingsMetadataRequest`` must match the + call that provided the page token. responses: '200': description: Success @@ -1182,10 +1386,10 @@ paths: unspecified, at most 200 settings will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. :param page_token: str (optional) - A page token, received from a previous `ListAccountUserPreferencesMetadataRequest` call. Provide + A page token, received from a previous ``ListAccountUserPreferencesMetadataRequest`` call. Provide this to retrieve the subsequent page. - When paginating, all other parameters provided to `ListAccountUserPreferencesMetadataRequest` must + When paginating, all other parameters provided to ``ListAccountUserPreferencesMetadataRequest`` must match the call that provided the page token. :returns: Iterator over :class:`SettingsMetadata` @@ -1218,9 +1422,9 @@ paths: type: string description: >- A page token, received from a previous - `ListAccountUserPreferencesMetadataRequest` call. Provide this to + ``ListAccountUserPreferencesMetadataRequest`` call. Provide this to retrieve the subsequent page. When paginating, all other parameters - provided to `ListAccountUserPreferencesMetadataRequest` must match + provided to ``ListAccountUserPreferencesMetadataRequest`` must match the call that provided the page token. responses: '200': @@ -1258,6 +1462,13 @@ components: type: array items: type: string + AllowedAppsUserApiScopesMessage: + type: object + properties: + allowed_scopes: + type: array + items: + type: string BooleanMessage: type: object properties: @@ -1324,6 +1535,31 @@ components: type: integer minutes: type: integer + CollaborationPlatformConnectivityMessage: + type: object + properties: + connectivity: + $ref: >- + #/components/schemas/CollaborationPlatformConnectivityMessageConnectivity + required: + - connectivity + description: >- + Controls which external collaboration platforms (Slack, Microsoft Teams) + can connect to a + workspace. Defaults to ALLOW_ALL. + GenieChatSharePolicy: + type: object + properties: + disable_make_chat_private: + type: boolean + disable_share_with_account_users: + type: boolean + description: >- + Workspace controls that restrict which Genie space chat sharing modes + are available to end + users. When disable_make_chat_private is true, users cannot make a Genie chat private. When + disable_share_with_account_users is true, users cannot share a Genie chat with all account + users. Defaults preserve current behavior (both false). IntegerMessage: type: object properties: @@ -1359,6 +1595,11 @@ components: items: $ref: '#/components/schemas/SettingsMetadata' description: List of all settings available via public APIs and their metadata + OperationalEmailCustomRecipientMessage: + type: object + properties: + email: + type: string PersonalComputeMessage: type: object properties: @@ -1389,6 +1630,12 @@ components: This is the setting value set by consumers, check effective_aibi_dashboard_embedding_approved_domains for final setting value. + allowed_apps_user_api_scopes: + $ref: '#/components/schemas/AllowedAppsUserApiScopesMessage' + description: >- + Setting value for allowed_apps_user_api_scopes setting. This is the + setting value set by consumers, check + effective_allowed_apps_user_api_scopes for final setting value. automatic_cluster_update_workspace: $ref: '#/components/schemas/ClusterAutoRestartMessage' description: >- @@ -1402,6 +1649,13 @@ components: Setting value for boolean type setting. This is the setting value set by consumers, check effective_boolean_val for final setting value. + collaboration_platform_connectivity: + $ref: '#/components/schemas/CollaborationPlatformConnectivityMessage' + description: >- + Setting value for collaboration_platform_connectivity setting. This + is the setting value set by consumers, check + effective_collaboration_platform_connectivity for final setting + value. effective_aibi_dashboard_embedding_access_policy: $ref: '#/components/schemas/AibiDashboardEmbeddingAccessPolicy' description: >- @@ -1415,6 +1669,12 @@ components: aibi_dashboard_embedding_approved_domains setting. This is the final effective value of setting. To set a value use aibi_dashboard_embedding_approved_domains. + effective_allowed_apps_user_api_scopes: + $ref: '#/components/schemas/AllowedAppsUserApiScopesMessage' + description: >- + Effective setting value for allowed_apps_user_api_scopes setting. + This is the final effective value of setting. To set a value use + allowed_apps_user_api_scopes. effective_automatic_cluster_update_workspace: $ref: '#/components/schemas/ClusterAutoRestartMessage' description: >- @@ -1426,11 +1686,29 @@ components: description: >- Effective setting value for boolean type setting. This is the final effective value of setting. To set a value use boolean_val. + effective_collaboration_platform_connectivity: + $ref: '#/components/schemas/CollaborationPlatformConnectivityMessage' + description: >- + Effective setting value for collaboration_platform_connectivity + setting. This is the final effective value of setting. To set a + value use collaboration_platform_connectivity. + effective_genie_chat_share_policy: + $ref: '#/components/schemas/GenieChatSharePolicy' + description: >- + Effective setting value for genie_chat_share_policy setting. This is + the final effective value of setting. To set a value use + genie_chat_share_policy. effective_integer_val: $ref: '#/components/schemas/IntegerMessage' description: >- Effective setting value for integer type setting. This is the final effective value of setting. To set a value use integer_val. + effective_operational_email_custom_recipient: + $ref: '#/components/schemas/OperationalEmailCustomRecipientMessage' + description: >- + Effective setting value for operational_email_custom_recipient + setting. This is the final effective value of setting. To set a + value use operational_email_custom_recipient. effective_personal_compute: $ref: '#/components/schemas/PersonalComputeMessage' description: >- @@ -1448,6 +1726,12 @@ components: description: >- Effective setting value for string type setting. This is the final effective value of setting. To set a value use string_val. + genie_chat_share_policy: + $ref: '#/components/schemas/GenieChatSharePolicy' + description: >- + Setting value for genie_chat_share_policy setting. This is the + setting value set by consumers, check + effective_genie_chat_share_policy for final setting value. integer_val: $ref: '#/components/schemas/IntegerMessage' description: >- @@ -1457,6 +1741,13 @@ components: name: type: string description: Name of the setting. + operational_email_custom_recipient: + $ref: '#/components/schemas/OperationalEmailCustomRecipientMessage' + description: >- + Setting value for operational_email_custom_recipient setting. This + is the setting value set by consumers, check + effective_operational_email_custom_recipient for final setting + value. personal_compute: $ref: '#/components/schemas/PersonalComputeMessage' description: >- @@ -1479,12 +1770,22 @@ components: properties: description: type: string + display_name: + type: string + description: >- + Human-readable display name for the setting or feature preview. This + field may be unset if no display name is available. docs_link: type: string description: Link to databricks documentation for the setting name: type: string description: Name of the setting. + preview_phase: + $ref: '#/components/schemas/PreviewPhase' + description: >- + Preview phase for feature preview settings. This field is not set + for non-preview settings. type: type: string description: >- @@ -1652,6 +1953,50 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CollaborationPlatformConnectivityMessageConnectivity: + type: string + x-enum: + - ALLOW_ALL + - ALLOW_SLACK + - ALLOW_TEAMS + - DENY_ALL + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. PersonalComputeMessagePersonalComputeMessageEnum: @@ -1673,6 +2018,25 @@ components: default policy before they will be able to create compute resources through that policy. + PreviewPhase: + type: string + x-enum: + - BETA + - GA + - GA_SOON + - PRIVATE_PREVIEW + - PUBLIC_PREVIEW + description: >- + Preview phase for settings that are feature previews. For settings that + are not feature + + previews, the preview_phase field is left unset. Mirrors only the + customer-facing phases + + surfaced in the UI; internal-only phases (DISABLED, DEV, + UNDER_MIGRATION, LAUNCHED, etc.) are + + not exposed here. RestrictWorkspaceAdminsMessageStatus: type: string x-enum: diff --git a/providers/src/databricks_workspace/v00.00.00000/provider.yaml b/providers/src/databricks_workspace/v00.00.00000/provider.yaml index 5b5a7fc8..277d8e50 100644 --- a/providers/src/databricks_workspace/v00.00.00000/provider.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/provider.yaml @@ -13,6 +13,17 @@ providerServices: description: >- OpenAPI specification for the Databricks agentbricks service (workspace-level APIs), generated from the Databricks Python SDK. + aisearch: + id: aisearch:v00.00.00000 + name: aisearch + preferred: true + service: + $ref: databricks_workspace/v00.00.00000/services/aisearch.yaml + title: Databricks Aisearch API (workspace) + version: v00.00.00000 + description: >- + OpenAPI specification for the Databricks aisearch service (workspace-level + APIs), generated from the Databricks Python SDK. apps: id: apps:v00.00.00000 name: apps @@ -24,6 +35,17 @@ providerServices: description: >- OpenAPI specification for the Databricks apps service (workspace-level APIs), generated from the Databricks Python SDK. + bundledeployments: + id: bundledeployments:v00.00.00000 + name: bundledeployments + preferred: true + service: + $ref: databricks_workspace/v00.00.00000/services/bundledeployments.yaml + title: Databricks Bundledeployments API (workspace) + version: v00.00.00000 + description: >- + OpenAPI specification for the Databricks bundledeployments service + (workspace-level APIs), generated from the Databricks Python SDK. catalog: id: catalog:v00.00.00000 name: catalog @@ -255,6 +277,17 @@ providerServices: description: >- OpenAPI specification for the Databricks sql service (workspace-level APIs), generated from the Databricks Python SDK. + supervisoragents: + id: supervisoragents:v00.00.00000 + name: supervisoragents + preferred: true + service: + $ref: databricks_workspace/v00.00.00000/services/supervisoragents.yaml + title: Databricks Supervisoragents API (workspace) + version: v00.00.00000 + description: >- + OpenAPI specification for the Databricks supervisoragents service + (workspace-level APIs), generated from the Databricks Python SDK. tags: id: tags:v00.00.00000 name: tags diff --git a/providers/src/databricks_workspace/v00.00.00000/services/agentbricks.yaml b/providers/src/databricks_workspace/v00.00.00000/services/agentbricks.yaml index c529cff5..416bd998 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/agentbricks.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/agentbricks.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks agentbricks service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.agentbricks servers: - url: https://{deployment_name}.cloud.databricks.com @@ -235,17 +235,17 @@ paths: The CustomLlm containing the fields which should be updated. :param update_mask: str The list of the CustomLlm fields to update. These should correspond to the values (or lack thereof) - present in `custom_llm`. + present in ``custom_llm``. The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`CustomLlm` parameters: @@ -336,16 +336,16 @@ paths: description: >- The list of the CustomLlm fields to update. These should correspond to the values (or lack thereof) present in - `custom_llm`. The field mask must be a single string, with + ``custom_llm``. The field mask must be a single string, with multiple fields separated by commas (no spaces). The field - path is relative to the resource object, using a dot (`.`) - to navigate sub-fields (e.g., `author.given_name`). + path is relative to the resource object, using a dot (``.``) + to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field - names. A field mask of `*` indicates full replacement. It’s - recommended to always explicitly list the fields being - updated and avoid using `*` wildcards, as it can lead to + names. A field mask of ``*`` indicates full replacement. + It’s recommended to always explicitly list the fields being + updated and avoid using ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - custom_llm diff --git a/providers/src/databricks_workspace/v00.00.00000/services/aisearch.yaml b/providers/src/databricks_workspace/v00.00.00000/services/aisearch.yaml new file mode 100644 index 00000000..78ec6005 --- /dev/null +++ b/providers/src/databricks_workspace/v00.00.00000/services/aisearch.yaml @@ -0,0 +1,2284 @@ +openapi: 3.0.0 +info: + title: Databricks Aisearch API (workspace) + description: >- + OpenAPI specification for the Databricks aisearch service (workspace-level + APIs), generated from the Databricks Python SDK. + version: 0.1.0 + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' + x-stackql-sdk-namespace: databricks.sdk.service.aisearch +servers: + - url: https://{deployment_name}.cloud.databricks.com + variables: + deployment_name: + description: The Databricks Workspace Deployment Name + default: dbc-abcd0123-a1bc +paths: + /api/2.0/ai-search/{parent}/endpoints: + post: + operationId: ai_search_create_endpoint + summary: Create a new AI Search endpoint. + tags: + - aisearch + - ai_search + description: |- + Create a new AI Search endpoint. + + :param parent: str + The Workspace where this Endpoint will be created. Format: ``workspaces/{workspace_id}`` + :param endpoint: :class:`Endpoint` + The Endpoint resource to create. Fields other than ``endpoint.name`` carry the desired + configuration; ``endpoint.name`` is server-assigned from ``parent`` and ``endpoint_id``. + :param endpoint_id: str (optional) + The user-supplied short name for the Endpoint, per AIP-133. The server composes the full + ``Endpoint.name`` as ``{parent}/endpoints/{endpoint_id}``. AIP-133 does not list ``endpoint_id`` as + a fields-may-be-required entry, so we annotate it OPTIONAL on the wire; the server still rejects + empty values with INVALID_PARAMETER_VALUE. + + :returns: :class:`Endpoint` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The Workspace where this Endpoint will be created. Format: + ``workspaces/{workspace_id}`` + - name: endpoint_id + in: query + required: false + schema: + type: string + description: >- + The user-supplied short name for the Endpoint, per AIP-133. The + server composes the full ``Endpoint.name`` as + ``{parent}/endpoints/{endpoint_id}``. AIP-133 does not list + ``endpoint_id`` as a fields-may-be-required entry, so we annotate it + OPTIONAL on the wire; the server still rejects empty values with + INVALID_PARAMETER_VALUE. + requestBody: + content: + application/json: + schema: + type: object + properties: + endpoint: + type: object + properties: + endpoint_type: + type: string + x-enum: + - STANDARD + - STORAGE_OPTIMIZED + description: >- + Type of endpoint. Required on create and immutable + thereafter. + budget_policy_id: + type: string + description: The user-selected budget policy id for the endpoint. + create_time: + type: string + format: date-time + description: Time the endpoint was created. + creator: + type: string + description: Creator of the endpoint + custom_tags: + type: array + items: + type: object + properties: + key: + type: string + description: Key field for an AI Search endpoint tag. + value: + type: string + description: >- + [Optional] Value field for an AI Search endpoint + tag. + required: + - key + description: >- + User-defined key/value tag attached to an AI Search + endpoint for cost attribution and access + control. + description: The custom tags assigned to the endpoint + effective_budget_policy_id: + type: string + description: The budget policy id applied to the endpoint + endpoint_status: + type: object + properties: + message: + type: string + description: >- + Human-readable detail about the endpoint's current + state or the reason for a state transition. + state: + type: string + x-enum: + - DELETED + - OFFLINE + - ONLINE + - PROVISIONING + - RED_STATE + - YELLOW_STATE + description: >- + Current lifecycle state of the endpoint. See + ``State`` for the meaning of each value. + description: Current status of the endpoint + id: + type: string + description: Unique identifier of the endpoint + index_count: + type: integer + description: Number of indexes on the endpoint + last_updated_user: + type: string + description: User who last updated the endpoint + name: + type: string + description: >- + Name of the AI Search endpoint. Server-assigned full + resource path + (``workspaces/{workspace}/endpoints/{endpoint}``) on + output. On create, the user-supplied short name is + conveyed via ``CreateEndpointRequest.endpoint_id``; the + server composes the full ``name`` and returns it on the + response. + replica_count: + type: integer + description: >- + The client-supplied desired number of replicas for the + endpoint, applied at create/update time. Mutually + exclusive with ``target_qps``. + scaling_info: + type: object + properties: + requested_target_qps: + type: integer + description: >- + The requested QPS target for the endpoint. + Best-effort; the system does not guarantee this QPS + will be achieved. + state: + type: string + x-enum: + - SCALING_CHANGE_APPLIED + - SCALING_CHANGE_IN_PROGRESS + - SCALING_CHANGE_UNSPECIFIED + description: The current state of the scaling change request. + description: Scaling information for the endpoint + target_qps: + type: integer + description: >- + Target QPS for the endpoint. Mutually exclusive with + ``replica_count``. Best-effort; the system does not + guarantee this QPS will be achieved. + throughput_info: + type: object + properties: + change_request_message: + type: string + description: >- + Additional information about the throughput change + request + change_request_state: + type: string + x-enum: + - CHANGE_ADJUSTED + - CHANGE_FAILED + - CHANGE_IN_PROGRESS + - CHANGE_REACHED_MAXIMUM + - CHANGE_REACHED_MINIMUM + - CHANGE_SUCCESS + description: >- + The state of the most recent throughput change + request + current_concurrency: + type: number + description: >- + The current concurrency (total CPU) allocated to the + endpoint + current_concurrency_utilization_percentage: + type: number + description: >- + The current utilization of concurrency as a + percentage (0-100) + current_num_replicas: + type: integer + description: >- + The current number of replicas allocated to the + endpoint + maximum_concurrency_allowed: + type: number + description: The maximum concurrency allowed for this endpoint + minimal_concurrency_allowed: + type: number + description: The minimum concurrency allowed for this endpoint + requested_concurrency: + type: number + description: >- + The requested concurrency (total CPU) for the + endpoint + requested_num_replicas: + type: integer + description: The requested number of replicas for the endpoint + description: Throughput information for the endpoint + update_time: + type: string + format: date-time + description: Time the endpoint was last updated. + usage_policy_id: + type: string + description: The usage policy id applied to the endpoint. + warnings: + type: array + items: + type: object + properties: + index_names: + type: array + items: + type: string + description: Indexes affected by this warning. + message: + type: string + description: Human-readable detail about the warning. + status_code: + type: string + x-enum: + - ENDPOINT_HAS_INDEX_WITHOUT_OPTIMIZED_ROUTE + - ENDPOINT_HAS_MANAGED_INDEX + - RERANKER_TEMPORARILY_UNAVAILABLE + description: Status code categorizing the warning. + description: >- + Advisory warning surfaced on an AI Search endpoint — a + non-fatal condition the customer should + be aware of (for example, a temporarily unavailable reranker or an index without an optimized + query route). + description: >- + Advisory warnings surfaced when target_qps is set on a + Standard endpoint. + required: + - endpoint_type + description: >- + The Endpoint resource to create. Fields other than + ``endpoint.name`` carry the desired configuration; + ``endpoint.name`` is server-assigned from ``parent`` and + ``endpoint_id``. + required: + - endpoint + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Endpoint' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + get: + operationId: ai_search_list_endpoints + summary: List AI Search endpoints in a workspace. + tags: + - aisearch + - ai_search + description: |- + List AI Search endpoints in a workspace. + + :param parent: str + The Workspace that owns this collection of endpoints. Format: ``workspaces/{workspace_id}`` + :param page_size: int (optional) + Best-effort upper bound on the number of results to return. Honored as an upper bound by the shim: + ``page_size`` only narrows the legacy backend's response, never widens it, so the practical cap is + ``min(page_size, legacy_fixed_page_size)``. + :param page_token: str (optional) + Page token from a previous response. If not provided, returns the first page. + + :returns: Iterator over :class:`Endpoint` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The Workspace that owns this collection of endpoints. Format: + ``workspaces/{workspace_id}`` + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Best-effort upper bound on the number of results to return. Honored + as an upper bound by the shim: ``page_size`` only narrows the legacy + backend's response, never widens it, so the practical cap is + ``min(page_size, legacy_fixed_page_size)``. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + Page token from a previous response. If not provided, returns the + first page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListEndpointsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{parent}/indexes: + post: + operationId: ai_search_create_index + summary: Create a new AI Search index. + tags: + - aisearch + - ai_search + description: |- + Create a new AI Search index. + + :param parent: str + The Endpoint where this Index will be created. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}`` + :param index: :class:`Index` + The Index resource to create. Fields other than ``index.name`` carry the desired configuration; + ``index.name`` is server-assigned from ``parent`` and ``index_id``. + :param index_id: str (optional) + The user-supplied Unity Catalog table name for the Index, per AIP-133. The server composes the full + ``Index.name`` as ``{parent}/indexes/{index_id}``. AIP-133 does not list ``index_id`` as a + fields-may-be-required entry, so we annotate it OPTIONAL on the wire; the server still rejects empty + values with INVALID_PARAMETER_VALUE. + + :returns: :class:`Index` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The Endpoint where this Index will be created. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}`` + - name: index_id + in: query + required: false + schema: + type: string + description: >- + The user-supplied Unity Catalog table name for the Index, per + AIP-133. The server composes the full ``Index.name`` as + ``{parent}/indexes/{index_id}``. AIP-133 does not list ``index_id`` + as a fields-may-be-required entry, so we annotate it OPTIONAL on the + wire; the server still rejects empty values with + INVALID_PARAMETER_VALUE. + requestBody: + content: + application/json: + schema: + type: object + properties: + index: + type: object + properties: + primary_key: + type: string + description: >- + Primary key of the index. Set on create and immutable + thereafter. + index_type: + type: string + x-enum: + - DELTA_SYNC + - DIRECT_ACCESS + description: >- + Type of index. Required on create and immutable + thereafter. + creator: + type: string + description: Creator of the index. + delta_sync_index_spec: + type: object + properties: + pipeline_type: + type: string + x-enum: + - CONTINUOUS + - TRIGGERED + description: >- + Pipeline execution mode. Required on create — the + backend rejects an unset value. Storage Optimized + endpoints accept only ``TRIGGERED``; Standard + endpoints accept both. No explicit ``stage`` — a + REQUIRED field staged below its service would be + dropped from combined specs while remaining in + ``required``, tripping the OpenAPI + required-vs-properties consistency check. The field + inherits the service's launch stage. + columns_to_sync: + type: array + items: + type: string + description: >- + [Optional] Select the columns to sync with the + index. If left blank, all columns from the source + table are synced. The primary key column and + embedding source or vector column are always synced. + embedding_source_columns: + type: array + items: + type: object + properties: + embedding_model_endpoint: + type: string + description: >- + Name of the embedding model endpoint, used by + default for both ingestion and querying. + model_endpoint_name_for_query: + type: string + description: >- + Name of the embedding model endpoint which, if + specified, is used for querying (not + ingestion). + name: + type: string + description: Name of the source column. + description: >- + Name of an embedding source column and its + associated embedding model endpoint. + description: The columns that contain the embedding source. + embedding_vector_columns: + type: array + items: + type: object + properties: + embedding_dimension: + type: integer + description: Dimension of the embedding vector. + name: + type: string + description: Name of the column. + description: Name and dimension of an embedding vector column. + description: The columns that contain the embedding vectors. + embedding_writeback_table: + type: string + description: >- + [Optional] Name of the Delta table to sync the index + contents and computed embeddings to. + pipeline_id: + type: string + description: >- + The ID of the pipeline that is used to sync the + index. + source_table: + type: string + description: The full name of the source Delta table. + required: + - pipeline_type + description: >- + Specification for a Delta Sync index. Set when + ``index_type`` is ``DELTA_SYNC``. + direct_access_index_spec: + type: object + properties: + embedding_source_columns: + type: array + items: + type: object + properties: + embedding_model_endpoint: + type: string + description: >- + Name of the embedding model endpoint, used by + default for both ingestion and querying. + model_endpoint_name_for_query: + type: string + description: >- + Name of the embedding model endpoint which, if + specified, is used for querying (not + ingestion). + name: + type: string + description: Name of the source column. + description: >- + Name of an embedding source column and its + associated embedding model endpoint. + description: The columns that contain the embedding source. + embedding_vector_columns: + type: array + items: + type: object + properties: + embedding_dimension: + type: integer + description: Dimension of the embedding vector. + name: + type: string + description: Name of the column. + description: Name and dimension of an embedding vector column. + description: The columns that contain the embedding vectors. + requested_schema_json: + type: string + description: >- + The index schema exactly as the user supplied it on + create, preserving the original type spellings (e.g. + ``integer``) rather than Unity Catalog's canonical + names (e.g. ``int``) that ``schema_json`` returns. + schema_json: + type: string + description: >- + The schema of the index in JSON format. Supported + types are ``integer``, ``long``, ``float``, + ``double``, ``boolean``, ``string``, ``date``, + ``timestamp``. Supported types for vector columns: + ``array``, ``array``. + description: >- + Specification for a Direct Access index. Set when + ``index_type`` is ``DIRECT_ACCESS``. + endpoint: + type: string + description: >- + Name of the endpoint associated with the index. Ignored + on create — the endpoint is taken from + ``CreateIndexRequest.parent``; populated only on output. + index_subtype: + type: string + x-enum: + - FULL_TEXT + - HYBRID + - VECTOR + description: >- + The subtype of the index. Set on create and immutable + thereafter. + name: + type: string + description: >- + Name of the AI Search index. Server-assigned full + resource path + (``workspaces/{workspace}/endpoints/{endpoint}/indexes/{index}``) + on output, where ``{index}`` is the index's Unity + Catalog table name. On create, the user-supplied UC + table name is conveyed via + ``CreateIndexRequest.index_id``; the server composes the + full ``name`` and returns it on the response. + status: + type: object + properties: + index_url: + type: string + description: >- + Index API URL used to perform operations on the + index. + indexed_row_count: + type: integer + description: Number of rows indexed. + message: + type: string + description: >- + Human-readable detail about the index's current + state. + ready: + type: boolean + description: Whether the index is ready for search. + description: Current status of the index. + required: + - primary_key + - index_type + description: >- + The Index resource to create. Fields other than + ``index.name`` carry the desired configuration; + ``index.name`` is server-assigned from ``parent`` and + ``index_id``. + required: + - index + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Index' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + get: + operationId: ai_search_list_indexes + summary: List AI Search indexes on an endpoint. + tags: + - aisearch + - ai_search + description: |- + List AI Search indexes on an endpoint. + + :param parent: str + The Endpoint that owns this collection of indexes. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}`` + :param debug_level: int (optional) + Opt-in debug level. When set to 1 or higher, the backend computes per-index routing eligibility and + populates ``can_use_optimized_route`` on each returned index. When unset (0), that field is left + unpopulated and the eligibility computation is skipped. Matches the ``debug_level`` convention on + the query path. + :param page_size: int (optional) + Best-effort upper bound on the number of results to return. Honored as an upper bound by the shim: + ``page_size`` only narrows the legacy backend's response, never widens it, so the practical cap is + ``min(page_size, legacy_fixed_page_size)``. + :param page_token: str (optional) + Page token from a previous response. If not provided, returns the first page. + + :returns: Iterator over :class:`Index` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The Endpoint that owns this collection of indexes. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}`` + - name: debug_level + in: query + required: false + schema: + type: integer + description: >- + Opt-in debug level. When set to 1 or higher, the backend computes + per-index routing eligibility and populates + ``can_use_optimized_route`` on each returned index. When unset (0), + that field is left unpopulated and the eligibility computation is + skipped. Matches the ``debug_level`` convention on the query path. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Best-effort upper bound on the number of results to return. Honored + as an upper bound by the shim: ``page_size`` only narrows the legacy + backend's response, never widens it, so the practical cap is + ``min(page_size, legacy_fixed_page_size)``. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + Page token from a previous response. If not provided, returns the + first page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListIndexesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{name}: + delete: + operationId: ai_search_delete_index + summary: Delete an AI Search index. + tags: + - aisearch + - ai_search + description: |- + Delete an AI Search index. + + :param name: str + Full resource name of the index to delete. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index to delete. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + get: + operationId: ai_search_get_index + summary: Get details for a single AI Search index. + tags: + - aisearch + - ai_search + description: |- + Get details for a single AI Search index. + + :param name: str + Full resource name of the index. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + + :returns: :class:`Index` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Index' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + patch: + operationId: ai_search_update_endpoint + summary: >- + Update an existing AI Search endpoint. Multi-bucket masks are supported + and dispatched in + tags: + - aisearch + - ai_search + description: >- + Update an existing AI Search endpoint. Multi-bucket masks are supported + and dispatched in + + deterministic bucket order: budget policy, custom tags, throughput, then + scaling/replicas. Per-bucket + + dispatch is not atomic across buckets — if a later bucket fails, earlier + buckets may already have + + been applied. + + + :param name: str + Name of the AI Search endpoint. Server-assigned full resource path + (``workspaces/{workspace}/endpoints/{endpoint}``) on output. On create, the user-supplied short name + is conveyed via ``CreateEndpointRequest.endpoint_id``; the server composes the full ``name`` and + returns it on the response. + :param endpoint: :class:`Endpoint` + The Endpoint resource to update. ``endpoint.name`` carries the full resource path. + :param update_mask: FieldMask + The list of fields to update. + + :returns: :class:`Endpoint` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Name of the AI Search endpoint. Server-assigned full resource path + (``workspaces/{workspace}/endpoints/{endpoint}``) on output. On + create, the user-supplied short name is conveyed via + ``CreateEndpointRequest.endpoint_id``; the server composes the full + ``name`` and returns it on the response. + - name: update_mask + in: query + required: true + schema: + type: object + description: The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + endpoint: + type: object + properties: + endpoint_type: + type: string + x-enum: + - STANDARD + - STORAGE_OPTIMIZED + description: >- + Type of endpoint. Required on create and immutable + thereafter. + budget_policy_id: + type: string + description: The user-selected budget policy id for the endpoint. + create_time: + type: string + format: date-time + description: Time the endpoint was created. + creator: + type: string + description: Creator of the endpoint + custom_tags: + type: array + items: + type: object + properties: + key: + type: string + description: Key field for an AI Search endpoint tag. + value: + type: string + description: >- + [Optional] Value field for an AI Search endpoint + tag. + required: + - key + description: >- + User-defined key/value tag attached to an AI Search + endpoint for cost attribution and access + control. + description: The custom tags assigned to the endpoint + effective_budget_policy_id: + type: string + description: The budget policy id applied to the endpoint + endpoint_status: + type: object + properties: + message: + type: string + description: >- + Human-readable detail about the endpoint's current + state or the reason for a state transition. + state: + type: string + x-enum: + - DELETED + - OFFLINE + - ONLINE + - PROVISIONING + - RED_STATE + - YELLOW_STATE + description: >- + Current lifecycle state of the endpoint. See + ``State`` for the meaning of each value. + description: Current status of the endpoint + id: + type: string + description: Unique identifier of the endpoint + index_count: + type: integer + description: Number of indexes on the endpoint + last_updated_user: + type: string + description: User who last updated the endpoint + name: + type: string + description: >- + Name of the AI Search endpoint. Server-assigned full + resource path + (``workspaces/{workspace}/endpoints/{endpoint}``) on + output. On create, the user-supplied short name is + conveyed via ``CreateEndpointRequest.endpoint_id``; the + server composes the full ``name`` and returns it on the + response. + replica_count: + type: integer + description: >- + The client-supplied desired number of replicas for the + endpoint, applied at create/update time. Mutually + exclusive with ``target_qps``. + scaling_info: + type: object + properties: + requested_target_qps: + type: integer + description: >- + The requested QPS target for the endpoint. + Best-effort; the system does not guarantee this QPS + will be achieved. + state: + type: string + x-enum: + - SCALING_CHANGE_APPLIED + - SCALING_CHANGE_IN_PROGRESS + - SCALING_CHANGE_UNSPECIFIED + description: The current state of the scaling change request. + description: Scaling information for the endpoint + target_qps: + type: integer + description: >- + Target QPS for the endpoint. Mutually exclusive with + ``replica_count``. Best-effort; the system does not + guarantee this QPS will be achieved. + throughput_info: + type: object + properties: + change_request_message: + type: string + description: >- + Additional information about the throughput change + request + change_request_state: + type: string + x-enum: + - CHANGE_ADJUSTED + - CHANGE_FAILED + - CHANGE_IN_PROGRESS + - CHANGE_REACHED_MAXIMUM + - CHANGE_REACHED_MINIMUM + - CHANGE_SUCCESS + description: >- + The state of the most recent throughput change + request + current_concurrency: + type: number + description: >- + The current concurrency (total CPU) allocated to the + endpoint + current_concurrency_utilization_percentage: + type: number + description: >- + The current utilization of concurrency as a + percentage (0-100) + current_num_replicas: + type: integer + description: >- + The current number of replicas allocated to the + endpoint + maximum_concurrency_allowed: + type: number + description: The maximum concurrency allowed for this endpoint + minimal_concurrency_allowed: + type: number + description: The minimum concurrency allowed for this endpoint + requested_concurrency: + type: number + description: >- + The requested concurrency (total CPU) for the + endpoint + requested_num_replicas: + type: integer + description: The requested number of replicas for the endpoint + description: Throughput information for the endpoint + update_time: + type: string + format: date-time + description: Time the endpoint was last updated. + usage_policy_id: + type: string + description: The usage policy id applied to the endpoint. + warnings: + type: array + items: + type: object + properties: + index_names: + type: array + items: + type: string + description: Indexes affected by this warning. + message: + type: string + description: Human-readable detail about the warning. + status_code: + type: string + x-enum: + - ENDPOINT_HAS_INDEX_WITHOUT_OPTIMIZED_ROUTE + - ENDPOINT_HAS_MANAGED_INDEX + - RERANKER_TEMPORARILY_UNAVAILABLE + description: Status code categorizing the warning. + description: >- + Advisory warning surfaced on an AI Search endpoint — a + non-fatal condition the customer should + be aware of (for example, a temporarily unavailable reranker or an index without an optimized + query route). + description: >- + Advisory warnings surfaced when target_qps is set on a + Standard endpoint. + required: + - endpoint_type + description: >- + The Endpoint resource to update. ``endpoint.name`` carries + the full resource path. + required: + - endpoint + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Endpoint' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{name}:query: + post: + operationId: ai_search_query_index + summary: >- + Query (search) an AI Search index. Read-only, so a read-scoped token may + invoke it. + tags: + - aisearch + - ai_search + description: >- + Query (search) an AI Search index. Read-only, so a read-scoped token may + invoke it. + + + :param name: str + Full resource name of the index to query. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + :param columns: List[str] + Column names to include in each result row. + :param columns_to_rerank: List[str] (optional) + Columns whose values are sent to the reranker. + :param facets: List[str] (optional) + Facets to compute over the matched results (e.g. ``"category TOP 5"``). + :param filters_json: str (optional) + JSON string describing query filters (e.g. ``{"id >": 5}``). + :param max_results: int (optional) + Maximum number of results to return (the legacy ``num_results``). Defaults to 10. + :param query_columns: List[str] (optional) + Text columns to search for ``query_text``. When empty, all text columns are searched. + :param query_text: str (optional) + Query text. Required for Delta Sync indexes that compute embeddings from a model endpoint. + :param query_type: str (optional) + Query type: ``ANN``, ``HYBRID``, or ``FULL_TEXT``. Defaults to ``ANN``. + :param query_vector: List[float] (optional) + Query vector. Required for Direct Access indexes and Delta Sync indexes with self-managed vectors. + :param reranker: :class:`RerankerConfig` (optional) + If set, results are reranked before being returned. + :param score_threshold: float (optional) + Score threshold for the approximate nearest-neighbor search. Defaults to 0.0. + :param sort_columns: List[str] (optional) + Sort clauses, e.g. ``["rating DESC", "price ASC"]``. Overrides relevance ordering. + + :returns: :class:`QueryIndexResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index to query. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + requestBody: + content: + application/json: + schema: + type: object + properties: + columns: + type: array + items: + type: string + description: Column names to include in each result row. + columns_to_rerank: + type: array + items: + type: string + description: Columns whose values are sent to the reranker. + facets: + type: array + items: + type: string + description: >- + Facets to compute over the matched results (e.g. ``"category + TOP 5"``). + filters_json: + type: string + description: 'JSON string describing query filters (e.g. ``{"id >": 5}``).' + max_results: + type: integer + description: >- + Maximum number of results to return (the legacy + ``num_results``). Defaults to 10. + query_columns: + type: array + items: + type: string + description: >- + Text columns to search for ``query_text``. When empty, all + text columns are searched. + query_text: + type: string + description: >- + Query text. Required for Delta Sync indexes that compute + embeddings from a model endpoint. + query_type: + type: string + description: >- + Query type: ``ANN``, ``HYBRID``, or ``FULL_TEXT``. Defaults + to ``ANN``. + query_vector: + type: array + items: + type: number + description: >- + Query vector. Required for Direct Access indexes and Delta + Sync indexes with self-managed vectors. + reranker: + type: object + properties: + model: + type: string + description: >- + Reranker identifier: "databricks_reranker" for the base + model, or a Model Serving endpoint name when + ``model_type`` is MODEL_TYPE_FINETUNED. + model_type: + type: string + x-enum: + - MODEL_TYPE_BASE + - MODEL_TYPE_FINETUNED + description: Discriminator for how ``model`` is interpreted. + parameters: + type: object + properties: + columns_to_rerank: + type: array + items: + type: string + description: >- + Columns whose values are concatenated and sent to + the reranker. + description: Parameters controlling reranking. + description: If set, results are reranked before being returned. + score_threshold: + type: number + description: >- + Score threshold for the approximate nearest-neighbor search. + Defaults to 0.0. + sort_columns: + type: array + items: + type: string + description: >- + Sort clauses, e.g. ``["rating DESC", "price ASC"]``. + Overrides relevance ordering. + required: + - columns + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/QueryIndexResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{name}:removeData: + post: + operationId: ai_search_remove_data + summary: Remove rows by primary key from a Direct Access AI Search index. + tags: + - aisearch + - ai_search + description: |- + Remove rows by primary key from a Direct Access AI Search index. + + :param name: str + Full resource name of the index. Must be a Direct Access index. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + :param primary_keys: List[str] + Primary keys of the rows to remove. + + :returns: :class:`RemoveDataResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index. Must be a Direct Access index. + Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + requestBody: + content: + application/json: + schema: + type: object + properties: + primary_keys: + type: array + items: + type: string + description: Primary keys of the rows to remove. + required: + - primary_keys + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/RemoveDataResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{name}:scan: + post: + operationId: ai_search_scan_index + summary: Scan (paginate over) the rows of an AI Search index. + tags: + - aisearch + - ai_search + description: |- + Scan (paginate over) the rows of an AI Search index. + + :param name: str + Full resource name of the index to scan. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + :param page_size: int (optional) + Maximum number of rows to return in this page. + :param page_token: str (optional) + Page token from a previous response; if unset, scanning starts from the beginning. + + :returns: :class:`ScanIndexResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index to scan. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + requestBody: + content: + application/json: + schema: + type: object + properties: + page_size: + type: integer + description: Maximum number of rows to return in this page. + page_token: + type: string + description: >- + Page token from a previous response; if unset, scanning + starts from the beginning. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ScanIndexResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{name}:sync: + post: + operationId: ai_search_sync_index + summary: >- + Synchronize a Delta Sync AI Search index with its source Delta table. + Applies only to Delta Sync + tags: + - aisearch + - ai_search + description: >- + Synchronize a Delta Sync AI Search index with its source Delta table. + Applies only to Delta Sync + + indexes; Direct Access indexes are written via the data-plane upsert + path. + + + :param name: str + Full resource name of the index to synchronize. Must be a Delta Sync index. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + + :returns: :class:`SyncIndexResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index to synchronize. Must be a Delta Sync + index. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SyncIndexResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI + /api/2.0/ai-search/{name}:upsertData: + post: + operationId: ai_search_upsert_data + summary: Upsert rows into a Direct Access AI Search index. + tags: + - aisearch + - ai_search + description: |- + Upsert rows into a Direct Access AI Search index. + + :param name: str + Full resource name of the index. Must be a Direct Access index. Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + :param inputs_json: str + JSON document describing the rows to upsert. + + :returns: :class:`UpsertDataResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name of the index. Must be a Direct Access index. + Format: + ``workspaces/{workspace_id}/endpoints/{endpoint_id}/indexes/{index_id}`` + requestBody: + content: + application/json: + schema: + type: object + properties: + inputs_json: + type: string + description: JSON document describing the rows to upsert. + required: + - inputs_json + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UpsertDataResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AiSearchAPI +components: + schemas: + ColumnInfo: + type: object + properties: + name: + type: string + description: Name of the column. + type_text: + type: string + description: Data type of the column (e.g., "string", "int", "array"). + description: >- + Column information (name and data type) for an index column. Surfaced on + ``Index.column_info``. + CustomTag: + type: object + properties: + key: + type: string + description: Key field for an AI Search endpoint tag. + value: + type: string + description: '[Optional] Value field for an AI Search endpoint tag.' + required: + - key + description: >- + User-defined key/value tag attached to an AI Search endpoint for cost + attribution and access + control. + DataModificationResult: + type: object + properties: + failed_primary_keys: + type: array + items: + type: string + description: Primary keys of rows that failed to process. + success_row_count: + type: integer + description: Count of rows processed successfully. + description: Per-row outcome of a data-plane upsert or delete operation. + DeltaSyncIndexSpec: + type: object + properties: + pipeline_type: + $ref: '#/components/schemas/PipelineType' + description: >- + Pipeline execution mode. Required on create — the backend rejects an + unset value. Storage Optimized endpoints accept only ``TRIGGERED``; + Standard endpoints accept both. No explicit ``stage`` — a REQUIRED + field staged below its service would be dropped from combined specs + while remaining in ``required``, tripping the OpenAPI + required-vs-properties consistency check. The field inherits the + service's launch stage. + columns_to_sync: + type: array + items: + type: string + description: >- + [Optional] Select the columns to sync with the index. If left blank, + all columns from the source table are synced. The primary key column + and embedding source or vector column are always synced. + embedding_source_columns: + type: array + items: + $ref: '#/components/schemas/EmbeddingSourceColumn' + description: The columns that contain the embedding source. + embedding_vector_columns: + type: array + items: + $ref: '#/components/schemas/EmbeddingVectorColumn' + description: The columns that contain the embedding vectors. + embedding_writeback_table: + type: string + description: >- + [Optional] Name of the Delta table to sync the index contents and + computed embeddings to. + pipeline_id: + type: string + description: The ID of the pipeline that is used to sync the index. + source_table: + type: string + description: The full name of the source Delta table. + required: + - pipeline_type + description: >- + Specification for a Delta Sync index — the index is kept in sync with a + source Delta table. + DirectAccessIndexSpec: + type: object + properties: + embedding_source_columns: + type: array + items: + $ref: '#/components/schemas/EmbeddingSourceColumn' + description: The columns that contain the embedding source. + embedding_vector_columns: + type: array + items: + $ref: '#/components/schemas/EmbeddingVectorColumn' + description: The columns that contain the embedding vectors. + requested_schema_json: + type: string + description: >- + The index schema exactly as the user supplied it on create, + preserving the original type spellings (e.g. ``integer``) rather + than Unity Catalog's canonical names (e.g. ``int``) that + ``schema_json`` returns. + schema_json: + type: string + description: >- + The schema of the index in JSON format. Supported types are + ``integer``, ``long``, ``float``, ``double``, ``boolean``, + ``string``, ``date``, ``timestamp``. Supported types for vector + columns: ``array``, ``array``. + description: >- + Specification for a Direct Access index — the customer manages vectors + and metadata directly. + EmbeddingSourceColumn: + type: object + properties: + embedding_model_endpoint: + type: string + description: >- + Name of the embedding model endpoint, used by default for both + ingestion and querying. + model_endpoint_name_for_query: + type: string + description: >- + Name of the embedding model endpoint which, if specified, is used + for querying (not ingestion). + name: + type: string + description: Name of the source column. + description: >- + Name of an embedding source column and its associated embedding model + endpoint. + EmbeddingVectorColumn: + type: object + properties: + embedding_dimension: + type: integer + description: Dimension of the embedding vector. + name: + type: string + description: Name of the column. + description: Name and dimension of an embedding vector column. + Endpoint: + type: object + properties: + endpoint_type: + $ref: '#/components/schemas/EndpointType' + description: Type of endpoint. Required on create and immutable thereafter. + budget_policy_id: + type: string + description: The user-selected budget policy id for the endpoint. + create_time: + type: string + format: date-time + description: Time the endpoint was created. + creator: + type: string + description: Creator of the endpoint + custom_tags: + type: array + items: + $ref: '#/components/schemas/CustomTag' + description: The custom tags assigned to the endpoint + effective_budget_policy_id: + type: string + description: The budget policy id applied to the endpoint + endpoint_status: + $ref: '#/components/schemas/EndpointStatus' + description: Current status of the endpoint + id: + type: string + description: Unique identifier of the endpoint + index_count: + type: integer + description: Number of indexes on the endpoint + last_updated_user: + type: string + description: User who last updated the endpoint + name: + type: string + description: >- + Name of the AI Search endpoint. Server-assigned full resource path + (``workspaces/{workspace}/endpoints/{endpoint}``) on output. On + create, the user-supplied short name is conveyed via + ``CreateEndpointRequest.endpoint_id``; the server composes the full + ``name`` and returns it on the response. + replica_count: + type: integer + description: >- + The client-supplied desired number of replicas for the endpoint, + applied at create/update time. Mutually exclusive with + ``target_qps``. + scaling_info: + $ref: '#/components/schemas/EndpointScalingInfo' + description: Scaling information for the endpoint + target_qps: + type: integer + description: >- + Target QPS for the endpoint. Mutually exclusive with + ``replica_count``. Best-effort; the system does not guarantee this + QPS will be achieved. + throughput_info: + $ref: '#/components/schemas/EndpointThroughputInfo' + description: Throughput information for the endpoint + update_time: + type: string + format: date-time + description: Time the endpoint was last updated. + usage_policy_id: + type: string + description: The usage policy id applied to the endpoint. + warnings: + type: array + items: + $ref: '#/components/schemas/Warning' + description: >- + Advisory warnings surfaced when target_qps is set on a Standard + endpoint. + required: + - endpoint_type + description: >- + An AI Search endpoint — compute infrastructure that hosts AI Search + indexes and serves queries + against them. Customers create, query, and delete endpoints; the system manages provisioning, + scaling, and health status. + EndpointScalingInfo: + type: object + properties: + requested_target_qps: + type: integer + description: >- + The requested QPS target for the endpoint. Best-effort; the system + does not guarantee this QPS will be achieved. + state: + $ref: '#/components/schemas/ScalingChangeState' + description: The current state of the scaling change request. + description: >- + Scaling information for a Storage Optimized endpoint — current scaling + state and the requested + QPS target the system is scaling toward. + EndpointStatus: + type: object + properties: + message: + type: string + description: >- + Human-readable detail about the endpoint's current state or the + reason for a state transition. + state: + $ref: '#/components/schemas/EndpointStatusState' + description: >- + Current lifecycle state of the endpoint. See ``State`` for the + meaning of each value. + description: >- + Lifecycle and health state of an AI Search endpoint, along with any + human-readable detail about + that state. + EndpointThroughputInfo: + type: object + properties: + change_request_message: + type: string + description: Additional information about the throughput change request + change_request_state: + $ref: '#/components/schemas/ThroughputChangeRequestState' + description: The state of the most recent throughput change request + current_concurrency: + type: number + description: The current concurrency (total CPU) allocated to the endpoint + current_concurrency_utilization_percentage: + type: number + description: The current utilization of concurrency as a percentage (0-100) + current_num_replicas: + type: integer + description: The current number of replicas allocated to the endpoint + maximum_concurrency_allowed: + type: number + description: The maximum concurrency allowed for this endpoint + minimal_concurrency_allowed: + type: number + description: The minimum concurrency allowed for this endpoint + requested_concurrency: + type: number + description: The requested concurrency (total CPU) for the endpoint + requested_num_replicas: + type: integer + description: The requested number of replicas for the endpoint + description: >- + Throughput information for an AI Search endpoint, including requested + and current concurrency + settings. + FacetResultData: + type: object + properties: + facet_array: + type: array + items: + type: array + items: + type: object + description: >- + Facet rows; each row is ``[facet_column_name, value_or_range, + count]``. + facet_row_count: + type: integer + description: Number of facet rows returned. + description: Facet aggregation rows returned by a query. + Index: + type: object + properties: + primary_key: + type: string + description: Primary key of the index. Set on create and immutable thereafter. + index_type: + $ref: '#/components/schemas/IndexType' + description: Type of index. Required on create and immutable thereafter. + creator: + type: string + description: Creator of the index. + delta_sync_index_spec: + $ref: '#/components/schemas/DeltaSyncIndexSpec' + description: >- + Specification for a Delta Sync index. Set when ``index_type`` is + ``DELTA_SYNC``. + direct_access_index_spec: + $ref: '#/components/schemas/DirectAccessIndexSpec' + description: >- + Specification for a Direct Access index. Set when ``index_type`` is + ``DIRECT_ACCESS``. + endpoint: + type: string + description: >- + Name of the endpoint associated with the index. Ignored on create — + the endpoint is taken from ``CreateIndexRequest.parent``; populated + only on output. + index_subtype: + $ref: '#/components/schemas/IndexSubtype' + description: The subtype of the index. Set on create and immutable thereafter. + name: + type: string + description: >- + Name of the AI Search index. Server-assigned full resource path + (``workspaces/{workspace}/endpoints/{endpoint}/indexes/{index}``) on + output, where ``{index}`` is the index's Unity Catalog table name. + On create, the user-supplied UC table name is conveyed via + ``CreateIndexRequest.index_id``; the server composes the full + ``name`` and returns it on the response. + status: + $ref: '#/components/schemas/IndexStatus' + description: Current status of the index. + required: + - primary_key + - index_type + description: >- + An AI Search index — a searchable collection of vectors and metadata + hosted on an AI Search + endpoint. Indexes are children of endpoints; customers create, get, list, and delete them. The + ``{index}`` segment of the resource name is the index's Unity Catalog table name. + IndexStatus: + type: object + properties: + index_url: + type: string + description: Index API URL used to perform operations on the index. + indexed_row_count: + type: integer + description: Number of rows indexed. + message: + type: string + description: Human-readable detail about the index's current state. + ready: + type: boolean + description: Whether the index is ready for search. + description: >- + Lifecycle and health state of an AI Search index, along with + human-readable detail about that + state and basic indexing progress. + ListEndpointsResponse: + type: object + properties: + endpoints: + type: array + items: + $ref: '#/components/schemas/Endpoint' + description: The endpoints in the workspace. + next_page_token: + type: string + description: >- + A token that can be used to get the next page of results. Empty when + there are no more results. + description: >- + Response for ListEndpoints carrying the page of endpoints and an + optional continuation token. + ListIndexesResponse: + type: object + properties: + indexes: + type: array + items: + $ref: '#/components/schemas/Index' + description: >- + The indexes on the endpoint. The field is named ``indexes`` (not the + irregular plural ``indices``) to satisfy core::0132, which derives + the response field name from the ListIndexes method. + core::0158::response-plural-first-field independently computes the + resource plural as ``indices`` and is satisfied via a scoped field + exception below. + next_page_token: + type: string + description: >- + A token that can be used to get the next page of results. Empty when + there are no more results. + description: >- + Response for ListIndexes carrying the page of indexes and an optional + continuation token. + QueryIndexResponse: + type: object + properties: + facet_result: + $ref: '#/components/schemas/FacetResultData' + description: Facet aggregation rows, when facets were requested. + manifest: + $ref: '#/components/schemas/ResultManifest' + description: Metadata describing the result columns. + result: + $ref: '#/components/schemas/ResultData' + description: The matched result rows. + description: >- + Response for QueryIndex carrying the matched rows and their column + metadata. + RemoveDataResponse: + type: object + properties: + result: + $ref: '#/components/schemas/DataModificationResult' + description: Per-row outcome of the delete. + status: + $ref: '#/components/schemas/DataModificationStatus' + description: Overall status of the delete. + description: Response for RemoveData. + RerankerConfig: + type: object + properties: + model: + type: string + description: >- + Reranker identifier: "databricks_reranker" for the base model, or a + Model Serving endpoint name when ``model_type`` is + MODEL_TYPE_FINETUNED. + model_type: + $ref: '#/components/schemas/RerankerConfigModelType' + description: Discriminator for how ``model`` is interpreted. + parameters: + $ref: '#/components/schemas/RerankerConfigRerankerParameters' + description: Parameters controlling reranking. + description: Configuration for reranking query results with a reranker model. + RerankerConfigRerankerParameters: + type: object + properties: + columns_to_rerank: + type: array + items: + type: string + description: Columns whose values are concatenated and sent to the reranker. + description: Parameters controlling how the reranker processes results. + ResultData: + type: object + properties: + data_array: + type: array + items: + type: array + items: + type: object + description: >- + Result rows; each row is a list of column values aligned with the + manifest columns. + row_count: + type: integer + description: Number of rows in the result set. + description: The rows of a query result set. + ResultManifest: + type: object + properties: + column_count: + type: integer + description: Number of columns in the result set. + columns: + type: array + items: + $ref: '#/components/schemas/ColumnInfo' + description: Information about each column in the result set. + facet_column_count: + type: integer + description: Number of columns in the facet result. + facet_columns: + type: array + items: + $ref: '#/components/schemas/ColumnInfo' + description: Information about each facet column. + description: Metadata describing the columns of a query result set. + ScanIndexResponse: + type: object + properties: + data: + type: array + items: + type: object + description: The rows in this page, each a struct of column name to value. + next_page_token: + type: string + description: Token for the next page; empty when the scan is exhausted. + description: >- + Response for ScanIndex carrying a page of rows and an optional + continuation token. + SyncIndexResponse: + type: object + properties: {} + description: >- + Response for SyncIndex. Empty today; reserved so future sync metadata + (e.g. an operation handle) + can be added without breaking the wire contract. + UpsertDataResponse: + type: object + properties: + result: + $ref: '#/components/schemas/DataModificationResult' + description: Per-row outcome of the upsert. + status: + $ref: '#/components/schemas/DataModificationStatus' + description: Overall status of the upsert. + description: Response for UpsertData. + Warning: + type: object + properties: + index_names: + type: array + items: + type: string + description: Indexes affected by this warning. + message: + type: string + description: Human-readable detail about the warning. + status_code: + $ref: '#/components/schemas/WarningStatusCode' + description: Status code categorizing the warning. + description: >- + Advisory warning surfaced on an AI Search endpoint — a non-fatal + condition the customer should + be aware of (for example, a temporarily unavailable reranker or an index without an optimized + query route). + DataModificationStatus: + type: string + x-enum: + - FAILURE + - PARTIAL_SUCCESS + - SUCCESS + description: >- + Overall outcome of a data-plane upsert or delete. Mirrors the legacy + + ``databricks.brickindexscheduler.UpsertDeleteDataStatus`` + value-for-value. + EndpointStatusState: + type: string + x-enum: + - DELETED + - OFFLINE + - ONLINE + - PROVISIONING + - RED_STATE + - YELLOW_STATE + description: >- + Lifecycle state of an AI Search endpoint, used by both Standard and + Storage Optimized SKUs. + EndpointType: + type: string + x-enum: + - STANDARD + - STORAGE_OPTIMIZED + description: Type of endpoint. + IndexSubtype: + type: string + x-enum: + - FULL_TEXT + - HYBRID + - VECTOR + description: >- + The subtype of the AI Search index, determining the indexing and + retrieval strategy. + + + - ``VECTOR``: Not a supported create value — do not select it. Use + ``HYBRID`` (vector + hybrid + search) or ``FULL_TEXT`` (full-text only). It is the proto2 default (``= 0``) solely to mirror + the legacy ``index_v2.proto`` enum value-for-value; it is not an offered index subtype. + - ``FULL_TEXT``: An index that uses full-text search without vector + embeddings. + + - ``HYBRID``: An index that uses vector embeddings for similarity search + and hybrid search. + IndexType: + type: string + x-enum: + - DELTA_SYNC + - DIRECT_ACCESS + description: >- + There are 2 types of AI Search indexes: + + + - ``DELTA_SYNC``: An index that automatically syncs with a source Delta + Table, automatically and + incrementally updating the index as the underlying data in the Delta Table changes. + - ``DIRECT_ACCESS``: An index that supports direct read and write of + vectors and metadata + through our REST and SDK APIs. With this model, the user manages index updates. + PipelineType: + type: string + x-enum: + - CONTINUOUS + - TRIGGERED + description: >- + Pipeline execution mode for a Delta Sync index. Required on create for + Delta Sync indexes; the + + legacy backend rejects an unset value with INVALID_PARAMETER_VALUE. + + + - ``TRIGGERED``: the pipeline stops after refreshing the source table + once, using the data + available when the update started. + - ``CONTINUOUS``: the pipeline processes new data as it arrives in the + source table to keep the + index fresh. + RerankerConfigModelType: + type: string + x-enum: + - MODEL_TYPE_BASE + - MODEL_TYPE_FINETUNED + description: How the ``model`` field is interpreted. + ScalingChangeState: + type: string + x-enum: + - SCALING_CHANGE_APPLIED + - SCALING_CHANGE_IN_PROGRESS + - SCALING_CHANGE_UNSPECIFIED + description: >- + State of the most recent scaling change request for a Storage Optimized + endpoint. + ThroughputChangeRequestState: + type: string + x-enum: + - CHANGE_ADJUSTED + - CHANGE_FAILED + - CHANGE_IN_PROGRESS + - CHANGE_REACHED_MAXIMUM + - CHANGE_REACHED_MINIMUM + - CHANGE_SUCCESS + description: >- + State of the most recent throughput change request issued against a + Storage Optimized endpoint. + + Surfaced on ``EndpointThroughputInfo.change_request_state``. + WarningStatusCode: + type: string + x-enum: + - ENDPOINT_HAS_INDEX_WITHOUT_OPTIMIZED_ROUTE + - ENDPOINT_HAS_MANAGED_INDEX + - RERANKER_TEMPORARILY_UNAVAILABLE + description: >- + Status code categorizing an advisory warning surfaced on an AI Search + endpoint. + x-stackQL-resources: + endpoints: + id: databricks_workspace.aisearch.endpoints + name: endpoints + title: Endpoints + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{parent}~1endpoints/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{parent}~1endpoints/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.endpoints + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/endpoints/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/endpoints/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/endpoints/methods/update' + delete: [] + replace: [] + indexes: + id: databricks_workspace.aisearch.indexes + name: indexes + title: Indexes + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{parent}~1indexes/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{parent}~1indexes/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.indexes + delete: + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + query: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}:query/post' + response: + mediaType: application/json + openAPIDocKey: '200' + remove_data: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}:removeData/post' + response: + mediaType: application/json + openAPIDocKey: '200' + scan: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}:scan/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sync: + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}:sync/post' + response: + mediaType: application/json + openAPIDocKey: '200' + upsert_data: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1ai-search~1{name}:upsertData/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/indexes/methods/list' + - $ref: '#/components/x-stackQL-resources/indexes/methods/get' + insert: + - $ref: '#/components/x-stackQL-resources/indexes/methods/create' + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/indexes/methods/delete' + replace: [] +x-stackQL-config: + pagination: + requestToken: + key: page_token + location: query + responseToken: + key: next_page_token + location: body diff --git a/providers/src/databricks_workspace/v00.00.00000/services/apps.yaml b/providers/src/databricks_workspace/v00.00.00000/services/apps.yaml index e5bd3bfc..54b47f44 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/apps.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/apps.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks apps service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.apps servers: - url: https://{deployment_name}.cloud.databricks.com @@ -106,6 +106,14 @@ paths: description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. required: - url - provider @@ -133,7 +141,7 @@ paths: source_code_path: type: string description: The workspace file system path of the source code used to create the app deployment. - This is different from `deployment_artifacts.source_code_path`, which is the path used by the + This is different from ``deployment_artifacts.source_code_path``, which is the path used by the deployed app. The former refers to the original source code location of the app in the workspace during deployment creation, whereas the latter provides a system generated stable snapshotted source code path used by the deployment. @@ -161,6 +169,9 @@ paths: properties: message: type: string + running_instances: + type: integer + description: The number of running instances of this application. state: type: string x-enum: @@ -171,11 +182,26 @@ paths: description: State of the application. budget_policy_id: type: string + compatibility_flags: + type: array + items: + type: string + description: 'Compatibility flags the customer is requesting for the app''s runtime environment. On + update, the submitted set must be a subset of ``effective_compatibility_flags`` (flags may only be + removed, never added). Supported flags: "PREINSTALLED_PACKAGES_LEGACY".' + compute_max_instances: + type: integer + description: Maximum number of app instances. Must be set together with ``compute_min_instances``. + compute_min_instances: + type: integer + description: Minimum number of app instances. Must be set together with ``compute_max_instances``. compute_size: type: string x-enum: - LARGE + - LIQUID - MEDIUM + - XLARGE description: |- Create a collection of name/value pairs. @@ -238,153 +264,66 @@ paths: creator: type: string description: The email of the user that created the app. - default_source_code_path: - type: string - description: The default workspace file system path of the source code from which app deployment are - created. This field tracks the workspace source code path of the last active deployment. - description: - type: string - description: The description of the app. - effective_budget_policy_id: - type: string - effective_usage_policy_id: - type: string - effective_user_api_scopes: - type: array - items: - type: string - description: The effective api scopes granted to the user access token. - git_repository: - type: object - properties: - url: - type: string - description: URL of the Git repository. - provider: - type: string - description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, - bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' - required: - - url - - provider - description: Git repository configuration for app deployments. When specified, deployments can reference - code from this repository by providing only the git reference (branch, tag, or commit). - id: - type: string - description: The unique identifier of the app. - oauth2_app_client_id: - type: string - oauth2_app_integration_id: - type: string - pending_deployment: + default_git_source: type: object properties: - command: - type: array - items: - type: string - create_time: - type: string - description: The creation time of the deployment. Formatted timestamp in ISO 6801. - creator: + branch: type: string - description: The email of the user creates the deployment. - deployment_artifacts: - type: object - properties: - source_code_path: - type: string - description: The deployment artifacts for an app. - deployment_id: + description: Git branch to checkout. + commit: type: string - description: The unique id of the deployment. - env_vars: - type: array - items: - type: object - properties: - name: - type: string - value: - type: string - description: The value for the environment variable. - value_from: - type: string - description: The name of an external Databricks resource that contains the value, such as - a secret or a database table. - description: The environment variables to set in the app runtime environment. This will override - the environment variables specified in the app.yaml file. - git_source: + description: Git commit SHA to checkout. + git_repository: type: object properties: - branch: - type: string - description: Git branch to checkout. - commit: - type: string - description: Git commit SHA to checkout. - git_repository: - type: object - properties: - url: - type: string - description: URL of the Git repository. - provider: - type: string - description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, - bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, - awsCodeCommit.' - required: - - url - - provider - description: Git repository configuration. Populated from the app's git_repository configuration. - resolved_commit: - type: string - description: The resolved commit SHA that was actually used for the deployment. This is populated - by the system after resolving the reference (branch, tag, or commit). If commit is specified - directly, this will match commit. If a branch or tag is specified, this contains the commit - SHA that the branch or tag pointed to at deployment time. - source_code_path: + url: type: string - description: Relative path to the app source code within the Git repository. If not specified, - the root of the repository is used. - tag: + description: URL of the Git repository. + provider: type: string - description: Git tag to checkout. - description: Git repository to use as the source for the app deployment. - mode: + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: type: string - x-enum: - - AUTO_SYNC - - SNAPSHOT - description: The mode of which the deployment will manage the source code. + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified directly, + this will match commit. If a branch or tag is specified, this contains the commit SHA that the + branch or tag pointed to at deployment time. source_code_path: type: string - description: The workspace file system path of the source code used to create the app deployment. - This is different from `deployment_artifacts.source_code_path`, which is the path used by the - deployed app. The former refers to the original source code location of the app in the workspace - during deployment creation, whereas the latter provides a system generated stable snapshotted - source code path used by the deployment. - status: - type: object - properties: - message: - type: string - state: - type: string - x-enum: - - CANCELLED - - FAILED - - IN_PROGRESS - - SUCCEEDED - description: State of the deployment. - description: Status and status message of the deployment - update_time: + description: Relative path to the app source code within the Git repository. If not specified, the + root of the repository is used. + tag: type: string - description: The update time of the deployment. Formatted timestamp in ISO 6801. - description: The pending deployment of the app. A deployment is considered pending when it is being - prepared for deployment to the app compute. - resources: + description: Git tag to checkout. + description: Complete git source specification including repository location and reference. + default_source_code_path: + type: string + description: The default workspace file system path of the source code from which app deployment are + created. This field tracks the workspace source code path of the last active deployment. + description: + type: string + description: The description of the app. + effective_budget_policy_id: + type: string + effective_compatibility_flags: + type: array + items: + type: string + description: The compatibility flags currently applied to the app. + effective_resources: type: array items: type: object @@ -393,7 +332,50 @@ paths: type: string app: type: object - properties: {} + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. database: type: object properties: @@ -776,175 +758,2361 @@ paths: - permission required: - name - description: Resources for the app. - service_principal_client_id: - type: string - service_principal_id: - type: integer - service_principal_name: - type: string - space: + description: Union of the app's own resources and the resources inherited from its space. Only populated + when the app belongs to a space that uses the group identity model. + effective_usage_policy_id: type: string - description: Name of the space this app belongs to. - telemetry_export_destinations: + effective_user_api_scopes: type: array items: - type: object - properties: - unity_catalog: - type: object - properties: - logs_table: - type: string - description: Unity Catalog table for OTEL logs. - metrics_table: - type: string - description: Unity Catalog table for OTEL metrics. - traces_table: - type: string - description: Unity Catalog table for OTEL traces (spans). - required: - - logs_table - - metrics_table - - traces_table - description: Unity Catalog Destinations for OTEL telemetry export. - description: A single telemetry export destination with its configuration and status. - update_time: - type: string - description: The update time of the app. Formatted timestamp in ISO 6801. - updater: - type: string - description: The email of the user that last updated the app. - url: + type: string + description: The effective api scopes granted to the user access token. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured in + the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant the + app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration for app deployments. When specified, deployments can reference + code from this repository by providing only the git reference (branch, tag, or commit). + git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified directly, + this will match commit. If a branch or tag is specified, this contains the commit SHA that the + branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, the + root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Complete git source specification including repository location and reference. + id: type: string - description: The URL of the app once it is deployed. - usage_policy_id: + description: The unique identifier of the app. + last_deployment_id: type: string - user_api_scopes: + description: The ID of the last deployment created for this app. + oauth2_app_client_id: + type: string + oauth2_app_integration_id: + type: string + pending_deployment: + type: object + properties: + command: + type: array + items: + type: string + create_time: + type: string + description: The creation time of the deployment. Formatted timestamp in ISO 6801. + creator: + type: string + description: The email of the user creates the deployment. + deployment_artifacts: + type: object + properties: + source_code_path: + type: string + description: The deployment artifacts for an app. + deployment_id: + type: string + description: The unique id of the deployment. + env_vars: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: The value for the environment variable. + value_from: + type: string + description: The name of an external Databricks resource that contains the value, such as + a secret or a database table. + description: The environment variables to set in the app runtime environment. This will override + the environment variables specified in the app.yaml file. + git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, + bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, + awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified + directly, this will match commit. If a branch or tag is specified, this contains the commit + SHA that the branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, + the root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Git repository to use as the source for the app deployment. + mode: + type: string + x-enum: + - AUTO_SYNC + - SNAPSHOT + description: The mode of which the deployment will manage the source code. + source_code_path: + type: string + description: The workspace file system path of the source code used to create the app deployment. + This is different from ``deployment_artifacts.source_code_path``, which is the path used by the + deployed app. The former refers to the original source code location of the app in the workspace + during deployment creation, whereas the latter provides a system generated stable snapshotted + source code path used by the deployment. + status: + type: object + properties: + message: + type: string + state: + type: string + x-enum: + - CANCELLED + - FAILED + - IN_PROGRESS + - SUCCEEDED + description: State of the deployment. + description: Status and status message of the deployment + update_time: + type: string + description: The update time of the deployment. Formatted timestamp in ISO 6801. + description: The pending deployment of the app. A deployment is considered pending when it is being + prepared for deployment to the app compute. + resources: type: array items: - type: string - required: - - name - required: - - app - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/App' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: AppsAPI - get: - operationId: apps_list - summary: Lists all apps in the workspace. - tags: - - apps - description: |- - Lists all apps in the workspace. + type: object + properties: + name: + type: string + app: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + database: + type: object + properties: + instance_name: + type: string + database_name: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - instance_name + - database_name + - permission + description: + type: string + description: Description of the App Resource. + experiment: + type: object + properties: + experiment_id: + type: string + permission: + type: string + x-enum: + - CAN_EDIT + - CAN_MANAGE + - CAN_READ + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - experiment_id + - permission + genie_space: + type: object + properties: + name: + type: string + space_id: + type: string + permission: + type: string + x-enum: + - CAN_EDIT + - CAN_MANAGE + - CAN_RUN + - CAN_VIEW + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - name + - space_id + - permission + job: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_MANAGE_RUN + - CAN_VIEW + - IS_OWNER + description: 'Permissions to grant on the Job. Supported permissions are: "CAN_MANAGE", "IS_OWNER", + "CAN_MANAGE_RUN", "CAN_VIEW".' + required: + - id + - permission + postgres: + type: object + properties: + branch: + type: string + database: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + secret: + type: object + properties: + scope: + type: string + key: + type: string + description: Key of the secret to grant permission on. + permission: + type: string + x-enum: + - MANAGE + - READ + - WRITE + description: 'Permission to grant on the secret scope. For secrets, only one permission is + allowed. Permission must be one of: "READ", "WRITE", "MANAGE".' + required: + - scope + - key + - permission + serving_endpoint: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + - CAN_VIEW + description: 'Permission to grant on the serving endpoint. Supported permissions are: "CAN_MANAGE", + "CAN_QUERY", "CAN_VIEW".' + required: + - name + - permission + sql_warehouse: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_USE + - IS_OWNER + description: 'Permission to grant on the SQL warehouse. Supported permissions are: "CAN_MANAGE", + "CAN_USE", "IS_OWNER".' + required: + - id + - permission + uc_securable: + type: object + properties: + securable_full_name: + type: string + securable_type: + type: string + x-enum: + - CONNECTION + - FUNCTION + - TABLE + - VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + permission: + type: string + x-enum: + - EXECUTE + - MODIFY + - READ_VOLUME + - SELECT + - USE_CONNECTION + - WRITE_VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + securable_kind: + type: string + description: The securable kind from Unity Catalog. See https://docs.databricks.com/api/workspace/tables/get#securable_kind_manifest-securable_kind. + required: + - securable_full_name + - securable_type + - permission + required: + - name + description: Resources for the app. + service_principal_client_id: + type: string + service_principal_id: + type: integer + service_principal_name: + type: string + source_code_path: + type: string + space: + type: string + description: Name of the space this app belongs to. + space_id: + type: string + description: The ID of the app space this app belongs to. None if app does not belong to a space. + telemetry_export_destinations: + type: array + items: + type: object + properties: + unity_catalog: + type: object + properties: + logs_table: + type: string + description: Unity Catalog table for OTEL logs. + metrics_table: + type: string + description: Unity Catalog table for OTEL metrics. + traces_table: + type: string + description: Unity Catalog table for OTEL traces (spans). + required: + - logs_table + - metrics_table + - traces_table + description: Unity Catalog Destinations for OTEL telemetry export. + description: A single telemetry export destination with its configuration and status. + thumbnail_url: + type: string + description: The URL of the thumbnail image for the app. + update_time: + type: string + description: The update time of the app. Formatted timestamp in ISO 6801. + updater: + type: string + description: The email of the user that last updated the app. + url: + type: string + description: The URL of the app once it is deployed. + usage_policy_id: + type: string + user_api_scopes: + type: array + items: + type: string + required: + - name + required: + - app + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/App' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + get: + operationId: apps_list + summary: Lists all apps in the workspace. + tags: + - apps + description: |- + Lists all apps in the workspace. + + :param page_size: int (optional) + Upper bound for items returned. + :param page_token: str (optional) + Pagination token to go to the next page of apps. Requests first page if absent. + :param space: str (optional) + Filter apps by app space name. When specified, only apps belonging to this space are returned. + + :returns: Iterator over :class:`App` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token to go to the next page of apps. Requests first page if absent. + - name: space + in: query + required: false + schema: + type: string + description: Filter apps by app space name. When specified, only apps belonging to this space are returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListAppsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + /api/2.0/app-spaces: + post: + operationId: apps_create_space + summary: Creates a new app space. + tags: + - apps + description: |- + Creates a new app space. + + :param space: :class:`Space` + + :returns: :class:`Operation` + requestBody: + content: + application/json: + schema: + type: object + properties: + space: + type: object + properties: + name: + type: string + create_time: + type: string + format: date-time + description: The creation time of the app space. Formatted timestamp in ISO 6801. + creator: + type: string + description: The email of the user that created the app space. + description: + type: string + description: The description of the app space. + effective_usage_policy_id: + type: string + description: The effective usage policy ID used by apps in the space. + effective_user_api_scopes: + type: array + items: + type: string + description: The effective api scopes granted to the user access token. + group_id: + type: string + description: The ID of the group attached to the app space. ASPs minted for apps in this space are added + as members of this group at create time, so the group's resource permissions flow through to every + app in the space. + id: + type: string + description: The unique identifier of the app space. + resources: + type: array + items: + type: object + properties: + name: + type: string + app: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + database: + type: object + properties: + instance_name: + type: string + database_name: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - instance_name + - database_name + - permission + description: + type: string + description: Description of the App Resource. + experiment: + type: object + properties: + experiment_id: + type: string + permission: + type: string + x-enum: + - CAN_EDIT + - CAN_MANAGE + - CAN_READ + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - experiment_id + - permission + genie_space: + type: object + properties: + name: + type: string + space_id: + type: string + permission: + type: string + x-enum: + - CAN_EDIT + - CAN_MANAGE + - CAN_RUN + - CAN_VIEW + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - name + - space_id + - permission + job: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_MANAGE_RUN + - CAN_VIEW + - IS_OWNER + description: 'Permissions to grant on the Job. Supported permissions are: "CAN_MANAGE", "IS_OWNER", + "CAN_MANAGE_RUN", "CAN_VIEW".' + required: + - id + - permission + postgres: + type: object + properties: + branch: + type: string + database: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + secret: + type: object + properties: + scope: + type: string + key: + type: string + description: Key of the secret to grant permission on. + permission: + type: string + x-enum: + - MANAGE + - READ + - WRITE + description: 'Permission to grant on the secret scope. For secrets, only one permission is + allowed. Permission must be one of: "READ", "WRITE", "MANAGE".' + required: + - scope + - key + - permission + serving_endpoint: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + - CAN_VIEW + description: 'Permission to grant on the serving endpoint. Supported permissions are: "CAN_MANAGE", + "CAN_QUERY", "CAN_VIEW".' + required: + - name + - permission + sql_warehouse: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_USE + - IS_OWNER + description: 'Permission to grant on the SQL warehouse. Supported permissions are: "CAN_MANAGE", + "CAN_USE", "IS_OWNER".' + required: + - id + - permission + uc_securable: + type: object + properties: + securable_full_name: + type: string + securable_type: + type: string + x-enum: + - CONNECTION + - FUNCTION + - TABLE + - VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + permission: + type: string + x-enum: + - EXECUTE + - MODIFY + - READ_VOLUME + - SELECT + - USE_CONNECTION + - WRITE_VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + securable_kind: + type: string + description: The securable kind from Unity Catalog. See https://docs.databricks.com/api/workspace/tables/get#securable_kind_manifest-securable_kind. + required: + - securable_full_name + - securable_type + - permission + required: + - name + description: Resources for the app space. Resources configured at the space level are available to all + apps in the space. + service_principal_client_id: + type: string + description: The service principal client ID for the app space. + service_principal_id: + type: integer + description: The service principal ID for the app space. + service_principal_name: + type: string + description: The service principal name for the app space. + status: + type: object + properties: + message: + type: string + state: + type: string + x-enum: + - SPACE_ACTIVE + - SPACE_CREATING + - SPACE_DELETED + - SPACE_DELETING + - SPACE_ERROR + - SPACE_UPDATING + description: The state of the app space. + description: The status of the app space. + telemetry_export_destinations: + type: array + items: + type: object + properties: + unity_catalog: + type: object + properties: + logs_table: + type: string + description: Unity Catalog table for OTEL logs. + metrics_table: + type: string + description: Unity Catalog table for OTEL metrics. + traces_table: + type: string + description: Unity Catalog table for OTEL traces (spans). + required: + - logs_table + - metrics_table + - traces_table + description: Unity Catalog Destinations for OTEL telemetry export. + description: A single telemetry export destination with its configuration and status. + description: Telemetry export destinations for all apps in this space. When configured, all apps in + the space export OTEL telemetry (logs, metrics, traces) to the specified Unity Catalog tables. + update_time: + type: string + format: date-time + description: The update time of the app space. Formatted timestamp in ISO 6801. + updater: + type: string + description: The email of the user that last updated the app space. + usage_policy_id: + type: string + description: The usage policy ID for managing cost at the space level. + user_api_scopes: + type: array + items: + type: string + description: OAuth scopes for apps in the space. + required: + - name + required: + - space + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + get: + operationId: apps_list_spaces + summary: Lists all app spaces in the workspace. + tags: + - apps + description: |- + Lists all app spaces in the workspace. + + :param page_size: int (optional) + Upper bound for items returned. + :param page_token: str (optional) + Pagination token to go to the next page of app spaces. Requests first page if absent. + + :returns: Iterator over :class:`Space` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token to go to the next page of app spaces. Requests first page if absent. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListSpacesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + /api/2.0/apps/{app_name}/update: + post: + operationId: apps_create_update + summary: Creates an app update and starts the update process. The update process is asynchronous and the status + tags: + - apps + description: |- + Creates an app update and starts the update process. The update process is asynchronous and the status + of the update can be checked with the GetAppUpdate method. + + :param app_name: str + :param update_mask: str + The field mask must be a single string, with multiple fields separated by commas (no spaces). The + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only + the entire collection field can be specified. Field names must exactly match the resource field + names. + + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. + :param app: :class:`App` (optional) + + :returns: + Long-running operation waiter for :class:`AppUpdate`. + See :method:wait_get_update_app_succeeded for more details. + parameters: + - name: app_name + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + update_mask: + type: string + description: The field mask must be a single string, with multiple fields separated by commas (no spaces). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). + Specification of elements in sequence or map fields is not allowed, as only the entire collection field + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates + full replacement. It’s recommended to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. + app: + type: object + properties: + name: + type: string + active_deployment: + type: object + properties: + command: + type: array + items: + type: string + create_time: + type: string + description: The creation time of the deployment. Formatted timestamp in ISO 6801. + creator: + type: string + description: The email of the user creates the deployment. + deployment_artifacts: + type: object + properties: + source_code_path: + type: string + description: The deployment artifacts for an app. + deployment_id: + type: string + description: The unique id of the deployment. + env_vars: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: The value for the environment variable. + value_from: + type: string + description: The name of an external Databricks resource that contains the value, such as + a secret or a database table. + description: The environment variables to set in the app runtime environment. This will override + the environment variables specified in the app.yaml file. + git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, + bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, + awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified + directly, this will match commit. If a branch or tag is specified, this contains the commit + SHA that the branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, + the root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Git repository to use as the source for the app deployment. + mode: + type: string + x-enum: + - AUTO_SYNC + - SNAPSHOT + description: The mode of which the deployment will manage the source code. + source_code_path: + type: string + description: The workspace file system path of the source code used to create the app deployment. + This is different from ``deployment_artifacts.source_code_path``, which is the path used by the + deployed app. The former refers to the original source code location of the app in the workspace + during deployment creation, whereas the latter provides a system generated stable snapshotted + source code path used by the deployment. + status: + type: object + properties: + message: + type: string + state: + type: string + x-enum: + - CANCELLED + - FAILED + - IN_PROGRESS + - SUCCEEDED + description: State of the deployment. + description: Status and status message of the deployment + update_time: + type: string + description: The update time of the deployment. Formatted timestamp in ISO 6801. + description: The active deployment of the app. A deployment is considered active when it has been deployed + to the app compute. + app_status: + type: object + properties: + message: + type: string + running_instances: + type: integer + description: The number of running instances of this application. + state: + type: string + x-enum: + - CRASHED + - DEPLOYING + - RUNNING + - UNAVAILABLE + description: State of the application. + budget_policy_id: + type: string + compatibility_flags: + type: array + items: + type: string + description: 'Compatibility flags the customer is requesting for the app''s runtime environment. On + update, the submitted set must be a subset of ``effective_compatibility_flags`` (flags may only be + removed, never added). Supported flags: "PREINSTALLED_PACKAGES_LEGACY".' + compute_max_instances: + type: integer + description: Maximum number of app instances. Must be set together with ``compute_min_instances``. + compute_min_instances: + type: integer + description: Minimum number of app instances. Must be set together with ``compute_max_instances``. + compute_size: + type: string + x-enum: + - LARGE + - LIQUID + - MEDIUM + - XLARGE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + compute_status: + type: object + properties: + active_instances: + type: integer + message: + type: string + description: Compute status message + state: + type: string + x-enum: + - ACTIVE + - DELETING + - ERROR + - STARTING + - STOPPED + - STOPPING + - UPDATING + description: State of the app compute. + create_time: + type: string + description: The creation time of the app. Formatted timestamp in ISO 6801. + creator: + type: string + description: The email of the user that created the app. + default_git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified directly, + this will match commit. If a branch or tag is specified, this contains the commit SHA that the + branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, the + root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Complete git source specification including repository location and reference. + default_source_code_path: + type: string + description: The default workspace file system path of the source code from which app deployment are + created. This field tracks the workspace source code path of the last active deployment. + description: + type: string + description: The description of the app. + effective_budget_policy_id: + type: string + effective_compatibility_flags: + type: array + items: + type: string + description: The compatibility flags currently applied to the app. + effective_resources: + type: array + items: + type: object + properties: + name: + type: string + app: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + database: + type: object + properties: + instance_name: + type: string + database_name: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - instance_name + - database_name + - permission + description: + type: string + description: Description of the App Resource. + experiment: + type: object + properties: + experiment_id: + type: string + permission: + type: string + x-enum: + - CAN_EDIT + - CAN_MANAGE + - CAN_READ + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - experiment_id + - permission + genie_space: + type: object + properties: + name: + type: string + space_id: + type: string + permission: + type: string + x-enum: + - CAN_EDIT + - CAN_MANAGE + - CAN_RUN + - CAN_VIEW + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + required: + - name + - space_id + - permission + job: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_MANAGE_RUN + - CAN_VIEW + - IS_OWNER + description: 'Permissions to grant on the Job. Supported permissions are: "CAN_MANAGE", "IS_OWNER", + "CAN_MANAGE_RUN", "CAN_VIEW".' + required: + - id + - permission + postgres: + type: object + properties: + branch: + type: string + database: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + secret: + type: object + properties: + scope: + type: string + key: + type: string + description: Key of the secret to grant permission on. + permission: + type: string + x-enum: + - MANAGE + - READ + - WRITE + description: 'Permission to grant on the secret scope. For secrets, only one permission is + allowed. Permission must be one of: "READ", "WRITE", "MANAGE".' + required: + - scope + - key + - permission + serving_endpoint: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + - CAN_VIEW + description: 'Permission to grant on the serving endpoint. Supported permissions are: "CAN_MANAGE", + "CAN_QUERY", "CAN_VIEW".' + required: + - name + - permission + sql_warehouse: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_USE + - IS_OWNER + description: 'Permission to grant on the SQL warehouse. Supported permissions are: "CAN_MANAGE", + "CAN_USE", "IS_OWNER".' + required: + - id + - permission + uc_securable: + type: object + properties: + securable_full_name: + type: string + securable_type: + type: string + x-enum: + - CONNECTION + - FUNCTION + - TABLE + - VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] - :param page_size: int (optional) - Upper bound for items returned. - :param page_token: str (optional) - Pagination token to go to the next page of apps. Requests first page if absent. - :param space: str (optional) - Filter apps by app space name. When specified, only apps belonging to this space are returned. + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + permission: + type: string + x-enum: + - EXECUTE + - MODIFY + - READ_VOLUME + - SELECT + - USE_CONNECTION + - WRITE_VOLUME + description: |- + Create a collection of name/value pairs. - :returns: Iterator over :class:`App` - parameters: - - name: page_size - in: query - required: false - schema: - type: integer - description: Upper bound for items returned. - - name: page_token - in: query - required: false - schema: - type: string - description: Pagination token to go to the next page of apps. Requests first page if absent. - - name: space - in: query - required: false - schema: - type: string - description: Filter apps by app space name. When specified, only apps belonging to this space are returned. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListAppsResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: AppsAPI - /api/2.0/app-spaces: - post: - operationId: apps_create_space - summary: Creates a new app space. - tags: - - apps - description: |- - Creates a new app space. + Example enumeration: - :param space: :class:`Space` + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 - :returns: :class:`Operation` - requestBody: - content: - application/json: - schema: - type: object - properties: - space: - type: object - properties: - name: - type: string - create_time: - type: string - format: date-time - description: The creation time of the app space. Formatted timestamp in ISO 6801. - creator: - type: string - description: The email of the user that created the app space. - description: - type: string - description: The description of the app space. + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + securable_kind: + type: string + description: The securable kind from Unity Catalog. See https://docs.databricks.com/api/workspace/tables/get#securable_kind_manifest-securable_kind. + required: + - securable_full_name + - securable_type + - permission + required: + - name + description: Union of the app's own resources and the resources inherited from its space. Only populated + when the app belongs to a space that uses the group identity model. effective_usage_policy_id: type: string - description: The effective usage policy ID used by apps in the space. effective_user_api_scopes: type: array items: type: string description: The effective api scopes granted to the user access token. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured in + the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant the + app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration for app deployments. When specified, deployments can reference + code from this repository by providing only the git reference (branch, tag, or commit). + git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified directly, + this will match commit. If a branch or tag is specified, this contains the commit SHA that the + branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, the + root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Complete git source specification including repository location and reference. id: type: string - description: The unique identifier of the app space. + description: The unique identifier of the app. + last_deployment_id: + type: string + description: The ID of the last deployment created for this app. + oauth2_app_client_id: + type: string + oauth2_app_integration_id: + type: string + pending_deployment: + type: object + properties: + command: + type: array + items: + type: string + create_time: + type: string + description: The creation time of the deployment. Formatted timestamp in ISO 6801. + creator: + type: string + description: The email of the user creates the deployment. + deployment_artifacts: + type: object + properties: + source_code_path: + type: string + description: The deployment artifacts for an app. + deployment_id: + type: string + description: The unique id of the deployment. + env_vars: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: The value for the environment variable. + value_from: + type: string + description: The name of an external Databricks resource that contains the value, such as + a secret or a database table. + description: The environment variables to set in the app runtime environment. This will override + the environment variables specified in the app.yaml file. + git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, + bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, + awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified + directly, this will match commit. If a branch or tag is specified, this contains the commit + SHA that the branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, + the root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Git repository to use as the source for the app deployment. + mode: + type: string + x-enum: + - AUTO_SYNC + - SNAPSHOT + description: The mode of which the deployment will manage the source code. + source_code_path: + type: string + description: The workspace file system path of the source code used to create the app deployment. + This is different from ``deployment_artifacts.source_code_path``, which is the path used by the + deployed app. The former refers to the original source code location of the app in the workspace + during deployment creation, whereas the latter provides a system generated stable snapshotted + source code path used by the deployment. + status: + type: object + properties: + message: + type: string + state: + type: string + x-enum: + - CANCELLED + - FAILED + - IN_PROGRESS + - SUCCEEDED + description: State of the deployment. + description: Status and status message of the deployment + update_time: + type: string + description: The update time of the deployment. Formatted timestamp in ISO 6801. + description: The pending deployment of the app. A deployment is considered pending when it is being + prepared for deployment to the app compute. resources: type: array items: @@ -954,7 +3122,50 @@ paths: type: string app: type: object - properties: {} + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. database: type: object properties: @@ -1337,59 +3548,73 @@ paths: - permission required: - name - description: Resources for the app space. Resources configured at the space level are available to all - apps in the space. + description: Resources for the app. service_principal_client_id: type: string - description: The service principal client ID for the app space. service_principal_id: type: integer - description: The service principal ID for the app space. service_principal_name: type: string - description: The service principal name for the app space. - status: - type: object - properties: - message: - type: string - state: - type: string - x-enum: - - SPACE_ACTIVE - - SPACE_CREATING - - SPACE_DELETED - - SPACE_DELETING - - SPACE_ERROR - - SPACE_UPDATING - description: The state of the app space. - description: The status of the app space. + source_code_path: + type: string + space: + type: string + description: Name of the space this app belongs to. + space_id: + type: string + description: The ID of the app space this app belongs to. None if app does not belong to a space. + telemetry_export_destinations: + type: array + items: + type: object + properties: + unity_catalog: + type: object + properties: + logs_table: + type: string + description: Unity Catalog table for OTEL logs. + metrics_table: + type: string + description: Unity Catalog table for OTEL metrics. + traces_table: + type: string + description: Unity Catalog table for OTEL traces (spans). + required: + - logs_table + - metrics_table + - traces_table + description: Unity Catalog Destinations for OTEL telemetry export. + description: A single telemetry export destination with its configuration and status. + thumbnail_url: + type: string + description: The URL of the thumbnail image for the app. update_time: type: string - format: date-time - description: The update time of the app space. Formatted timestamp in ISO 6801. + description: The update time of the app. Formatted timestamp in ISO 6801. updater: type: string - description: The email of the user that last updated the app space. + description: The email of the user that last updated the app. + url: + type: string + description: The URL of the app once it is deployed. usage_policy_id: type: string - description: The usage policy ID for managing cost at the space level. user_api_scopes: type: array items: type: string - description: OAuth scopes for apps in the space. required: - name required: - - space + - update_mask responses: '200': description: Success content: application/json: schema: - type: object + $ref: '#/components/schemas/AppUpdate' default: description: Error response content: @@ -1403,39 +3628,108 @@ paths: type: string x-stackql-sdk-source: AppsAPI get: - operationId: apps_list_spaces - summary: Lists all app spaces in the workspace. + operationId: apps_get_update + summary: Gets the status of an app update. tags: - apps description: |- - Lists all app spaces in the workspace. + Gets the status of an app update. - :param page_size: int (optional) - Upper bound for items returned. - :param page_token: str (optional) - Pagination token to go to the next page of app spaces. Requests first page if absent. + :param app_name: str + The name of the app. - :returns: Iterator over :class:`Space` + :returns: :class:`AppUpdate` parameters: - - name: page_size - in: query - required: false + - name: app_name + in: path + required: true schema: - type: integer - description: Upper bound for items returned. - - name: page_token - in: query - required: false + type: string + description: The name of the app. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AppUpdate' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + /api/2.0/apps/{name}: + delete: + operationId: apps_delete + summary: Deletes an app. + tags: + - apps + description: |- + Deletes an app. + + :param name: str + The name of the app. + + :returns: :class:`App` + parameters: + - name: name + in: path + required: true schema: type: string - description: Pagination token to go to the next page of app spaces. Requests first page if absent. + description: The name of the app. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/App' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + get: + operationId: apps_get + summary: Retrieves information for the app with the supplied name. + tags: + - apps + description: |- + Retrieves information for the app with the supplied name. + + :param name: str + The name of the app. + + :returns: :class:`App` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The name of the app. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListSpacesResponse' + $ref: '#/components/schemas/App' default: description: Error response content: @@ -1448,52 +3742,34 @@ paths: message: type: string x-stackql-sdk-source: AppsAPI - /api/2.0/apps/{app_name}/update: - post: - operationId: apps_create_update - summary: Creates an app update and starts the update process. The update process is asynchronous and the status + patch: + operationId: apps_update + summary: Updates the app with the supplied name. tags: - apps description: |- - Creates an app update and starts the update process. The update process is asynchronous and the status - of the update can be checked with the GetAppUpdate method. - - :param app_name: str - :param update_mask: str - The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only - the entire collection field can be specified. Field names must exactly match the resource field - names. + Updates the app with the supplied name. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. - :param app: :class:`App` (optional) + :param name: str + The name of the app. The name must contain only lowercase alphanumeric characters and hyphens. It + must be unique within the workspace. + :param app: :class:`App` - :returns: - Long-running operation waiter for :class:`AppUpdate`. - See :method:wait_get_update_app_succeeded for more details. + :returns: :class:`App` parameters: - - name: app_name + - name: name in: path required: true schema: type: string + description: The name of the app. The name must contain only lowercase alphanumeric characters and hyphens. It must + be unique within the workspace. requestBody: content: application/json: schema: type: object properties: - update_mask: - type: string - description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). - Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates - full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. app: type: object properties: @@ -1557,6 +3833,14 @@ paths: description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. required: - url - provider @@ -1584,7 +3868,7 @@ paths: source_code_path: type: string description: The workspace file system path of the source code used to create the app deployment. - This is different from `deployment_artifacts.source_code_path`, which is the path used by the + This is different from ``deployment_artifacts.source_code_path``, which is the path used by the deployed app. The former refers to the original source code location of the app in the workspace during deployment creation, whereas the latter provides a system generated stable snapshotted source code path used by the deployment. @@ -1612,359 +3896,176 @@ paths: properties: message: type: string - state: - type: string - x-enum: - - CRASHED - - DEPLOYING - - RUNNING - - UNAVAILABLE - description: State of the application. - budget_policy_id: - type: string - compute_size: - type: string - x-enum: - - LARGE - - MEDIUM - description: |- - Create a collection of name/value pairs. - - Example enumeration: - - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 - - Access them by: - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many members they have: - - >>> len(Color) - 3 - - >>> list(Color) - [, , ] - - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. - compute_status: - type: object - properties: - active_instances: + running_instances: type: integer - message: - type: string - description: Compute status message + description: The number of running instances of this application. state: type: string x-enum: - - ACTIVE - - DELETING - - ERROR - - STARTING - - STOPPED - - STOPPING - - UPDATING - description: State of the app compute. - create_time: - type: string - description: The creation time of the app. Formatted timestamp in ISO 6801. - creator: - type: string - description: The email of the user that created the app. - default_source_code_path: - type: string - description: The default workspace file system path of the source code from which app deployment are - created. This field tracks the workspace source code path of the last active deployment. - description: - type: string - description: The description of the app. - effective_budget_policy_id: - type: string - effective_usage_policy_id: - type: string - effective_user_api_scopes: - type: array - items: - type: string - description: The effective api scopes granted to the user access token. - git_repository: - type: object - properties: - url: - type: string - description: URL of the Git repository. - provider: - type: string - description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, - bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' - required: - - url - - provider - description: Git repository configuration for app deployments. When specified, deployments can reference - code from this repository by providing only the git reference (branch, tag, or commit). - id: - type: string - description: The unique identifier of the app. - oauth2_app_client_id: - type: string - oauth2_app_integration_id: - type: string - pending_deployment: - type: object - properties: - command: - type: array - items: - type: string - create_time: - type: string - description: The creation time of the deployment. Formatted timestamp in ISO 6801. - creator: - type: string - description: The email of the user creates the deployment. - deployment_artifacts: - type: object - properties: - source_code_path: - type: string - description: The deployment artifacts for an app. - deployment_id: - type: string - description: The unique id of the deployment. - env_vars: - type: array - items: - type: object - properties: - name: - type: string - value: - type: string - description: The value for the environment variable. - value_from: - type: string - description: The name of an external Databricks resource that contains the value, such as - a secret or a database table. - description: The environment variables to set in the app runtime environment. This will override - the environment variables specified in the app.yaml file. - git_source: - type: object - properties: - branch: - type: string - description: Git branch to checkout. - commit: - type: string - description: Git commit SHA to checkout. - git_repository: - type: object - properties: - url: - type: string - description: URL of the Git repository. - provider: - type: string - description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, - bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, - awsCodeCommit.' - required: - - url - - provider - description: Git repository configuration. Populated from the app's git_repository configuration. - resolved_commit: - type: string - description: The resolved commit SHA that was actually used for the deployment. This is populated - by the system after resolving the reference (branch, tag, or commit). If commit is specified - directly, this will match commit. If a branch or tag is specified, this contains the commit - SHA that the branch or tag pointed to at deployment time. - source_code_path: - type: string - description: Relative path to the app source code within the Git repository. If not specified, - the root of the repository is used. - tag: - type: string - description: Git tag to checkout. - description: Git repository to use as the source for the app deployment. - mode: - type: string - x-enum: - - AUTO_SYNC - - SNAPSHOT - description: The mode of which the deployment will manage the source code. - source_code_path: - type: string - description: The workspace file system path of the source code used to create the app deployment. - This is different from `deployment_artifacts.source_code_path`, which is the path used by the - deployed app. The former refers to the original source code location of the app in the workspace - during deployment creation, whereas the latter provides a system generated stable snapshotted - source code path used by the deployment. - status: - type: object - properties: - message: - type: string - state: - type: string - x-enum: - - CANCELLED - - FAILED - - IN_PROGRESS - - SUCCEEDED - description: State of the deployment. - description: Status and status message of the deployment - update_time: - type: string - description: The update time of the deployment. Formatted timestamp in ISO 6801. - description: The pending deployment of the app. A deployment is considered pending when it is being - prepared for deployment to the app compute. - resources: + - CRASHED + - DEPLOYING + - RUNNING + - UNAVAILABLE + description: State of the application. + budget_policy_id: + type: string + compatibility_flags: type: array items: - type: object - properties: - name: - type: string - app: - type: object - properties: {} - database: - type: object - properties: - instance_name: - type: string - database_name: - type: string - permission: - type: string - x-enum: - - CAN_CONNECT_AND_CREATE - description: |- - Create a collection of name/value pairs. + type: string + description: 'Compatibility flags the customer is requesting for the app''s runtime environment. On + update, the submitted set must be a subset of ``effective_compatibility_flags`` (flags may only be + removed, never added). Supported flags: "PREINSTALLED_PACKAGES_LEGACY".' + compute_max_instances: + type: integer + description: Maximum number of app instances. Must be set together with ``compute_min_instances``. + compute_min_instances: + type: integer + description: Minimum number of app instances. Must be set together with ``compute_max_instances``. + compute_size: + type: string + x-enum: + - LARGE + - LIQUID + - MEDIUM + - XLARGE + description: |- + Create a collection of name/value pairs. - Example enumeration: + Example enumeration: - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 - Access them by: + Access them by: - - attribute access: + - attribute access: - >>> Color.RED - + >>> Color.RED + - - value lookup: + - value lookup: - >>> Color(1) - + >>> Color(1) + - - name lookup: + - name lookup: - >>> Color['RED'] - + >>> Color['RED'] + - Enumerations can be iterated over, and know how many members they have: + Enumerations can be iterated over, and know how many members they have: - >>> len(Color) - 3 + >>> len(Color) + 3 - >>> list(Color) - [, , ] + >>> list(Color) + [, , ] - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. - required: - - instance_name - - database_name - - permission - description: + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + compute_status: + type: object + properties: + active_instances: + type: integer + message: + type: string + description: Compute status message + state: + type: string + x-enum: + - ACTIVE + - DELETING + - ERROR + - STARTING + - STOPPED + - STOPPING + - UPDATING + description: State of the app compute. + create_time: + type: string + description: The creation time of the app. Formatted timestamp in ISO 6801. + creator: + type: string + description: The email of the user that created the app. + default_git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified directly, + this will match commit. If a branch or tag is specified, this contains the commit SHA that the + branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, the + root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Complete git source specification including repository location and reference. + default_source_code_path: + type: string + description: The default workspace file system path of the source code from which app deployment are + created. This field tracks the workspace source code path of the last active deployment. + description: + type: string + description: The description of the app. + effective_budget_policy_id: + type: string + effective_compatibility_flags: + type: array + items: + type: string + description: The compatibility flags currently applied to the app. + effective_resources: + type: array + items: + type: object + properties: + name: type: string - description: Description of the App Resource. - experiment: - type: object - properties: - experiment_id: - type: string - permission: - type: string - x-enum: - - CAN_EDIT - - CAN_MANAGE - - CAN_READ - description: |- - Create a collection of name/value pairs. - - Example enumeration: - - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 - - Access them by: - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many members they have: - - >>> len(Color) - 3 - - >>> list(Color) - [, , ] - - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. - required: - - experiment_id - - permission - genie_space: + app: type: object properties: name: type: string - space_id: - type: string permission: type: string x-enum: - - CAN_EDIT - - CAN_MANAGE - - CAN_RUN - - CAN_VIEW + - CAN_USE description: |- Create a collection of name/value pairs. @@ -1990,45 +4091,24 @@ paths: - name lookup: >>> Color['RED'] - - - Enumerations can be iterated over, and know how many members they have: - - >>> len(Color) - 3 - - >>> list(Color) - [, , ] - - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. - required: - - name - - space_id - - permission - job: - type: object - properties: - id: - type: string - permission: - type: string - x-enum: - - CAN_MANAGE - - CAN_MANAGE_RUN - - CAN_VIEW - - IS_OWNER - description: 'Permissions to grant on the Job. Supported permissions are: "CAN_MANAGE", "IS_OWNER", - "CAN_MANAGE_RUN", "CAN_VIEW".' - required: - - id - - permission - postgres: + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + database: type: object properties: - branch: + instance_name: type: string - database: + database_name: type: string permission: type: string @@ -2071,70 +4151,24 @@ paths: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. - secret: - type: object - properties: - scope: - type: string - key: - type: string - description: Key of the secret to grant permission on. - permission: - type: string - x-enum: - - MANAGE - - READ - - WRITE - description: 'Permission to grant on the secret scope. For secrets, only one permission is - allowed. Permission must be one of: "READ", "WRITE", "MANAGE".' - required: - - scope - - key - - permission - serving_endpoint: - type: object - properties: - name: - type: string - permission: - type: string - x-enum: - - CAN_MANAGE - - CAN_QUERY - - CAN_VIEW - description: 'Permission to grant on the serving endpoint. Supported permissions are: "CAN_MANAGE", - "CAN_QUERY", "CAN_VIEW".' required: - - name + - instance_name + - database_name - permission - sql_warehouse: + description: + type: string + description: Description of the App Resource. + experiment: type: object properties: - id: + experiment_id: type: string permission: type: string x-enum: + - CAN_EDIT - CAN_MANAGE - - CAN_USE - - IS_OWNER - description: 'Permission to grant on the SQL warehouse. Supported permissions are: "CAN_MANAGE", - "CAN_USE", "IS_OWNER".' - required: - - id - - permission - uc_securable: - type: object - properties: - securable_full_name: - type: string - securable_type: - type: string - x-enum: - - CONNECTION - - FUNCTION - - TABLE - - VOLUME + - CAN_READ description: |- Create a collection of name/value pairs. @@ -2172,15 +4206,23 @@ paths: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + required: + - experiment_id + - permission + genie_space: + type: object + properties: + name: + type: string + space_id: + type: string permission: type: string x-enum: - - EXECUTE - - MODIFY - - READ_VOLUME - - SELECT - - USE_CONNECTION - - WRITE_VOLUME + - CAN_EDIT + - CAN_MANAGE + - CAN_RUN + - CAN_VIEW description: |- Create a collection of name/value pairs. @@ -2218,433 +4260,233 @@ paths: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. - securable_kind: + required: + - name + - space_id + - permission + job: + type: object + properties: + id: type: string - description: The securable kind from Unity Catalog. See https://docs.databricks.com/api/workspace/tables/get#securable_kind_manifest-securable_kind. + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_MANAGE_RUN + - CAN_VIEW + - IS_OWNER + description: 'Permissions to grant on the Job. Supported permissions are: "CAN_MANAGE", "IS_OWNER", + "CAN_MANAGE_RUN", "CAN_VIEW".' required: - - securable_full_name - - securable_type + - id - permission - required: - - name - description: Resources for the app. - service_principal_client_id: - type: string - service_principal_id: - type: integer - service_principal_name: - type: string - space: - type: string - description: Name of the space this app belongs to. - telemetry_export_destinations: - type: array - items: - type: object - properties: - unity_catalog: + postgres: type: object properties: - logs_table: + branch: type: string - description: Unity Catalog table for OTEL logs. - metrics_table: + database: type: string - description: Unity Catalog table for OTEL metrics. - traces_table: + permission: type: string - description: Unity Catalog table for OTEL traces (spans). - required: - - logs_table - - metrics_table - - traces_table - description: Unity Catalog Destinations for OTEL telemetry export. - description: A single telemetry export destination with its configuration and status. - update_time: - type: string - description: The update time of the app. Formatted timestamp in ISO 6801. - updater: - type: string - description: The email of the user that last updated the app. - url: - type: string - description: The URL of the app once it is deployed. - usage_policy_id: - type: string - user_api_scopes: - type: array - items: - type: string - required: - - name - required: - - update_mask - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/AppUpdate' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: AppsAPI - get: - operationId: apps_get_update - summary: Gets the status of an app update. - tags: - - apps - description: |- - Gets the status of an app update. + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. - :param app_name: str - The name of the app. + Example enumeration: - :returns: :class:`AppUpdate` - parameters: - - name: app_name - in: path - required: true - schema: - type: string - description: The name of the app. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/AppUpdate' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: AppsAPI - /api/2.0/apps/{name}: - delete: - operationId: apps_delete - summary: Deletes an app. - tags: - - apps - description: |- - Deletes an app. + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: - :param name: str - The name of the app. + >>> Color(1) + - :returns: :class:`App` - parameters: - - name: name - in: path - required: true - schema: - type: string - description: The name of the app. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/App' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: AppsAPI - get: - operationId: apps_get - summary: Retrieves information for the app with the supplied name. - tags: - - apps - description: |- - Retrieves information for the app with the supplied name. + - name lookup: - :param name: str - The name of the app. + >>> Color['RED'] + - :returns: :class:`App` - parameters: - - name: name - in: path - required: true - schema: - type: string - description: The name of the app. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/App' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: AppsAPI - patch: - operationId: apps_update - summary: Updates the app with the supplied name. - tags: - - apps - description: |- - Updates the app with the supplied name. + Enumerations can be iterated over, and know how many members they have: - :param name: str - The name of the app. The name must contain only lowercase alphanumeric characters and hyphens. It - must be unique within the workspace. - :param app: :class:`App` + >>> len(Color) + 3 - :returns: :class:`App` - parameters: - - name: name - in: path - required: true - schema: - type: string - description: The name of the app. The name must contain only lowercase alphanumeric characters and hyphens. It must - be unique within the workspace. - requestBody: - content: - application/json: - schema: - type: object - properties: - app: - type: object - properties: - name: - type: string - active_deployment: - type: object - properties: - command: - type: array - items: - type: string - create_time: - type: string - description: The creation time of the deployment. Formatted timestamp in ISO 6801. - creator: - type: string - description: The email of the user creates the deployment. - deployment_artifacts: - type: object - properties: - source_code_path: - type: string - description: The deployment artifacts for an app. - deployment_id: - type: string - description: The unique id of the deployment. - env_vars: - type: array - items: + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + secret: type: object properties: - name: + scope: type: string - value: + key: type: string - description: The value for the environment variable. - value_from: + description: Key of the secret to grant permission on. + permission: type: string - description: The name of an external Databricks resource that contains the value, such as - a secret or a database table. - description: The environment variables to set in the app runtime environment. This will override - the environment variables specified in the app.yaml file. - git_source: - type: object - properties: - branch: - type: string - description: Git branch to checkout. - commit: - type: string - description: Git commit SHA to checkout. - git_repository: - type: object - properties: - url: - type: string - description: URL of the Git repository. - provider: - type: string - description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, - bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, - awsCodeCommit.' - required: - - url - - provider - description: Git repository configuration. Populated from the app's git_repository configuration. - resolved_commit: - type: string - description: The resolved commit SHA that was actually used for the deployment. This is populated - by the system after resolving the reference (branch, tag, or commit). If commit is specified - directly, this will match commit. If a branch or tag is specified, this contains the commit - SHA that the branch or tag pointed to at deployment time. - source_code_path: - type: string - description: Relative path to the app source code within the Git repository. If not specified, - the root of the repository is used. - tag: - type: string - description: Git tag to checkout. - description: Git repository to use as the source for the app deployment. - mode: - type: string - x-enum: - - AUTO_SYNC - - SNAPSHOT - description: The mode of which the deployment will manage the source code. - source_code_path: - type: string - description: The workspace file system path of the source code used to create the app deployment. - This is different from `deployment_artifacts.source_code_path`, which is the path used by the - deployed app. The former refers to the original source code location of the app in the workspace - during deployment creation, whereas the latter provides a system generated stable snapshotted - source code path used by the deployment. - status: - type: object - properties: - message: - type: string - state: - type: string - x-enum: - - CANCELLED - - FAILED - - IN_PROGRESS - - SUCCEEDED - description: State of the deployment. - description: Status and status message of the deployment - update_time: - type: string - description: The update time of the deployment. Formatted timestamp in ISO 6801. - description: The active deployment of the app. A deployment is considered active when it has been deployed - to the app compute. - app_status: - type: object - properties: - message: - type: string - state: - type: string - x-enum: - - CRASHED - - DEPLOYING - - RUNNING - - UNAVAILABLE - description: State of the application. - budget_policy_id: - type: string - compute_size: - type: string - x-enum: - - LARGE - - MEDIUM - description: |- - Create a collection of name/value pairs. + x-enum: + - MANAGE + - READ + - WRITE + description: 'Permission to grant on the secret scope. For secrets, only one permission is + allowed. Permission must be one of: "READ", "WRITE", "MANAGE".' + required: + - scope + - key + - permission + serving_endpoint: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + - CAN_VIEW + description: 'Permission to grant on the serving endpoint. Supported permissions are: "CAN_MANAGE", + "CAN_QUERY", "CAN_VIEW".' + required: + - name + - permission + sql_warehouse: + type: object + properties: + id: + type: string + permission: + type: string + x-enum: + - CAN_MANAGE + - CAN_USE + - IS_OWNER + description: 'Permission to grant on the SQL warehouse. Supported permissions are: "CAN_MANAGE", + "CAN_USE", "IS_OWNER".' + required: + - id + - permission + uc_securable: + type: object + properties: + securable_full_name: + type: string + securable_type: + type: string + x-enum: + - CONNECTION + - FUNCTION + - TABLE + - VOLUME + description: |- + Create a collection of name/value pairs. - Example enumeration: + Example enumeration: - >>> class Color(Enum): - ... RED = 1 - ... BLUE = 2 - ... GREEN = 3 + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 - Access them by: + Access them by: - - attribute access: + - attribute access: - >>> Color.RED - + >>> Color.RED + - - value lookup: + - value lookup: - >>> Color(1) - + >>> Color(1) + - - name lookup: + - name lookup: - >>> Color['RED'] - + >>> Color['RED'] + - Enumerations can be iterated over, and know how many members they have: + Enumerations can be iterated over, and know how many members they have: - >>> len(Color) - 3 + >>> len(Color) + 3 - >>> list(Color) - [, , ] + >>> list(Color) + [, , ] - Methods can be added to enumerations, and members can have their own - attributes -- see the documentation for details. - compute_status: - type: object - properties: - active_instances: - type: integer - message: - type: string - description: Compute status message - state: - type: string - x-enum: - - ACTIVE - - DELETING - - ERROR - - STARTING - - STOPPED - - STOPPING - - UPDATING - description: State of the app compute. - create_time: - type: string - description: The creation time of the app. Formatted timestamp in ISO 6801. - creator: - type: string - description: The email of the user that created the app. - default_source_code_path: - type: string - description: The default workspace file system path of the source code from which app deployment are - created. This field tracks the workspace source code path of the last active deployment. - description: - type: string - description: The description of the app. - effective_budget_policy_id: - type: string + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + permission: + type: string + x-enum: + - EXECUTE + - MODIFY + - READ_VOLUME + - SELECT + - USE_CONNECTION + - WRITE_VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + securable_kind: + type: string + description: The securable kind from Unity Catalog. See https://docs.databricks.com/api/workspace/tables/get#securable_kind_manifest-securable_kind. + required: + - securable_full_name + - securable_type + - permission + required: + - name + description: Union of the app's own resources and the resources inherited from its space. Only populated + when the app belongs to a space that uses the group identity model. effective_usage_policy_id: type: string effective_user_api_scopes: @@ -2662,14 +4504,70 @@ paths: type: string description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured in + the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant the + app's service principal access to this repository. required: - url - provider description: Git repository configuration for app deployments. When specified, deployments can reference code from this repository by providing only the git reference (branch, tag, or commit). + git_source: + type: object + properties: + branch: + type: string + description: Git branch to checkout. + commit: + type: string + description: Git commit SHA to checkout. + git_repository: + type: object + properties: + url: + type: string + description: URL of the Git repository. + provider: + type: string + description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, + bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. + required: + - url + - provider + description: Git repository configuration. Populated from the app's git_repository configuration. + resolved_commit: + type: string + description: The resolved commit SHA that was actually used for the deployment. This is populated + by the system after resolving the reference (branch, tag, or commit). If commit is specified directly, + this will match commit. If a branch or tag is specified, this contains the commit SHA that the + branch or tag pointed to at deployment time. + source_code_path: + type: string + description: Relative path to the app source code within the Git repository. If not specified, the + root of the repository is used. + tag: + type: string + description: Git tag to checkout. + description: Complete git source specification including repository location and reference. id: type: string description: The unique identifier of the app. + last_deployment_id: + type: string + description: The ID of the last deployment created for this app. oauth2_app_client_id: type: string oauth2_app_integration_id: @@ -2732,6 +4630,14 @@ paths: description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. required: - url - provider @@ -2759,7 +4665,7 @@ paths: source_code_path: type: string description: The workspace file system path of the source code used to create the app deployment. - This is different from `deployment_artifacts.source_code_path`, which is the path used by the + This is different from ``deployment_artifacts.source_code_path``, which is the path used by the deployed app. The former refers to the original source code location of the app in the workspace during deployment creation, whereas the latter provides a system generated stable snapshotted source code path used by the deployment. @@ -2791,7 +4697,50 @@ paths: type: string app: type: object - properties: {} + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. database: type: object properties: @@ -3181,9 +5130,14 @@ paths: type: integer service_principal_name: type: string + source_code_path: + type: string space: type: string description: Name of the space this app belongs to. + space_id: + type: string + description: The ID of the app space this app belongs to. None if app does not belong to a space. telemetry_export_destinations: type: array items: @@ -3207,6 +5161,9 @@ paths: - traces_table description: Unity Catalog Destinations for OTEL telemetry export. description: A single telemetry export destination with its configuration and status. + thumbnail_url: + type: string + description: The URL of the thumbnail image for the app. update_time: type: string description: The update time of the app. Formatted timestamp in ISO 6801. @@ -3245,6 +5202,92 @@ paths: message: type: string x-stackql-sdk-source: AppsAPI + /api/2.0/apps/{name}/thumbnail: + delete: + operationId: apps_delete_app_thumbnail + summary: Deletes the thumbnail for an app. + tags: + - apps + description: |- + Deletes the thumbnail for an app. + + :param name: str + The name of the app. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The name of the app. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI + patch: + operationId: apps_update_app_thumbnail + summary: Updates the thumbnail for an app. + tags: + - apps + description: |- + Updates the thumbnail for an app. + + :param name: str + The name of the app. + :param app_thumbnail: :class:`AppThumbnail` (optional) + The app thumbnail to set. + + :returns: :class:`AppThumbnail` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The name of the app. + requestBody: + content: + application/json: + schema: + type: object + properties: + app_thumbnail: + type: object + properties: + thumbnail: + type: string + description: The thumbnail image bytes. + description: The app thumbnail to set. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AppThumbnail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: AppsAPI /api/2.0/app-spaces/{name}: delete: operationId: apps_delete_space @@ -3337,14 +5380,14 @@ paths: :param space: :class:`Space` :param update_mask: FieldMask The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`Operation` parameters: @@ -3361,11 +5404,11 @@ paths: schema: type: object description: The field mask must be a single string, with multiple fields separated by commas (no spaces). The field - path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). Specification - of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field - names must exactly match the resource field names. A field mask of `*` indicates full replacement. It’s recommended - to always explicitly list the fields being updated and avoid using `*` wildcards, as it can lead to unintended results - if the API changes in the future. + path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). + Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. + Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s + recommended to always explicitly list the fields being updated and avoid using ``*`` wildcards, as it can lead to + unintended results if the API changes in the future. requestBody: content: application/json: @@ -3395,6 +5438,11 @@ paths: items: type: string description: The effective api scopes granted to the user access token. + group_id: + type: string + description: The ID of the group attached to the app space. ASPs minted for apps in this space are added + as members of this group at create time, so the group's resource permissions flow through to every + app in the space. id: type: string description: The unique identifier of the app space. @@ -3407,7 +5455,50 @@ paths: type: string app: type: object - properties: {} + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. database: type: object properties: @@ -3817,6 +5908,31 @@ paths: - SPACE_UPDATING description: The state of the app space. description: The status of the app space. + telemetry_export_destinations: + type: array + items: + type: object + properties: + unity_catalog: + type: object + properties: + logs_table: + type: string + description: Unity Catalog table for OTEL logs. + metrics_table: + type: string + description: Unity Catalog table for OTEL metrics. + traces_table: + type: string + description: Unity Catalog table for OTEL traces (spans). + required: + - logs_table + - metrics_table + - traces_table + description: Unity Catalog Destinations for OTEL telemetry export. + description: A single telemetry export destination with its configuration and status. + description: Telemetry export destinations for all apps in this space. When configured, all apps in + the space export OTEL telemetry (logs, metrics, traces) to the specified Unity Catalog tables. update_time: type: string format: date-time @@ -3942,6 +6058,14 @@ paths: type: string description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured + in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant + the app's service principal access to this repository. required: - url - provider @@ -3969,7 +6093,7 @@ paths: source_code_path: type: string description: The workspace file system path of the source code used to create the app deployment. This - is different from `deployment_artifacts.source_code_path`, which is the path used by the deployed + is different from ``deployment_artifacts.source_code_path``, which is the path used by the deployed app. The former refers to the original source code location of the app in the workspace during deployment creation, whereas the latter provides a system generated stable snapshotted source code path used by the deployment. @@ -4491,6 +6615,16 @@ paths: name: type: string description: Name of the App Resource. + app_spec: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: 'Permission to grant on the app. Supported permission: "CAN_USE".' description: type: string description: Description of the App Resource. @@ -4591,6 +6725,54 @@ paths: attributes -- see the documentation for details. required: - permission + postgres_spec: + type: object + properties: + branch: + type: string + database: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. secret_spec: type: object properties: @@ -4752,6 +6934,7 @@ paths: x-enum: - EXECUTE - MANAGE + - MODIFY - READ_VOLUME - SELECT - USE_CONNECTION @@ -5027,6 +7210,16 @@ paths: name: type: string description: Name of the App Resource. + app_spec: + type: object + properties: + name: + type: string + permission: + type: string + x-enum: + - CAN_USE + description: 'Permission to grant on the app. Supported permission: "CAN_USE".' description: type: string description: Description of the App Resource. @@ -5127,6 +7320,54 @@ paths: attributes -- see the documentation for details. required: - permission + postgres_spec: + type: object + properties: + branch: + type: string + database: + type: string + permission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. secret_spec: type: object properties: @@ -5288,6 +7529,7 @@ paths: x-enum: - EXECUTE - MANAGE + - MODIFY - READ_VOLUME - SELECT - USE_CONNECTION @@ -5392,6 +7634,19 @@ components: $ref: '#/components/schemas/ApplicationStatus' budget_policy_id: type: string + compatibility_flags: + type: array + items: + type: string + description: 'Compatibility flags the customer is requesting for the app''s runtime environment. On update, the + submitted set must be a subset of ``effective_compatibility_flags`` (flags may only be removed, never added). + Supported flags: "PREINSTALLED_PACKAGES_LEGACY".' + compute_max_instances: + type: integer + description: Maximum number of app instances. Must be set together with ``compute_min_instances``. + compute_min_instances: + type: integer + description: Minimum number of app instances. Must be set together with ``compute_max_instances``. compute_size: $ref: '#/components/schemas/ComputeSize' compute_status: @@ -5402,6 +7657,8 @@ components: creator: type: string description: The email of the user that created the app. + default_git_source: + $ref: '#/components/schemas/GitSource' default_source_code_path: type: string description: The default workspace file system path of the source code from which app deployment are created. This @@ -5411,6 +7668,17 @@ components: description: The description of the app. effective_budget_policy_id: type: string + effective_compatibility_flags: + type: array + items: + type: string + description: The compatibility flags currently applied to the app. + effective_resources: + type: array + items: + $ref: '#/components/schemas/AppResource' + description: Union of the app's own resources and the resources inherited from its space. Only populated when the + app belongs to a space that uses the group identity model. effective_usage_policy_id: type: string effective_user_api_scopes: @@ -5422,9 +7690,14 @@ components: $ref: '#/components/schemas/GitRepository' description: Git repository configuration for app deployments. When specified, deployments can reference code from this repository by providing only the git reference (branch, tag, or commit). + git_source: + $ref: '#/components/schemas/GitSource' id: type: string description: The unique identifier of the app. + last_deployment_id: + type: string + description: The ID of the last deployment created for this app. oauth2_app_client_id: type: string oauth2_app_integration_id: @@ -5444,13 +7717,21 @@ components: type: integer service_principal_name: type: string + source_code_path: + type: string space: type: string description: Name of the space this app belongs to. + space_id: + type: string + description: The ID of the app space this app belongs to. None if app does not belong to a space. telemetry_export_destinations: type: array items: $ref: '#/components/schemas/TelemetryExportDestination' + thumbnail_url: + type: string + description: The URL of the thumbnail image for the app. update_time: type: string description: The update time of the app. Formatted timestamp in ISO 6801. @@ -5534,9 +7815,9 @@ components: source_code_path: type: string description: The workspace file system path of the source code used to create the app deployment. This is different - from `deployment_artifacts.source_code_path`, which is the path used by the deployed app. The former refers to - the original source code location of the app in the workspace during deployment creation, whereas the latter provides - a system generated stable snapshotted source code path used by the deployment. + from ``deployment_artifacts.source_code_path``, which is the path used by the deployed app. The former refers + to the original source code location of the app in the workspace during deployment creation, whereas the latter + provides a system generated stable snapshotted source code path used by the deployment. status: $ref: '#/components/schemas/AppDeploymentStatus' description: Status and status message of the deployment @@ -5576,6 +7857,14 @@ components: - version - name description: App manifest definition + AppManifestAppResourceAppSpec: + type: object + properties: + name: + type: string + permission: + $ref: '#/components/schemas/AppManifestAppResourceAppSpecAppPermission' + description: 'Permission to grant on the app. Supported permission: "CAN_USE".' AppManifestAppResourceExperimentSpec: type: object properties: @@ -5590,6 +7879,15 @@ components: $ref: '#/components/schemas/AppManifestAppResourceJobSpecJobPermission' required: - permission + AppManifestAppResourcePostgresSpec: + type: object + properties: + branch: + type: string + database: + type: string + permission: + $ref: '#/components/schemas/AppManifestAppResourcePostgresSpecPostgresPermission' AppManifestAppResourceSecretSpec: type: object properties: @@ -5610,6 +7908,8 @@ components: name: type: string description: Name of the App Resource. + app_spec: + $ref: '#/components/schemas/AppManifestAppResourceAppSpec' description: type: string description: Description of the App Resource. @@ -5617,6 +7917,8 @@ components: $ref: '#/components/schemas/AppManifestAppResourceExperimentSpec' job_spec: $ref: '#/components/schemas/AppManifestAppResourceJobSpec' + postgres_spec: + $ref: '#/components/schemas/AppManifestAppResourcePostgresSpec' secret_spec: $ref: '#/components/schemas/AppManifestAppResourceSecretSpec' serving_endpoint_spec: @@ -5708,7 +8010,11 @@ components: - name AppResourceApp: type: object - properties: {} + properties: + name: + type: string + permission: + $ref: '#/components/schemas/AppResourceAppAppPermission' AppResourceDatabase: type: object properties: @@ -5821,11 +8127,30 @@ components: - securable_full_name - securable_type - permission + AppThumbnail: + type: object + properties: + thumbnail: + type: string + description: The thumbnail image bytes. + description: The thumbnail for an app. AppUpdate: type: object properties: budget_policy_id: type: string + compatibility_flags: + type: array + items: + type: string + description: Echoes the compatibility flags submitted on the most recent update (the input set from ``App.compatibility_flags``). + To see the flags currently in effect on the app, read ``App.effective_compatibility_flags`` via GetApp. + compute_max_instances: + type: integer + description: Maximum number of app instances. Must be set together with ``compute_min_instances``. + compute_min_instances: + type: integer + description: Minimum number of app instances. Must be set together with ``compute_max_instances``. compute_size: $ref: '#/components/schemas/ComputeSize' description: @@ -5838,6 +8163,10 @@ components: $ref: '#/components/schemas/AppResource' status: $ref: '#/components/schemas/AppUpdateUpdateStatus' + telemetry_export_destinations: + type: array + items: + $ref: '#/components/schemas/TelemetryExportDestination' usage_policy_id: type: string user_api_scopes: @@ -5856,6 +8185,9 @@ components: properties: message: type: string + running_instances: + type: integer + description: The number of running instances of this application. state: $ref: '#/components/schemas/ApplicationState' description: State of the application. @@ -5941,6 +8273,13 @@ components: type: string description: 'Git provider. Case insensitive. Supported values: gitHub, gitHubEnterprise, bitbucketCloud, bitbucketServer, azureDevOpsServices, gitLab, gitLabEnterpriseEdition, awsCodeCommit.' + auto_deploy: + type: boolean + description: When true, automatically deploys the app on push events to the branch configured in the app's deployment_source.git_source. + caller_credential_id: + type: integer + description: ID of a personal access token Git credential owned by the caller, used to grant the app's service principal + access to this repository. required: - url - provider @@ -6014,8 +8353,8 @@ components: properties: done: type: boolean - description: If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, - and either `error` or `response` is available. + description: If the value is ``false``, it means the operation is still in progress. If ``true``, the operation + is completed, and either ``error`` or ``response`` is available. error: $ref: '#/components/schemas/DatabricksServiceExceptionWithDetailsProto' description: The error result of the operation in case of failure or cancellation. @@ -6026,7 +8365,7 @@ components: name: type: string description: The server-assigned name, which is only unique within the same service that originally returns it. - If you use the default HTTP mapping, the `name` should be a resource name ending with `operations/{unique_id}`. + If you use the default HTTP mapping, the ``name`` should be a resource name ending with ``operations/{unique_id}``. response: type: object description: The normal, successful response of the operation. @@ -6054,6 +8393,10 @@ components: items: type: string description: The effective api scopes granted to the user access token. + group_id: + type: string + description: The ID of the group attached to the app space. ASPs minted for apps in this space are added as members + of this group at create time, so the group's resource permissions flow through to every app in the space. id: type: string description: The unique identifier of the app space. @@ -6075,6 +8418,12 @@ components: status: $ref: '#/components/schemas/SpaceStatus' description: The status of the app space. + telemetry_export_destinations: + type: array + items: + $ref: '#/components/schemas/TelemetryExportDestination' + description: Telemetry export destinations for all apps in this space. When configured, all apps in the space export + OTEL telemetry (logs, metrics, traces) to the specified Unity Catalog tables. update_time: type: string format: date-time @@ -6105,6 +8454,8 @@ components: properties: description: type: string + group_id: + type: string resources: type: array items: @@ -6233,6 +8584,47 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + AppManifestAppResourceAppSpecAppPermission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. AppManifestAppResourceExperimentSpecExperimentPermission: @@ -6320,6 +8712,47 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + AppManifestAppResourcePostgresSpecPostgresPermission: + type: string + x-enum: + - CAN_CONNECT_AND_CREATE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. AppManifestAppResourceSecretSpecSecretPermission: @@ -6420,6 +8853,7 @@ components: x-enum: - EXECUTE - MANAGE + - MODIFY - READ_VOLUME - SELECT - USE_CONNECTION @@ -6511,6 +8945,47 @@ components: - CAN_MANAGE - CAN_USE description: Permission level + AppResourceAppAppPermission: + type: string + x-enum: + - CAN_USE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. AppResourceDatabaseDatabasePermission: type: string x-enum: @@ -6999,7 +9474,9 @@ components: type: string x-enum: - LARGE + - LIQUID - MEDIUM + - XLARGE description: |- Create a collection of name/value pairs. @@ -7281,21 +9758,6 @@ components: mediaType: application/json openAPIDocKey: '200' objectKey: $.apps - apps_create_space: - config: - requestBodyTranslate: - algorithm: naive - operation: - $ref: '#/paths/~1api~12.0~1app-spaces/post' - response: - mediaType: application/json - openAPIDocKey: '200' - apps_list_spaces: - operation: - $ref: '#/paths/~1api~12.0~1app-spaces/get' - response: - mediaType: application/json - openAPIDocKey: '200' delete: operation: $ref: '#/paths/~1api~12.0~1apps~1{name}/delete' @@ -7317,61 +9779,81 @@ components: response: mediaType: application/json openAPIDocKey: '200' - apps_delete_space: + start: operation: - $ref: '#/paths/~1api~12.0~1app-spaces~1{name}/delete' + $ref: '#/paths/~1api~12.0~1apps~1{name}~1start/post' response: mediaType: application/json openAPIDocKey: '200' - apps_get_space: + stop: operation: - $ref: '#/paths/~1api~12.0~1app-spaces~1{name}/get' + $ref: '#/paths/~1api~12.0~1apps~1{name}~1stop/post' response: mediaType: application/json openAPIDocKey: '200' - apps_update_space: + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/apps/methods/get' + - $ref: '#/components/x-stackQL-resources/apps/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/apps/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/apps/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/apps/methods/delete' + replace: [] + app_spaces: + id: databricks_workspace.apps.app_spaces + name: app_spaces + title: App Spaces + methods: + create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1app-spaces~1{name}/patch' + $ref: '#/paths/~1api~12.0~1app-spaces/post' response: mediaType: application/json openAPIDocKey: '200' - apps_get_space_operation: + list: operation: - $ref: '#/paths/~1api~12.0~1app-spaces~1{name}~1operation/get' + $ref: '#/paths/~1api~12.0~1app-spaces/get' response: mediaType: application/json openAPIDocKey: '200' - start: + objectKey: $.spaces + delete: operation: - $ref: '#/paths/~1api~12.0~1apps~1{name}~1start/post' + $ref: '#/paths/~1api~12.0~1app-spaces~1{name}/delete' response: mediaType: application/json openAPIDocKey: '200' - stop: + get: operation: - $ref: '#/paths/~1api~12.0~1apps~1{name}~1stop/post' + $ref: '#/paths/~1api~12.0~1app-spaces~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1app-spaces~1{name}/patch' response: mediaType: application/json openAPIDocKey: '200' sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/apps/methods/get' - - $ref: '#/components/x-stackQL-resources/apps/methods/apps_get_space' - - $ref: '#/components/x-stackQL-resources/apps/methods/apps_get_space_operation' - - $ref: '#/components/x-stackQL-resources/apps/methods/list' - - $ref: '#/components/x-stackQL-resources/apps/methods/apps_list_spaces' + - $ref: '#/components/x-stackQL-resources/app_spaces/methods/get' + - $ref: '#/components/x-stackQL-resources/app_spaces/methods/list' insert: - - $ref: '#/components/x-stackQL-resources/apps/methods/create' - - $ref: '#/components/x-stackQL-resources/apps/methods/apps_create_space' + - $ref: '#/components/x-stackQL-resources/app_spaces/methods/create' update: - - $ref: '#/components/x-stackQL-resources/apps/methods/update' - - $ref: '#/components/x-stackQL-resources/apps/methods/apps_update_space' + - $ref: '#/components/x-stackQL-resources/app_spaces/methods/update' delete: - - $ref: '#/components/x-stackQL-resources/apps/methods/delete' - - $ref: '#/components/x-stackQL-resources/apps/methods/apps_delete_space' + - $ref: '#/components/x-stackQL-resources/app_spaces/methods/delete' replace: [] app_updates: id: databricks_workspace.apps.app_updates @@ -7401,6 +9883,34 @@ components: update: [] delete: [] replace: [] + app_thumbnails: + id: databricks_workspace.apps.app_thumbnails + name: app_thumbnails + title: App Thumbnails + methods: + delete: + operation: + $ref: '#/paths/~1api~12.0~1apps~1{name}~1thumbnail/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1apps~1{name}~1thumbnail/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: [] + update: + - $ref: '#/components/x-stackQL-resources/app_thumbnails/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/app_thumbnails/methods/delete' + replace: [] app_deployments: id: databricks_workspace.apps.app_deployments name: app_deployments @@ -7493,6 +10003,24 @@ components: delete: [] replace: - $ref: '#/components/x-stackQL-resources/app_permissions/methods/set' + app_space_operations: + id: databricks_workspace.apps.app_space_operations + name: app_space_operations + title: App Space Operations + methods: + get: + operation: + $ref: '#/paths/~1api~12.0~1app-spaces~1{name}~1operation/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/app_space_operations/methods/get' + insert: [] + update: [] + delete: [] + replace: [] apps_settings: id: databricks_workspace.apps.apps_settings name: apps_settings diff --git a/providers/src/databricks_workspace/v00.00.00000/services/bundledeployments.yaml b/providers/src/databricks_workspace/v00.00.00000/services/bundledeployments.yaml new file mode 100644 index 00000000..9cb6f498 --- /dev/null +++ b/providers/src/databricks_workspace/v00.00.00000/services/bundledeployments.yaml @@ -0,0 +1,2137 @@ +openapi: 3.0.0 +info: + title: Databricks Bundledeployments API (workspace) + description: >- + OpenAPI specification for the Databricks bundledeployments service + (workspace-level APIs), generated from the Databricks Python SDK. + version: 0.1.0 + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' + x-stackql-sdk-namespace: databricks.sdk.service.bundledeployments +servers: + - url: https://{deployment_name}.cloud.databricks.com + variables: + deployment_name: + description: The Databricks Workspace Deployment Name + default: dbc-abcd0123-a1bc +paths: + /api/2.0/bundle/{name}/complete: + post: + operationId: bundle_deployments_complete_version + summary: Marks a version as complete and releases the deployment lock. + tags: + - bundledeployments + - bundle_deployments + description: |- + Marks a version as complete and releases the deployment lock. + + The server atomically: + + 1. Sets the version status to the provided terminal status. + 2. Sets ``complete_time`` to the current server timestamp. + 3. Releases the lock on the parent deployment. + 4. Updates the parent deployment's ``status`` and ``last_version_id``. + + :param name: str + The name of the version to complete. Format: deployments/{deployment_id}/versions/{version_id} + :param completion_reason: :class:`VersionComplete` + The reason for completing the version. Must be a terminal reason: VERSION_COMPLETE_SUCCESS, + VERSION_COMPLETE_FAILURE, or VERSION_COMPLETE_FORCE_ABORT. + :param force: bool (optional) + If true, force-completes the version even if the caller is not the original creator. The + completion_reason must be VERSION_COMPLETE_FORCE_ABORT when force is true. + + :returns: :class:`Version` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The name of the version to complete. Format: + deployments/{deployment_id}/versions/{version_id} + requestBody: + content: + application/json: + schema: + type: object + properties: + completion_reason: + type: string + x-enum: + - VERSION_COMPLETE_FAILURE + - VERSION_COMPLETE_FORCE_ABORT + - VERSION_COMPLETE_LEASE_EXPIRED + - VERSION_COMPLETE_SUCCESS + description: >- + The reason for completing the version. Must be a terminal + reason: VERSION_COMPLETE_SUCCESS, VERSION_COMPLETE_FAILURE, + or VERSION_COMPLETE_FORCE_ABORT. + force: + type: boolean + description: >- + If true, force-completes the version even if the caller is + not the original creator. The completion_reason must be + VERSION_COMPLETE_FORCE_ABORT when force is true. + required: + - completion_reason + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Version' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + /api/2.0/bundle/deployments: + post: + operationId: bundle_deployments_create_deployment + summary: Creates a new deployment in the workspace. + tags: + - bundledeployments + - bundle_deployments + description: |- + Creates a new deployment in the workspace. + + :param deployment: :class:`Deployment` + The deployment to create. The caller must set ``initial_parent_path``. Other fields are ignored on + input and populated by the service. + + :returns: :class:`Deployment` + requestBody: + content: + application/json: + schema: + type: object + properties: + deployment: + type: object + properties: + create_time: + type: string + format: date-time + description: When the deployment was created. + created_by: + type: string + description: >- + The user who created the deployment (email or principal + name). + deployment_mode: + type: string + x-enum: + - DEPLOYMENT_MODE_DEVELOPMENT + - DEPLOYMENT_MODE_PRODUCTION + description: >- + Bundle target deployment mode (development or + production), derived from the most recent version's + mode. + destroy_time: + type: string + format: date-time + description: >- + When deletion was recorded. Unset if deletion has not + been recorded. This response metadata does not determine + the deployment's lifecycle status. + destroyed_by: + type: string + description: >- + The user who destroyed the deployment (email or + principal name). Unset if the deployment has not been + destroyed. + display_name: + type: string + description: >- + Human-readable name for the deployment, up to 256 + characters. Output only: clients update it by setting + ``display_name`` when creating a version. + git_info: + type: object + properties: + branch: + type: string + description: Branch the source was deployed from. + commit: + type: string + description: Commit SHA of the deployed source. + origin_url: + type: string + description: URL of the git remote the source was deployed from. + description: >- + Git provenance of the deployment's source, derived from + the latest version. + initial_parent_path: + type: string + description: >- + The workspace path of the existing folder where the + deployment is initially created. Must be absolute and + canonical, with single separators, no ``.`` or ``..`` + segments, and no trailing slash unless the path is + ``/``. It may contain at most 24 path segments, + excluding an optional leading ``/Workspace`` segment. + The complete path may contain up to 1,024 characters, + and each segment may contain up to 511 characters. This + field is input only and is not returned in create, get, + or list responses. + last_successful_version_id: + type: string + description: >- + The version_id of the most recent version that completed + successfully. Unset until a version has completed + successfully. Unlike last_version_id, it is not advanced + when a version fails, so it always points at the last + known-good deployment state (or is unset if there has + never been one). + last_version_id: + type: string + description: The version_id of the most recent deployment version. + name: + type: string + description: >- + Resource name of the deployment. Format: + deployments/{deployment_id} + status: + type: string + x-enum: + - DEPLOYMENT_STATUS_ACTIVE + - DEPLOYMENT_STATUS_DELETED + - DEPLOYMENT_STATUS_FAILED + - DEPLOYMENT_STATUS_IN_PROGRESS + description: Current status of the deployment. + target_name: + type: string + description: >- + The bundle target name associated with this deployment. + Output only: it is denormalized from the latest version, + not set directly on the deployment. + update_time: + type: string + format: date-time + description: When the deployment was last updated. + updated_by: + type: string + description: >- + The user who most recently updated the deployment (email + or principal name). + workspace_info: + type: object + properties: + bundle_root_path: + type: string + description: >- + Path of the bundle root (the directory containing + databricks.yml) relative to git_folder_path. Empty + when the deployment is not from a Databricks Git + folder. + file_path: + type: string + description: >- + Absolute workspace path where the deployed bundle + files live. Mirrors the workspace.file_path field in + DABs bundle config. + git_folder_path: + type: string + description: >- + When deployed from a Databricks Git folder, the + absolute workspace path of that folder; empty for + local deploys. + root_path: + type: string + description: >- + Absolute workspace path of the deployment root — the + base path the deployed files live under. Mirrors + workspace.root_path in the DABs bundle config; + file_path is its files subdirectory. + source_linked: + type: boolean + description: >- + Whether files are served directly from the source + sync root instead of being copied into file_path. + description: >- + Workspace location of the deployment, derived from the + latest version. + description: >- + The deployment to create. The caller must set + ``initial_parent_path``. Other fields are ignored on input + and populated by the service. + required: + - deployment + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Deployment' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + get: + operationId: bundle_deployments_list_deployments + summary: Lists deployments in the workspace. + tags: + - bundledeployments + - bundle_deployments + description: |- + Lists deployments in the workspace. + + :param filter: str (optional) + A filter expression restricting which deployments are returned, in the style of AIP-160 + (https://google.aip.dev/160). The expression is a conjunction of one or more ``field operator + value`` terms joined by ``AND`` (case-insensitive); a deployment is returned only when it matches + every term. Whitespace around terms is ignored, and a value containing spaces must be wrapped in + double quotes. An unset or empty filter returns all deployments. Filtering applies only to live + deployments; deleted deployments are never returned regardless of the filter. + + Supported terms: + + - ``status = ``: exact match on the deployment status. The value is a ``DeploymentStatus`` + enum value, with or without the ``DEPLOYMENT_STATUS_`` prefix and case-insensitive (e.g. ``status + = ACTIVE``). + - ``deployment_mode = ``: exact match on the deployment mode. The value is a + ``DeploymentMode`` enum value, with or without the ``DEPLOYMENT_MODE_`` prefix and + case-insensitive (e.g. ``deployment_mode = DEVELOPMENT``). + - ``display_name = ""``: exact match on the display name. + - ``display_name : ""``: case-insensitive substring match on the display name. + + For example: ``status = ACTIVE AND display_name : "etl"``. + :param page_size: int (optional) + The maximum number of deployments to return. The service may return fewer than this value. If + unspecified, at most 20 deployments will be returned. The maximum value is 100; values above 100 + will be coerced to 100. + :param page_token: str (optional) + A page token, received from a previous ``ListDeployments`` call. Provide this to retrieve the + subsequent page. + + :returns: Iterator over :class:`Deployment` + parameters: + - name: filter + in: query + required: false + schema: + type: string + description: >- + A filter expression restricting which deployments are returned, in + the style of AIP-160 (https://google.aip.dev/160). The expression is + a conjunction of one or more ``field operator value`` terms joined + by ``AND`` (case-insensitive); a deployment is returned only when it + matches every term. Whitespace around terms is ignored, and a value + containing spaces must be wrapped in double quotes. An unset or + empty filter returns all deployments. Filtering applies only to live + deployments; deleted deployments are never returned regardless of + the filter. Supported terms: - ``status = ``: exact match on + the deployment status. The value is a ``DeploymentStatus`` enum + value, with or without the ``DEPLOYMENT_STATUS_`` prefix and + case-insensitive (e.g. ``status = ACTIVE``). - ``deployment_mode = + ``: exact match on the deployment mode. The value is a + ``DeploymentMode`` enum value, with or without the + ``DEPLOYMENT_MODE_`` prefix and case-insensitive (e.g. + ``deployment_mode = DEVELOPMENT``). - ``display_name = ""``: + exact match on the display name. - ``display_name : ""``: + case-insensitive substring match on the display name. For example: + ``status = ACTIVE AND display_name : "etl"``. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of deployments to return. The service may return + fewer than this value. If unspecified, at most 20 deployments will + be returned. The maximum value is 100; values above 100 will be + coerced to 100. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ``ListDeployments`` call. + Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListDeploymentsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + /api/2.0/bundle/{parent}/operations: + post: + operationId: bundle_deployments_create_operation + summary: Creates a resource operation under a version. + tags: + - bundledeployments + - bundle_deployments + description: >- + Creates a resource operation under a version. + + + The caller must provide a ``resource_key`` which becomes the final + component of the operation's name. + + If an operation with the same key already exists under the version, the + server returns + + ``ALREADY_EXISTS``. + + + On success the server also updates the corresponding deployment-level + Resource (creating it if this is + + the first operation for that resource_key, or removing it if action_type + is DELETE). + + + :param parent: str + The parent version where this operation will be recorded. Format: + deployments/{deployment_id}/versions/{version_id} + :param operation: :class:`Operation` + The resource operation to create. + :param resource_key: str + The key identifying the resource this operation applies to. Becomes the final component of the + operation's name. + + :returns: :class:`Operation` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The parent version where this operation will be recorded. Format: + deployments/{deployment_id}/versions/{version_id} + - name: resource_key + in: query + required: true + schema: + type: string + description: >- + The key identifying the resource this operation applies to. Becomes + the final component of the operation's name. + requestBody: + content: + application/json: + schema: + type: object + properties: + operation: + type: object + properties: + action_type: + type: string + x-enum: + - OPERATION_ACTION_TYPE_BIND + - OPERATION_ACTION_TYPE_BIND_AND_UPDATE + - OPERATION_ACTION_TYPE_CREATE + - OPERATION_ACTION_TYPE_DELETE + - OPERATION_ACTION_TYPE_INITIAL_REGISTER + - OPERATION_ACTION_TYPE_RECREATE + - OPERATION_ACTION_TYPE_RESIZE + - OPERATION_ACTION_TYPE_UPDATE + - OPERATION_ACTION_TYPE_UPDATE_WITH_ID + description: The type of operation performed on this resource. + status: + type: string + x-enum: + - OPERATION_STATUS_FAILED + - OPERATION_STATUS_SUCCEEDED + description: >- + Whether the operation succeeded or failed. Mutable: may + be updated after creation via UpdateOperation, e.g. when + an operation recorded as failed is retried and + eventually succeeds. A succeeded operation cannot carry + an ``error_message``. + create_time: + type: string + format: date-time + description: When the operation was recorded. + dashboard_metadata: + type: object + properties: + definition_path: + type: string + description: >- + Path of the file that declares this dashboard, + relative to the bundle's workspace.file_path + (Version.workspace_info.file_path) — join the two to + get the file's absolute workspace path. For now this + lives only on the dashboard metadata, and is a + single string because it was a single string + (``relative_path``) in the legacy bundle + metadata.json. We may generalize it in the future: + lifting it to a top-level field on + Resource/Operation (every resource type has a + definition location) and converting it to a repeated + field, since a resource can be declared across + multiple files/locations. + source_path: + type: string + description: >- + Path of the dashboard's source artifact (its + ``.lvdash.json``), relative to the deployment root. + description: >- + Dashboard-specific metadata; set only for dashboard + resources. + error_message: + type: string + description: >- + Error message if the operation failed. Set when status + is OPERATION_STATUS_FAILED. Captures the error + encountered while applying the resource to the + workspace. Mutable: may be updated after creation via + UpdateOperation; setting it to an empty string clears + it. After an update is applied, an operation whose + status is OPERATION_STATUS_SUCCEEDED cannot carry an + error_message. + name: + type: string + description: >- + Resource name of the operation. Format: + deployments/{deployment_id}/versions/{version_id}/operations/{resource_key} + resource_id: + type: string + description: >- + ID of the actual resource in the workspace (e.g. the job + ID, pipeline ID). Optional at creation: CREATE and + RECREATE operations produce a new resource whose ID is + not yet known when the operation is recorded. Mutable: + may be filled in (or corrected) later via + UpdateOperation once the ID is known. + resource_key: + type: string + description: >- + Resource identifier within the bundle (e.g. "jobs.foo", + "pipelines.bar", "jobs.foo.permissions", + "files."). Can be an arbitrary UTF-8 encoded + string key. This key links the operation to the + corresponding deployment-level Resource. + resource_type: + type: string + x-enum: + - DEPLOYMENT_RESOURCE_TYPE_ALERT + - DEPLOYMENT_RESOURCE_TYPE_APP + - DEPLOYMENT_RESOURCE_TYPE_CATALOG + - DEPLOYMENT_RESOURCE_TYPE_CLUSTER + - DEPLOYMENT_RESOURCE_TYPE_DASHBOARD + - DEPLOYMENT_RESOURCE_TYPE_DATABASE_CATALOG + - DEPLOYMENT_RESOURCE_TYPE_DATABASE_INSTANCE + - DEPLOYMENT_RESOURCE_TYPE_EXPERIMENT + - DEPLOYMENT_RESOURCE_TYPE_EXTERNAL_LOCATION + - DEPLOYMENT_RESOURCE_TYPE_JOB + - DEPLOYMENT_RESOURCE_TYPE_MODEL + - DEPLOYMENT_RESOURCE_TYPE_MODEL_SERVING_ENDPOINT + - DEPLOYMENT_RESOURCE_TYPE_PIPELINE + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_BRANCH + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_ENDPOINT + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_PROJECT + - DEPLOYMENT_RESOURCE_TYPE_QUALITY_MONITOR + - DEPLOYMENT_RESOURCE_TYPE_REGISTERED_MODEL + - DEPLOYMENT_RESOURCE_TYPE_SCHEMA + - DEPLOYMENT_RESOURCE_TYPE_SECRET_SCOPE + - DEPLOYMENT_RESOURCE_TYPE_SQL_WAREHOUSE + - DEPLOYMENT_RESOURCE_TYPE_SYNCED_DATABASE_TABLE + - DEPLOYMENT_RESOURCE_TYPE_VOLUME + description: >- + The type of the deployment resource this operation + applies to. Derived from the ``resource_key`` prefix + (e.g. "jobs" → JOB); the caller does not set this field. + sequence_id: + type: integer + description: >- + Monotonically increasing revision used for optimistic + concurrency control (the AIP-154 concurrency token for + this resource, realized as a sequence number rather than + an opaque etag). The server assigns 1 on creation and + increments it on every successful UpdateOperation. It is + OPTIONAL rather than OUTPUT_ONLY because it is + dual-purpose: CreateOperation/GetOperation return the + current value, and UpdateOperation reads the + caller-supplied value as a precondition. The caller must + echo the value it last observed; if it no longer matches + the server's value, the update is rejected with ABORTED + so the caller can re-read and retry. Ignored on + CreateOperation. + state: + type: object + description: >- + Serialized local config state after the operation. + Should be unset for delete operations. Mutable: may be + updated after creation via UpdateOperation. When + updating, the caller must echo the last-observed + ``sequence_id`` as a concurrency precondition. + update_time: + type: string + format: date-time + description: >- + When the operation was last updated. Set to + ``create_time`` when the operation is created and to the + server timestamp on each successful UpdateOperation. + required: + - action_type + - status + description: The resource operation to create. + required: + - operation + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Operation' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + get: + operationId: bundle_deployments_list_operations + summary: Lists resource operations under a version. + tags: + - bundledeployments + - bundle_deployments + description: |- + Lists resource operations under a version. + + :param parent: str + The parent version. Format: deployments/{deployment_id}/versions/{version_id} + :param page_size: int (optional) + The maximum number of operations to return. The service may return fewer than this value. If + unspecified, at most 50 operations will be returned. The maximum value is 1000; values above 1000 + will be coerced to 1000. + :param page_token: str (optional) + A page token, received from a previous ``ListOperations`` call. Provide this to retrieve the + subsequent page. + + :returns: Iterator over :class:`Operation` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The parent version. Format: + deployments/{deployment_id}/versions/{version_id} + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of operations to return. The service may return + fewer than this value. If unspecified, at most 50 operations will be + returned. The maximum value is 1000; values above 1000 will be + coerced to 1000. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ``ListOperations`` call. + Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListOperationsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + /api/2.0/bundle/{parent}/versions: + post: + operationId: bundle_deployments_create_version + summary: Creates a new version under a deployment. + tags: + - bundledeployments + - bundle_deployments + description: >- + Creates a new version under a deployment. + + + Creating a version acquires an exclusive lock on the deployment, + preventing concurrent deploys. The + + caller provides a ``version_id``, a numeric string that must be + numerically greater than the + + deployment's most recent version, and sets the version's + ``previous_version_id`` to the deployment's + + most recent version (leaving it unset for the first version), which the + server validates to detect + + concurrent deploys. + + + :param parent: str + The parent deployment where this version will be created. Format: deployments/{deployment_id} + :param version: :class:`Version` + The version to create. + :param version_id: str + The ID to use for the version, which becomes the final component of the version's resource name. A + numeric string (base-10, fits in a signed 64-bit integer) chosen by the caller; must be greater than + or equal to 1. Must be numerically greater than the deployment's most recent version (see + ``version.previous_version_id``); it does not need to start at 1 or increase by exactly 1. If the + value is not numerically greater, the server returns ``INVALID_PARAMETER_VALUE``. + + :returns: :class:`Version` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + The parent deployment where this version will be created. Format: + deployments/{deployment_id} + - name: version_id + in: query + required: true + schema: + type: string + description: >- + The ID to use for the version, which becomes the final component of + the version's resource name. A numeric string (base-10, fits in a + signed 64-bit integer) chosen by the caller; must be greater than or + equal to 1. Must be numerically greater than the deployment's most + recent version (see ``version.previous_version_id``); it does not + need to start at 1 or increase by exactly 1. If the value is not + numerically greater, the server returns ``INVALID_PARAMETER_VALUE``. + requestBody: + content: + application/json: + schema: + type: object + properties: + version: + type: object + properties: + cli_version: + type: string + description: CLI version used to initiate the version. + version_type: + type: string + x-enum: + - VERSION_TYPE_DEPLOY + - VERSION_TYPE_DESTROY + description: Type of version (deploy or destroy). + complete_time: + type: string + format: date-time + description: >- + When the version completed. Unset while the version is + in progress. + completed_by: + type: string + description: >- + The user who completed the version (email or principal + name). May differ from ``created_by`` when another user + force-completes the version. + completion_reason: + type: string + x-enum: + - VERSION_COMPLETE_FAILURE + - VERSION_COMPLETE_FORCE_ABORT + - VERSION_COMPLETE_LEASE_EXPIRED + - VERSION_COMPLETE_SUCCESS + description: >- + Why the version was completed. Unset while in progress. + Set when status transitions to COMPLETED. + create_time: + type: string + format: date-time + description: When the version was created. + created_by: + type: string + description: >- + The user who created the version (email or principal + name). + deployment_mode: + type: string + x-enum: + - DEPLOYMENT_MODE_DEVELOPMENT + - DEPLOYMENT_MODE_PRODUCTION + description: >- + Bundle target deployment mode (development or + production), captured at the time of this version. + display_name: + type: string + description: >- + Display name for the deployment, captured at the time of + this version. Up to 256 characters. When present, + creating the version updates the deployment display + name. An empty value clears it; an absent value leaves + the current deployment display name unchanged. + git_info: + type: object + properties: + branch: + type: string + description: Branch the source was deployed from. + commit: + type: string + description: Commit SHA of the deployed source. + origin_url: + type: string + description: URL of the git remote the source was deployed from. + description: >- + Git provenance of the source, captured at the time of + this version. + name: + type: string + description: >- + Resource name of the version. Format: + deployments/{deployment_id}/versions/{version_id} + previous_version_id: + type: string + description: >- + The version_id this version was created on top of — the + deployment's most recent version at creation time. Leave + unset when creating the first version (the deployment + has no prior versions). Set by the client on creation + and immutable thereafter. Acts as an + optimistic-concurrency precondition: the server requires + it to equal the deployment's current most-recent version + (and to be unset when the deployment has no versions) + and returns ``INVALID_PARAMETER_VALUE`` on mismatch, so + a deploy racing against a concurrent deploy is rejected + rather than silently overwriting it. + status: + type: string + x-enum: + - VERSION_STATUS_COMPLETED + - VERSION_STATUS_IN_PROGRESS + description: 'Status of the version: IN_PROGRESS or COMPLETED.' + target_name: + type: string + description: >- + Target name of the deployment, captured at the time of + this version. + version_id: + type: string + description: >- + Version identifier within the parent deployment, + assigned by the client on creation. A numeric string + (base-10, fits in a signed 64-bit integer) that is + greater than or equal to 1. Version IDs are strictly + increasing within a deployment but are not required to + start at 1 or to be contiguous. + workspace_info: + type: object + properties: + bundle_root_path: + type: string + description: >- + Path of the bundle root (the directory containing + databricks.yml) relative to git_folder_path. Empty + when the deployment is not from a Databricks Git + folder. + file_path: + type: string + description: >- + Absolute workspace path where the deployed bundle + files live. Mirrors the workspace.file_path field in + DABs bundle config. + git_folder_path: + type: string + description: >- + When deployed from a Databricks Git folder, the + absolute workspace path of that folder; empty for + local deploys. + root_path: + type: string + description: >- + Absolute workspace path of the deployment root — the + base path the deployed files live under. Mirrors + workspace.root_path in the DABs bundle config; + file_path is its files subdirectory. + source_linked: + type: boolean + description: >- + Whether files are served directly from the source + sync root instead of being copied into file_path. + description: >- + Workspace location of the deployment, captured at the + time of this version. + required: + - cli_version + - version_type + description: The version to create. + required: + - version + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Version' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + get: + operationId: bundle_deployments_list_versions + summary: >- + Lists versions under a deployment, ordered numerically by version_id + descending (most recent first). + tags: + - bundledeployments + - bundle_deployments + description: >- + Lists versions under a deployment, ordered numerically by version_id + descending (most recent first). + + + :param parent: str + The parent deployment. Format: deployments/{deployment_id} + :param page_size: int (optional) + The maximum number of versions to return. The service may return fewer than this value. If + unspecified, at most 20 versions will be returned. The maximum value is 100; values above 100 will + be coerced to 100. + :param page_token: str (optional) + A page token, received from a previous ``ListVersions`` call. Provide this to retrieve the + subsequent page. + + :returns: Iterator over :class:`Version` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent deployment. Format: deployments/{deployment_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of versions to return. The service may return + fewer than this value. If unspecified, at most 20 versions will be + returned. The maximum value is 100; values above 100 will be coerced + to 100. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ``ListVersions`` call. + Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListVersionsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + /api/2.0/bundle/{name}: + delete: + operationId: bundle_deployments_delete_deployment + summary: Deletes a deployment. + tags: + - bundledeployments + - bundle_deployments + description: |- + Deletes a deployment. + + :param name: str + Resource name of the deployment to delete. Format: deployments/{deployment_id} + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Resource name of the deployment to delete. Format: + deployments/{deployment_id} + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + get: + operationId: bundle_deployments_get_version + summary: Retrieves a version by its resource name. + tags: + - bundledeployments + - bundle_deployments + description: |- + Retrieves a version by its resource name. + + :param name: str + The name of the version to retrieve. Format: deployments/{deployment_id}/versions/{version_id} + + :returns: :class:`Version` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The name of the version to retrieve. Format: + deployments/{deployment_id}/versions/{version_id} + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Version' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + patch: + operationId: bundle_deployments_update_operation + summary: Updates a resource operation's mutable fields. + tags: + - bundledeployments + - bundle_deployments + description: >- + Updates a resource operation's mutable fields. + + + ``state``, ``error_message``, ``resource_id``, and ``status`` may be + updated, independently; + + ``update_mask`` must contain only those paths. All other fields are + immutable. The update is guarded + + by an optimistic-concurrency check: the caller sets + ``operation.sequence_id`` to the value it last + + observed, and the server rejects the update with ``ABORTED`` if the + operation has been modified since. + + On success the server increments ``sequence_id``; updates to ``state`` + and ``resource_id`` are + + mirrored onto the corresponding deployment-level Resource projection. + The parent version must be in + + progress, delete operations cannot be updated, and after the update is + applied a succeeded operation + + cannot carry an ``error_message``. + + + :param name: str + Resource name of the operation. Format: + deployments/{deployment_id}/versions/{version_id}/operations/{resource_key} + :param operation: :class:`Operation` + The operation to update. Its ``name`` selects the operation; the fields named in ``update_mask`` + carry the new values; and ``sequence_id`` carries the optimistic-concurrency precondition (see the + field docs on Operation). All other fields are ignored. + :param update_mask: FieldMask + The set of fields to update. Required; supported paths are ``state``, ``error_message``, + ``resource_id``, and ``status``. An empty mask or any other path is rejected with + INVALID_PARAMETER_VALUE. + + :returns: :class:`Operation` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Resource name of the operation. Format: + deployments/{deployment_id}/versions/{version_id}/operations/{resource_key} + - name: update_mask + in: query + required: true + schema: + type: object + description: >- + The set of fields to update. Required; supported paths are + ``state``, ``error_message``, ``resource_id``, and ``status``. An + empty mask or any other path is rejected with + INVALID_PARAMETER_VALUE. + requestBody: + content: + application/json: + schema: + type: object + properties: + operation: + type: object + properties: + action_type: + type: string + x-enum: + - OPERATION_ACTION_TYPE_BIND + - OPERATION_ACTION_TYPE_BIND_AND_UPDATE + - OPERATION_ACTION_TYPE_CREATE + - OPERATION_ACTION_TYPE_DELETE + - OPERATION_ACTION_TYPE_INITIAL_REGISTER + - OPERATION_ACTION_TYPE_RECREATE + - OPERATION_ACTION_TYPE_RESIZE + - OPERATION_ACTION_TYPE_UPDATE + - OPERATION_ACTION_TYPE_UPDATE_WITH_ID + description: The type of operation performed on this resource. + status: + type: string + x-enum: + - OPERATION_STATUS_FAILED + - OPERATION_STATUS_SUCCEEDED + description: >- + Whether the operation succeeded or failed. Mutable: may + be updated after creation via UpdateOperation, e.g. when + an operation recorded as failed is retried and + eventually succeeds. A succeeded operation cannot carry + an ``error_message``. + create_time: + type: string + format: date-time + description: When the operation was recorded. + dashboard_metadata: + type: object + properties: + definition_path: + type: string + description: >- + Path of the file that declares this dashboard, + relative to the bundle's workspace.file_path + (Version.workspace_info.file_path) — join the two to + get the file's absolute workspace path. For now this + lives only on the dashboard metadata, and is a + single string because it was a single string + (``relative_path``) in the legacy bundle + metadata.json. We may generalize it in the future: + lifting it to a top-level field on + Resource/Operation (every resource type has a + definition location) and converting it to a repeated + field, since a resource can be declared across + multiple files/locations. + source_path: + type: string + description: >- + Path of the dashboard's source artifact (its + ``.lvdash.json``), relative to the deployment root. + description: >- + Dashboard-specific metadata; set only for dashboard + resources. + error_message: + type: string + description: >- + Error message if the operation failed. Set when status + is OPERATION_STATUS_FAILED. Captures the error + encountered while applying the resource to the + workspace. Mutable: may be updated after creation via + UpdateOperation; setting it to an empty string clears + it. After an update is applied, an operation whose + status is OPERATION_STATUS_SUCCEEDED cannot carry an + error_message. + name: + type: string + description: >- + Resource name of the operation. Format: + deployments/{deployment_id}/versions/{version_id}/operations/{resource_key} + resource_id: + type: string + description: >- + ID of the actual resource in the workspace (e.g. the job + ID, pipeline ID). Optional at creation: CREATE and + RECREATE operations produce a new resource whose ID is + not yet known when the operation is recorded. Mutable: + may be filled in (or corrected) later via + UpdateOperation once the ID is known. + resource_key: + type: string + description: >- + Resource identifier within the bundle (e.g. "jobs.foo", + "pipelines.bar", "jobs.foo.permissions", + "files."). Can be an arbitrary UTF-8 encoded + string key. This key links the operation to the + corresponding deployment-level Resource. + resource_type: + type: string + x-enum: + - DEPLOYMENT_RESOURCE_TYPE_ALERT + - DEPLOYMENT_RESOURCE_TYPE_APP + - DEPLOYMENT_RESOURCE_TYPE_CATALOG + - DEPLOYMENT_RESOURCE_TYPE_CLUSTER + - DEPLOYMENT_RESOURCE_TYPE_DASHBOARD + - DEPLOYMENT_RESOURCE_TYPE_DATABASE_CATALOG + - DEPLOYMENT_RESOURCE_TYPE_DATABASE_INSTANCE + - DEPLOYMENT_RESOURCE_TYPE_EXPERIMENT + - DEPLOYMENT_RESOURCE_TYPE_EXTERNAL_LOCATION + - DEPLOYMENT_RESOURCE_TYPE_JOB + - DEPLOYMENT_RESOURCE_TYPE_MODEL + - DEPLOYMENT_RESOURCE_TYPE_MODEL_SERVING_ENDPOINT + - DEPLOYMENT_RESOURCE_TYPE_PIPELINE + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_BRANCH + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_ENDPOINT + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_PROJECT + - DEPLOYMENT_RESOURCE_TYPE_QUALITY_MONITOR + - DEPLOYMENT_RESOURCE_TYPE_REGISTERED_MODEL + - DEPLOYMENT_RESOURCE_TYPE_SCHEMA + - DEPLOYMENT_RESOURCE_TYPE_SECRET_SCOPE + - DEPLOYMENT_RESOURCE_TYPE_SQL_WAREHOUSE + - DEPLOYMENT_RESOURCE_TYPE_SYNCED_DATABASE_TABLE + - DEPLOYMENT_RESOURCE_TYPE_VOLUME + description: >- + The type of the deployment resource this operation + applies to. Derived from the ``resource_key`` prefix + (e.g. "jobs" → JOB); the caller does not set this field. + sequence_id: + type: integer + description: >- + Monotonically increasing revision used for optimistic + concurrency control (the AIP-154 concurrency token for + this resource, realized as a sequence number rather than + an opaque etag). The server assigns 1 on creation and + increments it on every successful UpdateOperation. It is + OPTIONAL rather than OUTPUT_ONLY because it is + dual-purpose: CreateOperation/GetOperation return the + current value, and UpdateOperation reads the + caller-supplied value as a precondition. The caller must + echo the value it last observed; if it no longer matches + the server's value, the update is rejected with ABORTED + so the caller can re-read and retry. Ignored on + CreateOperation. + state: + type: object + description: >- + Serialized local config state after the operation. + Should be unset for delete operations. Mutable: may be + updated after creation via UpdateOperation. When + updating, the caller must echo the last-observed + ``sequence_id`` as a concurrency precondition. + update_time: + type: string + format: date-time + description: >- + When the operation was last updated. Set to + ``create_time`` when the operation is created and to the + server timestamp on each successful UpdateOperation. + required: + - action_type + - status + description: >- + The operation to update. Its ``name`` selects the operation; + the fields named in ``update_mask`` carry the new values; + and ``sequence_id`` carries the optimistic-concurrency + precondition (see the field docs on Operation). All other + fields are ignored. + required: + - operation + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Operation' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + /api/2.0/bundle/{name}/heartbeat: + post: + operationId: bundle_deployments_heartbeat + summary: Sends a heartbeat to renew the lock held by a version. + tags: + - bundledeployments + - bundle_deployments + description: >- + Sends a heartbeat to renew the lock held by a version. + + + The server validates that the version is the active (non-terminal) + version on the parent deployment + + and resets the lock expiry. If the lock has already expired or the + version is no longer active, the + + server returns ``ABORTED``. + + + :param name: str + The version whose lock to renew. Format: deployments/{deployment_id}/versions/{version_id} + + :returns: :class:`HeartbeatResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The version whose lock to renew. Format: + deployments/{deployment_id}/versions/{version_id} + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/HeartbeatResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI + /api/2.0/bundle/{parent}/resources: + get: + operationId: bundle_deployments_list_resources + summary: Lists resources under a deployment. + tags: + - bundledeployments + - bundle_deployments + description: |- + Lists resources under a deployment. + + :param parent: str + The parent deployment. Format: deployments/{deployment_id} + :param page_size: int (optional) + The maximum number of resources to return. The service may return fewer than this value. If + unspecified, at most 50 resources will be returned. The maximum value is 1000; values above 1000 + will be coerced to 1000. + :param page_token: str (optional) + A page token, received from a previous ``ListResources`` call. Provide this to retrieve the + subsequent page. + + :returns: Iterator over :class:`Resource` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent deployment. Format: deployments/{deployment_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of resources to return. The service may return + fewer than this value. If unspecified, at most 50 resources will be + returned. The maximum value is 1000; values above 1000 will be + coerced to 1000. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ``ListResources`` call. + Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListResourcesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: BundleDeploymentsAPI +components: + schemas: + DashboardMetadata: + type: object + properties: + definition_path: + type: string + description: >- + Path of the file that declares this dashboard, relative to the + bundle's workspace.file_path (Version.workspace_info.file_path) — + join the two to get the file's absolute workspace path. For now this + lives only on the dashboard metadata, and is a single string because + it was a single string (``relative_path``) in the legacy bundle + metadata.json. We may generalize it in the future: lifting it to a + top-level field on Resource/Operation (every resource type has a + definition location) and converting it to a repeated field, since a + resource can be declared across multiple files/locations. + source_path: + type: string + description: >- + Path of the dashboard's source artifact (its ``.lvdash.json``), + relative to the deployment root. + description: >- + Dashboard-specific per-resource metadata. Set only for dashboard + resources. + Deployment: + type: object + properties: + create_time: + type: string + format: date-time + description: When the deployment was created. + created_by: + type: string + description: The user who created the deployment (email or principal name). + deployment_mode: + $ref: '#/components/schemas/DeploymentMode' + description: >- + Bundle target deployment mode (development or production), derived + from the most recent version's mode. + destroy_time: + type: string + format: date-time + description: >- + When deletion was recorded. Unset if deletion has not been recorded. + This response metadata does not determine the deployment's lifecycle + status. + destroyed_by: + type: string + description: >- + The user who destroyed the deployment (email or principal name). + Unset if the deployment has not been destroyed. + display_name: + type: string + description: >- + Human-readable name for the deployment, up to 256 characters. Output + only: clients update it by setting ``display_name`` when creating a + version. + git_info: + $ref: '#/components/schemas/GitInfo' + description: >- + Git provenance of the deployment's source, derived from the latest + version. + initial_parent_path: + type: string + description: >- + The workspace path of the existing folder where the deployment is + initially created. Must be absolute and canonical, with single + separators, no ``.`` or ``..`` segments, and no trailing slash + unless the path is ``/``. It may contain at most 24 path segments, + excluding an optional leading ``/Workspace`` segment. The complete + path may contain up to 1,024 characters, and each segment may + contain up to 511 characters. This field is input only and is not + returned in create, get, or list responses. + last_successful_version_id: + type: string + description: >- + The version_id of the most recent version that completed + successfully. Unset until a version has completed successfully. + Unlike last_version_id, it is not advanced when a version fails, so + it always points at the last known-good deployment state (or is + unset if there has never been one). + last_version_id: + type: string + description: The version_id of the most recent deployment version. + name: + type: string + description: 'Resource name of the deployment. Format: deployments/{deployment_id}' + status: + $ref: '#/components/schemas/DeploymentStatus' + description: Current status of the deployment. + target_name: + type: string + description: >- + The bundle target name associated with this deployment. Output only: + it is denormalized from the latest version, not set directly on the + deployment. + update_time: + type: string + format: date-time + description: When the deployment was last updated. + updated_by: + type: string + description: >- + The user who most recently updated the deployment (email or + principal name). + workspace_info: + $ref: '#/components/schemas/WorkspaceInfo' + description: >- + Workspace location of the deployment, derived from the latest + version. + description: A bundle deployment registered with the control plane. + GitInfo: + type: object + properties: + branch: + type: string + description: Branch the source was deployed from. + commit: + type: string + description: Commit SHA of the deployed source. + origin_url: + type: string + description: URL of the git remote the source was deployed from. + description: >- + Git provenance of a bundle's source, captured at deploy time. Lets + consumers link a deployed + resource back to its source in version control. + HeartbeatResponse: + type: object + properties: + expire_time: + type: string + format: date-time + description: The new lock expiry time after renewal. + description: Response for Heartbeat. + ListDeploymentsResponse: + type: object + properties: + deployments: + type: array + items: + $ref: '#/components/schemas/Deployment' + description: The deployments from the queried workspace. + next_page_token: + type: string + description: >- + A token, which can be sent as ``page_token`` to retrieve the next + page. If this field is omitted, there are no subsequent pages. + description: Response for ListDeployments. + ListOperationsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as ``page_token`` to retrieve the next + page. If this field is omitted, there are no subsequent pages. + operations: + type: array + items: + $ref: '#/components/schemas/Operation' + description: The resource operations under the specified version. + description: Response for ListOperations. + ListResourcesResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as ``page_token`` to retrieve the next + page. If this field is omitted, there are no subsequent pages. + resources: + type: array + items: + $ref: '#/components/schemas/Resource' + description: The resources under the specified deployment. + description: Response for ListResources. + ListVersionsResponse: + type: object + properties: + next_page_token: + type: string + description: >- + A token, which can be sent as ``page_token`` to retrieve the next + page. If this field is omitted, there are no subsequent pages. + versions: + type: array + items: + $ref: '#/components/schemas/Version' + description: The versions under the specified deployment. + description: Response for ListVersions. + Operation: + type: object + properties: + action_type: + $ref: '#/components/schemas/OperationActionType' + description: The type of operation performed on this resource. + status: + $ref: '#/components/schemas/OperationStatus' + description: >- + Whether the operation succeeded or failed. Mutable: may be updated + after creation via UpdateOperation, e.g. when an operation recorded + as failed is retried and eventually succeeds. A succeeded operation + cannot carry an ``error_message``. + create_time: + type: string + format: date-time + description: When the operation was recorded. + dashboard_metadata: + $ref: '#/components/schemas/DashboardMetadata' + description: Dashboard-specific metadata; set only for dashboard resources. + error_message: + type: string + description: >- + Error message if the operation failed. Set when status is + OPERATION_STATUS_FAILED. Captures the error encountered while + applying the resource to the workspace. Mutable: may be updated + after creation via UpdateOperation; setting it to an empty string + clears it. After an update is applied, an operation whose status is + OPERATION_STATUS_SUCCEEDED cannot carry an error_message. + name: + type: string + description: >- + Resource name of the operation. Format: + deployments/{deployment_id}/versions/{version_id}/operations/{resource_key} + resource_id: + type: string + description: >- + ID of the actual resource in the workspace (e.g. the job ID, + pipeline ID). Optional at creation: CREATE and RECREATE operations + produce a new resource whose ID is not yet known when the operation + is recorded. Mutable: may be filled in (or corrected) later via + UpdateOperation once the ID is known. + resource_key: + type: string + description: >- + Resource identifier within the bundle (e.g. "jobs.foo", + "pipelines.bar", "jobs.foo.permissions", "files."). Can be + an arbitrary UTF-8 encoded string key. This key links the operation + to the corresponding deployment-level Resource. + resource_type: + $ref: '#/components/schemas/DeploymentResourceType' + description: >- + The type of the deployment resource this operation applies to. + Derived from the ``resource_key`` prefix (e.g. "jobs" → JOB); the + caller does not set this field. + sequence_id: + type: integer + description: >- + Monotonically increasing revision used for optimistic concurrency + control (the AIP-154 concurrency token for this resource, realized + as a sequence number rather than an opaque etag). The server assigns + 1 on creation and increments it on every successful UpdateOperation. + It is OPTIONAL rather than OUTPUT_ONLY because it is dual-purpose: + CreateOperation/GetOperation return the current value, and + UpdateOperation reads the caller-supplied value as a precondition. + The caller must echo the value it last observed; if it no longer + matches the server's value, the update is rejected with ABORTED so + the caller can re-read and retry. Ignored on CreateOperation. + state: + type: object + description: >- + Serialized local config state after the operation. Should be unset + for delete operations. Mutable: may be updated after creation via + UpdateOperation. When updating, the caller must echo the + last-observed ``sequence_id`` as a concurrency precondition. + update_time: + type: string + format: date-time + description: >- + When the operation was last updated. Set to ``create_time`` when the + operation is created and to the server timestamp on each successful + UpdateOperation. + required: + - action_type + - status + description: >- + An operation on a single resource performed during a version. Operations + record the result of + applying a resource change to the workspace. Most fields are immutable once recorded; ``state``, + ``error_message``, ``resource_id``, and ``status`` may be updated afterwards (via + UpdateOperation), guarded by ``sequence_id`` for optimistic concurrency control. + Resource: + type: object + properties: + resource_type: + $ref: '#/components/schemas/DeploymentResourceType' + description: The type of the deployment resource. + dashboard_metadata: + $ref: '#/components/schemas/DashboardMetadata' + description: Dashboard-specific metadata; set only for dashboard resources. + last_action_type: + $ref: '#/components/schemas/OperationActionType' + description: The action performed on this resource during the last version. + last_version_id: + type: string + description: The version_id of the last version where this resource was updated. + name: + type: string + description: >- + Resource name. Format: + deployments/{deployment_id}/resources/{resource_key} + resource_id: + type: string + description: >- + ID that references the actual resource in the workspace (e.g. the + job ID, pipeline ID). + resource_key: + type: string + description: >- + Resource identifier within the bundle (e.g. "jobs.foo", + "pipelines.bar", "jobs.foo.permissions"). + state: + type: object + description: Serialized local config state (what the CLI deployed). + update_time: + type: string + format: date-time + description: >- + When the last operation that updated this resource's recorded state + was applied. Pairs with last_action_type and last_version_id (all + three advance together on that write). + required: + - resource_type + description: >- + A resource managed by a deployment. Resources are implicitly created, + updated, or deleted when + operations are recorded on a version. + Version: + type: object + properties: + cli_version: + type: string + description: CLI version used to initiate the version. + version_type: + $ref: '#/components/schemas/VersionType' + description: Type of version (deploy or destroy). + complete_time: + type: string + format: date-time + description: When the version completed. Unset while the version is in progress. + completed_by: + type: string + description: >- + The user who completed the version (email or principal name). May + differ from ``created_by`` when another user force-completes the + version. + completion_reason: + $ref: '#/components/schemas/VersionComplete' + description: >- + Why the version was completed. Unset while in progress. Set when + status transitions to COMPLETED. + create_time: + type: string + format: date-time + description: When the version was created. + created_by: + type: string + description: The user who created the version (email or principal name). + deployment_mode: + $ref: '#/components/schemas/DeploymentMode' + description: >- + Bundle target deployment mode (development or production), captured + at the time of this version. + display_name: + type: string + description: >- + Display name for the deployment, captured at the time of this + version. Up to 256 characters. When present, creating the version + updates the deployment display name. An empty value clears it; an + absent value leaves the current deployment display name unchanged. + git_info: + $ref: '#/components/schemas/GitInfo' + description: Git provenance of the source, captured at the time of this version. + name: + type: string + description: >- + Resource name of the version. Format: + deployments/{deployment_id}/versions/{version_id} + previous_version_id: + type: string + description: >- + The version_id this version was created on top of — the deployment's + most recent version at creation time. Leave unset when creating the + first version (the deployment has no prior versions). Set by the + client on creation and immutable thereafter. Acts as an + optimistic-concurrency precondition: the server requires it to equal + the deployment's current most-recent version (and to be unset when + the deployment has no versions) and returns + ``INVALID_PARAMETER_VALUE`` on mismatch, so a deploy racing against + a concurrent deploy is rejected rather than silently overwriting it. + status: + $ref: '#/components/schemas/VersionStatus' + description: 'Status of the version: IN_PROGRESS or COMPLETED.' + target_name: + type: string + description: Target name of the deployment, captured at the time of this version. + version_id: + type: string + description: >- + Version identifier within the parent deployment, assigned by the + client on creation. A numeric string (base-10, fits in a signed + 64-bit integer) that is greater than or equal to 1. Version IDs are + strictly increasing within a deployment but are not required to + start at 1 or to be contiguous. + workspace_info: + $ref: '#/components/schemas/WorkspaceInfo' + description: >- + Workspace location of the deployment, captured at the time of this + version. + required: + - cli_version + - version_type + description: >- + A single invocation of a deploy or destroy command against a deployment. + Creating a version + acquires an exclusive lock on the parent deployment. + WorkspaceInfo: + type: object + properties: + bundle_root_path: + type: string + description: >- + Path of the bundle root (the directory containing databricks.yml) + relative to git_folder_path. Empty when the deployment is not from a + Databricks Git folder. + file_path: + type: string + description: >- + Absolute workspace path where the deployed bundle files live. + Mirrors the workspace.file_path field in DABs bundle config. + git_folder_path: + type: string + description: >- + When deployed from a Databricks Git folder, the absolute workspace + path of that folder; empty for local deploys. + root_path: + type: string + description: >- + Absolute workspace path of the deployment root — the base path the + deployed files live under. Mirrors workspace.root_path in the DABs + bundle config; file_path is its files subdirectory. + source_linked: + type: boolean + description: >- + Whether files are served directly from the source sync root instead + of being copied into file_path. + description: Workspace location of a bundle deployment, captured at deploy time. + DeploymentMode: + type: string + x-enum: + - DEPLOYMENT_MODE_DEVELOPMENT + - DEPLOYMENT_MODE_PRODUCTION + description: >- + Bundle target deployment mode. Mirrors the ``mode`` field on a bundle + target in + + ``databricks.yml`` (see + https://docs.databricks.com/dev-tools/bundles/deployment-modes). + DeploymentResourceType: + type: string + x-enum: + - DEPLOYMENT_RESOURCE_TYPE_ALERT + - DEPLOYMENT_RESOURCE_TYPE_APP + - DEPLOYMENT_RESOURCE_TYPE_CATALOG + - DEPLOYMENT_RESOURCE_TYPE_CLUSTER + - DEPLOYMENT_RESOURCE_TYPE_DASHBOARD + - DEPLOYMENT_RESOURCE_TYPE_DATABASE_CATALOG + - DEPLOYMENT_RESOURCE_TYPE_DATABASE_INSTANCE + - DEPLOYMENT_RESOURCE_TYPE_EXPERIMENT + - DEPLOYMENT_RESOURCE_TYPE_EXTERNAL_LOCATION + - DEPLOYMENT_RESOURCE_TYPE_JOB + - DEPLOYMENT_RESOURCE_TYPE_MODEL + - DEPLOYMENT_RESOURCE_TYPE_MODEL_SERVING_ENDPOINT + - DEPLOYMENT_RESOURCE_TYPE_PIPELINE + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_BRANCH + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_ENDPOINT + - DEPLOYMENT_RESOURCE_TYPE_POSTGRES_PROJECT + - DEPLOYMENT_RESOURCE_TYPE_QUALITY_MONITOR + - DEPLOYMENT_RESOURCE_TYPE_REGISTERED_MODEL + - DEPLOYMENT_RESOURCE_TYPE_SCHEMA + - DEPLOYMENT_RESOURCE_TYPE_SECRET_SCOPE + - DEPLOYMENT_RESOURCE_TYPE_SQL_WAREHOUSE + - DEPLOYMENT_RESOURCE_TYPE_SYNCED_DATABASE_TABLE + - DEPLOYMENT_RESOURCE_TYPE_VOLUME + description: Type of a deployment resource. + DeploymentStatus: + type: string + x-enum: + - DEPLOYMENT_STATUS_ACTIVE + - DEPLOYMENT_STATUS_DELETED + - DEPLOYMENT_STATUS_FAILED + - DEPLOYMENT_STATUS_IN_PROGRESS + description: Status of a deployment. + OperationActionType: + type: string + x-enum: + - OPERATION_ACTION_TYPE_BIND + - OPERATION_ACTION_TYPE_BIND_AND_UPDATE + - OPERATION_ACTION_TYPE_CREATE + - OPERATION_ACTION_TYPE_DELETE + - OPERATION_ACTION_TYPE_INITIAL_REGISTER + - OPERATION_ACTION_TYPE_RECREATE + - OPERATION_ACTION_TYPE_RESIZE + - OPERATION_ACTION_TYPE_UPDATE + - OPERATION_ACTION_TYPE_UPDATE_WITH_ID + description: Type of action performed on a resource during a deployment. + OperationStatus: + type: string + x-enum: + - OPERATION_STATUS_FAILED + - OPERATION_STATUS_SUCCEEDED + description: Status of a resource operation. + VersionComplete: + type: string + x-enum: + - VERSION_COMPLETE_FAILURE + - VERSION_COMPLETE_FORCE_ABORT + - VERSION_COMPLETE_LEASE_EXPIRED + - VERSION_COMPLETE_SUCCESS + description: Reason why a version was completed. + VersionStatus: + type: string + x-enum: + - VERSION_STATUS_COMPLETED + - VERSION_STATUS_IN_PROGRESS + description: Status of a version. + VersionType: + type: string + x-enum: + - VERSION_TYPE_DEPLOY + - VERSION_TYPE_DESTROY + description: Type of version. + x-stackQL-resources: + versions: + id: databricks_workspace.bundledeployments.versions + name: versions + title: Versions + methods: + complete: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{name}~1complete/post' + response: + mediaType: application/json + openAPIDocKey: '200' + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{parent}~1versions/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{parent}~1versions/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.versions + get: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/versions/methods/list' + - $ref: '#/components/x-stackQL-resources/versions/methods/get' + insert: + - $ref: '#/components/x-stackQL-resources/versions/methods/create' + update: [] + delete: [] + replace: [] + deployments: + id: databricks_workspace.bundledeployments.deployments + name: deployments + title: Deployments + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1bundle~1deployments/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1deployments/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.deployments + delete: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + heartbeat: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{name}~1heartbeat/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/deployments/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/deployments/methods/create' + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/deployments/methods/delete' + replace: [] + operations: + id: databricks_workspace.bundledeployments.operations + name: operations + title: Operations + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{parent}~1operations/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{parent}~1operations/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.operations + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/operations/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/operations/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/operations/methods/update' + delete: [] + replace: [] + deployment_resources: + id: databricks_workspace.bundledeployments.deployment_resources + name: deployment_resources + title: Deployment Resources + methods: + list: + operation: + $ref: '#/paths/~1api~12.0~1bundle~1{parent}~1resources/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.resources + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/deployment_resources/methods/list' + insert: [] + update: [] + delete: [] + replace: [] +x-stackQL-config: + pagination: + requestToken: + key: page_token + location: query + responseToken: + key: next_page_token + location: body diff --git a/providers/src/databricks_workspace/v00.00.00000/services/catalog.yaml b/providers/src/databricks_workspace/v00.00.00000/services/catalog.yaml index 8ec7f335..05e05cf9 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/catalog.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/catalog.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks catalog service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.catalog servers: - url: https://{deployment_name}.cloud.databricks.com @@ -15,45 +15,312 @@ servers: description: The Databricks Workspace Deployment Name default: dbc-abcd0123-a1bc paths: - /api/2.1/unity-catalog/artifact-allowlists/{artifact_type.value}: - get: - operationId: artifact_allowlists_get + /api/2.1/unity-catalog/agent-services: + post: + operationId: ai_gateway_create_agent_service summary: >- - Get the artifact allowlist of a certain artifact type. The caller must - be a metastore admin or have + Creates an agent service in a Unity Catalog schema. An agent service is + a governed securable that tags: - catalog - - artifact_allowlists + - ai_gateway description: >- - Get the artifact allowlist of a certain artifact type. The caller must - be a metastore admin or have + Creates an agent service in a Unity Catalog schema. An agent service is + a governed securable that - the **MANAGE ALLOWLIST** privilege on the metastore. + registers an AI agent and exposes it for discovery, access control, and + auditing. The caller supplies + the leaf name in ``agent_service_id`` and the agent service type, which + is immutable after creation. - :param artifact_type: :class:`ArtifactType` - The artifact type of the allowlist. - :returns: :class:`ArtifactAllowlistInfo` + You must be the owner of the parent schema or have the + ``CREATE_SERVICE`` and ``USE_SCHEMA`` + + privileges on the parent schema and ``USE_CATALOG`` on the parent + catalog. + + + :param agent_service: :class:`AgentService` + The agent service to create. The server populates ``name`` from ``parent`` + ``agent_service_id``; + clients should leave it unset. + :param parent: str + Resource name of the parent schema. Format: ``schemas/{catalog}.{schema}``. Each ``{...}`` component + is capped at 255 characters individually. + :param agent_service_id: str + Leaf identifier for the agent service (the unqualified name within the parent schema, e.g. + "support_agent"). + + :returns: :class:`AgentService` parameters: - - name: artifact_type.value - in: path + - name: parent + in: query required: true schema: type: string - - name: artifact_type + description: >- + Resource name of the parent schema. Format: + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: agent_service_id in: query required: true schema: - $ref: '#/components/schemas/ArtifactType' - description: The artifact type of the allowlist. + type: string + description: >- + Leaf identifier for the agent service (the unqualified name within + the parent schema, e.g. "support_agent"). + requestBody: + content: + application/json: + schema: + type: object + properties: + agent_service: + type: object + properties: + agent_service_type: + type: string + x-enum: + - AGENT_SERVICE_TYPE_EXTERNAL + description: >- + Type of agent service. Required on create. Immutable + after creation. For future internal / genie / app kinds, + determines how the backing agent surface is provisioned. + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through + the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + type: object + properties: + base_path: + type: string + description: >- + Vendor base path appended to the connection url at + invocation time. Bedrock example: + ``/agents/AGENT123/agentAliases/ALIAS1/sessions``. + Empty when the agent platform does not require a + per-agent suffix. + inference_table: + type: object + properties: + parent: + type: string + description: >- + Parent UC schema where the inference table is + created. Format: ``schemas/{catalog}.{schema}``. + Set at create time and immutable thereafter; + changing it on an existing service is rejected. + disabled: + type: boolean + description: >- + Indicates whether payload logging is disabled + (opt-out). Unset means that payload logging is + active (the on-by-default state coincides with + the proto zero-value, so the server never fills + this field for a client that leaves it unset). + Set ``disabled = true`` to pause runtime logging + while keeping the sub-message attached + (preserving ``parent`` and ``table_name_prefix`` + for a later flip back to active). ``parent`` + remains required either way. + is_deleted: + type: boolean + description: >- + True when the bound inference TABLE has been + deleted but the parent service still references + it. The dangling reference is surfaced (not + silently dropped) so callers can see the broken + dependency. AI Gateway payload logging fails + closed in this state. + table: + type: string + description: >- + Resolved UC table for payload logs. Format: + ``tables/{catalog}.{schema}.{table}``. + table_name_prefix: + type: string + description: >- + Prefix for the inference-table's UC-registered + name. The actual leaf name UC stores is + ``_payload``; the + ``_payload`` suffix is appended automatically. + To find the actual UC table after Create, read + the ``table`` field on the response. Defaults to + ``_payload`` when unset. Set + at create time and immutable thereafter; + changing it on an existing service is rejected. + required: + - parent + description: >- + Inference table config for invocation payload + logging. Reuses the shared InferenceTableConfig + (same as ModelServiceConfig.inference_table). + model_service: + type: string + description: >- + Resource name of the model service backing this + agent. Empty when the agent platform manages its own + model selection (e.g. Bedrock Agents). Format: + ``model-services/{catalog}.{schema}.{model_service}``. + rate_limits: + type: array + items: + type: object + properties: + key: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: >- + Scope key. Determines whether ``principal`` is + required. + renewal_period: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period. + principal: + type: string + description: >- + Principal this limit applies to: user email, + group name, or service principal application + ID. Required unless ``key`` is + ``RATE_LIMIT_KEY_SERVICE``, + ``RATE_LIMIT_KEY_USER_DEFAULT``, or + ``RATE_LIMIT_KEY_REQUEST_TAG`` (which must not + set a principal). + request_tag_key: + type: string + description: >- + Request tag key this limit applies to. + Required when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden + otherwise. + request_tag_value: + type: string + description: >- + Request tag value this limit applies to. Only + valid when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``. Leave unset to + apply the limit to every value of + ``request_tag_key`` (an any-value default); a + set value is a specific override for that + value. + requests: + type: integer + description: >- + Max requests allowed within a renewal period. + Leave unset for no request limit. + tokens: + type: integer + description: >- + Max tokens allowed within a renewal period. + Leave unset for no token limit. + required: + - key + - renewal_period + description: >- + A rate limit applied to service requests. Leave + ``requests`` or ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. + description: >- + Per-principal rate limits applied to invocations + routed through this agent service. Repeated to + support per-USER / USER_GROUP / SERVICE_PRINCIPAL / + SERVICE / USER_DEFAULT scopes simultaneously, + mirroring McpServiceConfig.rate_limits. Empty when + no rate limit is configured. + source_connection: + type: object + properties: + name: + type: string + is_deleted: + type: boolean + required: + - name + description: External agent referenced via a UC Connection. + system_prompt: + type: string + description: System prompt prepended to invocations. Max 8 KB. + description: >- + Operational configuration: connection, base_path, + system_prompt, model_service, optional inference table, + optional rate limits. Required on CreateAgentService; on + UpdateAgentService it is required only when ``config`` + (or a ``config.*`` subpath) appears in ``update_mask``. + create_time: + type: string + format: date-time + description: When the agent service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: >- + The resolved owner of the agent service. Falls back to + the caller's identity when ``owner`` is not explicitly + set on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated + from the entity's state and returned on every read. To + use it as an if-match precondition on a mutation, echo + the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects + the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the agent service. + name: + type: string + description: >- + Resource name of the agent service. Format: + ``agent-services/{catalog}.{schema}.{agent_service}``. + Each ``{...}`` component is capped at 255 characters + individually. Server-derived on Create from ``parent`` + + ``agent_service_id``; required and immutable on + Update/Get/Delete. + owner: + type: string + description: >- + The owner of the agent service. Write-only; read owner + via effective_owner. + update_time: + type: string + format: date-time + description: When the agent service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: >- + The agent service to create. The server populates ``name`` + from ``parent`` + ``agent_service_id``; clients should leave + it unset. + required: + - agent_service responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ArtifactAllowlistInfo' + $ref: '#/components/schemas/AgentService' default: description: Error response content: @@ -65,93 +332,86 @@ paths: type: string message: type: string - x-stackql-sdk-source: ArtifactAllowlistsAPI - put: - operationId: artifact_allowlists_update + x-stackql-sdk-source: AiGatewayAPI + get: + operationId: ai_gateway_list_agent_services summary: >- - Set the artifact allowlist of a certain artifact type. The whole - artifact allowlist is replaced with + Lists the agent services in a Unity Catalog schema. Provide ``parent`` + as tags: - catalog - - artifact_allowlists + - ai_gateway description: >- - Set the artifact allowlist of a certain artifact type. The whole - artifact allowlist is replaced with + Lists the agent services in a Unity Catalog schema. Provide ``parent`` + as - the new allowlist. The caller must be a metastore admin or have the - **MANAGE ALLOWLIST** privilege on + ``schemas/{catalog}.{schema}``. Results are paginated; pass the returned + ``next_page_token`` to fetch - the metastore. + subsequent pages. - :param artifact_type: :class:`ArtifactType` - The artifact type of the allowlist. - :param artifact_matchers: List[:class:`ArtifactMatcher`] - A list of allowed artifact match patterns. - :param created_at: int (optional) - Time at which this artifact allowlist was set, in epoch milliseconds. - :param created_by: str (optional) - Username of the user who set the artifact allowlist. - :param metastore_id: str (optional) - Unique identifier of parent metastore. + Requires ``USE_CATALOG`` on the parent catalog and ``USE_SCHEMA`` on the + parent schema. Only agent - :returns: :class:`ArtifactAllowlistInfo` + services the caller can access (as owner or through ``EXECUTE``, + ``READ_METADATA``, or ``MANAGE``) are + + returned. + + + :param include_browse: bool (optional) + Whether to include agent services for which the principal can only access selective metadata. + :param page_size: int (optional) + Maximum number of agent services to return. Defaults to 100 when unset or 0; the maximum is 1000. + Use ``next_page_token`` to retrieve additional pages. + :param page_token: str (optional) + Opaque pagination token from a previous request. + :param parent: str (optional) + Resource name of the parent schema to list within, as ``schemas/{catalog}.{schema}``. Each ``{...}`` + component is capped at 255 characters individually. + + :returns: Iterator over :class:`AgentService` parameters: - - name: artifact_type.value - in: path - required: true + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include agent services for which the principal can only + access selective metadata. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Maximum number of agent services to return. Defaults to 100 when + unset or 0; the maximum is 1000. Use ``next_page_token`` to retrieve + additional pages. + - name: page_token + in: query + required: false schema: type: string - requestBody: - content: - application/json: - schema: - type: object - properties: - artifact_type: - type: string - x-enum: - - INIT_SCRIPT - - LIBRARY_JAR - - LIBRARY_MAVEN - description: The artifact type of the allowlist. - artifact_matchers: - type: array - items: - type: object - properties: - artifact: - type: string - match_type: - type: string - x-enum: - - PREFIX_MATCH - description: The pattern matching type of the artifact - required: - - artifact - - match_type - description: A list of allowed artifact match patterns. - created_at: - type: integer - description: >- - Time at which this artifact allowlist was set, in epoch - milliseconds. - created_by: - type: string - description: Username of the user who set the artifact allowlist. - metastore_id: - type: string - description: Unique identifier of parent metastore. - required: - - artifact_type - - artifact_matchers + description: Opaque pagination token from a previous request. + - name: parent + in: query + required: false + schema: + type: string + description: >- + Resource name of the parent schema to list within, as + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ArtifactAllowlistInfo' + $ref: '#/components/schemas/ListAgentServicesResponse' default: description: Error response content: @@ -163,85 +423,243 @@ paths: type: string message: type: string - x-stackql-sdk-source: ArtifactAllowlistsAPI - /api/2.1/unity-catalog/catalogs: + x-stackql-sdk-source: AiGatewayAPI + /api/2.1/unity-catalog/mcp-services: post: - operationId: catalogs_create + operationId: ai_gateway_create_mcp_service summary: >- - Creates a new catalog instance in the parent metastore if the caller is - a metastore admin or has the + Creates an MCP service in a Unity Catalog schema. An MCP (Model Context + Protocol) service is a tags: - catalog - - catalogs + - ai_gateway description: >- - Creates a new catalog instance in the parent metastore if the caller is - a metastore admin or has the + Creates an MCP service in a Unity Catalog schema. An MCP (Model Context + Protocol) service is a - **CREATE_CATALOG** privilege. + governed securable that registers an MCP server and exposes its tools + for discovery, access control, + and invocation. The caller supplies the leaf name in ``mcp_service_id``. - :param name: str - Name of catalog. - :param comment: str (optional) - User-provided free-form text description. - :param connection_name: str (optional) - The name of the connection to an external data source. - :param options: Dict[str,str] (optional) - A map of key-value properties attached to the securable. - :param properties: Dict[str,str] (optional) - A map of key-value properties attached to the securable. - :param provider_name: str (optional) - The name of delta sharing provider. - A Delta Sharing catalog is a catalog that is based on a Delta share on a remote sharing server. - :param share_name: str (optional) - The name of the share under the share provider. - :param storage_root: str (optional) - Storage root URL for managed tables within catalog. + You must be the owner of the parent schema or have the + ``CREATE_SERVICE`` and ``USE_SCHEMA`` - :returns: :class:`CatalogInfo` + privileges on the parent schema and ``USE_CATALOG`` on the parent + catalog. You also need + + ``USE_CONNECTION`` on the connection the MCP service references. + + + :param mcp_service: :class:`McpService` + The MCP service to create. The server populates ``name`` from ``parent`` + ``mcp_service_id``; + clients should leave it unset. + :param parent: str + Resource name of the parent schema. Format: ``schemas/{catalog}.{schema}``. Each ``{...}`` component + is capped at 255 characters individually. + :param mcp_service_id: str + Leaf identifier for the MCP service (the unqualified name within the parent schema, e.g. + "my_mcp_service"). + + :returns: :class:`McpService` + parameters: + - name: parent + in: query + required: true + schema: + type: string + description: >- + Resource name of the parent schema. Format: + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: mcp_service_id + in: query + required: true + schema: + type: string + description: >- + Leaf identifier for the MCP service (the unqualified name within the + parent schema, e.g. "my_mcp_service"). requestBody: content: application/json: schema: type: object properties: - name: - type: string - description: Name of catalog. - comment: - type: string - description: User-provided free-form text description. - connection_name: - type: string - description: The name of the connection to an external data source. - options: - type: object - description: A map of key-value properties attached to the securable. - properties: + mcp_service: type: object - description: A map of key-value properties attached to the securable. - provider_name: - type: string + properties: + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through + the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + type: object + properties: + include_tool_selectors: + type: array + items: + type: string + description: >- + Glob or exact-match patterns selecting which tools + from the MCP server to expose. Prefix match for + patterns with ``*``, exact match otherwise. An empty + list means all tools are included. Per-element max + 256 chars. + rate_limits: + type: array + items: + type: object + properties: + key: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: >- + Scope key. Determines whether ``principal`` is + required. + renewal_period: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period. + principal: + type: string + description: >- + Principal this limit applies to: user email, + group name, or service principal application + ID. Required unless ``key`` is + ``RATE_LIMIT_KEY_SERVICE``, + ``RATE_LIMIT_KEY_USER_DEFAULT``, or + ``RATE_LIMIT_KEY_REQUEST_TAG`` (which must not + set a principal). + request_tag_key: + type: string + description: >- + Request tag key this limit applies to. + Required when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden + otherwise. + request_tag_value: + type: string + description: >- + Request tag value this limit applies to. Only + valid when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``. Leave unset to + apply the limit to every value of + ``request_tag_key`` (an any-value default); a + set value is a specific override for that + value. + requests: + type: integer + description: >- + Max requests allowed within a renewal period. + Leave unset for no request limit. + tokens: + type: integer + description: >- + Max tokens allowed within a renewal period. + Leave unset for no token limit. + required: + - key + - renewal_period + description: >- + A rate limit applied to service requests. Leave + ``requests`` or ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. + description: >- + Per-principal rate limits applied to tool + invocations routed through this MCP service. + Repeated to support per-USER / USER_GROUP / + SERVICE_PRINCIPAL / SERVICE / USER_DEFAULT scopes + simultaneously, mirroring the + ``ModelServiceConfig.rate_limits`` shape. Empty when + no rate limit is configured. + source_connection: + type: object + properties: + name: + type: string + is_deleted: + type: boolean + required: + - name + description: UC Connection referencing the MCP server. + description: >- + Operational configuration: connection, tool selectors, + rate limit. Required on CreateMcpService; on + UpdateMcpService it is required only when ``config`` (or + a ``config.*`` subpath) appears in ``update_mask``. + create_time: + type: string + format: date-time + description: When the MCP service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: >- + The resolved owner of the MCP service. Falls back to the + caller's identity when ``owner`` is not explicitly set + on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated + from the entity's state and returned on every read. To + use it as an if-match precondition on a mutation, echo + the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects + the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the MCP service. + name: + type: string + description: >- + Resource name of the MCP service. Format: + ``mcp-services/{catalog}.{schema}.{mcp_service}``. Each + ``{...}`` component is capped at 255 characters + individually. Server-derived on Create from ``parent`` + + ``mcp_service_id``; required and immutable on + Update/Get/Delete. + owner: + type: string + description: >- + The owner of the MCP service. Write-only; read owner via + effective_owner. + update_time: + type: string + format: date-time + description: When the MCP service was last modified. + updated_by: + type: string + description: Identity of the last updater. description: >- - The name of delta sharing provider. A Delta Sharing catalog - is a catalog that is based on a Delta share on a remote - sharing server. - share_name: - type: string - description: The name of the share under the share provider. - storage_root: - type: string - description: Storage root URL for managed tables within catalog. + The MCP service to create. The server populates ``name`` + from ``parent`` + ``mcp_service_id``; clients should leave + it unset. required: - - name + - mcp_service responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/CatalogInfo' + $ref: '#/components/schemas/McpService' default: description: Error response content: @@ -253,62 +671,45 @@ paths: type: string message: type: string - x-stackql-sdk-source: CatalogsAPI + x-stackql-sdk-source: AiGatewayAPI get: - operationId: catalogs_list - summary: >- - Gets an array of catalogs in the metastore. If the caller is the - metastore admin, all catalogs will be + operationId: ai_gateway_list_mcp_services + summary: Lists the MCP services in a Unity Catalog schema. Provide ``parent`` as tags: - catalog - - catalogs + - ai_gateway description: >- - Gets an array of catalogs in the metastore. If the caller is the - metastore admin, all catalogs will be - - retrieved. Otherwise, only catalogs owned by the caller (or for which - the caller has the - - **USE_CATALOG** privilege) will be retrieved. There is no guarantee of a - specific ordering of the + Lists the MCP services in a Unity Catalog schema. Provide ``parent`` as - elements in the array. - - - NOTE: we recommend using max_results=0 to use the paginated version of - this API. Unpaginated calls + ``schemas/{catalog}.{schema}``. Results are paginated; pass the returned + ``next_page_token`` to fetch - will be deprecated soon. + subsequent pages. - PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page - may contain zero results while + Requires ``USE_CATALOG`` on the parent catalog and ``USE_SCHEMA`` on the + parent schema. Only MCP - still providing a next_page_token. Clients must continue reading pages - until next_page_token is + services the caller can access (as owner or through ``EXECUTE``, + ``READ_METADATA``, or ``MANAGE``) are - absent, which is the only indication that the end of results has been - reached. + returned. :param include_browse: bool (optional) - Whether to include catalogs in the response for which the principal can only access selective - metadata for - :param include_unbound: bool (optional) - Whether to include catalogs not bound to the workspace. Effective only if the user has permission to - update the catalog–workspace binding. - :param max_results: int (optional) - Maximum number of catalogs to return. - when set to 0, the page length is set to a server configured - value (recommended); - when set to a value greater than 0, the page length is the minimum of this - value and a server configured value; - when set to a value less than 0, an invalid parameter error - is returned; - If not set, all valid catalogs are returned (not recommended). - Note: The number of - returned catalogs might be less than the specified max_results size, even zero. The only definitive - indication that no further catalogs can be fetched is when the next_page_token is unset from the - response. + Whether to include MCP services for which the principal can only access selective metadata. + :param page_size: int (optional) + Maximum number of MCP services to return. Defaults to 100 when unset or 0; the maximum is 1000. Use + ``next_page_token`` to retrieve additional pages. :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. - - :returns: Iterator over :class:`CatalogInfo` + Opaque pagination token from a previous request. + :param parent: str (optional) + Resource name of the parent schema to list within, as ``schemas/{catalog}.{schema}``. Each ``{...}`` + component is capped at 255 characters individually. + :param view: :class:`ListMcpServicesRequestView` (optional) + View selector controlling which fields are populated per row. + + :returns: Iterator over :class:`McpService` parameters: - name: include_browse in: query @@ -316,46 +717,45 @@ paths: schema: type: boolean description: >- - Whether to include catalogs in the response for which the principal - can only access selective metadata for - - name: include_unbound + Whether to include MCP services for which the principal can only + access selective metadata. + - name: page_size in: query required: false schema: - type: boolean + type: integer description: >- - Whether to include catalogs not bound to the workspace. Effective - only if the user has permission to update the catalog–workspace - binding. - - name: max_results + Maximum number of MCP services to return. Defaults to 100 when unset + or 0; the maximum is 1000. Use ``next_page_token`` to retrieve + additional pages. + - name: page_token in: query required: false schema: - type: integer - description: >- - Maximum number of catalogs to return. - when set to 0, the page - length is set to a server configured value (recommended); - when set - to a value greater than 0, the page length is the minimum of this - value and a server configured value; - when set to a value less than - 0, an invalid parameter error is returned; - If not set, all valid - catalogs are returned (not recommended). - Note: The number of - returned catalogs might be less than the specified max_results size, - even zero. The only definitive indication that no further catalogs - can be fetched is when the next_page_token is unset from the - response. - - name: page_token + type: string + description: Opaque pagination token from a previous request. + - name: parent in: query required: false schema: type: string - description: Opaque pagination token to go to next page based on previous query. + description: >- + Resource name of the parent schema to list within, as + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: view + in: query + required: false + schema: + $ref: '#/components/schemas/ListMcpServicesRequestView' + description: View selector controlling which fields are populated per row. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListCatalogsResponse' + $ref: '#/components/schemas/ListMcpServicesResponse' default: description: Error response content: @@ -367,199 +767,963 @@ paths: type: string message: type: string - x-stackql-sdk-source: CatalogsAPI - /api/2.1/unity-catalog/catalogs/{name}: - delete: - operationId: catalogs_delete + x-stackql-sdk-source: AiGatewayAPI + /api/2.1/unity-catalog/model-provider-services: + post: + operationId: ai_gateway_create_model_provider_service summary: >- - Deletes the catalog that matches the supplied name. The caller must be a - metastore admin or the owner + Creates a model provider service in a Unity Catalog schema. A model + provider service is a governed tags: - catalog - - catalogs + - ai_gateway description: >- - Deletes the catalog that matches the supplied name. The caller must be a - metastore admin or the owner + Creates a model provider service in a Unity Catalog schema. A model + provider service is a governed - of the catalog. + connection to an external model provider (for example OpenAI, Azure + OpenAI, or Amazon Bedrock) that + model services reference to invoke that provider. The caller supplies + the leaf name in - :param name: str - The name of the catalog. - :param force: bool (optional) - Force deletion even if the catalog is not empty. - parameters: - - name: name - in: path - required: true - schema: - type: string - description: The name of the catalog. - - name: force - in: query - required: false - schema: - type: boolean - description: Force deletion even if the catalog is not empty. - responses: - '200': - description: Success - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: CatalogsAPI - get: - operationId: catalogs_get - summary: >- - Gets the specified catalog in a metastore. The caller must be a - metastore admin, the owner of the - tags: - - catalog - - catalogs - description: >- - Gets the specified catalog in a metastore. The caller must be a - metastore admin, the owner of the + ``model_provider_service_id``. - catalog, or a user that has the **USE_CATALOG** privilege set for their - account. + You must be the owner of the parent schema or have the + ``CREATE_SERVICE`` and ``USE_SCHEMA`` - :param name: str - The name of the catalog. - :param include_browse: bool (optional) - Whether to include catalogs in the response for which the principal can only access selective - metadata for + privileges on the parent schema and ``USE_CATALOG`` on the parent + catalog. - :returns: :class:`CatalogInfo` + + :param model_provider_service: :class:`ModelProviderService` + The model provider service to create. The server populates ``name`` from ``parent`` + + ``model_provider_service_id``; clients should leave it unset. + :param parent: str + Resource name of the parent schema. Format: ``schemas/{catalog}.{schema}``. Each ``{...}`` component + is capped at 255 characters individually. + :param model_provider_service_id: str + Leaf identifier for the provider service (the unqualified name within the parent schema, e.g. + "openai_prod"). + + :returns: :class:`ModelProviderService` parameters: - - name: name - in: path + - name: parent + in: query required: true schema: type: string - description: The name of the catalog. - - name: include_browse + description: >- + Resource name of the parent schema. Format: + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: model_provider_service_id in: query - required: false + required: true schema: - type: boolean + type: string description: >- - Whether to include catalogs in the response for which the principal - can only access selective metadata for - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/CatalogInfo' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: CatalogsAPI - patch: - operationId: catalogs_update - summary: >- - Updates the catalog that matches the supplied name. The caller must be - either the owner of the - tags: - - catalog - - catalogs - description: >- - Updates the catalog that matches the supplied name. The caller must be - either the owner of the - - catalog, or a metastore admin (when changing the owner field of the - catalog). - - - :param name: str - The name of the catalog. - :param comment: str (optional) - User-provided free-form text description. - :param enable_predictive_optimization: - :class:`EnablePredictiveOptimization` (optional) - Whether predictive optimization should be enabled for this object and objects under it. - :param isolation_mode: :class:`CatalogIsolationMode` (optional) - Whether the current securable is accessible from all workspaces or a specific set of workspaces. - :param new_name: str (optional) - New name for the catalog. - :param options: Dict[str,str] (optional) - A map of key-value properties attached to the securable. - :param owner: str (optional) - Username of current owner of catalog. - :param properties: Dict[str,str] (optional) - A map of key-value properties attached to the securable. - - :returns: :class:`CatalogInfo` - parameters: - - name: name - in: path - required: true - schema: - type: string - description: The name of the catalog. + Leaf identifier for the provider service (the unqualified name + within the parent schema, e.g. "openai_prod"). requestBody: content: application/json: schema: type: object properties: - comment: - type: string - description: User-provided free-form text description. - enable_predictive_optimization: - type: string - x-enum: - - DISABLE - - ENABLE - - INHERIT - description: >- - Whether predictive optimization should be enabled for this - object and objects under it. - isolation_mode: - type: string - x-enum: - - ISOLATED - - OPEN - description: >- - Whether the current securable is accessible from all - workspaces or a specific set of workspaces. - new_name: - type: string - description: New name for the catalog. - options: - type: object - description: A map of key-value properties attached to the securable. - owner: - type: string - description: Username of current owner of catalog. - properties: + model_provider_service: type: object - description: A map of key-value properties attached to the securable. + properties: + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through + the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + type: object + properties: + allow_all_targets: + type: boolean + description: >- + When true, accepts any model exposed by the upstream + provider; ``targets`` is not required and does not + restrict routability. When false, only models listed + in ``targets`` are routable. + amazon_bedrock: + type: object + properties: + direct: + type: object + properties: + aws_access_key: + type: object + properties: + access_key_id: + type: string + description: >- + AWS access key ID. Required on Create + when using access-key auth. Treated as + username-equivalent (not a secret + value): round-trips on reads and is + scrubbed from audit logs. + secret_access_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + AWS secret access key paired with + ``access_key_id``. Required on Create + when using access-key auth. Supplied as + inline plaintext via + ``ProviderSecret.plaintext``. + description: >- + AWS access-key-pair auth. Mutually exclusive + with ``service_credential``. Supersedes the + flat ``aws_access_key_id`` / + ``aws_secret_access_key`` fields. + aws_access_key_id: + type: string + description: >- + Deprecated flat AWS access key ID. + Superseded by + ``aws_access_key.access_key_id``. Kept for + one migration cycle; the handler mirrors it + to/from ``aws_access_key``. Treated as + username-equivalent (not a secret value): + round-trips on reads and is scrubbed from + audit logs. + aws_secret_access_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Deprecated flat AWS secret access key. + Superseded by + ``aws_access_key.secret_access_key``. Kept + for one migration cycle; the handler mirrors + it to/from ``aws_access_key``. Supplied as + inline plaintext via + ``ProviderSecret.plaintext``. + region: + type: string + description: >- + AWS region where the Bedrock endpoint is + hosted (e.g., ``us-east-1``). Required on + Create. + service_credential: + type: object + properties: + name: + type: string + description: >- + Resource name of the bound UC service + credential, in the AIP-122 form + ``credentials/{name}`` (a + metastore-level single-part credential + name). On create the caller supplies the + name here. On read it reflects the + credential's current name at read time. + required: + - name + description: >- + Reference to a UC service credential + authorizing Bedrock requests. On Create the + caller supplies ``service_credential.name`` + in the AIP-122 resource-name form + ``credentials/{name}``. Required on Create + when using UC-service-credential auth; + mutually exclusive with ``aws_access_key``. + The credential is referenced by name; its + value is not carried here. On read the + resolved ``id`` and ``is_deleted`` are also + populated. Only supported on AWS-hosted + workspaces; Create requests from other + clouds are rejected with + INVALID_PARAMETER_VALUE. + description: |- + Direct form of Amazon Bedrock provider config. + + Authentication is one of two mutually exclusive modes, exactly one of which must be supplied on + Create: + + - Access keys: set ``aws_access_key``, leave ``service_credential`` unset. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``aws_access_key`` unset. The credential value lives in UC and + is referenced by name, not held on this message. Setting more than one mode is rejected. + description: Amazon Bedrock provider configuration. + anthropic: + type: object + properties: + direct: + type: object + properties: + api_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Anthropic API key. Required on Create. Sent + as the ``x-api-key`` header on outbound + requests. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + description: >- + Direct (inline-credentials) form: caller + supplies the API key in the request body. + Required on Create unless ``relayed`` is set. + relayed: + type: object + properties: + plan_type: + type: string + x-enum: + - ANTHROPIC_RELAYED_PLAN_TYPE_MAX + - >- + ANTHROPIC_RELAYED_PLAN_TYPE_TEAM_ENTERPRISE + description: >- + Which Anthropic subscription tier the + relayed token belongs to. Optional; when + unset the MPS gets the full governance + surface (see TEAM_ENTERPRISE). Immutable + after Create, so the tier cannot be flipped + in place. + description: >- + Relayed (credential-less) form: no Anthropic + credential is stored. Each inference request + instead carries the caller's own OAuth token, + which the platform forwards to Anthropic on + outbound requests. Mutually exclusive with + ``direct``; no ``api_key`` is required or + persisted. + description: >- + Anthropic provider configuration. Exactly one of + ``direct`` or ``relayed`` must be set on + Create; the two are mutually exclusive. + azure_openai: + type: object + properties: + direct: + type: object + properties: + api_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Azure OpenAI API key. Mutually exclusive + with the Entra and service-credential modes. + Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + base_url: + type: string + description: >- + Full Azure OpenAI endpoint base URL, e.g. + ``https://myresource.openai.azure.com``. + Required on Create. + client_id: + type: string + description: >- + Deprecated flat Entra client ID. Superseded + by ``entra_service_principal.client_id``. + Kept for one migration cycle; the handler + mirrors it to/from + ``entra_service_principal``. + client_secret: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Deprecated flat Entra client secret. + Superseded by + ``entra_service_principal.client_secret``. + Kept for one migration cycle; the handler + mirrors it to/from + ``entra_service_principal``. Supplied as + inline plaintext via + ``ProviderSecret.plaintext``. + entra_service_principal: + type: object + properties: + client_id: + type: string + description: >- + Entra ID client (application) ID. + Required on Create. + client_secret: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Entra ID client secret. Supplied as + inline plaintext via + ``ProviderSecret.plaintext``. + tenant_id: + type: string + description: >- + Entra ID (Azure AD) tenant ID. Required + on Create. + description: >- + Entra ID (service principal) auth. Mutually + exclusive with ``api_key`` and + ``service_credential``. Supersedes the flat + ``tenant_id`` / ``client_id`` / + ``client_secret`` fields. + service_credential: + type: object + properties: + name: + type: string + description: >- + Resource name of the bound UC service + credential, in the AIP-122 form + ``credentials/{name}`` (a + metastore-level single-part credential + name). On create the caller supplies the + name here. On read it reflects the + credential's current name at read time. + required: + - name + description: >- + Reference to a UC service credential + authorizing Azure OpenAI requests. On Create + the caller supplies + ``service_credential.name`` in the AIP-122 + resource-name form ``credentials/{name}``. + Required on Create when using + UC-service-credential auth; mutually + exclusive with ``api_key`` and + ``entra_service_principal``. The credential + is referenced by name; its value is not + carried here. On read the resolved ``id`` + and ``is_deleted`` are also populated. Only + supported on Azure-hosted workspaces; Create + requests from other clouds are rejected with + INVALID_PARAMETER_VALUE. + tenant_id: + type: string + description: >- + Deprecated flat Entra tenant ID. Superseded + by ``entra_service_principal.tenant_id``. + Kept for one migration cycle; the handler + mirrors it to/from + ``entra_service_principal``. + description: >- + Direct form of Azure OpenAI provider config. + Exactly one of three mutually-exclusive auth + modes + must be supplied on Create: + + - API key: set ``api_key``, leave ``entra_service_principal`` and ``service_credential`` unset. + - Entra ID (service principal): set ``entra_service_principal``, leave ``api_key`` and + ``service_credential`` unset. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``api_key`` and ``entra_service_principal`` unset. The + credential value lives in UC and is referenced by name, not held on this message. Only + supported on Azure-hosted workspaces. Setting more than one mode is rejected. + description: Azure OpenAI provider configuration. + custom: + type: object + properties: + direct: + type: object + properties: + api_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Bearer token forwarded as the + ``Authorization: Bearer ...`` header on + outbound requests. Supplied as inline + plaintext via ``ProviderSecret.plaintext``. + Set this for bearer-token auth. + base_url: + type: string + description: >- + Endpoint URL of the OpenAI-compatible + service (e.g., + ``https://api.example.com/v1``). Required on + Create. + description: |- + Direct form of custom provider config. + + Authentication is one of two mutually exclusive modes, exactly one of which must be supplied on + Create: + + - Bearer: set ``api_key``, leave ``header_auth`` unset. The secret is forwarded as + ``Authorization: Bearer ``. + - Header: set ``header_auth``, leave ``api_key`` unset. The secret is forwarded as + ``: ``. Setting both modes or neither mode is rejected. + description: >- + Custom provider configuration: arbitrary HTTP + endpoint with bearer-token auth. + forward_headers: + type: boolean + description: >- + Whether to forward incoming request headers to the + upstream provider. Applies to managed (multi-model) + requests as well as passthrough requests served by + this provider service. Governance-level decision by + the provider service owner; not selectable per + inference call. + forward_query_parameters: + type: boolean + description: >- + Whether to forward incoming request query parameters + to the upstream provider. Same trust-boundary + semantics as ``forward_headers``. + forward_unmanaged_paths: + type: boolean + description: >- + Whether to forward request paths that fall outside + this service's managed API set to the upstream + provider as opaque passthrough. When true, requests + addressed to subpaths not recognized by the managed + API surface are proxied to the upstream provider + over the same provider connection. When false, only + managed-API paths are served. Governance-level + decision by the provider service owner; expanding + this expands the trust boundary that the + ModelProviderService exposes. + gemini_enterprise: + type: object + properties: + direct: + type: object + properties: + api_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Google Gemini Enterprise API key. Required + on Create. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + project_id: + type: string + description: >- + GCP project ID hosting the Gemini Enterprise + endpoint. Required on Create. + region: + type: string + description: >- + GCP region of the Gemini Enterprise endpoint + (e.g., ``us-central1``). Required on Create. + description: >- + Direct form of Gemini Enterprise provider + config. + description: Gemini Enterprise provider configuration. + inference_table: + type: object + properties: + parent: + type: string + description: >- + Parent UC schema where the inference table is + created. Format: ``schemas/{catalog}.{schema}``. + Set at create time and immutable thereafter; + changing it on an existing service is rejected. + disabled: + type: boolean + description: >- + Indicates whether payload logging is disabled + (opt-out). Unset means that payload logging is + active (the on-by-default state coincides with + the proto zero-value, so the server never fills + this field for a client that leaves it unset). + Set ``disabled = true`` to pause runtime logging + while keeping the sub-message attached + (preserving ``parent`` and ``table_name_prefix`` + for a later flip back to active). ``parent`` + remains required either way. + is_deleted: + type: boolean + description: >- + True when the bound inference TABLE has been + deleted but the parent service still references + it. The dangling reference is surfaced (not + silently dropped) so callers can see the broken + dependency. AI Gateway payload logging fails + closed in this state. + table: + type: string + description: >- + Resolved UC table for payload logs. Format: + ``tables/{catalog}.{schema}.{table}``. + table_name_prefix: + type: string + description: >- + Prefix for the inference-table's UC-registered + name. The actual leaf name UC stores is + ``_payload``; the + ``_payload`` suffix is appended automatically. + To find the actual UC table after Create, read + the ``table`` field on the response. Defaults to + ``_payload`` when unset. Set + at create time and immutable thereafter; + changing it on an existing service is rejected. + required: + - parent + description: >- + Inference table configuration for payload logging + when this provider service is invoked directly. When + it is invoked through a model service, the model + service's own inference table captures the + invocation instead. Mirrors + ``ModelServiceConfig.inference_table`` / + ``AgentServiceConfig.inference_table``. + microsoft_foundry: + type: object + properties: + direct: + type: object + properties: + api_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Microsoft AI Foundry API key. Mutually + exclusive with the Entra and + service-credential modes. Supplied as inline + plaintext via ``ProviderSecret.plaintext``. + base_url: + type: string + description: >- + Microsoft AI Foundry endpoint URL. Required + on Create. + client_id: + type: string + description: >- + Deprecated flat Entra client ID. Superseded + by ``entra_service_principal.client_id``. + Kept for one migration cycle; the handler + mirrors it to/from + ``entra_service_principal``. + client_secret: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Deprecated flat Entra client secret. + Superseded by + ``entra_service_principal.client_secret``. + Kept for one migration cycle; the handler + mirrors it to/from + ``entra_service_principal``. Supplied as + inline plaintext via + ``ProviderSecret.plaintext``. + entra_service_principal: + type: object + properties: + client_id: + type: string + description: >- + Entra ID client (application) ID. + Required on Create. + client_secret: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + Entra ID client secret. Supplied as + inline plaintext via + ``ProviderSecret.plaintext``. + tenant_id: + type: string + description: >- + Entra ID (Azure AD) tenant ID. Required + on Create. + description: >- + Entra ID (service principal) auth. Mutually + exclusive with ``api_key`` and + ``service_credential``. Supersedes the flat + ``tenant_id`` / ``client_id`` / + ``client_secret`` fields. + service_credential: + type: object + properties: + name: + type: string + description: >- + Resource name of the bound UC service + credential, in the AIP-122 form + ``credentials/{name}`` (a + metastore-level single-part credential + name). On create the caller supplies the + name here. On read it reflects the + credential's current name at read time. + required: + - name + description: >- + Reference to a UC service credential + authorizing Microsoft Foundry requests. On + Create the caller supplies + ``service_credential.name`` in the AIP-122 + resource-name form ``credentials/{name}``. + Required on Create when using + UC-service-credential auth; mutually + exclusive with ``api_key`` and + ``entra_service_principal``. The credential + is referenced by name; its value is not + carried here. On read the resolved ``id`` + and ``is_deleted`` are also populated. Only + supported on Azure-hosted workspaces; Create + requests from other clouds are rejected with + INVALID_PARAMETER_VALUE. + tenant_id: + type: string + description: >- + Deprecated flat Entra tenant ID. Superseded + by ``entra_service_principal.tenant_id``. + Kept for one migration cycle; the handler + mirrors it to/from + ``entra_service_principal``. + description: >- + Direct form of Microsoft Foundry provider + config. + + Authentication is one of three mutually exclusive modes, exactly one of which must be supplied + on Create: + + - API key: set ``api_key``, leave ``entra_service_principal`` and ``service_credential`` unset. + - Entra ID (service principal): set ``entra_service_principal``, leave ``api_key`` and + ``service_credential`` unset. AI Gateway exchanges these for an Entra bearer token on outbound + requests via the OAuth2 client-credentials grant. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``api_key`` and ``entra_service_principal`` unset. The + credential value lives in UC and is referenced by name, not held on this message. Only + supported on Azure-hosted workspaces. Setting more than one mode is rejected. + description: Microsoft Foundry provider configuration. + openai: + type: object + properties: + direct: + type: object + properties: + api_key: + type: object + properties: + plaintext: + type: string + description: >- + Inline plaintext credential. INPUT_ONLY: + the value never round-trips on reads. + Get and List responses omit + ``plaintext``; the field's presence in + the read shape only indicates that a + secret is configured. + description: >- + OpenAI API key. Required on Create. Supplied + as inline plaintext via + ``ProviderSecret.plaintext``. + base_url: + type: string + description: >- + Optional custom base URL. Defaults to + ``https://api.openai.com/v1``. Use for + OpenAI-API-compatible third-party endpoints + or in-network proxies. + organization: + type: string + description: >- + Optional OpenAI organization ID. When set, + the platform forwards it as the + ``OpenAI-Organization`` header. + description: >- + Direct (inline-credentials) form of the OpenAI + provider config. + description: OpenAI provider configuration. + provider_type: + type: string + x-enum: + - EXTERNAL_MODEL_PROVIDER_TYPE_AMAZON_BEDROCK + - EXTERNAL_MODEL_PROVIDER_TYPE_ANTHROPIC + - EXTERNAL_MODEL_PROVIDER_TYPE_AZURE_OPENAI + - EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM + - EXTERNAL_MODEL_PROVIDER_TYPE_GEMINI_ENTERPRISE + - EXTERNAL_MODEL_PROVIDER_TYPE_MICROSOFT_FOUNDRY + - EXTERNAL_MODEL_PROVIDER_TYPE_OPENAI + description: >- + Provider type discriminator. Required at create + time; immutable after. Determines which variant of + the ``provider`` oneof must be set. May not be + changed via Update; attempts to include + ``config.provider_type`` in + ``UpdateModelProviderServiceRequest.update_mask`` + are rejected. Required on CreateModelProviderService + and immutable thereafter. + rate_limits: + type: array + items: + type: object + properties: + key: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: >- + Scope key. Determines whether ``principal`` is + required. + renewal_period: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period. + principal: + type: string + description: >- + Principal this limit applies to: user email, + group name, or service principal application + ID. Required unless ``key`` is + ``RATE_LIMIT_KEY_SERVICE``, + ``RATE_LIMIT_KEY_USER_DEFAULT``, or + ``RATE_LIMIT_KEY_REQUEST_TAG`` (which must not + set a principal). + request_tag_key: + type: string + description: >- + Request tag key this limit applies to. + Required when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden + otherwise. + request_tag_value: + type: string + description: >- + Request tag value this limit applies to. Only + valid when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``. Leave unset to + apply the limit to every value of + ``request_tag_key`` (an any-value default); a + set value is a specific override for that + value. + requests: + type: integer + description: >- + Max requests allowed within a renewal period. + Leave unset for no request limit. + tokens: + type: integer + description: >- + Max tokens allowed within a renewal period. + Leave unset for no token limit. + required: + - key + - renewal_period + description: >- + A rate limit applied to service requests. Leave + ``requests`` or ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. + description: >- + Rate limits applied when this provider service is + invoked directly. When it is invoked through a model + service, the model service's own ``rate_limits`` + apply instead. Mirrors + ``ModelServiceConfig.rate_limits`` / + ``McpServiceConfig.rate_limits``. + targets: + type: array + items: + type: object + properties: + model: + type: string + description: >- + Provider-side model identifier (e.g. "gpt-5", + "claude-opus-4-7"). This is a string on the + LLM provider's side, not a UC entity. The UC + governance hook for external destinations is + the ModelProviderService referenced by + ``ExternalModelConfig.model_provider_service``, + not the model itself. + native_api_types: + type: array + items: + type: string + description: >- + Provider-native API types the model supports + (e.g. "openai/v1/chat/completions"). Used by + the platform for request/response translation + from the unified API type. At most 64 entries + of at most 256 characters each; the list is + persisted into the destination binding's + bounded storage envelope. + required: + - model + description: >- + Model target configuration for an external model + destination. + description: >- + Routing targets this provider service exposes + (provider-side model identifier + unified API types + per entry). Required (>=1) when ``allow_all_targets + = false``; optional and additive when + ``allow_all_targets = true``. References from + ``ExternalModelConfig.target`` must match an entry + here unless ``allow_all_targets = true``. + description: >- + Behavioral configuration: provider connection, model + catalog, and passthrough policy. See + ``ModelProviderServiceConfig`` for the per-field + contract. Required on CreateModelProviderService; on + Update it is required only when ``config`` (or a + ``config.*`` subpath) appears in ``update_mask``. + create_time: + type: string + format: date-time + description: When the provider service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: >- + The resolved owner of the model provider service. Falls + back to the caller's identity when ``owner`` is not + explicitly set on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated + from the entity's state and returned on every read. To + use it as an if-match precondition on a mutation, echo + the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects + the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the provider service. + name: + type: string + description: >- + Resource name of the provider service. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at 255 characters + individually. Server-derived on Create from ``parent`` + + ``model_provider_service_id``; required and immutable on + Update/Get/Delete. + owner: + type: string + description: >- + The owner of the model provider service. Write-only; + read owner via effective_owner. + update_time: + type: string + format: date-time + description: When the provider service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: >- + The model provider service to create. The server populates + ``name`` from ``parent`` + ``model_provider_service_id``; + clients should leave it unset. + required: + - model_provider_service responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/CatalogInfo' + $ref: '#/components/schemas/ModelProviderService' default: description: Error response content: @@ -571,94 +1735,94 @@ paths: type: string message: type: string - x-stackql-sdk-source: CatalogsAPI - /api/2.1/unity-catalog/connections: - post: - operationId: connections_create - summary: Creates a new connection + x-stackql-sdk-source: AiGatewayAPI + get: + operationId: ai_gateway_list_model_provider_services + summary: >- + Lists the model provider services in a Unity Catalog schema. Provide + ``parent`` as tags: - catalog - - connections + - ai_gateway description: >- - Creates a new connection + Lists the model provider services in a Unity Catalog schema. Provide + ``parent`` as + ``schemas/{catalog}.{schema}``. Results are paginated; pass the returned + ``next_page_token`` to fetch - Creates a new connection to an external data source. It allows users to - specify connection details and + subsequent pages. - configurations for interaction with the external server. + Requires ``USE_CATALOG`` on the parent catalog and ``USE_SCHEMA`` on the + parent schema. Only model - :param name: str - Name of the connection. - :param connection_type: :class:`ConnectionType` - The type of connection. - :param options: Dict[str,str] - A map of key-value properties attached to the securable. - :param comment: str (optional) - User-provided free-form text description. - :param properties: Dict[str,str] (optional) - A map of key-value properties attached to the securable. - :param read_only: bool (optional) - If the connection is read only. + provider services the caller can access (as owner or through + ``EXECUTE``, ``READ_METADATA``, or - :returns: :class:`ConnectionInfo` - requestBody: - content: - application/json: - schema: - type: object - properties: - name: - type: string - description: Name of the connection. - connection_type: - type: string - x-enum: - - BIGQUERY - - DATABRICKS - - GA4_RAW_DATA - - GLUE - - HIVE_METASTORE - - HTTP - - MYSQL - - ORACLE - - POSTGRESQL - - POWER_BI - - REDSHIFT - - SALESFORCE - - SALESFORCE_DATA_CLOUD - - SERVICENOW - - SNOWFLAKE - - SQLDW - - SQLSERVER - - TERADATA - - UNKNOWN_CONNECTION_TYPE - - WORKDAY_RAAS - description: The type of connection. - options: - type: object - description: A map of key-value properties attached to the securable. - comment: - type: string - description: User-provided free-form text description. - properties: - type: object - description: A map of key-value properties attached to the securable. - read_only: - type: boolean - description: If the connection is read only. - required: - - name - - connection_type - - options + ``MANAGE``) are returned. + + + :param include_browse: bool (optional) + Whether to include provider services for which the principal can only access selective metadata. + :param page_size: int (optional) + Maximum number of provider services to return. Defaults to 100 when unset or 0; the maximum is 1000. + Use ``next_page_token`` to retrieve additional pages. + :param page_token: str (optional) + Opaque pagination token from a previous request. + :param parent: str (optional) + Resource name of the parent schema to list within, as ``schemas/{catalog}.{schema}``. Each ``{...}`` + component is capped at 255 characters individually. + :param view: :class:`ListModelProviderServicesRequestView` (optional) + View selector controlling which fields are populated per row. + + :returns: Iterator over :class:`ModelProviderService` + parameters: + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include provider services for which the principal can + only access selective metadata. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Maximum number of provider services to return. Defaults to 100 when + unset or 0; the maximum is 1000. Use ``next_page_token`` to retrieve + additional pages. + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token from a previous request. + - name: parent + in: query + required: false + schema: + type: string + description: >- + Resource name of the parent schema to list within, as + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: view + in: query + required: false + schema: + $ref: '#/components/schemas/ListModelProviderServicesRequestView' + description: View selector controlling which fields are populated per row. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ConnectionInfo' + $ref: '#/components/schemas/ListModelProviderServicesResponse' default: description: Error response content: @@ -670,68 +1834,634 @@ paths: type: string message: type: string - x-stackql-sdk-source: ConnectionsAPI - get: - operationId: connections_list - summary: List all connections. + x-stackql-sdk-source: AiGatewayAPI + /api/2.1/unity-catalog/model-services: + post: + operationId: ai_gateway_create_model_service + summary: >- + Creates a model service in a Unity Catalog schema. A model service is a + governed AI Gateway endpoint tags: - catalog - - connections + - ai_gateway description: >- - List all connections. - - - NOTE: we recommend using max_results=0 to use the paginated version of - this API. Unpaginated calls + Creates a model service in a Unity Catalog schema. A model service is a + governed AI Gateway endpoint - will be deprecated soon. + that routes inference requests to one or more model destinations. The + caller supplies the leaf name in + ``model_service_id``. - PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page - may contain zero results while - still providing a next_page_token. Clients must continue reading pages - until next_page_token is + You must be the owner of the parent schema or have the + ``CREATE_SERVICE`` and ``USE_SCHEMA`` - absent, which is the only indication that the end of results has been - reached. + privileges on the parent schema and ``USE_CATALOG`` on the parent + catalog. - :param max_results: int (optional) - Maximum number of connections to return. - If not set, all connections are returned (not - recommended). - when set to a value greater than 0, the page length is the minimum of this value and - a server configured value; - when set to 0, the page length is set to a server configured value - (recommended); - when set to a value less than 0, an invalid parameter error is returned; - :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. + :param model_service: :class:`ModelService` + The model service to create. The server populates ``name`` from ``parent`` + ``model_service_id``; + clients should leave it unset. + :param parent: str + Resource name of the parent schema. Format: ``schemas/{catalog}.{schema}``. Each ``{...}`` component + is capped at 255 characters individually. + :param model_service_id: str + Leaf identifier for the model service (the unqualified name within the parent schema, e.g. + "my_model_service"). - :returns: Iterator over :class:`ConnectionInfo` + :returns: :class:`ModelService` parameters: - - name: max_results + - name: parent in: query - required: false + required: true schema: - type: integer + type: string description: >- - Maximum number of connections to return. - If not set, all - connections are returned (not recommended). - when set to a value - greater than 0, the page length is the minimum of this value and a - server configured value; - when set to 0, the page length is set to - a server configured value (recommended); - when set to a value less - than 0, an invalid parameter error is returned; - - name: page_token + Resource name of the parent schema. Format: + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: model_service_id in: query - required: false + required: true schema: type: string - description: Opaque pagination token to go to next page based on previous query. + description: >- + Leaf identifier for the model service (the unqualified name within + the parent schema, e.g. "my_model_service"). + requestBody: + content: + application/json: + schema: + type: object + properties: + model_service: + type: object + properties: + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through + the BROWSE privilege. + comment: + type: string + description: User-provided description. + config: + type: object + properties: + inference_table: + type: object + properties: + parent: + type: string + description: >- + Parent UC schema where the inference table is + created. Format: ``schemas/{catalog}.{schema}``. + Set at create time and immutable thereafter; + changing it on an existing service is rejected. + disabled: + type: boolean + description: >- + Indicates whether payload logging is disabled + (opt-out). Unset means that payload logging is + active (the on-by-default state coincides with + the proto zero-value, so the server never fills + this field for a client that leaves it unset). + Set ``disabled = true`` to pause runtime logging + while keeping the sub-message attached + (preserving ``parent`` and ``table_name_prefix`` + for a later flip back to active). ``parent`` + remains required either way. + is_deleted: + type: boolean + description: >- + True when the bound inference TABLE has been + deleted but the parent service still references + it. The dangling reference is surfaced (not + silently dropped) so callers can see the broken + dependency. AI Gateway payload logging fails + closed in this state. + table: + type: string + description: >- + Resolved UC table for payload logs. Format: + ``tables/{catalog}.{schema}.{table}``. + table_name_prefix: + type: string + description: >- + Prefix for the inference-table's UC-registered + name. The actual leaf name UC stores is + ``_payload``; the + ``_payload`` suffix is appended automatically. + To find the actual UC table after Create, read + the ``table`` field on the response. Defaults to + ``_payload`` when unset. Set + at create time and immutable thereafter; + changing it on an existing service is rejected. + required: + - parent + description: Inference table config for payload logging. + rate_limits: + type: array + items: + type: object + properties: + key: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: >- + Scope key. Determines whether ``principal`` is + required. + renewal_period: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period. + principal: + type: string + description: >- + Principal this limit applies to: user email, + group name, or service principal application + ID. Required unless ``key`` is + ``RATE_LIMIT_KEY_SERVICE``, + ``RATE_LIMIT_KEY_USER_DEFAULT``, or + ``RATE_LIMIT_KEY_REQUEST_TAG`` (which must not + set a principal). + request_tag_key: + type: string + description: >- + Request tag key this limit applies to. + Required when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden + otherwise. + request_tag_value: + type: string + description: >- + Request tag value this limit applies to. Only + valid when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``. Leave unset to + apply the limit to every value of + ``request_tag_key`` (an any-value default); a + set value is a specific override for that + value. + requests: + type: integer + description: >- + Max requests allowed within a renewal period. + Leave unset for no request limit. + tokens: + type: integer + description: >- + Max tokens allowed within a renewal period. + Leave unset for no token limit. + required: + - key + - renewal_period + description: >- + A rate limit applied to service requests. Leave + ``requests`` or ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. + description: >- + Rate limits applied to requests routed through this + model service. + routing: + type: object + properties: + destinations: + type: array + items: + type: object + properties: + name: + type: string + description: >- + User-facing label for this destination, + used in routing references. + destination_type: + type: string + x-enum: + - >- + DESTINATION_TYPE_EXTERNAL_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PAY_PER_TOKEN_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PROVISIONED_THROUGHPUT_FOUNDATION_MODEL + description: >- + Backing-model category. Determines which + oneof variant is populated. + external_model_config: + type: object + properties: + model_provider_service: + type: string + description: >- + Resource name of the governed + ModelProviderService that owns provider + auth and provider-specific + configuration. The referenced + ModelProviderService also carries the + provider type, so this message does not + surface it directly. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at + 255 characters individually. + target: + type: object + properties: + model: + type: string + description: >- + Provider-side model identifier (e.g. + "gpt-5", "claude-opus-4-7"). This is a + string on the LLM provider's side, not a + UC entity. The UC governance hook for + external destinations is the + ModelProviderService referenced by + ``ExternalModelConfig.model_provider_service``, + not the model itself. + native_api_types: + type: array + items: + type: string + description: >- + Provider-native API types the model + supports (e.g. + "openai/v1/chat/completions"). Used by + the platform for request/response + translation from the unified API type. + At most 64 entries of at most 256 + characters each; the list is persisted + into the destination binding's bounded + storage envelope. + required: + - model + description: >- + Routing target for the destination: the + provider-side model selected from the + referenced ModelProviderService's + ``targets`` catalog, plus the unified + API types the platform should translate + to/from at request time. + required: + - model_provider_service + - target + description: >- + Configuration for an + external-foundation-model destination. + Provider auth and provider-specific + cloud configuration are owned by a separate, governed ModelProviderService entity referenced via + ``model_provider_service``; the platform resolves the provider at invocation time. + is_deleted: + type: boolean + description: >- + True when the destination's backing UC + entity (MODEL for foundation-model + destinations, MODEL_PROVIDER_SERVICE for + external destinations) has been deleted + but the destination row still references + it. The dangling destination is surfaced + (not silently dropped) so callers can see + the broken routing. Inference traffic + through this destination fails closed + (BAD_REQUEST / FAILED_PRECONDITION). + pay_per_token_config: + type: object + properties: + model: + type: string + description: >- + Resource name of the UC model. Format: + ``models/{catalog}.{schema}.{model}``. + required: + - model + description: >- + Configuration for a pay-per-token + foundation-model destination. Identifies + the foundation model + by its UC resource name; the platform resolves it to a Model Serving endpoint at request time. + provisioned_throughput_config: + type: object + properties: + model_serving_endpoint: + type: string + description: >- + Name of the backing Model Serving + endpoint serving the provisioned- + throughput foundation model, as the + AIP-122 typed resource name + ``serving-endpoints/{name}``. The same + UC model can be served on multiple Model + Serving endpoints (different throughput + / region / config); the caller picks + which one this destination routes to. + The endpoint must exist at create time. + model: + type: string + description: >- + UC model FQN of the model served by the + backing endpoint (e.g., + ``system.ai.databricks-claude-opus-4-6``). + Resolved from Model Serving at + Create/Update time. + required: + - model_serving_endpoint + description: >- + Configuration for a provisioned-throughput + foundation-model destination. References a + pre-existing Model Serving endpoint that serves the model; sizing (provisioned throughput, burst + scaling, model version) is owned by the Model Serving endpoint itself, not by this message. + traffic_percentage: + type: integer + description: >- + Share of traffic sent to this destination, + 0-100. Optional on fallback destinations; + see FallbackConfig. + required: + - name + - destination_type + description: >- + A destination the model service can route + traffic to. Exactly one of the per-type + configs inside + ``type_config`` must be set, and it must match ``destination_type``. + description: >- + Primary routing destinations. At most 10 are + allowed. At least one is required on + CreateModelService; on UpdateModelService it is + required only when ``config.routing`` (or a + ``config.routing.*`` subpath) appears in + ``update_mask``. + fallback: + type: object + properties: + destinations: + type: array + items: + type: object + properties: + name: + type: string + description: >- + User-facing label for this destination, + used in routing references. + destination_type: + type: string + x-enum: + - >- + DESTINATION_TYPE_EXTERNAL_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PAY_PER_TOKEN_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PROVISIONED_THROUGHPUT_FOUNDATION_MODEL + description: >- + Backing-model category. Determines which + oneof variant is populated. + external_model_config: + type: object + properties: + model_provider_service: + type: string + description: >- + Resource name of the governed + ModelProviderService that owns provider + auth and provider-specific + configuration. The referenced + ModelProviderService also carries the + provider type, so this message does not + surface it directly. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at + 255 characters individually. + target: + type: object + properties: + model: + type: string + description: >- + Provider-side model identifier (e.g. + "gpt-5", "claude-opus-4-7"). This is a + string on the LLM provider's side, not a + UC entity. The UC governance hook for + external destinations is the + ModelProviderService referenced by + ``ExternalModelConfig.model_provider_service``, + not the model itself. + native_api_types: + type: array + items: + type: string + description: >- + Provider-native API types the model + supports (e.g. + "openai/v1/chat/completions"). Used by + the platform for request/response + translation from the unified API type. + At most 64 entries of at most 256 + characters each; the list is persisted + into the destination binding's bounded + storage envelope. + required: + - model + description: >- + Routing target for the destination: the + provider-side model selected from the + referenced ModelProviderService's + ``targets`` catalog, plus the unified + API types the platform should translate + to/from at request time. + required: + - model_provider_service + - target + description: >- + Configuration for an + external-foundation-model destination. + Provider auth and provider-specific + cloud configuration are owned by a separate, governed ModelProviderService entity referenced via + ``model_provider_service``; the platform resolves the provider at invocation time. + is_deleted: + type: boolean + description: >- + True when the destination's backing UC + entity (MODEL for foundation-model + destinations, MODEL_PROVIDER_SERVICE for + external destinations) has been deleted + but the destination row still references + it. The dangling destination is surfaced + (not silently dropped) so callers can + see the broken routing. Inference + traffic through this destination fails + closed (BAD_REQUEST / + FAILED_PRECONDITION). + pay_per_token_config: + type: object + properties: + model: + type: string + description: >- + Resource name of the UC model. Format: + ``models/{catalog}.{schema}.{model}``. + required: + - model + description: >- + Configuration for a pay-per-token + foundation-model destination. Identifies + the foundation model + by its UC resource name; the platform resolves it to a Model Serving endpoint at request time. + provisioned_throughput_config: + type: object + properties: + model_serving_endpoint: + type: string + description: >- + Name of the backing Model Serving + endpoint serving the provisioned- + throughput foundation model, as the + AIP-122 typed resource name + ``serving-endpoints/{name}``. The same + UC model can be served on multiple Model + Serving endpoints (different throughput + / region / config); the caller picks + which one this destination routes to. + The endpoint must exist at create time. + model: + type: string + description: >- + UC model FQN of the model served by the + backing endpoint (e.g., + ``system.ai.databricks-claude-opus-4-6``). + Resolved from Model Serving at + Create/Update time. + required: + - model_serving_endpoint + description: >- + Configuration for a + provisioned-throughput foundation-model + destination. References a + pre-existing Model Serving endpoint that serves the model; sizing (provisioned throughput, burst + scaling, model version) is owned by the Model Serving endpoint itself, not by this message. + traffic_percentage: + type: integer + description: >- + Share of traffic sent to this + destination, 0-100. Optional on fallback + destinations; see FallbackConfig. + required: + - name + - destination_type + description: >- + A destination the model service can route + traffic to. Exactly one of the per-type + configs inside + ``type_config`` must be set, and it must match ``destination_type``. + description: >- + Ordered list of fallback destinations. + Traversal is in list order; the attempt + count is the length of the list. At most 5 + are allowed. + description: >- + Fallback routing config, applied after primary + destinations fail. + first_token_timeout: + type: string + description: >- + Timeout for the first token of a streaming + response. If a destination does not return its + first token within this duration, AI Gateway + aborts the attempt and fails over to the next + destination. Applies to streaming requests only. + Leave unset for no first-token timeout. + traffic_splitting: + type: object + properties: {} + description: >- + Marker message selecting request-based traffic + splitting. Traffic is distributed according to + each destination's traffic_percentage value; no + configuration lives on this message itself. + description: >- + Routing configuration: destinations, routing + strategy, and fallback. + description: >- + Operational configuration: destinations, routing, rate + limits, inference table. Required on CreateModelService; + on UpdateModelService it is required only when + ``config`` (or a ``config.*`` subpath) appears in + ``update_mask``. + create_time: + type: string + format: date-time + description: When the model service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: >- + The resolved owner of the ModelService. Falls back to + the caller's identity when ``owner`` is not explicitly + set on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated + from the entity's state and returned on every read. To + use it as an if-match precondition on a mutation, echo + the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects + the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the model service. + name: + type: string + description: >- + Resource name of the model service. Format: + ``model-services/{catalog}.{schema}.{model_service}``. + Each ``{...}`` component is capped at 255 characters + individually. Server-derived on Create from ``parent`` + + ``model_service_id``; required and immutable on + Update/Get/Delete. + owner: + type: string + description: >- + The owner of the model service. Write-only; read owner + via effective_owner. + supported_api_types: + type: array + items: + type: string + description: >- + Unified API types this endpoint supports (e.g. "chat", + "embeddings", "completions"). Derived from the + destinations' backing models / providers at read time. + update_time: + type: string + format: date-time + description: When the model service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: >- + The model service to create. The server populates ``name`` + from ``parent`` + ``model_service_id``; clients should leave + it unset. + required: + - model_service responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListConnectionsResponse' + $ref: '#/components/schemas/ModelService' default: description: Error response content: @@ -743,29 +2473,94 @@ paths: type: string message: type: string - x-stackql-sdk-source: ConnectionsAPI - /api/2.1/unity-catalog/connections/{name}: - delete: - operationId: connections_delete - summary: Deletes the connection that matches the supplied name. + x-stackql-sdk-source: AiGatewayAPI + get: + operationId: ai_gateway_list_model_services + summary: >- + Lists the model services in a Unity Catalog schema. Provide ``parent`` + as tags: - catalog - - connections - description: |- - Deletes the connection that matches the supplied name. + - ai_gateway + description: >- + Lists the model services in a Unity Catalog schema. Provide ``parent`` + as - :param name: str - The name of the connection to be deleted. + ``schemas/{catalog}.{schema}``. Results are paginated; pass the returned + ``next_page_token`` to fetch + + subsequent pages. + + + Requires ``USE_CATALOG`` on the parent catalog and ``USE_SCHEMA`` on the + parent schema. Only model + + services the caller can access (as owner or through ``EXECUTE``, + ``READ_METADATA``, or ``MANAGE``) are + + returned. + + + :param include_browse: bool (optional) + Whether to include model services for which the principal can only access selective metadata. + :param page_size: int (optional) + Maximum number of model services to return. Defaults to 100 when unset or 0; the maximum is 1000. + Use ``next_page_token`` to retrieve additional pages. + :param page_token: str (optional) + Opaque pagination token from a previous request. + :param parent: str (optional) + Resource name of the parent schema to list within, as ``schemas/{catalog}.{schema}``. Each ``{...}`` + component is capped at 255 characters individually. + :param view: :class:`ListModelServicesRequestView` (optional) + View selector controlling which fields are populated per row. + + :returns: Iterator over :class:`ModelService` parameters: - - name: name - in: path - required: true + - name: include_browse + in: query + required: false schema: - type: string - description: The name of the connection to be deleted. + type: boolean + description: >- + Whether to include model services for which the principal can only + access selective metadata. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Maximum number of model services to return. Defaults to 100 when + unset or 0; the maximum is 1000. Use ``next_page_token`` to retrieve + additional pages. + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token from a previous request. + - name: parent + in: query + required: false + schema: + type: string + description: >- + Resource name of the parent schema to list within, as + ``schemas/{catalog}.{schema}``. Each ``{...}`` component is capped + at 255 characters individually. + - name: view + in: query + required: false + schema: + $ref: '#/components/schemas/ListModelServicesRequestView' + description: View selector controlling which fields are populated per row. responses: '200': description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListModelServicesResponse' default: description: Error response content: @@ -777,34 +2572,57 @@ paths: type: string message: type: string - x-stackql-sdk-source: ConnectionsAPI - get: - operationId: connections_get - summary: Gets a connection from it's name. + x-stackql-sdk-source: AiGatewayAPI + /api/2.1/unity-catalog/{name}: + delete: + operationId: ai_gateway_delete_model_service + summary: >- + Deletes the model service identified by its resource name. Optionally + supply an ``etag`` to make the tags: - catalog - - connections - description: |- - Gets a connection from it's name. + - ai_gateway + description: >- + Deletes the model service identified by its resource name. Optionally + supply an ``etag`` to make the - :param name: str - Name of the connection. + delete conditional on the model service not having changed since it was + read. - :returns: :class:`ConnectionInfo` + + You must be the owner of the model service or have ``MANAGE`` on it, + plus ``USE_CATALOG`` on the + + parent catalog and ``USE_SCHEMA`` on the parent schema. + + + :param name: str + Resource name of the model service. Format: ``model-services/{catalog}.{schema}.{model_service}``. + Each ``{...}`` component is capped at 255 characters individually. + :param etag: str (optional) + If-match precondition: when set, the delete proceeds only if the current server-side etag matches. + Empty means unconditional delete. parameters: - name: name in: path required: true schema: type: string - description: Name of the connection. + description: >- + Resource name of the model service. Format: + ``model-services/{catalog}.{schema}.{model_service}``. Each + ``{...}`` component is capped at 255 characters individually. + - name: etag + in: query + required: false + schema: + type: string + description: >- + If-match precondition: when set, the delete proceeds only if the + current server-side etag matches. Empty means unconditional delete. responses: '200': description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ConnectionInfo' default: description: Error response content: @@ -816,57 +2634,56 @@ paths: type: string message: type: string - x-stackql-sdk-source: ConnectionsAPI - patch: - operationId: connections_update - summary: Updates the connection that matches the supplied name. + x-stackql-sdk-source: AiGatewayAPI + get: + operationId: ai_gateway_get_model_service + summary: Returns the model service identified by its resource name. tags: - catalog - - connections - description: |- - Updates the connection that matches the supplied name. + - ai_gateway + description: >- + Returns the model service identified by its resource name. + + + You must be the owner of the model service or have ``EXECUTE``, + ``READ_METADATA``, or ``MANAGE`` on + + it, plus ``USE_CATALOG`` on the parent catalog and ``USE_SCHEMA`` on the + parent schema. + :param name: str - Name of the connection. - :param options: Dict[str,str] - A map of key-value properties attached to the securable. - :param new_name: str (optional) - New name for the connection. - :param owner: str (optional) - Username of current owner of the connection. + Resource name of the model service. Format: ``model-services/{catalog}.{schema}.{model_service}``. + Each ``{...}`` component is capped at 255 characters individually. + :param include_browse: bool (optional) + Whether to include model services for which the principal can only access selective metadata. - :returns: :class:`ConnectionInfo` + :returns: :class:`ModelService` parameters: - name: name in: path required: true schema: type: string - description: Name of the connection. - requestBody: - content: - application/json: - schema: - type: object - properties: - options: - type: object - description: A map of key-value properties attached to the securable. - new_name: - type: string - description: New name for the connection. - owner: - type: string - description: Username of current owner of the connection. - required: - - options + description: >- + Resource name of the model service. Format: + ``model-services/{catalog}.{schema}.{model_service}``. Each + ``{...}`` component is capped at 255 characters individually. + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include model services for which the principal can only + access selective metadata. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ConnectionInfo' + $ref: '#/components/schemas/ModelService' default: description: Error response content: @@ -878,95 +2695,650 @@ paths: type: string message: type: string - x-stackql-sdk-source: ConnectionsAPI - /api/2.1/unity-catalog/entity-tag-assignments: - post: - operationId: entity_tag_assignments_create - summary: Creates a tag assignment for an Unity Catalog entity. + x-stackql-sdk-source: AiGatewayAPI + patch: + operationId: ai_gateway_update_model_service + summary: >- + Updates a model service. Only the fields named in ``update_mask`` are + changed; the resource name is tags: - catalog - - entity_tag_assignments + - ai_gateway description: >- - Creates a tag assignment for an Unity Catalog entity. + Updates a model service. Only the fields named in ``update_mask`` are + changed; the resource name is + immutable. Optionally supply an ``etag`` to make the update conditional + on the model service not - To add tags to Unity Catalog entities, you must own the entity or have - the following privileges: - - - **APPLY TAG** on the entity - **USE SCHEMA** on the entity's parent - schema - **USE CATALOG** on the - - entity's parent catalog - - - To add a governed tag to Unity Catalog entities, you must also have the - **ASSIGN** or **MANAGE** - - permission on the tag policy. See [Manage tag policy permissions]. + having changed since it was read. - [Manage tag policy permissions]: - https://docs.databricks.com/aws/en/admin/tag-policies/manage-permissions + You must be the owner of the model service or have ``MANAGE`` on it, + plus ``USE_CATALOG`` on the - - :param tag_assignment: :class:`EntityTagAssignment` + parent catalog and ``USE_SCHEMA`` on the parent schema. - :returns: :class:`EntityTagAssignment` + :param name: str + Resource name of the model service. Format: ``model-services/{catalog}.{schema}.{model_service}``. + Each ``{...}`` component is capped at 255 characters individually. Server-derived on Create from + ``parent`` + ``model_service_id``; required and immutable on Update/Get/Delete. + :param model_service: :class:`ModelService` + The model service with the updated field values. ``name`` identifies the resource + (``model-services/{catalog}.{schema}.{model_service}``); only fields listed in ``update_mask`` are + applied. + :param update_mask: FieldMask + The list of fields to update. The framework validates each path against the ``model_service`` field + above. Wildcard paths (``paths: ["*"]``) are not supported; list each field path explicitly. + :param etag: str (optional) + If-match precondition: when set, the update proceeds only if the current server-side etag matches. + Empty means an unconditional update. + + :returns: :class:`ModelService` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Resource name of the model service. Format: + ``model-services/{catalog}.{schema}.{model_service}``. Each + ``{...}`` component is capped at 255 characters individually. + Server-derived on Create from ``parent`` + ``model_service_id``; + required and immutable on Update/Get/Delete. + - name: update_mask + in: query + required: true + schema: + type: object + description: >- + The list of fields to update. The framework validates each path + against the ``model_service`` field above. Wildcard paths (``paths: + ["*"]``) are not supported; list each field path explicitly. + - name: etag + in: query + required: false + schema: + type: string + description: >- + If-match precondition: when set, the update proceeds only if the + current server-side etag matches. Empty means an unconditional + update. requestBody: content: application/json: schema: type: object properties: - tag_assignment: + model_service: type: object properties: - entity_name: + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through + the BROWSE privilege. + comment: type: string + description: User-provided description. + config: + type: object + properties: + inference_table: + type: object + properties: + parent: + type: string + description: >- + Parent UC schema where the inference table is + created. Format: ``schemas/{catalog}.{schema}``. + Set at create time and immutable thereafter; + changing it on an existing service is rejected. + disabled: + type: boolean + description: >- + Indicates whether payload logging is disabled + (opt-out). Unset means that payload logging is + active (the on-by-default state coincides with + the proto zero-value, so the server never fills + this field for a client that leaves it unset). + Set ``disabled = true`` to pause runtime logging + while keeping the sub-message attached + (preserving ``parent`` and ``table_name_prefix`` + for a later flip back to active). ``parent`` + remains required either way. + is_deleted: + type: boolean + description: >- + True when the bound inference TABLE has been + deleted but the parent service still references + it. The dangling reference is surfaced (not + silently dropped) so callers can see the broken + dependency. AI Gateway payload logging fails + closed in this state. + table: + type: string + description: >- + Resolved UC table for payload logs. Format: + ``tables/{catalog}.{schema}.{table}``. + table_name_prefix: + type: string + description: >- + Prefix for the inference-table's UC-registered + name. The actual leaf name UC stores is + ``_payload``; the + ``_payload`` suffix is appended automatically. + To find the actual UC table after Create, read + the ``table`` field on the response. Defaults to + ``_payload`` when unset. Set + at create time and immutable thereafter; + changing it on an existing service is rejected. + required: + - parent + description: Inference table config for payload logging. + rate_limits: + type: array + items: + type: object + properties: + key: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: >- + Scope key. Determines whether ``principal`` is + required. + renewal_period: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period. + principal: + type: string + description: >- + Principal this limit applies to: user email, + group name, or service principal application + ID. Required unless ``key`` is + ``RATE_LIMIT_KEY_SERVICE``, + ``RATE_LIMIT_KEY_USER_DEFAULT``, or + ``RATE_LIMIT_KEY_REQUEST_TAG`` (which must not + set a principal). + request_tag_key: + type: string + description: >- + Request tag key this limit applies to. + Required when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden + otherwise. + request_tag_value: + type: string + description: >- + Request tag value this limit applies to. Only + valid when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``. Leave unset to + apply the limit to every value of + ``request_tag_key`` (an any-value default); a + set value is a specific override for that + value. + requests: + type: integer + description: >- + Max requests allowed within a renewal period. + Leave unset for no request limit. + tokens: + type: integer + description: >- + Max tokens allowed within a renewal period. + Leave unset for no token limit. + required: + - key + - renewal_period + description: >- + A rate limit applied to service requests. Leave + ``requests`` or ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. + description: >- + Rate limits applied to requests routed through this + model service. + routing: + type: object + properties: + destinations: + type: array + items: + type: object + properties: + name: + type: string + description: >- + User-facing label for this destination, + used in routing references. + destination_type: + type: string + x-enum: + - >- + DESTINATION_TYPE_EXTERNAL_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PAY_PER_TOKEN_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PROVISIONED_THROUGHPUT_FOUNDATION_MODEL + description: >- + Backing-model category. Determines which + oneof variant is populated. + external_model_config: + type: object + properties: + model_provider_service: + type: string + description: >- + Resource name of the governed + ModelProviderService that owns provider + auth and provider-specific + configuration. The referenced + ModelProviderService also carries the + provider type, so this message does not + surface it directly. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at + 255 characters individually. + target: + type: object + properties: + model: + type: string + description: >- + Provider-side model identifier (e.g. + "gpt-5", "claude-opus-4-7"). This is a + string on the LLM provider's side, not a + UC entity. The UC governance hook for + external destinations is the + ModelProviderService referenced by + ``ExternalModelConfig.model_provider_service``, + not the model itself. + native_api_types: + type: array + items: + type: string + description: >- + Provider-native API types the model + supports (e.g. + "openai/v1/chat/completions"). Used by + the platform for request/response + translation from the unified API type. + At most 64 entries of at most 256 + characters each; the list is persisted + into the destination binding's bounded + storage envelope. + required: + - model + description: >- + Routing target for the destination: the + provider-side model selected from the + referenced ModelProviderService's + ``targets`` catalog, plus the unified + API types the platform should translate + to/from at request time. + required: + - model_provider_service + - target + description: >- + Configuration for an + external-foundation-model destination. + Provider auth and provider-specific + cloud configuration are owned by a separate, governed ModelProviderService entity referenced via + ``model_provider_service``; the platform resolves the provider at invocation time. + is_deleted: + type: boolean + description: >- + True when the destination's backing UC + entity (MODEL for foundation-model + destinations, MODEL_PROVIDER_SERVICE for + external destinations) has been deleted + but the destination row still references + it. The dangling destination is surfaced + (not silently dropped) so callers can see + the broken routing. Inference traffic + through this destination fails closed + (BAD_REQUEST / FAILED_PRECONDITION). + pay_per_token_config: + type: object + properties: + model: + type: string + description: >- + Resource name of the UC model. Format: + ``models/{catalog}.{schema}.{model}``. + required: + - model + description: >- + Configuration for a pay-per-token + foundation-model destination. Identifies + the foundation model + by its UC resource name; the platform resolves it to a Model Serving endpoint at request time. + provisioned_throughput_config: + type: object + properties: + model_serving_endpoint: + type: string + description: >- + Name of the backing Model Serving + endpoint serving the provisioned- + throughput foundation model, as the + AIP-122 typed resource name + ``serving-endpoints/{name}``. The same + UC model can be served on multiple Model + Serving endpoints (different throughput + / region / config); the caller picks + which one this destination routes to. + The endpoint must exist at create time. + model: + type: string + description: >- + UC model FQN of the model served by the + backing endpoint (e.g., + ``system.ai.databricks-claude-opus-4-6``). + Resolved from Model Serving at + Create/Update time. + required: + - model_serving_endpoint + description: >- + Configuration for a provisioned-throughput + foundation-model destination. References a + pre-existing Model Serving endpoint that serves the model; sizing (provisioned throughput, burst + scaling, model version) is owned by the Model Serving endpoint itself, not by this message. + traffic_percentage: + type: integer + description: >- + Share of traffic sent to this destination, + 0-100. Optional on fallback destinations; + see FallbackConfig. + required: + - name + - destination_type + description: >- + A destination the model service can route + traffic to. Exactly one of the per-type + configs inside + ``type_config`` must be set, and it must match ``destination_type``. + description: >- + Primary routing destinations. At most 10 are + allowed. At least one is required on + CreateModelService; on UpdateModelService it is + required only when ``config.routing`` (or a + ``config.routing.*`` subpath) appears in + ``update_mask``. + fallback: + type: object + properties: + destinations: + type: array + items: + type: object + properties: + name: + type: string + description: >- + User-facing label for this destination, + used in routing references. + destination_type: + type: string + x-enum: + - >- + DESTINATION_TYPE_EXTERNAL_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PAY_PER_TOKEN_FOUNDATION_MODEL + - >- + DESTINATION_TYPE_PROVISIONED_THROUGHPUT_FOUNDATION_MODEL + description: >- + Backing-model category. Determines which + oneof variant is populated. + external_model_config: + type: object + properties: + model_provider_service: + type: string + description: >- + Resource name of the governed + ModelProviderService that owns provider + auth and provider-specific + configuration. The referenced + ModelProviderService also carries the + provider type, so this message does not + surface it directly. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at + 255 characters individually. + target: + type: object + properties: + model: + type: string + description: >- + Provider-side model identifier (e.g. + "gpt-5", "claude-opus-4-7"). This is a + string on the LLM provider's side, not a + UC entity. The UC governance hook for + external destinations is the + ModelProviderService referenced by + ``ExternalModelConfig.model_provider_service``, + not the model itself. + native_api_types: + type: array + items: + type: string + description: >- + Provider-native API types the model + supports (e.g. + "openai/v1/chat/completions"). Used by + the platform for request/response + translation from the unified API type. + At most 64 entries of at most 256 + characters each; the list is persisted + into the destination binding's bounded + storage envelope. + required: + - model + description: >- + Routing target for the destination: the + provider-side model selected from the + referenced ModelProviderService's + ``targets`` catalog, plus the unified + API types the platform should translate + to/from at request time. + required: + - model_provider_service + - target + description: >- + Configuration for an + external-foundation-model destination. + Provider auth and provider-specific + cloud configuration are owned by a separate, governed ModelProviderService entity referenced via + ``model_provider_service``; the platform resolves the provider at invocation time. + is_deleted: + type: boolean + description: >- + True when the destination's backing UC + entity (MODEL for foundation-model + destinations, MODEL_PROVIDER_SERVICE for + external destinations) has been deleted + but the destination row still references + it. The dangling destination is surfaced + (not silently dropped) so callers can + see the broken routing. Inference + traffic through this destination fails + closed (BAD_REQUEST / + FAILED_PRECONDITION). + pay_per_token_config: + type: object + properties: + model: + type: string + description: >- + Resource name of the UC model. Format: + ``models/{catalog}.{schema}.{model}``. + required: + - model + description: >- + Configuration for a pay-per-token + foundation-model destination. Identifies + the foundation model + by its UC resource name; the platform resolves it to a Model Serving endpoint at request time. + provisioned_throughput_config: + type: object + properties: + model_serving_endpoint: + type: string + description: >- + Name of the backing Model Serving + endpoint serving the provisioned- + throughput foundation model, as the + AIP-122 typed resource name + ``serving-endpoints/{name}``. The same + UC model can be served on multiple Model + Serving endpoints (different throughput + / region / config); the caller picks + which one this destination routes to. + The endpoint must exist at create time. + model: + type: string + description: >- + UC model FQN of the model served by the + backing endpoint (e.g., + ``system.ai.databricks-claude-opus-4-6``). + Resolved from Model Serving at + Create/Update time. + required: + - model_serving_endpoint + description: >- + Configuration for a + provisioned-throughput foundation-model + destination. References a + pre-existing Model Serving endpoint that serves the model; sizing (provisioned throughput, burst + scaling, model version) is owned by the Model Serving endpoint itself, not by this message. + traffic_percentage: + type: integer + description: >- + Share of traffic sent to this + destination, 0-100. Optional on fallback + destinations; see FallbackConfig. + required: + - name + - destination_type + description: >- + A destination the model service can route + traffic to. Exactly one of the per-type + configs inside + ``type_config`` must be set, and it must match ``destination_type``. + description: >- + Ordered list of fallback destinations. + Traversal is in list order; the attempt + count is the length of the list. At most 5 + are allowed. + description: >- + Fallback routing config, applied after primary + destinations fail. + first_token_timeout: + type: string + description: >- + Timeout for the first token of a streaming + response. If a destination does not return its + first token within this duration, AI Gateway + aborts the attempt and fails over to the next + destination. Applies to streaming requests only. + Leave unset for no first-token timeout. + traffic_splitting: + type: object + properties: {} + description: >- + Marker message selecting request-based traffic + splitting. Traffic is distributed according to + each destination's traffic_percentage value; no + configuration lives on this message itself. + description: >- + Routing configuration: destinations, routing + strategy, and fallback. description: >- - The fully qualified name of the entity to which the tag - is assigned - tag_key: + Operational configuration: destinations, routing, rate + limits, inference table. Required on CreateModelService; + on UpdateModelService it is required only when + ``config`` (or a ``config.*`` subpath) appears in + ``update_mask``. + create_time: type: string - description: The key of the tag - entity_type: + format: date-time + description: When the model service was created. + created_by: + type: string + description: Creator identity. + effective_owner: type: string description: >- - The type of the entity to which the tag is assigned. - Allowed values are: catalogs, schemas, tables, columns, - volumes. - source_type: + The resolved owner of the ModelService. Falls back to + the caller's identity when ``owner`` is not explicitly + set on creation. + etag: type: string - x-enum: - - TAG_ASSIGNMENT_SOURCE_TYPE_SYSTEM_DATA_CLASSIFICATION description: >- - The source type of the tag assignment, e.g., - user-assigned or system-assigned - tag_value: + Optimistic concurrency control token. Server-generated + from the entity's state and returned on every read. To + use it as an if-match precondition on a mutation, echo + the last-read value back via the dedicated ``etag`` + field on the Update / Delete request; the server rejects + the mutation if the stored etag differs. + metastore_id: type: string - description: The value of the tag + description: Metastore hosting the model service. + name: + type: string + description: >- + Resource name of the model service. Format: + ``model-services/{catalog}.{schema}.{model_service}``. + Each ``{...}`` component is capped at 255 characters + individually. Server-derived on Create from ``parent`` + + ``model_service_id``; required and immutable on + Update/Get/Delete. + owner: + type: string + description: >- + The owner of the model service. Write-only; read owner + via effective_owner. + supported_api_types: + type: array + items: + type: string + description: >- + Unified API types this endpoint supports (e.g. "chat", + "embeddings", "completions"). Derived from the + destinations' backing models / providers at read time. update_time: type: string format: date-time - description: The timestamp when the tag assignment was last updated + description: When the model service was last modified. updated_by: type: string - description: The user or principal who updated the tag assignment - required: - - entity_name - - tag_key - - entity_type - description: Represents a tag assignment to an entity + description: Identity of the last updater. + description: >- + The model service with the updated field values. ``name`` + identifies the resource + (``model-services/{catalog}.{schema}.{model_service}``); + only fields listed in ``update_mask`` are applied. required: - - tag_assignment + - model_service responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/EntityTagAssignment' + $ref: '#/components/schemas/ModelService' default: description: Error response content: @@ -978,68 +3350,46 @@ paths: type: string message: type: string - x-stackql-sdk-source: EntityTagAssignmentsAPI - /api/2.1/unity-catalog/entity-tag-assignments/{entity_type}/{entity_name}/tags/{tag_key}: - delete: - operationId: entity_tag_assignments_delete - summary: Deletes a tag assignment for an Unity Catalog entity by its key. + x-stackql-sdk-source: AiGatewayAPI + /api/2.1/unity-catalog/artifact-allowlists/{artifact_type.value}: + get: + operationId: artifact_allowlists_get + summary: >- + Get the artifact allowlist of a certain artifact type. The caller must + be a metastore admin or have tags: - catalog - - entity_tag_assignments + - artifact_allowlists description: >- - Deletes a tag assignment for an Unity Catalog entity by its key. - - - To delete tags from Unity Catalog entities, you must own the entity or - have the following privileges: - - - **APPLY TAG** on the entity - **USE_SCHEMA** on the entity's parent - schema - **USE_CATALOG** on the - - entity's parent catalog - - - To delete a governed tag from Unity Catalog entities, you must also have - the **ASSIGN** or **MANAGE** - - permission on the tag policy. See [Manage tag policy permissions]. + Get the artifact allowlist of a certain artifact type. The caller must + be a metastore admin or have + the **MANAGE ALLOWLIST** privilege on the metastore. - [Manage tag policy permissions]: - https://docs.databricks.com/aws/en/admin/tag-policies/manage-permissions + :param artifact_type: :class:`ArtifactType` + The artifact type of the allowlist. - :param entity_type: str - The type of the entity to which the tag is assigned. Allowed values are: catalogs, schemas, tables, - columns, volumes. - :param entity_name: str - The fully qualified name of the entity to which the tag is assigned - :param tag_key: str - Required. The key of the tag to delete + :returns: :class:`ArtifactAllowlistInfo` parameters: - - name: entity_type - in: path - required: true - schema: - type: string - description: >- - The type of the entity to which the tag is assigned. Allowed values - are: catalogs, schemas, tables, columns, volumes. - - name: entity_name + - name: artifact_type.value in: path required: true schema: type: string - description: The fully qualified name of the entity to which the tag is assigned - - name: tag_key - in: path + - name: artifact_type + in: query required: true schema: - type: string - description: Required. The key of the tag to delete + $ref: '#/components/schemas/ArtifactType' + description: The artifact type of the allowlist. responses: '200': description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ArtifactAllowlistInfo' default: description: Error response content: @@ -1051,53 +3401,93 @@ paths: type: string message: type: string - x-stackql-sdk-source: EntityTagAssignmentsAPI - get: - operationId: entity_tag_assignments_get - summary: Gets a tag assignment for an Unity Catalog entity by tag key. + x-stackql-sdk-source: ArtifactAllowlistsAPI + put: + operationId: artifact_allowlists_update + summary: >- + Set the artifact allowlist of a certain artifact type. The whole + artifact allowlist is replaced with tags: - catalog - - entity_tag_assignments - description: |- - Gets a tag assignment for an Unity Catalog entity by tag key. + - artifact_allowlists + description: >- + Set the artifact allowlist of a certain artifact type. The whole + artifact allowlist is replaced with - :param entity_type: str - The type of the entity to which the tag is assigned. Allowed values are: catalogs, schemas, tables, - columns, volumes. - :param entity_name: str - The fully qualified name of the entity to which the tag is assigned - :param tag_key: str - Required. The key of the tag + the new allowlist. The caller must be a metastore admin or have the + **MANAGE ALLOWLIST** privilege on - :returns: :class:`EntityTagAssignment` + the metastore. + + + :param artifact_type: :class:`ArtifactType` + The artifact type of the allowlist. + :param artifact_matchers: List[:class:`ArtifactMatcher`] + A list of allowed artifact match patterns. + :param created_at: int (optional) + Time at which this artifact allowlist was set, in epoch milliseconds. + :param created_by: str (optional) + Username of the user who set the artifact allowlist. + :param metastore_id: str (optional) + Unique identifier of parent metastore. + + :returns: :class:`ArtifactAllowlistInfo` parameters: - - name: entity_type - in: path - required: true - schema: - type: string - description: >- - The type of the entity to which the tag is assigned. Allowed values - are: catalogs, schemas, tables, columns, volumes. - - name: entity_name - in: path - required: true - schema: - type: string - description: The fully qualified name of the entity to which the tag is assigned - - name: tag_key + - name: artifact_type.value in: path required: true schema: type: string - description: Required. The key of the tag + requestBody: + content: + application/json: + schema: + type: object + properties: + artifact_type: + type: string + x-enum: + - INIT_SCRIPT + - LIBRARY_JAR + - LIBRARY_MAVEN + description: The artifact type of the allowlist. + artifact_matchers: + type: array + items: + type: object + properties: + artifact: + type: string + match_type: + type: string + x-enum: + - PREFIX_MATCH + description: The pattern matching type of the artifact + required: + - artifact + - match_type + description: A list of allowed artifact match patterns. + created_at: + type: integer + description: >- + Time at which this artifact allowlist was set, in epoch + milliseconds. + created_by: + type: string + description: Username of the user who set the artifact allowlist. + metastore_id: + type: string + description: Unique identifier of parent metastore. + required: + - artifact_type + - artifact_matchers responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/EntityTagAssignment' + $ref: '#/components/schemas/ArtifactAllowlistInfo' default: description: Error response content: @@ -1109,148 +3499,225 @@ paths: type: string message: type: string - x-stackql-sdk-source: EntityTagAssignmentsAPI - patch: - operationId: entity_tag_assignments_update - summary: Updates an existing tag assignment for an Unity Catalog entity. + x-stackql-sdk-source: ArtifactAllowlistsAPI + /api/2.1/unity-catalog/catalogs: + post: + operationId: catalogs_create + summary: >- + Creates a new catalog instance in the parent metastore if the caller is + a metastore admin or has the tags: - catalog - - entity_tag_assignments + - catalogs description: >- - Updates an existing tag assignment for an Unity Catalog entity. - - - To update tags to Unity Catalog entities, you must own the entity or - have the following privileges: - - - **APPLY TAG** on the entity - **USE SCHEMA** on the entity's parent - schema - **USE CATALOG** on the - - entity's parent catalog - - - To update a governed tag to Unity Catalog entities, you must also have - the **ASSIGN** or **MANAGE** - - permission on the tag policy. See [Manage tag policy permissions]. - - - [Manage tag policy permissions]: - https://docs.databricks.com/aws/en/admin/tag-policies/manage-permissions + Creates a new catalog instance in the parent metastore if the caller is + a metastore admin or has the + **CREATE_CATALOG** privilege. - :param entity_type: str - The type of the entity to which the tag is assigned. Allowed values are: catalogs, schemas, tables, - columns, volumes. - :param entity_name: str - The fully qualified name of the entity to which the tag is assigned - :param tag_key: str - The key of the tag - :param tag_assignment: :class:`EntityTagAssignment` - :param update_mask: str - The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only - the entire collection field can be specified. Field names must exactly match the resource field - names. + :param name: str + Name of catalog. + :param catalog_federation_config: :class:`CatalogFederationConfig` + (optional) + Federation-only configuration, present only for FOREIGN catalogs. + :param comment: str (optional) + User-provided free-form text description. + :param connection_name: str (optional) + The name of the connection to an external data source. + :param conversion_info: :class:`ConversionInfo` (optional) + Status of conversion of FOREIGN catalog to UC Native catalog. + :param custom_max_retention_hours: int (optional) + Custom maximum retention period in hours for the catalog + :param dr_replication_info: :class:`DrReplicationInfo` (optional) + Disaster Recovery replication state snapshot. + :param managed_encryption_settings: :class:`EncryptionSettings` + (optional) + Control CMK encryption for managed catalog data + :param options: Dict[str,str] (optional) + A map of key-value properties attached to the securable. + :param properties: Dict[str,str] (optional) + A map of key-value properties attached to the securable. + :param provider_name: str (optional) + The name of delta sharing provider. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A Delta Sharing catalog is a catalog that is based on a Delta share on a remote sharing server. + :param share_name: str (optional) + The name of the share under the share provider. + :param storage_root: str (optional) + Storage root URL for managed tables within catalog. - :returns: :class:`EntityTagAssignment` - parameters: - - name: entity_type - in: path - required: true - schema: - type: string - description: >- - The type of the entity to which the tag is assigned. Allowed values - are: catalogs, schemas, tables, columns, volumes. - - name: entity_name - in: path - required: true - schema: - type: string - description: The fully qualified name of the entity to which the tag is assigned - - name: tag_key - in: path - required: true - schema: - type: string - description: The key of the tag - - name: update_mask - in: query - required: true - schema: - type: string - description: >- - The field mask must be a single string, with multiple fields - separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map - fields is not allowed, as only the entire collection field can be - specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. + :returns: :class:`CatalogInfo` requestBody: content: application/json: schema: type: object properties: - tag_assignment: + name: + type: string + description: Name of catalog. + catalog_federation_config: type: object properties: - entity_name: - type: string - description: >- - The fully qualified name of the entity to which the tag - is assigned - tag_key: - type: string - description: The key of the tag - entity_type: - type: string - description: >- - The type of the entity to which the tag is assigned. - Allowed values are: catalogs, schemas, tables, columns, - volumes. - source_type: + schema_filter_type: type: string x-enum: - - TAG_ASSIGNMENT_SOURCE_TYPE_SYSTEM_DATA_CLASSIFICATION + - SCHEMA_FILTER_TYPE_EXCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE_ALL description: >- - The source type of the tag assignment, e.g., - user-assigned or system-assigned - tag_value: - type: string - description: The value of the tag - update_time: + Controls how the catalog's schema filter items are + treated. + description: >- + Federation-only configuration, present only for FOREIGN + catalogs. + comment: + type: string + description: User-provided free-form text description. + connection_name: + type: string + description: The name of the connection to an external data source. + conversion_info: + type: object + properties: + state: type: string - format: date-time - description: The timestamp when the tag assignment was last updated - updated_by: + x-enum: + - COMPLETED + - IN_PROGRESS + description: The conversion state of the resource. + description: >- + Status of conversion of FOREIGN catalog to UC Native + catalog. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the catalog + dr_replication_info: + type: object + properties: + last_failover_time_ms: + type: integer + description: >- + Wall-clock epoch milliseconds when this catalog was last + promoted to primary via failover or failback. Set by DR + Manager. Used by Predictive Optimization to suppress + operations until sufficient workload history + accumulates. + replicated_entities: type: string - description: The user or principal who updated the tag assignment - required: - - entity_name - - tag_key - - entity_type - description: Represents a tag assignment to an entity + description: >- + See + https://docs.google.com/document/d/1X0A_3hMhzuS2V1E3zB0x5wxPsFx70bVYK5rHep2AjW8. + status: + type: string + x-enum: + - DR_REPLICATION_STATUS_PRIMARY + - DR_REPLICATION_STATUS_SECONDARY + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + description: Disaster Recovery replication state snapshot. + managed_encryption_settings: + type: object + properties: + azure_encryption_settings: + type: object + properties: + azure_tenant_id: + type: string + azure_cmk_access_connector_id: + type: string + azure_cmk_managed_identity_id: + type: string + required: + - azure_tenant_id + description: >- + optional Azure settings - only required if an Azure CMK + is used. + azure_key_vault_key_id: + type: string + description: the AKV URL in Azure, null otherwise. + customer_managed_key_id: + type: string + description: the CMK uuid in AWS and GCP, null otherwise. + description: Control CMK encryption for managed catalog data + options: + type: object + description: A map of key-value properties attached to the securable. + properties: + type: object + description: A map of key-value properties attached to the securable. + provider_name: + type: string + description: >- + The name of delta sharing provider. A Delta Sharing catalog + is a catalog that is based on a Delta share on a remote + sharing server. + share_name: + type: string + description: The name of the share under the share provider. + storage_root: + type: string + description: Storage root URL for managed tables within catalog. required: - - tag_assignment + - name responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/EntityTagAssignment' + $ref: '#/components/schemas/CatalogInfo' default: description: Error response content: @@ -1262,74 +3729,112 @@ paths: type: string message: type: string - x-stackql-sdk-source: EntityTagAssignmentsAPI - /api/2.1/unity-catalog/entity-tag-assignments/{entity_type}/{entity_name}/tags: + x-stackql-sdk-source: CatalogsAPI get: - operationId: entity_tag_assignments_list - summary: List tag assignments for an Unity Catalog entity + operationId: catalogs_list + summary: >- + Gets an array of catalogs in the metastore. If the caller is the + metastore admin, all catalogs will be tags: - catalog - - entity_tag_assignments + - catalogs description: >- - List tag assignments for an Unity Catalog entity + Gets an array of catalogs in the metastore. If the caller is the + metastore admin, all catalogs will be + retrieved. Otherwise, only catalogs owned by the caller (or for which + the caller has the - PAGINATION BEHAVIOR: The API is by default paginated, a page may contain - zero results while still + **USE_CATALOG** privilege) will be retrieved. There is no guarantee of a + specific ordering of the - providing a next_page_token. Clients must continue reading pages until - next_page_token is absent, + elements in the array. - which is the only indication that the end of results has been reached. + NOTE: we recommend using max_results=0 to use the paginated version of + this API. Unpaginated calls - :param entity_type: str - The type of the entity to which the tag is assigned. Allowed values are: catalogs, schemas, tables, - columns, volumes. - :param entity_name: str - The fully qualified name of the entity to which the tag is assigned + will be deprecated soon. + + + PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page + may contain zero results while + + still providing a next_page_token. Clients must continue reading pages + until next_page_token is + + absent, which is the only indication that the end of results has been + reached. + + + :param include_browse: bool (optional) + Whether to include catalogs in the response for which the principal can only access selective + metadata for + :param include_unbound: bool (optional) + Whether to include catalogs not bound to the workspace. Effective only if the user has permission to + update the catalog–workspace binding. :param max_results: int (optional) - Optional. Maximum number of tag assignments to return in a single page + Maximum number of catalogs to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid catalogs are returned (not recommended). + - Note: The number of returned catalogs might be less than the specified max_results size, even + zero. The only definitive indication that no further catalogs can be fetched is when the + next_page_token is unset from the response. :param page_token: str (optional) - Optional. Pagination token to retrieve the next page of results + Opaque pagination token to go to next page based on previous query. - :returns: Iterator over :class:`EntityTagAssignment` + :returns: Iterator over :class:`CatalogInfo` parameters: - - name: entity_type - in: path - required: true + - name: include_browse + in: query + required: false schema: - type: string + type: boolean description: >- - The type of the entity to which the tag is assigned. Allowed values - are: catalogs, schemas, tables, columns, volumes. - - name: entity_name - in: path - required: true + Whether to include catalogs in the response for which the principal + can only access selective metadata for + - name: include_unbound + in: query + required: false schema: - type: string - description: The fully qualified name of the entity to which the tag is assigned + type: boolean + description: >- + Whether to include catalogs not bound to the workspace. Effective + only if the user has permission to update the catalog–workspace + binding. - name: max_results in: query required: false schema: type: integer description: >- - Optional. Maximum number of tag assignments to return in a single - page + Maximum number of catalogs to return. - when set to 0, the page + length is set to a server configured value (recommended); - when set + to a value greater than 0, the page length is the minimum of this + value and a server configured value; - when set to a value less than + 0, an invalid parameter error is returned; - If not set, all valid + catalogs are returned (not recommended). - Note: The number of + returned catalogs might be less than the specified max_results size, + even zero. The only definitive indication that no further catalogs + can be fetched is when the next_page_token is unset from the + response. - name: page_token in: query required: false schema: type: string - description: Optional. Pagination token to retrieve the next page of results + description: Opaque pagination token to go to next page based on previous query. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListEntityTagAssignmentsResponse' + $ref: '#/components/schemas/ListCatalogsResponse' default: description: Error response content: @@ -1341,155 +3846,40 @@ paths: type: string message: type: string - x-stackql-sdk-source: EntityTagAssignmentsAPI - /api/2.0/lineage-tracking/external-lineage: - post: - operationId: external_lineage_create_external_lineage_relationship + x-stackql-sdk-source: CatalogsAPI + /api/2.1/unity-catalog/catalogs/{name}: + delete: + operationId: catalogs_delete summary: >- - Creates an external lineage relationship between a Databricks or - external metadata object and another + Deletes the catalog that matches the supplied name. The caller must be a + metastore admin or the owner tags: - catalog - - external_lineage + - catalogs description: >- - Creates an external lineage relationship between a Databricks or - external metadata object and another - - external metadata object. - + Deletes the catalog that matches the supplied name. The caller must be a + metastore admin or the owner - :param external_lineage_relationship: - :class:`CreateRequestExternalLineage` + of the catalog. - :returns: :class:`ExternalLineageRelationship` - requestBody: - content: - application/json: - schema: - type: object - properties: - external_lineage_relationship: - type: object - properties: - source: - type: object - properties: - external_metadata: - type: object - properties: - name: - type: string - model_version: - type: object - properties: - name: - type: string - version: - type: string - path: - type: object - properties: - url: - type: string - table: - type: object - properties: - name: - type: string - target: - type: object - properties: - external_metadata: - type: object - properties: - name: - type: string - model_version: - type: object - properties: - name: - type: string - version: - type: string - path: - type: object - properties: - url: - type: string - table: - type: object - properties: - name: - type: string - description: Target object of the external lineage relationship. - columns: - type: array - items: - type: object - properties: - source: - type: string - target: - type: string - description: >- - List of column relationships between source and target - objects. - id: - type: string - description: Unique identifier of the external lineage relationship. - properties: - type: object - description: >- - Key-value properties associated with the external - lineage relationship. - required: - - source - - target - required: - - external_lineage_relationship - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ExternalLineageRelationship' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: ExternalLineageAPI - delete: - operationId: external_lineage_delete_external_lineage_relationship - summary: >- - Deletes an external lineage relationship between a Databricks or - external metadata object and another - tags: - - catalog - - external_lineage - description: >- - Deletes an external lineage relationship between a Databricks or - external metadata object and another - - external metadata object. - - - :param external_lineage_relationship: - :class:`DeleteRequestExternalLineage` + :param name: str + The name of the catalog. + :param force: bool (optional) + Force deletion even if the catalog is not empty. parameters: - - name: external_lineage_relationship - in: query + - name: name + in: path required: true schema: - $ref: '#/components/schemas/DeleteRequestExternalLineage' + type: string + description: The name of the catalog. + - name: force + in: query + required: false + schema: + type: boolean + description: Force deletion even if the catalog is not empty. responses: '200': description: Success @@ -1504,74 +3894,52 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLineageAPI + x-stackql-sdk-source: CatalogsAPI get: - operationId: external_lineage_list_external_lineage_relationships + operationId: catalogs_get summary: >- - Lists external lineage relationships of a Databricks object or external - metadata given a supplied + Gets the specified catalog in a metastore. The caller must be a + metastore admin, the owner of the tags: - catalog - - external_lineage + - catalogs description: >- - Lists external lineage relationships of a Databricks object or external - metadata given a supplied + Gets the specified catalog in a metastore. The caller must be a + metastore admin, the owner of the - direction. + catalog, or a user that has the **USE_CATALOG** privilege set for their + account. - :param object_info: :class:`ExternalLineageObject` - The object to query external lineage relationships for. Since this field is a query parameter, - please flatten the nested fields. For example, if the object is a table, the query parameter should - look like: `object_info.table.name=main.sales.customers` - :param lineage_direction: :class:`LineageDirection` - The lineage direction to filter on. - :param page_size: int (optional) - Specifies the maximum number of external lineage relationships to return in a single response. The - value must be less than or equal to 1000. - :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. + :param name: str + The name of the catalog. + :param include_browse: bool (optional) + Whether to include catalogs in the response for which the principal can only access selective + metadata for - :returns: Iterator over :class:`ExternalLineageInfo` + :returns: :class:`CatalogInfo` parameters: - - name: object_info - in: query - required: true - schema: - $ref: '#/components/schemas/ExternalLineageObject' - description: >- - The object to query external lineage relationships for. Since this - field is a query parameter, please flatten the nested fields. For - example, if the object is a table, the query parameter should look - like: `object_info.table.name=main.sales.customers` - - name: lineage_direction - in: query + - name: name + in: path required: true schema: - $ref: '#/components/schemas/LineageDirection' - description: The lineage direction to filter on. - - name: page_size + type: string + description: The name of the catalog. + - name: include_browse in: query required: false schema: - type: integer + type: boolean description: >- - Specifies the maximum number of external lineage relationships to - return in a single response. The value must be less than or equal to - 1000. - - name: page_token - in: query - required: false - schema: - type: string - description: Opaque pagination token to go to next page based on previous query. + Whether to include catalogs in the response for which the principal + can only access selective metadata for responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListExternalLineageRelationshipsResponse' + $ref: '#/components/schemas/CatalogInfo' default: description: Error response content: @@ -1583,146 +3951,235 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLineageAPI + x-stackql-sdk-source: CatalogsAPI patch: - operationId: external_lineage_update_external_lineage_relationship + operationId: catalogs_update summary: >- - Updates an external lineage relationship between a Databricks or - external metadata object and another + Updates the catalog that matches the supplied name. The caller must be + either the owner of the tags: - catalog - - external_lineage + - catalogs description: >- - Updates an external lineage relationship between a Databricks or - external metadata object and another - - external metadata object. - + Updates the catalog that matches the supplied name. The caller must be + either the owner of the - :param external_lineage_relationship: - :class:`UpdateRequestExternalLineage` + catalog, or a metastore admin (when changing the owner field of the + catalog). - :param update_mask: str - The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only - the entire collection field can be specified. Field names must exactly match the resource field - names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + :param name: str + The name of the catalog. + :param catalog_federation_config: :class:`CatalogFederationConfig` + (optional) + Federation-only configuration, present only for FOREIGN catalogs. + :param comment: str (optional) + User-provided free-form text description. + :param conversion_info: :class:`ConversionInfo` (optional) + Status of conversion of FOREIGN catalog to UC Native catalog. + :param custom_max_retention_hours: int (optional) + Custom maximum retention period in hours for the catalog + :param dr_replication_info: :class:`DrReplicationInfo` (optional) + Disaster Recovery replication state snapshot. + :param enable_predictive_optimization: + :class:`EnablePredictiveOptimization` (optional) + Whether predictive optimization should be enabled for this object and objects under it. + :param isolation_mode: :class:`CatalogIsolationMode` (optional) + Whether the current securable is accessible from all workspaces or a specific set of workspaces. + :param managed_encryption_settings: :class:`EncryptionSettings` + (optional) + Control CMK encryption for managed catalog data + :param new_name: str (optional) + New name for the catalog. + :param options: Dict[str,str] (optional) + A map of key-value properties attached to the securable. + :param owner: str (optional) + Username of current owner of catalog. + :param properties: Dict[str,str] (optional) + A map of key-value properties attached to the securable. - :returns: :class:`ExternalLineageRelationship` + :returns: :class:`CatalogInfo` parameters: - - name: update_mask - in: query + - name: name + in: path required: true schema: type: string - description: >- - The field mask must be a single string, with multiple fields - separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map - fields is not allowed, as only the entire collection field can be - specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. + description: The name of the catalog. requestBody: content: application/json: schema: type: object properties: - external_lineage_relationship: + catalog_federation_config: type: object properties: - source: - type: object - properties: - external_metadata: - type: object - properties: - name: - type: string - model_version: - type: object - properties: - name: - type: string - version: - type: string - path: - type: object - properties: - url: - type: string - table: - type: object - properties: - name: - type: string - target: - type: object - properties: - external_metadata: - type: object - properties: - name: - type: string - model_version: - type: object - properties: - name: - type: string - version: - type: string - path: - type: object - properties: - url: - type: string - table: - type: object - properties: - name: - type: string - description: Target object of the external lineage relationship. - columns: - type: array - items: - type: object - properties: - source: - type: string - target: - type: string + schema_filter_type: + type: string + x-enum: + - SCHEMA_FILTER_TYPE_EXCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE_ALL description: >- - List of column relationships between source and target - objects. - id: + Controls how the catalog's schema filter items are + treated. + description: >- + Federation-only configuration, present only for FOREIGN + catalogs. + comment: + type: string + description: User-provided free-form text description. + conversion_info: + type: object + properties: + state: type: string - description: Unique identifier of the external lineage relationship. - properties: + x-enum: + - COMPLETED + - IN_PROGRESS + description: The conversion state of the resource. + description: >- + Status of conversion of FOREIGN catalog to UC Native + catalog. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the catalog + dr_replication_info: + type: object + properties: + last_failover_time_ms: + type: integer + description: >- + Wall-clock epoch milliseconds when this catalog was last + promoted to primary via failover or failback. Set by DR + Manager. Used by Predictive Optimization to suppress + operations until sufficient workload history + accumulates. + replicated_entities: + type: string + description: >- + See + https://docs.google.com/document/d/1X0A_3hMhzuS2V1E3zB0x5wxPsFx70bVYK5rHep2AjW8. + status: + type: string + x-enum: + - DR_REPLICATION_STATUS_PRIMARY + - DR_REPLICATION_STATUS_SECONDARY + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + description: Disaster Recovery replication state snapshot. + enable_predictive_optimization: + type: string + x-enum: + - DISABLE + - ENABLE + - INHERIT + description: >- + Whether predictive optimization should be enabled for this + object and objects under it. + isolation_mode: + type: string + x-enum: + - ISOLATED + - OPEN + - OPEN_IN_ACCOUNT + description: >- + Whether the current securable is accessible from all + workspaces or a specific set of workspaces. + managed_encryption_settings: + type: object + properties: + azure_encryption_settings: type: object + properties: + azure_tenant_id: + type: string + azure_cmk_access_connector_id: + type: string + azure_cmk_managed_identity_id: + type: string + required: + - azure_tenant_id description: >- - Key-value properties associated with the external - lineage relationship. - required: - - source - - target - required: - - external_lineage_relationship + optional Azure settings - only required if an Azure CMK + is used. + azure_key_vault_key_id: + type: string + description: the AKV URL in Azure, null otherwise. + customer_managed_key_id: + type: string + description: the CMK uuid in AWS and GCP, null otherwise. + description: Control CMK encryption for managed catalog data + new_name: + type: string + description: New name for the catalog. + options: + type: object + description: A map of key-value properties attached to the securable. + owner: + type: string + description: Username of current owner of catalog. + properties: + type: object + description: A map of key-value properties attached to the securable. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalLineageRelationship' + $ref: '#/components/schemas/CatalogInfo' default: description: Error response content: @@ -1734,55 +4191,46 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLineageAPI - /api/2.1/unity-catalog/external-locations: + x-stackql-sdk-source: CatalogsAPI + /api/2.1/unity-catalog/connections: post: - operationId: external_locations_create - summary: >- - Creates a new external location entry in the metastore. The caller must - be a metastore admin or have + operationId: connections_create + summary: Creates a new connection tags: - catalog - - external_locations + - connections description: >- - Creates a new external location entry in the metastore. The caller must - be a metastore admin or have + Creates a new connection - the **CREATE_EXTERNAL_LOCATION** privilege on both the metastore and the - associated storage - credential. + Creates a new connection to an external data source. It allows users to + specify connection details and + + configurations for interaction with the external server. :param name: str - Name of the external location. - :param url: str - Path URL of the external location. - :param credential_name: str - Name of the storage credential used with this location. + Name of the connection. + :param connection_type: :class:`ConnectionType` + The type of connection. + :param options: Dict[str,str] + A map of key-value properties attached to the securable. :param comment: str (optional) User-provided free-form text description. - :param effective_enable_file_events: bool (optional) - The effective value of `enable_file_events` after applying server-side defaults. - :param enable_file_events: bool (optional) - Whether to enable file events on this external location. Default to `true`. Set to `false` to - disable file events. The actual applied value may differ due to server-side defaults; check - `effective_enable_file_events` for the effective state. - :param encryption_details: :class:`EncryptionDetails` (optional) - - :param fallback: bool (optional) - Indicates whether fallback mode is enabled for this external location. When fallback mode is - enabled, the access to the location falls back to cluster credentials if UC credentials are not - sufficient. - :param file_event_queue: :class:`FileEventQueue` (optional) - File event queue settings. If `enable_file_events` is not `false`, must be defined and have exactly - one of the documented properties. + :param environment_settings: :class:`EnvironmentSettings` (optional) + [Create,Update:OPT] Connection environment settings as EnvironmentSettings object. + :param parent: str (optional) + Parent schema for schema-level connections, in format "schemas/{catalog}.{schema}". Absent for + metastore-level (L1) connections. + :param properties: Dict[str,str] (optional) + A map of key-value properties attached to the securable. :param read_only: bool (optional) - Indicates whether the external location is read-only. - :param skip_validation: bool (optional) - Skips validation of the storage credential associated with the external location. + If the connection is read only. + :param secrets: Dict[str,str] (optional) + A map of option names to UC Secret references. Keys are connection option names (same as in + OptionsKVPairs) and values are UC Secret fully qualified names. - :returns: :class:`ExternalLocationInfo` + :returns: :class:`ConnectionInfo` requestBody: content: application/json: @@ -1791,184 +4239,146 @@ paths: properties: name: type: string - description: Name of the external location. - url: - type: string - description: Path URL of the external location. - credential_name: + description: Name of the connection. + connection_type: type: string - description: Name of the storage credential used with this location. + x-enum: + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AMPLITUDE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS + - BIGLAKE + - BIGQUERY + - CERIDIAN_DAYFORCE + - CONFLUENCE + - DATABRICKS + - DELIGHTED + - DYNAMICS365 + - EPIC_CLARITY + - FRESHSERVICE + - FRONT + - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB + - GLUE + - GMAIL + - GONG + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_CLOUD_LAKEHOUSE + - GOOGLE_WORKSPACE + - GURU + - HIBOB + - HIVE_METASTORE + - HTTP + - HUBSPOT + - ICEBERG_REST + - IRONCLAD + - JDBC + - KINESIS + - LINEAR + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MONDAY_COM + - MYSQL + - NETSKOPE_LOGS + - NOTION + - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PALANTIR + - PARTNERSTACK + - PENDO + - POSTGRESQL + - POWER_BI + - PUBSUB + - QUICKBOOKS + - REDSHIFT + - SALESFORCE + - SALESFORCE_DATA_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID + - SERVICENOW + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SMARTSHEET + - SNAPCHAT_ADS + - SNOWFLAKE + - SPLUNK + - SQLDW + - SQLSERVER + - SQUARE + - TERADATA + - UNKNOWN_CONNECTION_TYPE + - VERKADA + - WORKDAY_RAAS + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS + description: The type of connection. + options: + type: object + description: A map of key-value properties attached to the securable. comment: type: string description: User-provided free-form text description. - effective_enable_file_events: - type: boolean - description: >- - The effective value of `enable_file_events` after applying - server-side defaults. - enable_file_events: - type: boolean - description: >- - Whether to enable file events on this external location. - Default to `true`. Set to `false` to disable file events. - The actual applied value may differ due to server-side - defaults; check `effective_enable_file_events` for the - effective state. - encryption_details: + environment_settings: type: object properties: - sse_encryption_details: - type: object - properties: - algorithm: - type: string - x-enum: - - AWS_SSE_KMS - - AWS_SSE_S3 - description: >- - Sets the value of the 'x-amz-server-side-encryption' - header in S3 request. - aws_kms_key_arn: - type: string - description: >- - Optional. The ARN of the SSE-KMS key used with the - S3 location, when algorithm = "SSE-KMS". Sets the - value of the - 'x-amz-server-side-encryption-aws-kms-key-id' - header. - description: >- - Server-Side Encryption properties for clients - communicating with AWS s3. + environment_version: + type: string + java_dependencies: + type: array + items: + type: string description: >- - Encryption options that apply to clients connecting to cloud - storage. - fallback: - type: boolean + [Create,Update:OPT] Connection environment settings as + EnvironmentSettings object. + parent: + type: string description: >- - Indicates whether fallback mode is enabled for this external - location. When fallback mode is enabled, the access to the - location falls back to cluster credentials if UC credentials - are not sufficient. - file_event_queue: + Parent schema for schema-level connections, in format + "schemas/{catalog}.{schema}". Absent for metastore-level + (L1) connections. + properties: type: object - properties: - managed_aqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format https://{storage - account}.queue.core.windows.net/{queue name} Only - required for provided_aqs. - resource_group: - type: string - description: >- - Optional resource group for the queue, event grid - subscription, and external location storage account. - Only required for locations with a service principal - storage credential - subscription_id: - type: string - description: >- - Optional subscription id for the queue, event grid - subscription, and external location storage account. - Required for locations with a service principal - storage credential - managed_pubsub: - type: object - properties: - managed_resource_id: - type: string - subscription_name: - type: string - description: >- - The Pub/Sub subscription name in the format - projects/{project}/subscriptions/{subscription - name}. Only required for provided_pubsub. - managed_sqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format - https://sqs.{region}.amazonaws.com/{account - id}/{queue name}. Only required for provided_sqs. - provided_aqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format https://{storage - account}.queue.core.windows.net/{queue name} Only - required for provided_aqs. - resource_group: - type: string - description: >- - Optional resource group for the queue, event grid - subscription, and external location storage account. - Only required for locations with a service principal - storage credential - subscription_id: - type: string - description: >- - Optional subscription id for the queue, event grid - subscription, and external location storage account. - Required for locations with a service principal - storage credential - provided_pubsub: - type: object - properties: - managed_resource_id: - type: string - subscription_name: - type: string - description: >- - The Pub/Sub subscription name in the format - projects/{project}/subscriptions/{subscription - name}. Only required for provided_pubsub. - provided_sqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format - https://sqs.{region}.amazonaws.com/{account - id}/{queue name}. Only required for provided_sqs. - description: >- - File event queue settings. If `enable_file_events` is not - `false`, must be defined and have exactly one of the - documented properties. + description: A map of key-value properties attached to the securable. read_only: type: boolean - description: Indicates whether the external location is read-only. - skip_validation: - type: boolean + description: If the connection is read only. + secrets: + type: object description: >- - Skips validation of the storage credential associated with - the external location. + A map of option names to UC Secret references. Keys are + connection option names (same as in OptionsKVPairs) and + values are UC Secret fully qualified names. required: - name - - url - - credential_name + - connection_type + - options responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalLocationInfo' + $ref: '#/components/schemas/ConnectionInfo' default: description: Error response content: @@ -1980,24 +4390,15 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLocationsAPI + x-stackql-sdk-source: ConnectionsAPI get: - operationId: external_locations_list - summary: >- - Gets an array of external locations (__ExternalLocationInfo__ objects) - from the metastore. The caller + operationId: connections_list + summary: List all connections. tags: - catalog - - external_locations + - connections description: >- - Gets an array of external locations (__ExternalLocationInfo__ objects) - from the metastore. The caller - - must be a metastore admin, the owner of the external location, or a user - that has some privilege on - - the external location. There is no guarantee of a specific ordering of - the elements in the array. + List all connections. NOTE: we recommend using max_results=0 to use the paginated version of @@ -2016,64 +4417,55 @@ paths: reached. - :param include_browse: bool (optional) - Whether to include external locations in the response for which the principal can only access - selective metadata for - :param include_unbound: bool (optional) - Whether to include external locations not bound to the workspace. Effective only if the user has - permission to update the location–workspace binding. :param max_results: int (optional) - Maximum number of external locations to return. If not set, all the external locations are returned - (not recommended). - when set to a value greater than 0, the page length is the minimum of this - value and a server configured value; - when set to 0, the page length is set to a server configured - value (recommended); - when set to a value less than 0, an invalid parameter error is returned; + Maximum number of connections to return. + + - If not set, all connections are returned (not recommended). + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. + :param parent: str (optional) + Optional. Parent schema filter for listing schema-level connections, in format + "schemas/{catalog}.{schema}". - :returns: Iterator over :class:`ExternalLocationInfo` + :returns: Iterator over :class:`ConnectionInfo` parameters: - - name: include_browse - in: query - required: false - schema: - type: boolean - description: >- - Whether to include external locations in the response for which the - principal can only access selective metadata for - - name: include_unbound - in: query - required: false - schema: - type: boolean - description: >- - Whether to include external locations not bound to the workspace. - Effective only if the user has permission to update the - location–workspace binding. - name: max_results in: query required: false schema: type: integer description: >- - Maximum number of external locations to return. If not set, all the - external locations are returned (not recommended). - when set to a - value greater than 0, the page length is the minimum of this value - and a server configured value; - when set to 0, the page length is - set to a server configured value (recommended); - when set to a - value less than 0, an invalid parameter error is returned; + Maximum number of connections to return. - If not set, all + connections are returned (not recommended). - when set to a value + greater than 0, the page length is the minimum of this value and a + server configured value; - when set to 0, the page length is set to + a server configured value (recommended); - when set to a value less + than 0, an invalid parameter error is returned; - name: page_token in: query required: false schema: type: string description: Opaque pagination token to go to next page based on previous query. + - name: parent + in: query + required: false + schema: + type: string + description: >- + Optional. Parent schema filter for listing schema-level connections, + in format "schemas/{catalog}.{schema}". responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListExternalLocationsResponse' + $ref: '#/components/schemas/ListConnectionsResponse' default: description: Error response content: @@ -2085,42 +4477,26 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLocationsAPI - /api/2.1/unity-catalog/external-locations/{name}: + x-stackql-sdk-source: ConnectionsAPI + /api/2.1/unity-catalog/connections/{name}: delete: - operationId: external_locations_delete - summary: >- - Deletes the specified external location from the metastore. The caller - must be the owner of the + operationId: connections_delete + summary: Deletes the connection that matches the supplied name. tags: - catalog - - external_locations - description: >- - Deletes the specified external location from the metastore. The caller - must be the owner of the - - external location. - + - connections + description: |- + Deletes the connection that matches the supplied name. :param name: str - Name of the external location. - :param force: bool (optional) - Force deletion even if there are dependent external tables or mounts. + The name of the connection to be deleted. parameters: - name: name in: path required: true schema: type: string - description: Name of the external location. - - name: force - in: query - required: false - schema: - type: boolean - description: >- - Force deletion even if there are dependent external tables or - mounts. + description: The name of the connection to be deleted. responses: '200': description: Success @@ -2135,52 +4511,34 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLocationsAPI + x-stackql-sdk-source: ConnectionsAPI get: - operationId: external_locations_get - summary: >- - Gets an external location from the metastore. The caller must be either - a metastore admin, the owner + operationId: connections_get + summary: Gets a connection from it's name. tags: - catalog - - external_locations - description: >- - Gets an external location from the metastore. The caller must be either - a metastore admin, the owner - - of the external location, or a user that has some privilege on the - external location. - + - connections + description: |- + Gets a connection from it's name. :param name: str - Name of the external location. - :param include_browse: bool (optional) - Whether to include external locations in the response for which the principal can only access - selective metadata for + Name of the connection. - :returns: :class:`ExternalLocationInfo` + :returns: :class:`ConnectionInfo` parameters: - name: name in: path required: true schema: type: string - description: Name of the external location. - - name: include_browse - in: query - required: false - schema: - type: boolean - description: >- - Whether to include external locations in the response for which the - principal can only access selective metadata for + description: Name of the connection. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalLocationInfo' + $ref: '#/components/schemas/ConnectionInfo' default: description: Error response content: @@ -2192,316 +4550,80 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLocationsAPI + x-stackql-sdk-source: ConnectionsAPI patch: - operationId: external_locations_update - summary: >- - Updates an external location in the metastore. The caller must be the - owner of the external location, + operationId: connections_update + summary: Updates the connection that matches the supplied name. tags: - catalog - - external_locations - description: >- - Updates an external location in the metastore. The caller must be the - owner of the external location, - - or be a metastore admin. In the second case, the admin can only update - the name of the external - - location. - + - connections + description: |- + Updates the connection that matches the supplied name. :param name: str - Name of the external location. - :param comment: str (optional) - User-provided free-form text description. - :param credential_name: str (optional) - Name of the storage credential used with this location. - :param effective_enable_file_events: bool (optional) - The effective value of `enable_file_events` after applying server-side defaults. - :param enable_file_events: bool (optional) - Whether to enable file events on this external location. Default to `true`. Set to `false` to - disable file events. The actual applied value may differ due to server-side defaults; check - `effective_enable_file_events` for the effective state. - :param encryption_details: :class:`EncryptionDetails` (optional) - - :param fallback: bool (optional) - Indicates whether fallback mode is enabled for this external location. When fallback mode is - enabled, the access to the location falls back to cluster credentials if UC credentials are not - sufficient. - :param file_event_queue: :class:`FileEventQueue` (optional) - File event queue settings. If `enable_file_events` is not `false`, must be defined and have exactly - one of the documented properties. - :param force: bool (optional) - Force update even if changing url invalidates dependent external tables or mounts. - :param isolation_mode: :class:`IsolationMode` (optional) - + Name of the connection. + :param options: Dict[str,str] + A map of key-value properties attached to the securable. + :param environment_settings: :class:`EnvironmentSettings` (optional) + [Create,Update:OPT] Connection environment settings as EnvironmentSettings object. :param new_name: str (optional) - New name for the external location. + New name for the connection. :param owner: str (optional) - The owner of the external location. - :param read_only: bool (optional) - Indicates whether the external location is read-only. - :param skip_validation: bool (optional) - Skips validation of the storage credential associated with the external location. - :param url: str (optional) - Path URL of the external location. + Username of current owner of the connection. + :param secrets: Dict[str,str] (optional) + A map of option names to UC Secret references. Keys are connection option names (same as in + OptionsKVPairs) and values are UC Secret fully qualified names. - :returns: :class:`ExternalLocationInfo` + :returns: :class:`ConnectionInfo` parameters: - name: name in: path required: true schema: type: string - description: Name of the external location. + description: Name of the connection. requestBody: content: application/json: schema: type: object properties: - comment: - type: string - description: User-provided free-form text description. - credential_name: - type: string - description: Name of the storage credential used with this location. - effective_enable_file_events: - type: boolean - description: >- - The effective value of `enable_file_events` after applying - server-side defaults. - enable_file_events: - type: boolean - description: >- - Whether to enable file events on this external location. - Default to `true`. Set to `false` to disable file events. - The actual applied value may differ due to server-side - defaults; check `effective_enable_file_events` for the - effective state. - encryption_details: + options: type: object - properties: - sse_encryption_details: - type: object - properties: - algorithm: - type: string - x-enum: - - AWS_SSE_KMS - - AWS_SSE_S3 - description: >- - Sets the value of the 'x-amz-server-side-encryption' - header in S3 request. - aws_kms_key_arn: - type: string - description: >- - Optional. The ARN of the SSE-KMS key used with the - S3 location, when algorithm = "SSE-KMS". Sets the - value of the - 'x-amz-server-side-encryption-aws-kms-key-id' - header. - description: >- - Server-Side Encryption properties for clients - communicating with AWS s3. - description: >- - Encryption options that apply to clients connecting to cloud - storage. - fallback: - type: boolean - description: >- - Indicates whether fallback mode is enabled for this external - location. When fallback mode is enabled, the access to the - location falls back to cluster credentials if UC credentials - are not sufficient. - file_event_queue: + description: A map of key-value properties attached to the securable. + environment_settings: type: object properties: - managed_aqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format https://{storage - account}.queue.core.windows.net/{queue name} Only - required for provided_aqs. - resource_group: - type: string - description: >- - Optional resource group for the queue, event grid - subscription, and external location storage account. - Only required for locations with a service principal - storage credential - subscription_id: - type: string - description: >- - Optional subscription id for the queue, event grid - subscription, and external location storage account. - Required for locations with a service principal - storage credential - managed_pubsub: - type: object - properties: - managed_resource_id: - type: string - subscription_name: - type: string - description: >- - The Pub/Sub subscription name in the format - projects/{project}/subscriptions/{subscription - name}. Only required for provided_pubsub. - managed_sqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format - https://sqs.{region}.amazonaws.com/{account - id}/{queue name}. Only required for provided_sqs. - provided_aqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format https://{storage - account}.queue.core.windows.net/{queue name} Only - required for provided_aqs. - resource_group: - type: string - description: >- - Optional resource group for the queue, event grid - subscription, and external location storage account. - Only required for locations with a service principal - storage credential - subscription_id: - type: string - description: >- - Optional subscription id for the queue, event grid - subscription, and external location storage account. - Required for locations with a service principal - storage credential - provided_pubsub: - type: object - properties: - managed_resource_id: - type: string - subscription_name: - type: string - description: >- - The Pub/Sub subscription name in the format - projects/{project}/subscriptions/{subscription - name}. Only required for provided_pubsub. - provided_sqs: - type: object - properties: - managed_resource_id: - type: string - queue_url: - type: string - description: >- - The AQS queue url in the format - https://sqs.{region}.amazonaws.com/{account - id}/{queue name}. Only required for provided_sqs. - description: >- - File event queue settings. If `enable_file_events` is not - `false`, must be defined and have exactly one of the - documented properties. - force: - type: boolean + environment_version: + type: string + java_dependencies: + type: array + items: + type: string description: >- - Force update even if changing url invalidates dependent - external tables or mounts. - isolation_mode: + [Create,Update:OPT] Connection environment settings as + EnvironmentSettings object. + new_name: type: string - x-enum: - - ISOLATION_MODE_ISOLATED - - ISOLATION_MODE_OPEN + description: New name for the connection. + owner: + type: string + description: Username of current owner of the connection. + secrets: + type: object description: >- - Create a collection of name/value pairs. - - - Example enumeration: - - - >>> class Color(Enum): - - ... RED = 1 - - ... BLUE = 2 - - ... GREEN = 3 - - - Access them by: - - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many members - they have: - - - >>> len(Color) - - 3 - - - >>> list(Color) - - [, , ] - - - Methods can be added to enumerations, and members can have - their own - - attributes -- see the documentation for details. - new_name: - type: string - description: New name for the external location. - owner: - type: string - description: The owner of the external location. - read_only: - type: boolean - description: Indicates whether the external location is read-only. - skip_validation: - type: boolean - description: >- - Skips validation of the storage credential associated with - the external location. - url: - type: string - description: Path URL of the external location. + A map of option names to UC Secret references. Keys are + connection option names (same as in OptionsKVPairs) and + values are UC Secret fully qualified names. + required: + - options responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalLocationInfo' + $ref: '#/components/schemas/ConnectionInfo' default: description: Error response content: @@ -2513,127 +4635,95 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalLocationsAPI - /api/2.0/lineage-tracking/external-metadata: + x-stackql-sdk-source: ConnectionsAPI + /api/2.1/unity-catalog/entity-tag-assignments: post: - operationId: external_metadata_create_external_metadata - summary: >- - Creates a new external metadata object in the parent metastore if the - caller is a metastore admin or + operationId: entity_tag_assignments_create + summary: Creates a tag assignment for an Unity Catalog entity. tags: - catalog - - external_metadata + - entity_tag_assignments description: >- - Creates a new external metadata object in the parent metastore if the - caller is a metastore admin or + Creates a tag assignment for an Unity Catalog entity. - has the **CREATE_EXTERNAL_METADATA** privilege. Grants **BROWSE** to all - account users upon creation - by default. + To add tags to Unity Catalog entities, you must own the entity or have + the following privileges: - :param external_metadata: :class:`ExternalMetadata` + - **APPLY TAG** on the entity + - **USE SCHEMA** on the entity's parent schema - :returns: :class:`ExternalMetadata` + - **USE CATALOG** on the entity's parent catalog + + + To add a governed tag to Unity Catalog entities, you must also have the + **ASSIGN** or **MANAGE** + + permission on the tag policy. See `Manage tag policy permissions + + `__. + + + :param tag_assignment: :class:`EntityTagAssignment` + + + :returns: :class:`EntityTagAssignment` requestBody: content: application/json: schema: type: object properties: - external_metadata: + tag_assignment: type: object properties: - name: - type: string - system_type: - type: string - x-enum: - - AMAZON_REDSHIFT - - AZURE_SYNAPSE - - CONFLUENT - - DATABRICKS - - GOOGLE_BIGQUERY - - KAFKA - - LOOKER - - MICROSOFT_FABRIC - - MICROSOFT_SQL_SERVER - - MONGODB - - MYSQL - - ORACLE - - OTHER - - POSTGRESQL - - POWER_BI - - SALESFORCE - - SAP - - SERVICENOW - - SNOWFLAKE - - STREAM_NATIVE - - TABLEAU - - TERADATA - - WORKDAY - description: Type of external system. - entity_type: + entity_name: type: string - description: Type of entity within the external system. - columns: - type: array - items: - type: string description: >- - List of columns associated with the external metadata - object. - create_time: - type: string - description: Time at which this external metadata object was created. - created_by: - type: string - description: Username of external metadata object creator. - description: - type: string - description: User-provided free-form text description. - id: + The fully qualified name of the entity to which the tag + is assigned + tag_key: type: string - description: Unique identifier of the external metadata object. - metastore_id: + description: The key of the tag + entity_type: type: string - description: Unique identifier of parent metastore. - owner: + description: The type of the entity to which the tag is assigned. + inherited: + type: boolean + description: Boolean which indicates whether this tag is inherited. + source_type: type: string - description: Owner of the external metadata object. - properties: - type: object + x-enum: + - TAG_ASSIGNMENT_SOURCE_TYPE_SYSTEM_DATA_CLASSIFICATION description: >- - A map of key-value properties attached to the external - metadata object. + The source type of the tag assignment, e.g., + user-assigned or system-assigned + tag_value: + type: string + description: The value of the tag update_time: type: string - description: >- - Time at which this external metadata object was last - modified. + format: date-time + description: The timestamp when the tag assignment was last updated updated_by: type: string - description: >- - Username of user who last modified external metadata - object. - url: - type: string - description: URL associated with the external metadata object. + description: The user or principal who updated the tag assignment required: - - name - - system_type + - entity_name + - tag_key - entity_type + description: Represents a tag assignment to an entity required: - - external_metadata + - tag_assignment responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalMetadata' + $ref: '#/components/schemas/EntityTagAssignment' default: description: Error response content: @@ -2645,93 +4735,62 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalMetadataAPI - get: - operationId: external_metadata_list_external_metadata - summary: >- - Gets an array of external metadata objects in the metastore. If the - caller is the metastore admin, all + x-stackql-sdk-source: EntityTagAssignmentsAPI + /api/2.1/unity-catalog/entity-tag-assignments/{entity_type}/{entity_name}/tags/{tag_key}: + delete: + operationId: entity_tag_assignments_delete + summary: Deletes a tag assignment for an Unity Catalog entity by its key. tags: - catalog - - external_metadata + - entity_tag_assignments description: >- - Gets an array of external metadata objects in the metastore. If the - caller is the metastore admin, all + Deletes a tag assignment for an Unity Catalog entity by its key. - external metadata objects will be retrieved. Otherwise, only external - metadata objects that the caller - has **BROWSE** on will be retrieved. There is no guarantee of a specific - ordering of the elements in + To delete tags from Unity Catalog entities, you must own the entity or + have the following privileges: - the array. + - **APPLY TAG** on the entity - :param page_size: int (optional) - Specifies the maximum number of external metadata objects to return in a single response. The value - must be less than or equal to 1000. - :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. + - **USE_SCHEMA** on the entity's parent schema - :returns: Iterator over :class:`ExternalMetadata` + - **USE_CATALOG** on the entity's parent catalog + + + To delete a governed tag from Unity Catalog entities, you must also have + the **ASSIGN** or **MANAGE** + + permission on the tag policy. See `Manage tag policy permissions + + `__. + + + :param entity_type: str + The type of the entity to which the tag is assigned. + :param entity_name: str + The fully qualified name of the entity to which the tag is assigned + :param tag_key: str + Required. The key of the tag to delete parameters: - - name: page_size - in: query - required: false - schema: - type: integer - description: >- - Specifies the maximum number of external metadata objects to return - in a single response. The value must be less than or equal to 1000. - - name: page_token - in: query - required: false + - name: entity_type + in: path + required: true schema: type: string - description: Opaque pagination token to go to next page based on previous query. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListExternalMetadataResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: ExternalMetadataAPI - /api/2.0/lineage-tracking/external-metadata/{name}: - delete: - operationId: external_metadata_delete_external_metadata - summary: >- - Deletes the external metadata object that matches the supplied name. The - caller must be a metastore - tags: - - catalog - - external_metadata - description: >- - Deletes the external metadata object that matches the supplied name. The - caller must be a metastore - - admin, the owner of the external metadata object, or a user that has the - **MANAGE** privilege. - - - :param name: str - parameters: - - name: name + description: The type of the entity to which the tag is assigned. + - name: entity_name in: path required: true schema: type: string + description: The fully qualified name of the entity to which the tag is assigned + - name: tag_key + in: path + required: true + schema: + type: string + description: Required. The key of the tag to delete responses: '200': description: Success @@ -2746,40 +4805,58 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalMetadataAPI + x-stackql-sdk-source: EntityTagAssignmentsAPI get: - operationId: external_metadata_get_external_metadata - summary: >- - Gets the specified external metadata object in a metastore. The caller - must be a metastore admin, the + operationId: entity_tag_assignments_get + summary: Gets a tag assignment for an Unity Catalog entity by tag key. tags: - catalog - - external_metadata - description: >- - Gets the specified external metadata object in a metastore. The caller - must be a metastore admin, the - - owner of the external metadata object, or a user that has the **BROWSE** - privilege. - - - :param name: str + - entity_tag_assignments + description: |- + Gets a tag assignment for an Unity Catalog entity by tag key. + :param entity_type: str + The type of the entity to which the tag is assigned. + :param entity_name: str + The fully qualified name of the entity to which the tag is assigned + :param tag_key: str + Required. The key of the tag + :param include_inherited: bool (optional) + Boolean which indicates whether this tag is inherited. - :returns: :class:`ExternalMetadata` + :returns: :class:`EntityTagAssignment` parameters: - - name: name + - name: entity_type + in: path + required: true + schema: + type: string + description: The type of the entity to which the tag is assigned. + - name: entity_name + in: path + required: true + schema: + type: string + description: The fully qualified name of the entity to which the tag is assigned + - name: tag_key in: path required: true schema: type: string + description: Required. The key of the tag + - name: include_inherited + in: query + required: false + schema: + type: boolean + description: Boolean which indicates whether this tag is inherited. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalMetadata' + $ref: '#/components/schemas/EntityTagAssignment' default: description: Error response content: @@ -2791,51 +4868,75 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalMetadataAPI + x-stackql-sdk-source: EntityTagAssignmentsAPI patch: - operationId: external_metadata_update_external_metadata - summary: >- - Updates the external metadata object that matches the supplied name. The - caller can only update either + operationId: entity_tag_assignments_update + summary: Updates an existing tag assignment for an Unity Catalog entity. tags: - catalog - - external_metadata + - entity_tag_assignments description: >- - Updates the external metadata object that matches the supplied name. The - caller can only update either + Updates an existing tag assignment for an Unity Catalog entity. - the owner or other metadata fields in one request. The caller must be a - metastore admin, the owner of - the external metadata object, or a user that has the **MODIFY** - privilege. If the caller is updating + To update tags to Unity Catalog entities, you must own the entity or + have the following privileges: - the owner, they must also have the **MANAGE** privilege. + - **APPLY TAG** on the entity + + - **USE SCHEMA** on the entity's parent schema + + - **USE CATALOG** on the entity's parent catalog + + + To update a governed tag to Unity Catalog entities, you must also have + the **ASSIGN** or **MANAGE** + + permission on the tag policy. See `Manage tag policy permissions + + `__. - :param name: str - Name of the external metadata object. - :param external_metadata: :class:`ExternalMetadata` + + :param entity_type: str + The type of the entity to which the tag is assigned. + :param entity_name: str + The fully qualified name of the entity to which the tag is assigned + :param tag_key: str + The key of the tag + :param tag_assignment: :class:`EntityTagAssignment` :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. - :returns: :class:`ExternalMetadata` + :returns: :class:`EntityTagAssignment` parameters: - - name: name + - name: entity_type in: path required: true schema: type: string - description: Name of the external metadata object. + description: The type of the entity to which the tag is assigned. + - name: entity_name + in: path + required: true + schema: + type: string + description: The fully qualified name of the entity to which the tag is assigned + - name: tag_key + in: path + required: true + schema: + type: string + description: The key of the tag - name: update_mask in: query required: true @@ -2844,111 +4945,68 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. requestBody: content: application/json: schema: type: object properties: - external_metadata: + tag_assignment: type: object properties: - name: - type: string - system_type: - type: string - x-enum: - - AMAZON_REDSHIFT - - AZURE_SYNAPSE - - CONFLUENT - - DATABRICKS - - GOOGLE_BIGQUERY - - KAFKA - - LOOKER - - MICROSOFT_FABRIC - - MICROSOFT_SQL_SERVER - - MONGODB - - MYSQL - - ORACLE - - OTHER - - POSTGRESQL - - POWER_BI - - SALESFORCE - - SAP - - SERVICENOW - - SNOWFLAKE - - STREAM_NATIVE - - TABLEAU - - TERADATA - - WORKDAY - description: Type of external system. - entity_type: + entity_name: type: string - description: Type of entity within the external system. - columns: - type: array - items: - type: string description: >- - List of columns associated with the external metadata - object. - create_time: - type: string - description: Time at which this external metadata object was created. - created_by: - type: string - description: Username of external metadata object creator. - description: - type: string - description: User-provided free-form text description. - id: + The fully qualified name of the entity to which the tag + is assigned + tag_key: type: string - description: Unique identifier of the external metadata object. - metastore_id: + description: The key of the tag + entity_type: type: string - description: Unique identifier of parent metastore. - owner: + description: The type of the entity to which the tag is assigned. + inherited: + type: boolean + description: Boolean which indicates whether this tag is inherited. + source_type: type: string - description: Owner of the external metadata object. - properties: - type: object + x-enum: + - TAG_ASSIGNMENT_SOURCE_TYPE_SYSTEM_DATA_CLASSIFICATION description: >- - A map of key-value properties attached to the external - metadata object. + The source type of the tag assignment, e.g., + user-assigned or system-assigned + tag_value: + type: string + description: The value of the tag update_time: type: string - description: >- - Time at which this external metadata object was last - modified. + format: date-time + description: The timestamp when the tag assignment was last updated updated_by: type: string - description: >- - Username of user who last modified external metadata - object. - url: - type: string - description: URL associated with the external metadata object. + description: The user or principal who updated the tag assignment required: - - name - - system_type + - entity_name + - tag_key - entity_type + description: Represents a tag assignment to an entity required: - - external_metadata + - tag_assignment responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ExternalMetadata' + $ref: '#/components/schemas/EntityTagAssignment' default: description: Error response content: @@ -2960,399 +5018,214 @@ paths: type: string message: type: string - x-stackql-sdk-source: ExternalMetadataAPI - /api/2.1/unity-catalog/functions: - post: - operationId: functions_create - summary: '**WARNING: This API is experimental and will change in future versions**' + x-stackql-sdk-source: EntityTagAssignmentsAPI + /api/2.1/unity-catalog/entity-tag-assignments/{entity_type}/{entity_name}/tags: + get: + operationId: entity_tag_assignments_list + summary: List tag assignments for an Unity Catalog entity tags: - catalog - - functions + - entity_tag_assignments description: >- - **WARNING: This API is experimental and will change in future versions** + List tag assignments for an Unity Catalog entity - Creates a new function + PAGINATION BEHAVIOR: The API is by default paginated, a page may contain + zero results while still + + providing a next_page_token. Clients must continue reading pages until + next_page_token is absent, + which is the only indication that the end of results has been reached. - The user must have the following permissions in order for the function - to be created: - - **USE_CATALOG** on the function's parent catalog - **USE_SCHEMA** and - **CREATE_FUNCTION** on the + :param entity_type: str + The type of the entity to which the tag is assigned. + :param entity_name: str + The fully qualified name of the entity to which the tag is assigned + :param include_inherited: bool (optional) + Boolean which indicates whether this tag is inherited. + :param max_results: int (optional) + Optional. Maximum number of tag assignments to return in a single page + :param page_token: str (optional) + Optional. Pagination token to retrieve the next page of results + + :returns: Iterator over :class:`EntityTagAssignment` + parameters: + - name: entity_type + in: path + required: true + schema: + type: string + description: The type of the entity to which the tag is assigned. + - name: entity_name + in: path + required: true + schema: + type: string + description: The fully qualified name of the entity to which the tag is assigned + - name: include_inherited + in: query + required: false + schema: + type: boolean + description: Boolean which indicates whether this tag is inherited. + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Optional. Maximum number of tag assignments to return in a single + page + - name: page_token + in: query + required: false + schema: + type: string + description: Optional. Pagination token to retrieve the next page of results + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListEntityTagAssignmentsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: EntityTagAssignmentsAPI + /api/2.0/lineage-tracking/external-lineage: + post: + operationId: external_lineage_create_external_lineage_relationship + summary: >- + Creates an external lineage relationship between a Databricks or + external metadata object and another + tags: + - catalog + - external_lineage + description: >- + Creates an external lineage relationship between a Databricks or + external metadata object and another + + external metadata object. - function's parent schema + :param external_lineage_relationship: + :class:`CreateRequestExternalLineage` - :param function_info: :class:`CreateFunction` - Partial __FunctionInfo__ specifying the function to be created. - :returns: :class:`FunctionInfo` + :returns: :class:`ExternalLineageRelationship` requestBody: content: application/json: schema: type: object properties: - function_info: + external_lineage_relationship: type: object properties: - name: - type: string - catalog_name: - type: string - description: Name of parent Catalog. - schema_name: - type: string - description: Name of parent Schema relative to its parent Catalog. - input_params: + source: type: object properties: - parameters: - type: array - items: - type: object - properties: - name: - type: string - type_text: - type: string - description: Full data type spec, SQL/catalogString text. - type_name: - type: string - x-enum: - - ARRAY - - BINARY - - BOOLEAN - - BYTE - - CHAR - - DATE - - DECIMAL - - DOUBLE - - FLOAT - - GEOGRAPHY - - GEOMETRY - - INT - - INTERVAL - - LONG - - MAP - - 'NULL' - - SHORT - - STRING - - STRUCT - - TABLE_TYPE - - TIMESTAMP - - TIMESTAMP_NTZ - - USER_DEFINED_TYPE - - VARIANT - description: Name of type (INT, STRUCT, MAP, etc.) - position: - type: integer - description: >- - Ordinal position of column (starting at - position 0). - comment: - type: string - description: User-provided free-form text description. - parameter_default: - type: string - description: Default value of the parameter. - parameter_mode: - type: string - x-enum: - - IN - description: Function parameter mode. - parameter_type: - type: string - x-enum: - - COLUMN - - PARAM - description: Function parameter type. - type_interval_type: - type: string - description: Format of IntervalType. - type_json: - type: string - description: Full data type spec, JSON-serialized. - type_precision: - type: integer - description: >- - Digits of precision; required on Create for - DecimalTypes. - type_scale: - type: integer - description: >- - Digits to right of decimal; Required on Create - for DecimalTypes. - required: - - name - - type_text - - type_name - - position - description: Function input parameters. - data_type: + external_metadata: + type: object + properties: + name: + type: string + model_version: + type: object + properties: + name: + type: string + version: + type: string + path: + type: object + properties: + url: + type: string + table: + type: object + properties: + name: + type: string + volume: + type: object + properties: + name: + type: string + target: + type: object + properties: + external_metadata: + type: object + properties: + name: + type: string + model_version: + type: object + properties: + name: + type: string + version: + type: string + path: + type: object + properties: + url: + type: string + table: + type: object + properties: + name: + type: string + volume: + type: object + properties: + name: + type: string + description: Target object of the external lineage relationship. + columns: + type: array + items: + type: object + properties: + source: + type: string + target: + type: string + description: >- + List of column relationships between source and target + objects. + id: type: string - x-enum: - - ARRAY - - BINARY - - BOOLEAN - - BYTE - - CHAR - - DATE - - DECIMAL - - DOUBLE - - FLOAT - - GEOGRAPHY - - GEOMETRY - - INT - - INTERVAL - - LONG - - MAP - - 'NULL' - - SHORT - - STRING - - STRUCT - - TABLE_TYPE - - TIMESTAMP - - TIMESTAMP_NTZ - - USER_DEFINED_TYPE - - VARIANT - description: Scalar function return data type. - full_data_type: - type: string - description: Pretty printed function data type. - routine_body: - type: string - x-enum: - - EXTERNAL - - SQL - description: >- - Function language. When **EXTERNAL** is used, the - language of the routine function should be specified in - the **external_language** field, and the - **return_params** of the function cannot be used (as - **TABLE** return type is not supported), and the - **sql_data_access** field must be **NO_SQL**. - routine_definition: - type: string - description: Function body. - parameter_style: - type: string - x-enum: - - S - description: Function parameter style. **S** is the value for SQL. - is_deterministic: - type: boolean - description: Whether the function is deterministic. - sql_data_access: - type: string - x-enum: - - CONTAINS_SQL - - NO_SQL - - READS_SQL_DATA - description: Function SQL data access. - is_null_call: - type: boolean - description: Function null call. - security_type: - type: string - x-enum: - - DEFINER - description: Function security type. - specific_name: - type: string - description: Specific name of the function; Reserved for future use. - comment: - type: string - description: User-provided free-form text description. - external_language: - type: string - description: External function language. - external_name: - type: string - description: External function name. + description: Unique identifier of the external lineage relationship. properties: - type: string - description: >- - JSON-serialized key-value pair map, encoded (escaped) as - a string. - return_params: - type: object - properties: - parameters: - type: array - items: - type: object - properties: - name: - type: string - type_text: - type: string - description: Full data type spec, SQL/catalogString text. - type_name: - type: string - x-enum: - - ARRAY - - BINARY - - BOOLEAN - - BYTE - - CHAR - - DATE - - DECIMAL - - DOUBLE - - FLOAT - - GEOGRAPHY - - GEOMETRY - - INT - - INTERVAL - - LONG - - MAP - - 'NULL' - - SHORT - - STRING - - STRUCT - - TABLE_TYPE - - TIMESTAMP - - TIMESTAMP_NTZ - - USER_DEFINED_TYPE - - VARIANT - description: Name of type (INT, STRUCT, MAP, etc.) - position: - type: integer - description: >- - Ordinal position of column (starting at - position 0). - comment: - type: string - description: User-provided free-form text description. - parameter_default: - type: string - description: Default value of the parameter. - parameter_mode: - type: string - x-enum: - - IN - description: Function parameter mode. - parameter_type: - type: string - x-enum: - - COLUMN - - PARAM - description: Function parameter type. - type_interval_type: - type: string - description: Format of IntervalType. - type_json: - type: string - description: Full data type spec, JSON-serialized. - type_precision: - type: integer - description: >- - Digits of precision; required on Create for - DecimalTypes. - type_scale: - type: integer - description: >- - Digits to right of decimal; Required on Create - for DecimalTypes. - required: - - name - - type_text - - type_name - - position - description: Table function return parameters. - routine_dependencies: type: object - properties: - dependencies: - type: array - items: - type: object - properties: - connection: - type: object - properties: - connection_name: - type: string - description: >- - Full name of the dependent connection, in - the form of __connection_name__. - description: >- - A connection that is dependent on a SQL - object. - credential: - type: object - properties: - credential_name: - type: string - description: >- - Full name of the dependent credential, in - the form of __credential_name__. - description: >- - A credential that is dependent on a SQL - object. - function: - type: object - properties: - function_full_name: - type: string - description: >- - Full name of the dependent function, in - the form of - __catalog_name__.__schema_name__.__function_name__. - required: - - function_full_name - description: A function that is dependent on a SQL object. - table: - type: object - properties: - table_full_name: - type: string - description: >- - Full name of the dependent table, in the - form of - __catalog_name__.__schema_name__.__table_name__. - required: - - table_full_name - description: A table that is dependent on a SQL object. - description: >- - A dependency of a SQL object. One of the following - fields must be defined: __table__, - __function__, __connection__, or __credential__. - description: Array of dependencies. - description: function dependencies. - sql_path: - type: string description: >- - List of schemes whose objects can be referenced without - qualification. + Key-value properties associated with the external + lineage relationship. required: - - name - - catalog_name - - schema_name - - input_params - - data_type - - full_data_type - - routine_body - - routine_definition - - parameter_style - - is_deterministic - - sql_data_access - - is_null_call - - security_type - - specific_name - description: >- - Partial __FunctionInfo__ specifying the function to be - created. + - source + - target required: - - function_info + - external_lineage_relationship responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/FunctionInfo' + $ref: '#/components/schemas/ExternalLineageRelationship' default: description: Error response content: @@ -3364,109 +5237,33 @@ paths: type: string message: type: string - x-stackql-sdk-source: FunctionsAPI - get: - operationId: functions_list + x-stackql-sdk-source: ExternalLineageAPI + delete: + operationId: external_lineage_delete_external_lineage_relationship summary: >- - List functions within the specified parent catalog and schema. If the - user is a metastore admin, all + Deletes an external lineage relationship between a Databricks or + external metadata object and another tags: - catalog - - functions + - external_lineage description: >- - List functions within the specified parent catalog and schema. If the - user is a metastore admin, all - - functions are returned in the output list. Otherwise, the user must have - the **USE_CATALOG** privilege - - on the catalog and the **USE_SCHEMA** privilege on the schema, and the - output list contains only - - functions for which either the user has the **EXECUTE** privilege or the - user is the owner. There is - - no guarantee of a specific ordering of the elements in the array. - - - NOTE: we recommend using max_results=0 to use the paginated version of - this API. Unpaginated calls - - will be deprecated soon. - - - PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page - may contain zero results while - - still providing a next_page_token. Clients must continue reading pages - until next_page_token is - - absent, which is the only indication that the end of results has been - reached. + Deletes an external lineage relationship between a Databricks or + external metadata object and another + external metadata object. - :param catalog_name: str - Name of parent catalog for functions of interest. - :param schema_name: str - Parent schema of functions. - :param include_browse: bool (optional) - Whether to include functions in the response for which the principal can only access selective - metadata for - :param max_results: int (optional) - Maximum number of functions to return. If not set, all the functions are returned (not recommended). - - when set to a value greater than 0, the page length is the minimum of this value and a server - configured value; - when set to 0, the page length is set to a server configured value - (recommended); - when set to a value less than 0, an invalid parameter error is returned; - :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. - :returns: Iterator over :class:`FunctionInfo` + :param external_lineage_relationship: + :class:`DeleteRequestExternalLineage` parameters: - - name: catalog_name - in: query - required: true - schema: - type: string - description: Name of parent catalog for functions of interest. - - name: schema_name + - name: external_lineage_relationship in: query required: true schema: - type: string - description: Parent schema of functions. - - name: include_browse - in: query - required: false - schema: - type: boolean - description: >- - Whether to include functions in the response for which the principal - can only access selective metadata for - - name: max_results - in: query - required: false - schema: - type: integer - description: >- - Maximum number of functions to return. If not set, all the functions - are returned (not recommended). - when set to a value greater than - 0, the page length is the minimum of this value and a server - configured value; - when set to 0, the page length is set to a - server configured value (recommended); - when set to a value less - than 0, an invalid parameter error is returned; - - name: page_token - in: query - required: false - schema: - type: string - description: Opaque pagination token to go to next page based on previous query. + $ref: '#/components/schemas/DeleteRequestExternalLineage' responses: '200': description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListFunctionsResponse' default: description: Error response content: @@ -3478,126 +5275,74 @@ paths: type: string message: type: string - x-stackql-sdk-source: FunctionsAPI - /api/2.1/unity-catalog/functions/{name}: - delete: - operationId: functions_delete + x-stackql-sdk-source: ExternalLineageAPI + get: + operationId: external_lineage_list_external_lineage_relationships summary: >- - Deletes the function that matches the supplied name. For the deletion to - succeed, the user must + Lists external lineage relationships of a Databricks object or external + metadata given a supplied tags: - catalog - - functions + - external_lineage description: >- - Deletes the function that matches the supplied name. For the deletion to - succeed, the user must - - satisfy one of the following conditions: - Is the owner of the - function's parent catalog - Is the - - owner of the function's parent schema and have the **USE_CATALOG** - privilege on its parent catalog - + Lists external lineage relationships of a Databricks object or external + metadata given a supplied - Is the owner of the function itself and have both the **USE_CATALOG** - privilege on its parent catalog + direction. - and the **USE_SCHEMA** privilege on its parent schema + :param object_info: :class:`ExternalLineageObject` + The object to query external lineage relationships for. Since this field is a query parameter, + please flatten the nested fields. For example, if the object is a table, the query parameter should + look like: ``object_info.table.name=main.sales.customers`` + :param lineage_direction: :class:`LineageDirection` + The lineage direction to filter on. + :param page_size: int (optional) + Specifies the maximum number of external lineage relationships to return in a single response. The + value must be less than or equal to 1000. + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. - :param name: str - The fully-qualified name of the function (of the form - __catalog_name__.__schema_name__.__function__name__) . - :param force: bool (optional) - Force deletion even if the function is notempty. + :returns: Iterator over :class:`ExternalLineageInfo` parameters: - - name: name - in: path + - name: object_info + in: query required: true schema: - type: string + $ref: '#/components/schemas/ExternalLineageObject' description: >- - The fully-qualified name of the function (of the form - __catalog_name__.__schema_name__.__function__name__) . - - name: force + The object to query external lineage relationships for. Since this + field is a query parameter, please flatten the nested fields. For + example, if the object is a table, the query parameter should look + like: ``object_info.table.name=main.sales.customers`` + - name: lineage_direction in: query - required: false - schema: - type: boolean - description: Force deletion even if the function is notempty. - responses: - '200': - description: Success - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: FunctionsAPI - get: - operationId: functions_get - summary: >- - Gets a function from within a parent catalog and schema. For the fetch - to succeed, the user must - tags: - - catalog - - functions - description: >- - Gets a function from within a parent catalog and schema. For the fetch - to succeed, the user must - - satisfy one of the following requirements: - Is a metastore admin - Is - an owner of the function's - - parent catalog - Have the **USE_CATALOG** privilege on the function's - parent catalog and be the owner - - of the function - Have the **USE_CATALOG** privilege on the function's - parent catalog, the - - **USE_SCHEMA** privilege on the function's parent schema, and the - **EXECUTE** privilege on the - - function itself - - - :param name: str - The fully-qualified name of the function (of the form - __catalog_name__.__schema_name__.__function__name__). - :param include_browse: bool (optional) - Whether to include functions in the response for which the principal can only access selective - metadata for - - :returns: :class:`FunctionInfo` - parameters: - - name: name - in: path required: true schema: - type: string - description: >- - The fully-qualified name of the function (of the form - __catalog_name__.__schema_name__.__function__name__). - - name: include_browse + $ref: '#/components/schemas/LineageDirection' + description: The lineage direction to filter on. + - name: page_size in: query required: false schema: - type: boolean + type: integer description: >- - Whether to include functions in the response for which the principal - can only access selective metadata for + Specifies the maximum number of external lineage relationships to + return in a single response. The value must be less than or equal to + 1000. + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/FunctionInfo' + $ref: '#/components/schemas/ListExternalLineageRelationshipsResponse' default: description: Error response content: @@ -3609,66 +5354,156 @@ paths: type: string message: type: string - x-stackql-sdk-source: FunctionsAPI + x-stackql-sdk-source: ExternalLineageAPI patch: - operationId: functions_update + operationId: external_lineage_update_external_lineage_relationship summary: >- - Updates the function that matches the supplied name. Only the owner of - the function can be updated. If + Updates an external lineage relationship between a Databricks or + external metadata object and another tags: - catalog - - functions + - external_lineage description: >- - Updates the function that matches the supplied name. Only the owner of - the function can be updated. If - - the user is not a metastore admin, the user must be a member of the - group that is the new function - - owner. - Is a metastore admin - Is the owner of the function's parent - catalog - Is the owner of the + Updates an external lineage relationship between a Databricks or + external metadata object and another - function's parent schema and has the **USE_CATALOG** privilege on its - parent catalog - Is the owner of + external metadata object. - the function itself and has the **USE_CATALOG** privilege on its parent - catalog as well as the - **USE_SCHEMA** privilege on the function's parent schema. + :param external_lineage_relationship: + :class:`UpdateRequestExternalLineage` + :param update_mask: str + The field mask must be a single string, with multiple fields separated by commas (no spaces). The + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only + the entire collection field can be specified. Field names must exactly match the resource field + names. - :param name: str - The fully-qualified name of the function (of the form - __catalog_name__.__schema_name__.__function__name__). - :param owner: str (optional) - Username of current owner of the function. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. - :returns: :class:`FunctionInfo` + :returns: :class:`ExternalLineageRelationship` parameters: - - name: name - in: path + - name: update_mask + in: query required: true schema: type: string description: >- - The fully-qualified name of the function (of the form - __catalog_name__.__schema_name__.__function__name__). + The field mask must be a single string, with multiple fields + separated by commas (no spaces). The field path is relative to the + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map + fields is not allowed, as only the entire collection field can be + specified. Field names must exactly match the resource field names. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. requestBody: content: application/json: schema: type: object properties: - owner: - type: string - description: Username of current owner of the function. - responses: - '200': - description: Success - content: + external_lineage_relationship: + type: object + properties: + source: + type: object + properties: + external_metadata: + type: object + properties: + name: + type: string + model_version: + type: object + properties: + name: + type: string + version: + type: string + path: + type: object + properties: + url: + type: string + table: + type: object + properties: + name: + type: string + volume: + type: object + properties: + name: + type: string + target: + type: object + properties: + external_metadata: + type: object + properties: + name: + type: string + model_version: + type: object + properties: + name: + type: string + version: + type: string + path: + type: object + properties: + url: + type: string + table: + type: object + properties: + name: + type: string + volume: + type: object + properties: + name: + type: string + description: Target object of the external lineage relationship. + columns: + type: array + items: + type: object + properties: + source: + type: string + target: + type: string + description: >- + List of column relationships between source and target + objects. + id: + type: string + description: Unique identifier of the external lineage relationship. + properties: + type: object + description: >- + Key-value properties associated with the external + lineage relationship. + required: + - source + - target + required: + - external_lineage_relationship + responses: + '200': + description: Success + content: application/json: schema: - $ref: '#/components/schemas/FunctionInfo' + $ref: '#/components/schemas/ExternalLineageRelationship' default: description: Error response content: @@ -3680,150 +5515,2701 @@ paths: type: string message: type: string - x-stackql-sdk-source: FunctionsAPI - /api/2.1/unity-catalog/permissions/{securable_type}/{full_name}: - get: - operationId: grants_get + x-stackql-sdk-source: ExternalLineageAPI + /api/2.1/unity-catalog/external-locations: + post: + operationId: external_locations_create summary: >- - Gets the permissions for a securable. Does not include inherited - permissions. + Creates a new external location entry in the metastore. The caller must + be a metastore admin or have tags: - catalog - - grants + - external_locations description: >- - Gets the permissions for a securable. Does not include inherited - permissions. - - - NOTE: we recommend using max_results=0 to use the paginated version of - this API. Unpaginated calls - - will be deprecated soon. - - - PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page - may contain zero results while - - still providing a next_page_token. Clients must continue reading pages - until next_page_token is - - absent, which is the only indication that the end of results has been - reached. + Creates a new external location entry in the metastore. The caller must + be a metastore admin or have + the **CREATE_EXTERNAL_LOCATION** privilege on both the metastore and the + associated storage - :param securable_type: str - Type of securable. - :param full_name: str - Full name of securable. - :param max_results: int (optional) - Specifies the maximum number of privileges to return (page length). Every PrivilegeAssignment - present in a single page response is guaranteed to contain all the privileges granted on the - requested Securable for the respective principal. + credential. - If not set, all the permissions are returned. If set to - lesser than 0: invalid parameter error - - 0: page length is set to a server configured value - lesser than 150 but greater than 0: invalid - parameter error (this is to ensure that server is able to return at least one complete - PrivilegeAssignment in a single page response) - greater than (or equal to) 150: page length is the - minimum of this value and a server configured value - :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. - :param principal: str (optional) - If provided, only the permissions for the specified principal (user or group) are returned. - :returns: :class:`GetPermissionsResponse` - parameters: - - name: securable_type - in: path - required: true - schema: - type: string - description: Type of securable. - - name: full_name - in: path - required: true - schema: - type: string - description: Full name of securable. - - name: max_results - in: query - required: false - schema: - type: integer - description: >- - Specifies the maximum number of privileges to return (page length). - Every PrivilegeAssignment present in a single page response is - guaranteed to contain all the privileges granted on the requested - Securable for the respective principal. If not set, all the - permissions are returned. If set to - lesser than 0: invalid - parameter error - 0: page length is set to a server configured value - - lesser than 150 but greater than 0: invalid parameter error (this - is to ensure that server is able to return at least one complete - PrivilegeAssignment in a single page response) - greater than (or - equal to) 150: page length is the minimum of this value and a server - configured value - - name: page_token - in: query - required: false - schema: - type: string - description: Opaque pagination token to go to next page based on previous query. - - name: principal - in: query - required: false - schema: - type: string - description: >- - If provided, only the permissions for the specified principal (user - or group) are returned. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/GetPermissionsResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: GrantsAPI - patch: - operationId: grants_update - summary: Updates the permissions for a securable. - tags: - - catalog - - grants - description: |- - Updates the permissions for a securable. + :param name: str + Name of the external location. + :param url: str + Path URL of the external location. + :param credential_name: str + Name of the storage credential used with this location. + :param comment: str (optional) + User-provided free-form text description. + :param effective_enable_file_events: bool (optional) + The effective value of ``enable_file_events`` after applying server-side defaults. + :param effective_file_event_queue: :class:`FileEventQueue` (optional) + The effective file event queue configuration after applying server-side defaults. Always populated + when a queue is provisioned, regardless of whether the user explicitly set ``enable_file_events``. + Use this field instead of ``file_event_queue`` for reading the actual queue state. + :param enable_file_events: bool (optional) + Whether to enable file events on this external location. Default to ``true``. Set to ``false`` to + disable file events. The actual applied value may differ due to server-side defaults; check + ``effective_enable_file_events`` for the effective state. + :param encryption_details: :class:`EncryptionDetails` (optional) - :param securable_type: str - Type of securable. - :param full_name: str - Full name of securable. - :param changes: List[:class:`PermissionsChange`] (optional) - Array of permissions change objects. + :param fallback: bool (optional) + Indicates whether fallback mode is enabled for this external location. When fallback mode is + enabled, the access to the location falls back to cluster credentials if UC credentials are not + sufficient. + :param file_event_queue: :class:`FileEventQueue` (optional) + File event queue settings. If ``enable_file_events`` is not ``false``, must be defined and have + exactly one of the documented properties. + :param read_only: bool (optional) + Indicates whether the external location is read-only. + :param skip_validation: bool (optional) + Skips validation of the storage credential associated with the external location. - :returns: :class:`UpdatePermissionsResponse` - parameters: - - name: securable_type - in: path - required: true - schema: - type: string - description: Type of securable. - - name: full_name - in: path - required: true - schema: - type: string - description: Full name of securable. + :returns: :class:`ExternalLocationInfo` + requestBody: + content: + application/json: + schema: + type: object + properties: + name: + type: string + description: Name of the external location. + url: + type: string + description: Path URL of the external location. + credential_name: + type: string + description: Name of the storage credential used with this location. + comment: + type: string + description: User-provided free-form text description. + effective_enable_file_events: + type: boolean + description: >- + The effective value of ``enable_file_events`` after applying + server-side defaults. + effective_file_event_queue: + type: object + properties: + managed_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + managed_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + managed_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + managed_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + provided_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + provided_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + provided_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + provided_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + description: >- + The effective file event queue configuration after applying + server-side defaults. Always populated when a queue is + provisioned, regardless of whether the user explicitly set + ``enable_file_events``. Use this field instead of + ``file_event_queue`` for reading the actual queue state. + enable_file_events: + type: boolean + description: >- + Whether to enable file events on this external location. + Default to ``true``. Set to ``false`` to disable file + events. The actual applied value may differ due to + server-side defaults; check ``effective_enable_file_events`` + for the effective state. + encryption_details: + type: object + properties: + sse_encryption_details: + type: object + properties: + algorithm: + type: string + x-enum: + - AWS_SSE_KMS + - AWS_SSE_S3 + description: >- + Sets the value of the 'x-amz-server-side-encryption' + header in S3 request. + aws_kms_key_arn: + type: string + description: >- + Optional. The ARN of the SSE-KMS key used with the + S3 location, when algorithm = "SSE-KMS". Sets the + value of the + 'x-amz-server-side-encryption-aws-kms-key-id' + header. + description: >- + Server-Side Encryption properties for clients + communicating with AWS s3. + description: >- + Encryption options that apply to clients connecting to cloud + storage. + fallback: + type: boolean + description: >- + Indicates whether fallback mode is enabled for this external + location. When fallback mode is enabled, the access to the + location falls back to cluster credentials if UC credentials + are not sufficient. + file_event_queue: + type: object + properties: + managed_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + managed_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + managed_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + managed_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + provided_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + provided_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + provided_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + provided_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + description: >- + File event queue settings. If ``enable_file_events`` is not + ``false``, must be defined and have exactly one of the + documented properties. + read_only: + type: boolean + description: Indicates whether the external location is read-only. + skip_validation: + type: boolean + description: >- + Skips validation of the storage credential associated with + the external location. + required: + - name + - url + - credential_name + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalLocationInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalLocationsAPI + get: + operationId: external_locations_list + summary: >- + Gets an array of external locations (**ExternalLocationInfo** objects) + from the metastore. The caller + tags: + - catalog + - external_locations + description: >- + Gets an array of external locations (**ExternalLocationInfo** objects) + from the metastore. The caller + + must be a metastore admin, the owner of the external location, or a user + that has some privilege on + + the external location. There is no guarantee of a specific ordering of + the elements in the array. + + + NOTE: we recommend using max_results=0 to use the paginated version of + this API. Unpaginated calls + + will be deprecated soon. + + + PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page + may contain zero results while + + still providing a next_page_token. Clients must continue reading pages + until next_page_token is + + absent, which is the only indication that the end of results has been + reached. + + + :param include_browse: bool (optional) + Whether to include external locations in the response for which the principal can only access + selective metadata for + :param include_unbound: bool (optional) + Whether to include external locations not bound to the workspace. Effective only if the user has + permission to update the location–workspace binding. + :param max_results: int (optional) + Maximum number of external locations to return. If not set, all the external locations are returned + (not recommended). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. + + :returns: Iterator over :class:`ExternalLocationInfo` + parameters: + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include external locations in the response for which the + principal can only access selective metadata for + - name: include_unbound + in: query + required: false + schema: + type: boolean + description: >- + Whether to include external locations not bound to the workspace. + Effective only if the user has permission to update the + location–workspace binding. + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Maximum number of external locations to return. If not set, all the + external locations are returned (not recommended). - when set to a + value greater than 0, the page length is the minimum of this value + and a server configured value; - when set to 0, the page length is + set to a server configured value (recommended); - when set to a + value less than 0, an invalid parameter error is returned; + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListExternalLocationsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalLocationsAPI + /api/2.1/unity-catalog/external-locations/{name}: + delete: + operationId: external_locations_delete + summary: >- + Deletes the specified external location from the metastore. The caller + must be the owner of the + tags: + - catalog + - external_locations + description: >- + Deletes the specified external location from the metastore. The caller + must be the owner of the + + external location. + + + :param name: str + Name of the external location. + :param force: bool (optional) + Force deletion even if there are dependent external tables or mounts. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the external location. + - name: force + in: query + required: false + schema: + type: boolean + description: >- + Force deletion even if there are dependent external tables or + mounts. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalLocationsAPI + get: + operationId: external_locations_get + summary: >- + Gets an external location from the metastore. The caller must be either + a metastore admin, the owner + tags: + - catalog + - external_locations + description: >- + Gets an external location from the metastore. The caller must be either + a metastore admin, the owner + + of the external location, or a user that has some privilege on the + external location. + + + :param name: str + Name of the external location. + :param include_browse: bool (optional) + Whether to include external locations in the response for which the principal can only access + selective metadata for + + :returns: :class:`ExternalLocationInfo` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the external location. + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include external locations in the response for which the + principal can only access selective metadata for + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalLocationInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalLocationsAPI + patch: + operationId: external_locations_update + summary: >- + Updates an external location in the metastore. The caller must be the + owner of the external location, + tags: + - catalog + - external_locations + description: >- + Updates an external location in the metastore. The caller must be the + owner of the external location, + + or be a metastore admin. In the second case, the admin can only update + the name of the external + + location. + + + :param name: str + Name of the external location. + :param comment: str (optional) + User-provided free-form text description. + :param credential_name: str (optional) + Name of the storage credential used with this location. + :param effective_enable_file_events: bool (optional) + The effective value of ``enable_file_events`` after applying server-side defaults. + :param effective_file_event_queue: :class:`FileEventQueue` (optional) + The effective file event queue configuration after applying server-side defaults. Always populated + when a queue is provisioned, regardless of whether the user explicitly set ``enable_file_events``. + Use this field instead of ``file_event_queue`` for reading the actual queue state. + :param enable_file_events: bool (optional) + Whether to enable file events on this external location. Default to ``true``. Set to ``false`` to + disable file events. The actual applied value may differ due to server-side defaults; check + ``effective_enable_file_events`` for the effective state. + :param encryption_details: :class:`EncryptionDetails` (optional) + + :param fallback: bool (optional) + Indicates whether fallback mode is enabled for this external location. When fallback mode is + enabled, the access to the location falls back to cluster credentials if UC credentials are not + sufficient. + :param file_event_queue: :class:`FileEventQueue` (optional) + File event queue settings. If ``enable_file_events`` is not ``false``, must be defined and have + exactly one of the documented properties. + :param force: bool (optional) + Force update even if changing url invalidates dependent external tables or mounts. + :param isolation_mode: :class:`IsolationMode` (optional) + + :param new_name: str (optional) + New name for the external location. + :param owner: str (optional) + The owner of the external location. + :param read_only: bool (optional) + Indicates whether the external location is read-only. + :param skip_validation: bool (optional) + Skips validation of the storage credential associated with the external location. + :param url: str (optional) + Path URL of the external location. + + :returns: :class:`ExternalLocationInfo` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the external location. + requestBody: + content: + application/json: + schema: + type: object + properties: + comment: + type: string + description: User-provided free-form text description. + credential_name: + type: string + description: Name of the storage credential used with this location. + effective_enable_file_events: + type: boolean + description: >- + The effective value of ``enable_file_events`` after applying + server-side defaults. + effective_file_event_queue: + type: object + properties: + managed_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + managed_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + managed_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + managed_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + provided_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + provided_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + provided_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + provided_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + description: >- + The effective file event queue configuration after applying + server-side defaults. Always populated when a queue is + provisioned, regardless of whether the user explicitly set + ``enable_file_events``. Use this field instead of + ``file_event_queue`` for reading the actual queue state. + enable_file_events: + type: boolean + description: >- + Whether to enable file events on this external location. + Default to ``true``. Set to ``false`` to disable file + events. The actual applied value may differ due to + server-side defaults; check ``effective_enable_file_events`` + for the effective state. + encryption_details: + type: object + properties: + sse_encryption_details: + type: object + properties: + algorithm: + type: string + x-enum: + - AWS_SSE_KMS + - AWS_SSE_S3 + description: >- + Sets the value of the 'x-amz-server-side-encryption' + header in S3 request. + aws_kms_key_arn: + type: string + description: >- + Optional. The ARN of the SSE-KMS key used with the + S3 location, when algorithm = "SSE-KMS". Sets the + value of the + 'x-amz-server-side-encryption-aws-kms-key-id' + header. + description: >- + Server-Side Encryption properties for clients + communicating with AWS s3. + description: >- + Encryption options that apply to clients connecting to cloud + storage. + fallback: + type: boolean + description: >- + Indicates whether fallback mode is enabled for this external + location. When fallback mode is enabled, the access to the + location falls back to cluster credentials if UC credentials + are not sufficient. + file_event_queue: + type: object + properties: + managed_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + managed_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + managed_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + managed_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + provided_aqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format https://{storage + account}.queue.core.windows.net/{queue name} Only + required for provided_aqs. + resource_group: + type: string + description: >- + Optional resource group for the queue, event grid + subscription, and external location storage account. + Only required for locations with a service principal + storage credential + subscription_id: + type: string + description: >- + Optional subscription id for the queue, event grid + subscription, and external location storage account. + Required for locations with a service principal + storage credential + provided_onelake: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file + events. Defaults to "$Default" when unset. + Recommended for provided_onelake: create a dedicated + consumer group on the Event Hub for file events to + avoid contending with the customer's other + consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only + required for provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs + namespace, e.g. + {yournamespace}.servicebus.windows.net. Only + required for provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file + events managed resources. + description: >- + File event queue for OneLake (Microsoft Fabric) + locations. Events flow through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). + provided_pubsub: + type: object + properties: + managed_resource_id: + type: string + subscription_name: + type: string + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription + name}. Only required for provided_pubsub. + provided_sqs: + type: object + properties: + managed_resource_id: + type: string + queue_url: + type: string + description: >- + The AQS queue url in the format + https://sqs.{region}.amazonaws.com/{account + id}/{queue name}. Only required for provided_sqs. + description: >- + File event queue settings. If ``enable_file_events`` is not + ``false``, must be defined and have exactly one of the + documented properties. + force: + type: boolean + description: >- + Force update even if changing url invalidates dependent + external tables or mounts. + isolation_mode: + type: string + x-enum: + - ISOLATION_MODE_ISOLATED + - ISOLATION_MODE_OPEN + - ISOLATION_MODE_OPEN_IN_ACCOUNT + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members + they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can have + their own + + attributes -- see the documentation for details. + new_name: + type: string + description: New name for the external location. + owner: + type: string + description: The owner of the external location. + read_only: + type: boolean + description: Indicates whether the external location is read-only. + skip_validation: + type: boolean + description: >- + Skips validation of the storage credential associated with + the external location. + url: + type: string + description: Path URL of the external location. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalLocationInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalLocationsAPI + /api/2.0/lineage-tracking/external-metadata: + post: + operationId: external_metadata_create_external_metadata + summary: >- + Creates a new external metadata object in the parent metastore if the + caller is a metastore admin or + tags: + - catalog + - external_metadata + description: >- + Creates a new external metadata object in the parent metastore if the + caller is a metastore admin or + + has the **CREATE_EXTERNAL_METADATA** privilege. Grants **BROWSE** to all + account users upon creation + + by default. + + + :param external_metadata: :class:`ExternalMetadata` + + + :returns: :class:`ExternalMetadata` + requestBody: + content: + application/json: + schema: + type: object + properties: + external_metadata: + type: object + properties: + name: + type: string + system_type: + type: string + x-enum: + - AMAZON_REDSHIFT + - AZURE_SYNAPSE + - CONFLUENT + - DATABRICKS + - GOOGLE_BIGQUERY + - KAFKA + - LOOKER + - MICROSOFT_FABRIC + - MICROSOFT_SQL_SERVER + - MONGODB + - MYSQL + - ORACLE + - OTHER + - POSTGRESQL + - POWER_BI + - SALESFORCE + - SAP + - SERVICENOW + - SNOWFLAKE + - STREAM_NATIVE + - TABLEAU + - TERADATA + - WORKDAY + description: Type of external system. + entity_type: + type: string + description: Type of entity within the external system. + columns: + type: array + items: + type: string + description: >- + List of columns associated with the external metadata + object. + create_time: + type: string + description: Time at which this external metadata object was created. + created_by: + type: string + description: Username of external metadata object creator. + description: + type: string + description: User-provided free-form text description. + id: + type: string + description: Unique identifier of the external metadata object. + metastore_id: + type: string + description: Unique identifier of parent metastore. + owner: + type: string + description: Owner of the external metadata object. + properties: + type: object + description: >- + A map of key-value properties attached to the external + metadata object. + update_time: + type: string + description: >- + Time at which this external metadata object was last + modified. + updated_by: + type: string + description: >- + Username of user who last modified external metadata + object. + url: + type: string + description: URL associated with the external metadata object. + required: + - name + - system_type + - entity_type + required: + - external_metadata + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalMetadata' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalMetadataAPI + get: + operationId: external_metadata_list_external_metadata + summary: >- + Gets an array of external metadata objects in the metastore. If the + caller is the metastore admin, all + tags: + - catalog + - external_metadata + description: >- + Gets an array of external metadata objects in the metastore. If the + caller is the metastore admin, all + + external metadata objects will be retrieved. Otherwise, only external + metadata objects that the caller + + has **BROWSE** on will be retrieved. There is no guarantee of a specific + ordering of the elements in + + the array. + + + :param page_size: int (optional) + Specifies the maximum number of external metadata objects to return in a single response. The value + must be less than or equal to 1000. + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. + + :returns: Iterator over :class:`ExternalMetadata` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Specifies the maximum number of external metadata objects to return + in a single response. The value must be less than or equal to 1000. + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListExternalMetadataResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalMetadataAPI + /api/2.0/lineage-tracking/external-metadata/{name}: + delete: + operationId: external_metadata_delete_external_metadata + summary: >- + Deletes the external metadata object that matches the supplied name. The + caller must be a metastore + tags: + - catalog + - external_metadata + description: >- + Deletes the external metadata object that matches the supplied name. The + caller must be a metastore + + admin, the owner of the external metadata object, or a user that has the + **MANAGE** privilege. + + + :param name: str + parameters: + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalMetadataAPI + get: + operationId: external_metadata_get_external_metadata + summary: >- + Gets the specified external metadata object in a metastore. The caller + must be a metastore admin, the + tags: + - catalog + - external_metadata + description: >- + Gets the specified external metadata object in a metastore. The caller + must be a metastore admin, the + + owner of the external metadata object, or a user that has the **BROWSE** + privilege. + + + :param name: str + + + :returns: :class:`ExternalMetadata` + parameters: + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalMetadata' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalMetadataAPI + patch: + operationId: external_metadata_update_external_metadata + summary: >- + Updates the external metadata object that matches the supplied name. The + caller can only update either + tags: + - catalog + - external_metadata + description: >- + Updates the external metadata object that matches the supplied name. The + caller can only update either + + the owner or other metadata fields in one request. The caller must be a + metastore admin, the owner of + + the external metadata object, or a user that has the **MODIFY** + privilege. If the caller is updating + + the owner, they must also have the **MANAGE** privilege. + + + :param name: str + Name of the external metadata object. + :param external_metadata: :class:`ExternalMetadata` + + :param update_mask: str + The field mask must be a single string, with multiple fields separated by commas (no spaces). The + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only + the entire collection field can be specified. Field names must exactly match the resource field + names. + + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. + + :returns: :class:`ExternalMetadata` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the external metadata object. + - name: update_mask + in: query + required: true + schema: + type: string + description: >- + The field mask must be a single string, with multiple fields + separated by commas (no spaces). The field path is relative to the + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map + fields is not allowed, as only the entire collection field can be + specified. Field names must exactly match the resource field names. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. + requestBody: + content: + application/json: + schema: + type: object + properties: + external_metadata: + type: object + properties: + name: + type: string + system_type: + type: string + x-enum: + - AMAZON_REDSHIFT + - AZURE_SYNAPSE + - CONFLUENT + - DATABRICKS + - GOOGLE_BIGQUERY + - KAFKA + - LOOKER + - MICROSOFT_FABRIC + - MICROSOFT_SQL_SERVER + - MONGODB + - MYSQL + - ORACLE + - OTHER + - POSTGRESQL + - POWER_BI + - SALESFORCE + - SAP + - SERVICENOW + - SNOWFLAKE + - STREAM_NATIVE + - TABLEAU + - TERADATA + - WORKDAY + description: Type of external system. + entity_type: + type: string + description: Type of entity within the external system. + columns: + type: array + items: + type: string + description: >- + List of columns associated with the external metadata + object. + create_time: + type: string + description: Time at which this external metadata object was created. + created_by: + type: string + description: Username of external metadata object creator. + description: + type: string + description: User-provided free-form text description. + id: + type: string + description: Unique identifier of the external metadata object. + metastore_id: + type: string + description: Unique identifier of parent metastore. + owner: + type: string + description: Owner of the external metadata object. + properties: + type: object + description: >- + A map of key-value properties attached to the external + metadata object. + update_time: + type: string + description: >- + Time at which this external metadata object was last + modified. + updated_by: + type: string + description: >- + Username of user who last modified external metadata + object. + url: + type: string + description: URL associated with the external metadata object. + required: + - name + - system_type + - entity_type + required: + - external_metadata + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalMetadata' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExternalMetadataAPI + /api/2.1/unity-catalog/functions: + post: + operationId: functions_create + summary: '**WARNING: This API is experimental and will change in future versions**' + tags: + - catalog + - functions + description: >- + **WARNING: This API is experimental and will change in future versions** + + + Creates a new function + + + The user must have the following permissions in order for the function + to be created: + + + - **USE_CATALOG** on the function's parent catalog + + - **USE_SCHEMA** and **CREATE_FUNCTION** on the function's parent schema + + + :param function_info: :class:`CreateFunction` + Partial **FunctionInfo** specifying the function to be created. + + :returns: :class:`FunctionInfo` + requestBody: + content: + application/json: + schema: + type: object + properties: + function_info: + type: object + properties: + name: + type: string + catalog_name: + type: string + description: Name of parent Catalog. + schema_name: + type: string + description: Name of parent Schema relative to its parent Catalog. + input_params: + type: object + properties: + parameters: + type: array + items: + type: object + properties: + name: + type: string + type_text: + type: string + description: Full data type spec, SQL/catalogString text. + type_name: + type: string + x-enum: + - ARRAY + - BINARY + - BOOLEAN + - BYTE + - CHAR + - DATE + - DECIMAL + - DOUBLE + - FILE + - FLOAT + - GEOGRAPHY + - GEOMETRY + - INT + - INTERVAL + - LONG + - MAP + - 'NULL' + - SHORT + - STRING + - STRUCT + - TABLEREF_TYPE + - TABLE_TYPE + - TIME + - TIMESTAMP + - TIMESTAMP_NTZ + - USER_DEFINED_TYPE + - VARIANT + description: Name of type (INT, STRUCT, MAP, etc.) + position: + type: integer + description: >- + Ordinal position of column (starting at + position 0). + comment: + type: string + description: User-provided free-form text description. + parameter_default: + type: string + description: Default value of the parameter. + parameter_mode: + type: string + x-enum: + - IN + description: Function parameter mode. + parameter_type: + type: string + x-enum: + - COLUMN + - PARAM + description: Function parameter type. + type_interval_type: + type: string + description: Format of IntervalType. + type_json: + type: string + description: Full data type spec, JSON-serialized. + type_precision: + type: integer + description: >- + Digits of precision; required on Create for + DecimalTypes. + type_scale: + type: integer + description: >- + Digits to right of decimal; Required on Create + for DecimalTypes. + required: + - name + - type_text + - type_name + - position + description: Function input parameters. + data_type: + type: string + x-enum: + - ARRAY + - BINARY + - BOOLEAN + - BYTE + - CHAR + - DATE + - DECIMAL + - DOUBLE + - FILE + - FLOAT + - GEOGRAPHY + - GEOMETRY + - INT + - INTERVAL + - LONG + - MAP + - 'NULL' + - SHORT + - STRING + - STRUCT + - TABLEREF_TYPE + - TABLE_TYPE + - TIME + - TIMESTAMP + - TIMESTAMP_NTZ + - USER_DEFINED_TYPE + - VARIANT + description: Scalar function return data type. + full_data_type: + type: string + description: Pretty printed function data type. + routine_body: + type: string + x-enum: + - EXTERNAL + - SQL + description: >- + Function language. When **EXTERNAL** is used, the + language of the routine function should be specified in + the **external_language** field, and the + **return_params** of the function cannot be used (as + **TABLE** return type is not supported), and the + **sql_data_access** field must be **NO_SQL**. + routine_definition: + type: string + description: Function body. + parameter_style: + type: string + x-enum: + - S + description: Function parameter style. **S** is the value for SQL. + is_deterministic: + type: boolean + description: Whether the function is deterministic. + sql_data_access: + type: string + x-enum: + - CONTAINS_SQL + - NO_SQL + - READS_SQL_DATA + description: Function SQL data access. + is_null_call: + type: boolean + description: Function null call. + security_type: + type: string + x-enum: + - DEFINER + description: Function security type. + specific_name: + type: string + description: Specific name of the function; Reserved for future use. + comment: + type: string + description: User-provided free-form text description. + external_language: + type: string + description: External function language. + external_name: + type: string + description: External function name. + properties: + type: string + description: >- + JSON-serialized key-value pair map, encoded (escaped) as + a string. + return_params: + type: object + properties: + parameters: + type: array + items: + type: object + properties: + name: + type: string + type_text: + type: string + description: Full data type spec, SQL/catalogString text. + type_name: + type: string + x-enum: + - ARRAY + - BINARY + - BOOLEAN + - BYTE + - CHAR + - DATE + - DECIMAL + - DOUBLE + - FILE + - FLOAT + - GEOGRAPHY + - GEOMETRY + - INT + - INTERVAL + - LONG + - MAP + - 'NULL' + - SHORT + - STRING + - STRUCT + - TABLEREF_TYPE + - TABLE_TYPE + - TIME + - TIMESTAMP + - TIMESTAMP_NTZ + - USER_DEFINED_TYPE + - VARIANT + description: Name of type (INT, STRUCT, MAP, etc.) + position: + type: integer + description: >- + Ordinal position of column (starting at + position 0). + comment: + type: string + description: User-provided free-form text description. + parameter_default: + type: string + description: Default value of the parameter. + parameter_mode: + type: string + x-enum: + - IN + description: Function parameter mode. + parameter_type: + type: string + x-enum: + - COLUMN + - PARAM + description: Function parameter type. + type_interval_type: + type: string + description: Format of IntervalType. + type_json: + type: string + description: Full data type spec, JSON-serialized. + type_precision: + type: integer + description: >- + Digits of precision; required on Create for + DecimalTypes. + type_scale: + type: integer + description: >- + Digits to right of decimal; Required on Create + for DecimalTypes. + required: + - name + - type_text + - type_name + - position + description: Table function return parameters. + routine_dependencies: + type: object + properties: + dependencies: + type: array + items: + type: object + properties: + connection: + type: object + properties: + connection_name: + type: string + description: >- + Full name of the dependent connection, in + the form of **connection_name**. + description: >- + A connection that is dependent on a SQL + object. + credential: + type: object + properties: + credential_name: + type: string + description: >- + Full name of the dependent credential, in + the form of **credential_name**. + description: >- + A credential that is dependent on a SQL + object. + function: + type: object + properties: + function_full_name: + type: string + description: >- + Full name of the dependent function, in + the form of + **catalog_name**.**schema_name**.**function_name**. + required: + - function_full_name + description: A function that is dependent on a SQL object. + metastore_name: + type: string + description: >- + Metastore context for the dependency. Global + UC server responses currently populate this + with the metastore UUID string so clients can + disambiguate dependencies with identical names + across metastores. + secret: + type: object + properties: + secret_full_name: + type: string + description: >- + Full name of the dependent secret, in the + form of + **catalog_name**.**schema_name**.**secret_name**. + required: + - secret_full_name + description: A dependency on a Unity Catalog secret. + table: + type: object + properties: + table_full_name: + type: string + description: >- + Full name of the dependent table, in the + form of + **catalog_name**.**schema_name**.**table_name**. + required: + - table_full_name + description: A table that is dependent on a SQL object. + volume: + type: object + properties: + volume_full_name: + type: string + description: >- + Full name of the dependent volume, in the + form of + **catalog_name**.**schema_name**.**volume_name**. + required: + - volume_full_name + description: A dependency on a Unity Catalog volume. + description: >- + A dependency of a SQL object. One of the following + fields must be defined: **table**, + **function**, **connection**, **credential**, **volume**, or **secret**. + description: Array of dependencies. + description: function dependencies. + sql_path: + type: string + description: >- + List of schemes whose objects can be referenced without + qualification. + required: + - name + - catalog_name + - schema_name + - input_params + - data_type + - full_data_type + - routine_body + - routine_definition + - parameter_style + - is_deterministic + - sql_data_access + - is_null_call + - security_type + - specific_name + description: >- + Partial **FunctionInfo** specifying the function to be + created. + required: + - function_info + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FunctionInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FunctionsAPI + get: + operationId: functions_list + summary: >- + List functions within the specified parent catalog and schema. If the + user is a metastore admin, all + tags: + - catalog + - functions + description: >- + List functions within the specified parent catalog and schema. If the + user is a metastore admin, all + + functions are returned in the output list. Otherwise, the user must have + the **USE_CATALOG** privilege + + on the catalog and the **USE_SCHEMA** privilege on the schema, and the + output list contains only + + functions for which either the user has the **EXECUTE** privilege or the + user is the owner. There is + + no guarantee of a specific ordering of the elements in the array. + + + NOTE: we recommend using max_results=0 to use the paginated version of + this API. Unpaginated calls + + will be deprecated soon. + + + PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page + may contain zero results while + + still providing a next_page_token. Clients must continue reading pages + until next_page_token is + + absent, which is the only indication that the end of results has been + reached. + + + :param catalog_name: str + Name of parent catalog for functions of interest. + :param schema_name: str + Parent schema of functions. + :param include_browse: bool (optional) + Whether to include functions in the response for which the principal can only access selective + metadata for + :param max_results: int (optional) + Maximum number of functions to return. If not set, all the functions are returned (not recommended). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. + + :returns: Iterator over :class:`FunctionInfo` + parameters: + - name: catalog_name + in: query + required: true + schema: + type: string + description: Name of parent catalog for functions of interest. + - name: schema_name + in: query + required: true + schema: + type: string + description: Parent schema of functions. + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include functions in the response for which the principal + can only access selective metadata for + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Maximum number of functions to return. If not set, all the functions + are returned (not recommended). - when set to a value greater than + 0, the page length is the minimum of this value and a server + configured value; - when set to 0, the page length is set to a + server configured value (recommended); - when set to a value less + than 0, an invalid parameter error is returned; + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListFunctionsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FunctionsAPI + /api/2.1/unity-catalog/functions/{name}: + delete: + operationId: functions_delete + summary: >- + Deletes the function that matches the supplied name. For the deletion to + succeed, the user must + tags: + - catalog + - functions + description: >- + Deletes the function that matches the supplied name. For the deletion to + succeed, the user must + + satisfy one of the following conditions: + + + - Is the owner of the function's parent catalog + + - Is the owner of the function's parent schema and have the + **USE_CATALOG** privilege on its parent + catalog + - Is the owner of the function itself and have both the **USE_CATALOG** + privilege on its parent + catalog and the **USE_SCHEMA** privilege on its parent schema + + :param name: str + The fully-qualified name of the function (of the form + **catalog_name**.**schema_name**.**function__name**) . + :param force: bool (optional) + Force deletion even if the function is notempty. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The fully-qualified name of the function (of the form + **catalog_name**.**schema_name**.**function__name**) . + - name: force + in: query + required: false + schema: + type: boolean + description: Force deletion even if the function is notempty. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FunctionsAPI + get: + operationId: functions_get + summary: >- + Gets a function from within a parent catalog and schema. For the fetch + to succeed, the user must + tags: + - catalog + - functions + description: >- + Gets a function from within a parent catalog and schema. For the fetch + to succeed, the user must + + satisfy one of the following requirements: + + + - Is a metastore admin + + - Is an owner of the function's parent catalog + + - Have the **USE_CATALOG** privilege on the function's parent catalog + and be the owner of the function + + - Have the **USE_CATALOG** privilege on the function's parent catalog, + the **USE_SCHEMA** privilege on + the function's parent schema, and the **EXECUTE** privilege on the function itself + + :param name: str + The fully-qualified name of the function (of the form + **catalog_name**.**schema_name**.**function__name**). + :param include_browse: bool (optional) + Whether to include functions in the response for which the principal can only access selective + metadata for + + :returns: :class:`FunctionInfo` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The fully-qualified name of the function (of the form + **catalog_name**.**schema_name**.**function__name**). + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include functions in the response for which the principal + can only access selective metadata for + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FunctionInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FunctionsAPI + patch: + operationId: functions_update + summary: >- + Updates the function that matches the supplied name. Only the owner of + the function can be updated. If + tags: + - catalog + - functions + description: >- + Updates the function that matches the supplied name. Only the owner of + the function can be updated. If + + the user is not a metastore admin, the user must be a member of the + group that is the new function + + owner. + + + - Is a metastore admin + + - Is the owner of the function's parent catalog + + - Is the owner of the function's parent schema and has the + **USE_CATALOG** privilege on its parent + catalog + - Is the owner of the function itself and has the **USE_CATALOG** + privilege on its parent catalog as + well as the **USE_SCHEMA** privilege on the function's parent schema. + + :param name: str + The fully-qualified name of the function (of the form + **catalog_name**.**schema_name**.**function__name**). + :param owner: str (optional) + Username of current owner of the function. + + :returns: :class:`FunctionInfo` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The fully-qualified name of the function (of the form + **catalog_name**.**schema_name**.**function__name**). + requestBody: + content: + application/json: + schema: + type: object + properties: + owner: + type: string + description: Username of current owner of the function. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FunctionInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FunctionsAPI + /api/2.1/unity-catalog/permissions/{securable_type}/{full_name}: + get: + operationId: grants_get + summary: >- + Gets the permissions for a securable. Does not include inherited + permissions. + tags: + - catalog + - grants + description: >- + Gets the permissions for a securable. Does not include inherited + permissions. + + + NOTE: we recommend using max_results=0 to use the paginated version of + this API. Unpaginated calls + + will be deprecated soon. + + + PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page + may contain zero results while + + still providing a next_page_token. Clients must continue reading pages + until next_page_token is + + absent, which is the only indication that the end of results has been + reached. + + + :param securable_type: str + Type of securable. + :param full_name: str + Full name of securable. + :param include_deleted_principals: bool (optional) + Optional. If true, also return privilege assignments whose principals have been deleted. + :param max_results: int (optional) + Specifies the maximum number of privileges to return (page length). Every PrivilegeAssignment + present in a single page response is guaranteed to contain all the privileges granted on the + requested Securable for the respective principal. + + If not set, all the permissions are returned. If set to + + - lesser than 0: invalid parameter error + - 0: page length is set to a server configured value + - lesser than 150 but greater than 0: invalid parameter error (this is to ensure that server is able + to return at least one complete PrivilegeAssignment in a single page response) + - greater than (or equal to) 150: page length is the minimum of this value and a server configured + value + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. + :param principal: str (optional) + If provided, only the permissions for the specified principal (user or group) are returned. + + :returns: :class:`GetPermissionsResponse` + parameters: + - name: securable_type + in: path + required: true + schema: + type: string + description: Type of securable. + - name: full_name + in: path + required: true + schema: + type: string + description: Full name of securable. + - name: include_deleted_principals + in: query + required: false + schema: + type: boolean + description: >- + Optional. If true, also return privilege assignments whose + principals have been deleted. + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Specifies the maximum number of privileges to return (page length). + Every PrivilegeAssignment present in a single page response is + guaranteed to contain all the privileges granted on the requested + Securable for the respective principal. If not set, all the + permissions are returned. If set to - lesser than 0: invalid + parameter error - 0: page length is set to a server configured value + - lesser than 150 but greater than 0: invalid parameter error (this + is to ensure that server is able to return at least one complete + PrivilegeAssignment in a single page response) - greater than (or + equal to) 150: page length is the minimum of this value and a server + configured value + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. + - name: principal + in: query + required: false + schema: + type: string + description: >- + If provided, only the permissions for the specified principal (user + or group) are returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GetPermissionsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GrantsAPI + patch: + operationId: grants_update + summary: Updates the permissions for a securable. + tags: + - catalog + - grants + description: |- + Updates the permissions for a securable. + + :param securable_type: str + Type of securable. + :param full_name: str + Full name of securable. + :param changes: List[:class:`PermissionsChange`] (optional) + Array of permissions change objects. + :param omit_permissions_in_response: bool (optional) + Optional, default false. Specifies whether all the permissions should be returned in the response. + + :returns: :class:`UpdatePermissionsResponse` + parameters: + - name: securable_type + in: path + required: true + schema: + type: string + description: Type of securable. + - name: full_name + in: path + required: true + schema: + type: string + description: Full name of securable. requestBody: content: application/json: @@ -3845,40 +8231,70 @@ paths: - APPLY_TAG - BROWSE - CREATE + - CREATE_CASE_COLLECTION - CREATE_CATALOG - CREATE_CLEAN_ROOM - CREATE_CONNECTION + - CREATE_DATASOURCE - CREATE_EXTERNAL_LOCATION - CREATE_EXTERNAL_TABLE - CREATE_EXTERNAL_VOLUME + - CREATE_FEATURE + - CREATE_FLOW - CREATE_FOREIGN_CATALOG - CREATE_FOREIGN_SECURABLE - CREATE_FUNCTION - CREATE_MANAGED_STORAGE - CREATE_MATERIALIZED_VIEW + - CREATE_MEMORY_STORE - CREATE_MODEL - CREATE_PROVIDER - CREATE_RECIPIENT + - CREATE_RULE - CREATE_SCHEMA + - CREATE_SEMANTIC_GRAPH + - CREATE_SERVICE - CREATE_SERVICE_CREDENTIAL - CREATE_SHARE + - CREATE_SKILL - CREATE_STORAGE_CREDENTIAL + - CREATE_STREAM - CREATE_TABLE - CREATE_VIEW - CREATE_VOLUME + - DELETE + - DELETE_EVENTS + - DELETE_SECURITY_DATA - EXECUTE - EXECUTE_CLEAN_ROOM_TASK - EXTERNAL_USE_SCHEMA + - INSERT + - INSERT_SECURITY_DATA - MANAGE + - MANAGE_ACCESS + - MANAGE_ACCESS_CONTROL - MANAGE_ALLOWLIST + - MANAGE_GRANTS - MODIFY - MODIFY_CLEAN_ROOM + - READ_EVENTS + - READ_FEATURE - READ_FILES + - READ_FLOW + - READ_MEMORY_STORE + - READ_METADATA - READ_PRIVATE_FILES + - READ_SECURITY_DATA + - READ_SEMANTIC_GRAPH + - READ_SKILL + - READ_STREAM - READ_VOLUME - REFRESH - SELECT - SET_SHARE_PERMISSION + - UPDATE + - UPDATE_EVENTS + - UPDATE_SECURITY_DATA - USAGE - USE_CATALOG - USE_CONNECTION @@ -3887,8 +8303,16 @@ paths: - USE_RECIPIENT - USE_SCHEMA - USE_SHARE + - USE_VOLUME + - VIEW_ADMIN_METADATA + - VIEW_METADATA + - VIEW_OBJECT - WRITE_FILES + - WRITE_FLOW + - WRITE_MEMORY_STORE - WRITE_PRIVATE_FILES + - WRITE_SEMANTIC_GRAPH + - WRITE_SKILL - WRITE_VOLUME description: >- Create a collection of name/value pairs. @@ -3948,6 +8372,17 @@ paths: The principal whose privileges we are changing. Only one of principal or principal_id should be specified, never both at the same time. + principal_id: + type: integer + description: >- + An opaque internal ID that identifies the principal + whose privileges should be removed. This field is + intended for removing privileges associated with a + deleted user. When set, only the entries specified in + the remove field are processed; any entries in the add + field will be rejected. Only one of principal or + principal_id should be specified, never both at the + same time. remove: type: array items: @@ -3958,40 +8393,70 @@ paths: - APPLY_TAG - BROWSE - CREATE + - CREATE_CASE_COLLECTION - CREATE_CATALOG - CREATE_CLEAN_ROOM - CREATE_CONNECTION + - CREATE_DATASOURCE - CREATE_EXTERNAL_LOCATION - CREATE_EXTERNAL_TABLE - CREATE_EXTERNAL_VOLUME + - CREATE_FEATURE + - CREATE_FLOW - CREATE_FOREIGN_CATALOG - CREATE_FOREIGN_SECURABLE - CREATE_FUNCTION - CREATE_MANAGED_STORAGE - CREATE_MATERIALIZED_VIEW + - CREATE_MEMORY_STORE - CREATE_MODEL - CREATE_PROVIDER - CREATE_RECIPIENT + - CREATE_RULE - CREATE_SCHEMA + - CREATE_SEMANTIC_GRAPH + - CREATE_SERVICE - CREATE_SERVICE_CREDENTIAL - CREATE_SHARE + - CREATE_SKILL - CREATE_STORAGE_CREDENTIAL + - CREATE_STREAM - CREATE_TABLE - CREATE_VIEW - CREATE_VOLUME + - DELETE + - DELETE_EVENTS + - DELETE_SECURITY_DATA - EXECUTE - EXECUTE_CLEAN_ROOM_TASK - EXTERNAL_USE_SCHEMA + - INSERT + - INSERT_SECURITY_DATA - MANAGE + - MANAGE_ACCESS + - MANAGE_ACCESS_CONTROL - MANAGE_ALLOWLIST + - MANAGE_GRANTS - MODIFY - MODIFY_CLEAN_ROOM + - READ_EVENTS + - READ_FEATURE - READ_FILES + - READ_FLOW + - READ_MEMORY_STORE + - READ_METADATA - READ_PRIVATE_FILES + - READ_SECURITY_DATA + - READ_SEMANTIC_GRAPH + - READ_SKILL + - READ_STREAM - READ_VOLUME - REFRESH - SELECT - SET_SHARE_PERMISSION + - UPDATE + - UPDATE_EVENTS + - UPDATE_SECURITY_DATA - USAGE - USE_CATALOG - USE_CONNECTION @@ -4000,8 +8465,16 @@ paths: - USE_RECIPIENT - USE_SCHEMA - USE_SHARE + - USE_VOLUME + - VIEW_ADMIN_METADATA + - VIEW_METADATA + - VIEW_OBJECT - WRITE_FILES + - WRITE_FLOW + - WRITE_MEMORY_STORE - WRITE_PRIVATE_FILES + - WRITE_SEMANTIC_GRAPH + - WRITE_SKILL - WRITE_VOLUME description: >- Create a collection of name/value pairs. @@ -4057,13 +8530,250 @@ paths: attributes -- see the documentation for details. description: The set of privileges to remove. description: Array of permissions change objects. + omit_permissions_in_response: + type: boolean + description: >- + Optional, default false. Specifies whether all the + permissions should be returned in the response. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UpdatePermissionsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GrantsAPI + /api/2.1/unity-catalog/effective-permissions/{securable_type}/{full_name}: + get: + operationId: grants_get_effective + summary: >- + Gets the effective permissions for a securable. Includes inherited + permissions from any parent + tags: + - catalog + - grants + description: >- + Gets the effective permissions for a securable. Includes inherited + permissions from any parent + + securables. + + + NOTE: we recommend using max_results=0 to use the paginated version of + this API. Unpaginated calls + + will be deprecated soon. + + + PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page + may contain zero results while + + still providing a next_page_token. Clients must continue reading pages + until next_page_token is + + absent, which is the only indication that the end of results has been + reached. + + + :param securable_type: str + Type of securable. + :param full_name: str + Full name of securable. + :param max_results: int (optional) + Specifies the maximum number of privileges to return (page length). Every + EffectivePrivilegeAssignment present in a single page response is guaranteed to contain all the + effective privileges granted on (or inherited by) the requested Securable for the respective + principal. + + If not set, all the effective permissions are returned. If set to + + - lesser than 0: invalid parameter error + - 0: page length is set to a server configured value + - lesser than 150 but greater than 0: invalid parameter error (this is to ensure that server is able + to return at least one complete EffectivePrivilegeAssignment in a single page response) + - greater than (or equal to) 150: page length is the minimum of this value and a server configured + value + :param page_token: str (optional) + Opaque token for the next page of results (pagination). + :param principal: str (optional) + If provided, only the effective permissions for the specified principal (user or group) are + returned. + + :returns: :class:`EffectivePermissionsList` + parameters: + - name: securable_type + in: path + required: true + schema: + type: string + description: Type of securable. + - name: full_name + in: path + required: true + schema: + type: string + description: Full name of securable. + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Specifies the maximum number of privileges to return (page length). + Every EffectivePrivilegeAssignment present in a single page response + is guaranteed to contain all the effective privileges granted on (or + inherited by) the requested Securable for the respective principal. + If not set, all the effective permissions are returned. If set to - + lesser than 0: invalid parameter error - 0: page length is set to a + server configured value - lesser than 150 but greater than 0: + invalid parameter error (this is to ensure that server is able to + return at least one complete EffectivePrivilegeAssignment in a + single page response) - greater than (or equal to) 150: page length + is the minimum of this value and a server configured value + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque token for the next page of results (pagination). + - name: principal + in: query + required: false + schema: + type: string + description: >- + If provided, only the effective permissions for the specified + principal (user or group) are returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/EffectivePermissionsList' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GrantsAPI + /api/2.1/unity-catalog/privilege-assignments/{securable_type}/{full_name}: + get: + operationId: grants_list + summary: >- + Lists the privilege assignments for a securable. Does not include + inherited privileges. Paginated + tags: + - catalog + - grants + description: >- + Lists the privilege assignments for a securable. Does not include + inherited privileges. Paginated + + version of Get Permissions API. + + + :param securable_type: str + Type of securable. + :param full_name: str + Full name of securable. + :param include_deleted_principals: bool (optional) + Optional. If true, also return privilege assignments whose principals have been deleted. + :param page_size: int (optional) + Specifies the maximum number of privilege assignments to return (page length). Every + PrivilegeAssignment present in a single page response is guaranteed to contain all the privileges + granted on the requested Securable for the respective principal. + + If not set, page length is the server configured value. If set to + + - lesser than 0: invalid parameter error + - 0: page length is set to a server configured value + - lesser than 150 but greater than 0: invalid parameter error (this is to ensure that server is able + to return at least one complete PrivilegeAssignment in a single page response) + - greater than (or equal to) 150: page length is the minimum of this value and a server configured + value + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. + :param principal: str (optional) + If provided, only the permissions for the specified principal (user or group) are returned. + + :returns: Iterator over :class:`PrivilegeAssignment` + parameters: + - name: securable_type + in: path + required: true + schema: + type: string + description: Type of securable. + - name: full_name + in: path + required: true + schema: + type: string + description: Full name of securable. + - name: include_deleted_principals + in: query + required: false + schema: + type: boolean + description: >- + Optional. If true, also return privilege assignments whose + principals have been deleted. + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + Specifies the maximum number of privilege assignments to return + (page length). Every PrivilegeAssignment present in a single page + response is guaranteed to contain all the privileges granted on the + requested Securable for the respective principal. If not set, page + length is the server configured value. If set to - lesser than 0: + invalid parameter error - 0: page length is set to a server + configured value - lesser than 150 but greater than 0: invalid + parameter error (this is to ensure that server is able to return at + least one complete PrivilegeAssignment in a single page response) - + greater than (or equal to) 150: page length is the minimum of this + value and a server configured value + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. + - name: principal + in: query + required: false + schema: + type: string + description: >- + If provided, only the permissions for the specified principal (user + or group) are returned. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/UpdatePermissionsResponse' + $ref: '#/components/schemas/GetPermissionsResponse' default: description: Error response content: @@ -4076,60 +8786,47 @@ paths: message: type: string x-stackql-sdk-source: GrantsAPI - /api/2.1/unity-catalog/effective-permissions/{securable_type}/{full_name}: + /api/2.1/unity-catalog/effective-privilege-assignments/{securable_type}/{full_name}: get: - operationId: grants_get_effective + operationId: grants_list_effective summary: >- - Gets the effective permissions for a securable. Includes inherited - permissions from any parent + Lists the effective privilege assignments for a securable. Includes + inherited privileges. Paginated tags: - catalog - grants description: >- - Gets the effective permissions for a securable. Includes inherited - permissions from any parent - - securables. - - - NOTE: we recommend using max_results=0 to use the paginated version of - this API. Unpaginated calls - - will be deprecated soon. - - - PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page - may contain zero results while - - still providing a next_page_token. Clients must continue reading pages - until next_page_token is + Lists the effective privilege assignments for a securable. Includes + inherited privileges. Paginated - absent, which is the only indication that the end of results has been - reached. + version of Get Effective Permissions API. :param securable_type: str Type of securable. :param full_name: str Full name of securable. - :param max_results: int (optional) - Specifies the maximum number of privileges to return (page length). Every + :param page_size: int (optional) + Specifies the maximum number of privilege assignments to return (page length). Every EffectivePrivilegeAssignment present in a single page response is guaranteed to contain all the effective privileges granted on (or inherited by) the requested Securable for the respective principal. - If not set, all the effective permissions are returned. If set to - lesser than 0: invalid parameter - error - 0: page length is set to a server configured value - lesser than 150 but greater than 0: - invalid parameter error (this is to ensure that server is able to return at least one complete - EffectivePrivilegeAssignment in a single page response) - greater than (or equal to) 150: page - length is the minimum of this value and a server configured value + If not set, a server-configured default is used. If set to + + - lesser than 0: invalid parameter error + - 0: page length is set to a server configured value + - lesser than 150 but greater than 0: invalid parameter error (this is to ensure that server is able + to return at least one complete EffectivePrivilegeAssignment in a single page response) + - greater than (or equal to) 150: page length is the minimum of this value and a server configured + value :param page_token: str (optional) - Opaque token for the next page of results (pagination). + Opaque pagination token to go to next page based on previous query. :param principal: str (optional) If provided, only the effective permissions for the specified principal (user or group) are returned. - :returns: :class:`EffectivePermissionsList` + :returns: Iterator over :class:`EffectivePrivilegeAssignment` parameters: - name: securable_type in: path @@ -4143,29 +8840,30 @@ paths: schema: type: string description: Full name of securable. - - name: max_results + - name: page_size in: query required: false schema: type: integer description: >- - Specifies the maximum number of privileges to return (page length). - Every EffectivePrivilegeAssignment present in a single page response - is guaranteed to contain all the effective privileges granted on (or - inherited by) the requested Securable for the respective principal. - If not set, all the effective permissions are returned. If set to - - lesser than 0: invalid parameter error - 0: page length is set to a - server configured value - lesser than 150 but greater than 0: - invalid parameter error (this is to ensure that server is able to - return at least one complete EffectivePrivilegeAssignment in a - single page response) - greater than (or equal to) 150: page length - is the minimum of this value and a server configured value + Specifies the maximum number of privilege assignments to return + (page length). Every EffectivePrivilegeAssignment present in a + single page response is guaranteed to contain all the effective + privileges granted on (or inherited by) the requested Securable for + the respective principal. If not set, a server-configured default is + used. If set to - lesser than 0: invalid parameter error - 0: page + length is set to a server configured value - lesser than 150 but + greater than 0: invalid parameter error (this is to ensure that + server is able to return at least one complete + EffectivePrivilegeAssignment in a single page response) - greater + than (or equal to) 150: page length is the minimum of this value and + a server configured value - name: page_token in: query required: false schema: type: string - description: Opaque token for the next page of results (pagination). + description: Opaque pagination token to go to next page based on previous query. - name: principal in: query required: false @@ -4198,15 +8896,15 @@ paths: operationId: metastores_assign summary: >- Creates a new metastore assignment. If an assignment for the same - __workspace_id__ exists, it will be + **workspace_id** exists, it will be tags: - catalog - metastores description: >- Creates a new metastore assignment. If an assignment for the same - __workspace_id__ exists, it will be + **workspace_id** exists, it will be - overwritten by the new __metastore_id__ and __default_catalog_name__. + overwritten by the new **metastore_id** and **default_catalog_name**. The caller must be an account admin. @@ -4308,18 +9006,18 @@ paths: operationId: metastores_update_assignment summary: >- Updates a metastore assignment. This operation can be used to update - __metastore_id__ or + **metastore_id** or tags: - catalog - metastores description: >- Updates a metastore assignment. This operation can be used to update - __metastore_id__ or + **metastore_id** or - __default_catalog_name__ for a specified Workspace, if the Workspace is + **default_catalog_name** for a specified Workspace, if the Workspace is already assigned a metastore. - The caller must be an account admin to update __metastore_id__; + The caller must be an account admin to update **metastore_id**; otherwise, the caller can be a Workspace admin. @@ -4382,10 +9080,10 @@ paths: Creates a new metastore based on a provided name and optional storage root path. By default (if the - __owner__ field is not set), the owner of the new metastore is the user + **owner** field is not set), the owner of the new metastore is the user calling the - __createMetastore__ API. If the __owner__ field is set to the empty + **createMetastore** API. If the **owner** field is set to the empty string (**""**), the ownership is assigned to the System User instead. @@ -4396,7 +9094,7 @@ paths: :param external_access_enabled: bool (optional) Whether to allow non-DBR clients to directly access entities under the metastore. :param region: str (optional) - Cloud region which the metastore serves (e.g., `us-west-2`, `westus`). + Cloud region which the metastore serves (e.g., ``us-west-2``, ``westus``). :param storage_root: str (optional) The storage root URL for metastore @@ -4418,8 +9116,8 @@ paths: region: type: string description: >- - Cloud region which the metastore serves (e.g., `us-west-2`, - `westus`). + Cloud region which the metastore serves (e.g., + ``us-west-2``, ``westus``). storage_root: type: string description: The storage root URL for metastore @@ -4447,13 +9145,13 @@ paths: get: operationId: metastores_list summary: >- - Gets an array of the available metastores (as __MetastoreInfo__ + Gets an array of the available metastores (as **MetastoreInfo** objects). The caller must be an admin tags: - catalog - metastores description: >- - Gets an array of the available metastores (as __MetastoreInfo__ + Gets an array of the available metastores (as **MetastoreInfo** objects). The caller must be an admin to retrieve this info. There is no guarantee of a specific ordering of @@ -4477,13 +9175,16 @@ paths: :param max_results: int (optional) - Maximum number of metastores to return. - when set to a value greater than 0, the page length is the - minimum of this value and a server configured value; - when set to 0, the page length is set to a - server configured value (recommended); - when set to a value less than 0, an invalid parameter error - is returned; - If not set, all the metastores are returned (not recommended). - Note: The number of - returned metastores might be less than the specified max_results size, even zero. The only - definitive indication that no further metastores can be fetched is when the next_page_token is unset - from the response. + Maximum number of metastores to return. + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all the metastores are returned (not recommended). + - Note: The number of returned metastores might be less than the specified max_results size, even + zero. The only definitive indication that no further metastores can be fetched is when the + next_page_token is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -4652,13 +9353,13 @@ paths: operationId: metastores_update summary: >- Updates information for a specific metastore. The caller must be a - metastore admin. If the __owner__ + metastore admin. If the **owner** tags: - catalog - metastores description: >- Updates information for a specific metastore. The caller must be a - metastore admin. If the __owner__ + metastore admin. If the **owner** field is set to the empty string (**""**), the ownership is updated to the System User. @@ -4680,7 +9381,7 @@ paths: :param owner: str (optional) The owner of the metastore. :param privilege_model_version: str (optional) - Privilege model version of the metastore, of the form `major.minor` (e.g., `1.0`). + Privilege model version of the metastore, of the form ``major.minor`` (e.g., ``1.0``). :param storage_root_credential_id: str (optional) UUID of storage credential to access the metastore storage_root. @@ -4728,7 +9429,7 @@ paths: type: string description: >- Privilege model version of the metastore, of the form - `major.minor` (e.g., `1.0`). + ``major.minor`` (e.g., ``1.0``). storage_root_credential_id: type: string description: >- @@ -5079,7 +9780,7 @@ paths: type: string description: >- Full name of the dependent connection, in the - form of __connection_name__. + form of **connection_name**. description: A connection that is dependent on a SQL object. credential: type: object @@ -5088,7 +9789,7 @@ paths: type: string description: >- Full name of the dependent credential, in the - form of __credential_name__. + form of **credential_name**. description: A credential that is dependent on a SQL object. function: type: object @@ -5098,10 +9799,30 @@ paths: description: >- Full name of the dependent function, in the form of - __catalog_name__.__schema_name__.__function_name__. + **catalog_name**.**schema_name**.**function_name**. required: - function_full_name description: A function that is dependent on a SQL object. + metastore_name: + type: string + description: >- + Metastore context for the dependency. Global UC + server responses currently populate this with the + metastore UUID string so clients can disambiguate + dependencies with identical names across + metastores. + secret: + type: object + properties: + secret_full_name: + type: string + description: >- + Full name of the dependent secret, in the form + of + **catalog_name**.**schema_name**.**secret_name**. + required: + - secret_full_name + description: A dependency on a Unity Catalog secret. table: type: object properties: @@ -5110,14 +9831,26 @@ paths: description: >- Full name of the dependent table, in the form of - __catalog_name__.__schema_name__.__table_name__. + **catalog_name**.**schema_name**.**table_name**. required: - table_full_name description: A table that is dependent on a SQL object. + volume: + type: object + properties: + volume_full_name: + type: string + description: >- + Full name of the dependent volume, in the form + of + **catalog_name**.**schema_name**.**volume_name**. + required: + - volume_full_name + description: A dependency on a Unity Catalog volume. description: >- A dependency of a SQL object. One of the following - fields must be defined: __table__, - __function__, __connection__, or __credential__. + fields must be defined: **table**, + **function**, **connection**, **credential**, **volume**, or **secret**. description: Array of dependencies. description: >- Model version dependencies, for feature-store packaged @@ -5401,7 +10134,7 @@ paths: metastore admin is able to list all the model versions. A regular user needs to be the owner or have - the **EXECUTE** privilege on the parent registered model to recieve the + the **EXECUTE** privilege on the parent registered model to receive the model versions in the response. For the latter case, the caller must also be the owner or have @@ -5433,10 +10166,13 @@ paths: metadata for :param max_results: int (optional) Maximum number of model versions to return. If not set, the page length is set to a server - configured value (100, as of 1/3/2024). - when set to a value greater than 0, the page length is the - minimum of this value and a server configured value(1000, as of 1/3/2024); - when set to 0, the page - length is set to a server configured value (100, as of 1/3/2024) (recommended); - when set to a - value less than 0, an invalid parameter error is returned; + configured value (100, as of 1/3/2024). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value(1000, as of 1/3/2024); + - when set to 0, the page length is set to a server configured value (100, as of 1/3/2024) + (recommended); + - when set to a value less than 0, an invalid parameter error is returned; :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -5768,38 +10504,496 @@ paths: Progress of the active data synchronization pipeline. description: >- - Detailed status of an online table. Shown if the - online table is in the ONLINE_TRIGGERED_UPDATE - or the ONLINE_NO_PENDING_UPDATE state. - description: Online Table data synchronization status - table_serving_url: + Detailed status of an online table. Shown if the + online table is in the ONLINE_TRIGGERED_UPDATE + or the ONLINE_NO_PENDING_UPDATE state. + description: Online Table data synchronization status + table_serving_url: + type: string + description: Data serving REST API URL for this table + unity_catalog_provisioning_state: + type: string + x-enum: + - ACTIVE + - DEGRADED + - DELETING + - FAILED + - PROVISIONING + - UPDATING + description: >- + The provisioning state of the online table entity in + Unity Catalog. This is distinct from the state of the + data synchronization pipeline (i.e. the table may be in + "ACTIVE" but the pipeline may be in "PROVISIONING" as it + runs asynchronously). + description: Specification of the online table to be created. + required: + - table + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/OnlineTable' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: OnlineTablesAPI + /api/2.0/online-tables/{name}: + delete: + operationId: online_tables_delete + summary: >- + Delete an online table. Warning: This will delete all the data in the + online table. If the source + tags: + - catalog + - online_tables + description: >- + Delete an online table. Warning: This will delete all the data in the + online table. If the source + + Delta table was deleted or modified since this Online Table was created, + this will lose the data + + forever! + + + :param name: str + Full three-part (catalog, schema, table) name of the table. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Full three-part (catalog, schema, table) name of the table. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: OnlineTablesAPI + get: + operationId: online_tables_get + summary: Get information about an existing online table and its status. + tags: + - catalog + - online_tables + description: |- + Get information about an existing online table and its status. + + :param name: str + Full three-part (catalog, schema, table) name of the table. + + :returns: :class:`OnlineTable` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Full three-part (catalog, schema, table) name of the table. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/OnlineTable' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: OnlineTablesAPI + /api/2.1/unity-catalog/policies: + post: + operationId: policies_create_policy + summary: >- + Creates a new policy on a securable. The new policy applies to the + securable and all its descendants. + tags: + - catalog + - policies + description: >- + Creates a new policy on a securable. The new policy applies to the + securable and all its descendants. + + + :param policy_info: :class:`PolicyInfo` + Required. The policy to create. + + :returns: :class:`PolicyInfo` + requestBody: + content: + application/json: + schema: + type: object + properties: + policy_info: + type: object + properties: + to_principals: + type: array + items: + type: string + for_securable_type: + type: string + x-enum: + - CATALOG + - CLEAN_ROOM + - CONNECTION + - CREDENTIAL + - EXTERNAL_LOCATION + - EXTERNAL_METADATA + - FUNCTION + - METASTORE + - PIPELINE + - PROVIDER + - RECIPIENT + - SCHEMA + - SHARE + - STAGING_TABLE + - STORAGE_CREDENTIAL + - TABLE + - VOLUME + description: >- + Type of securables that the policy should take effect + on. Only ``TABLE`` is supported at this moment. Required + on create and optional on update. + policy_type: + type: string + x-enum: + - POLICY_TYPE_COLUMN_MASK + - POLICY_TYPE_DENY + - POLICY_TYPE_GRANT + - POLICY_TYPE_ROW_FILTER + description: Type of the policy. Required on create. + column_mask: + type: object + properties: + function_name: + type: string + on_column: + type: string + description: >- + The alias of the column to be masked. The alias must + refer to one of matched columns. The values of the + column is passed to the column mask function as the + first argument. Required on create and update. + using: + type: array + items: + type: object + properties: + alias: + type: string + constant: + type: string + description: A constant literal. + function_arg_expression: + type: object + properties: + tag_introspection: + type: object + properties: + column_tag_value: + type: object + properties: + column_alias: + type: string + description: >- + The alias from MATCH COLUMNS that + identifies the column. + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - column_alias + - tag_key + description: >- + Extracts the value of a column-level + tag. + tag_value: + type: object + properties: + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - tag_key + description: >- + Extracts the value of a securable-level + tag. + description: >- + An expression that introspects tags at + query time. + description: >- + An expression evaluated at query time. Wraps + per-request expression variants (e.g., tag + introspection) so new variants can be added + without extending the FunctionArgument oneof. + description: >- + Optional list of column aliases or constant literals + to be passed as additional arguments to the column + mask function. The type of each column should match + the positional argument of the column mask function. + required: + - function_name + - on_column + description: >- + Options for column mask policies. Valid only if + ``policy_type`` is ``POLICY_TYPE_COLUMN_MASK``. Required + on create and optional on update. When specified on + update, the new options will replace the existing + options as a whole. + comment: type: string - description: Data serving REST API URL for this table - unity_catalog_provisioning_state: + description: Optional description of the policy. + created_at: + type: integer + description: >- + Time at which the policy was created, in epoch + milliseconds. Output only. + created_by: + type: string + description: >- + Username of the user who created the policy. Output + only. + deny: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges + description: >- + Options for deny policies. Valid only if ``policy_type`` + is ``POLICY_TYPE_DENY``. Required on create and optional + on update. When specified on update, the new options + will replace the existing options as a whole. + except_principals: + type: array + items: + type: string + description: >- + Optional list of user or group names that should be + excluded from the policy. + grant: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges + description: >- + Options for grant policies. Valid only if + ``policy_type`` is ``POLICY_TYPE_GRANT``. Required on + create and optional on update. When specified on update, + the new options will replace the existing options as a + whole. + id: + type: string + description: >- + Unique identifier of the policy. This field is output + only and is generated by the system. + match_columns: + type: array + items: + type: object + properties: + alias: + type: string + condition: + type: string + description: >- + The condition expression used to match a table + column. + description: >- + Optional list of condition expressions used to match + table columns. Only valid when ``for_securable_type`` is + ``TABLE``. When specified, the policy only applies to + tables whose columns satisfy all match conditions. + name: + type: string + description: >- + Name of the policy. Required on create and optional on + update. To rename the policy, set ``name`` to a + different value on update. + on_securable_fullname: + type: string + description: >- + Full name of the securable on which the policy is + defined. Required on create. + on_securable_type: type: string x-enum: - - ACTIVE - - DEGRADED - - DELETING - - FAILED - - PROVISIONING - - UPDATING + - CATALOG + - CLEAN_ROOM + - CONNECTION + - CREDENTIAL + - EXTERNAL_LOCATION + - EXTERNAL_METADATA + - FUNCTION + - METASTORE + - PIPELINE + - PROVIDER + - RECIPIENT + - SCHEMA + - SHARE + - STAGING_TABLE + - STORAGE_CREDENTIAL + - TABLE + - VOLUME description: >- - The provisioning state of the online table entity in - Unity Catalog. This is distinct from the state of the - data synchronization pipeline (i.e. the table may be in - "ACTIVE" but the pipeline may be in "PROVISIONING" as it - runs asynchronously). - description: Specification of the online table to be created. + Type of the securable on which the policy is defined. + Only ``CATALOG``, ``SCHEMA`` and ``TABLE`` are supported + at this moment. Required on create. + row_filter: + type: object + properties: + function_name: + type: string + using: + type: array + items: + type: object + properties: + alias: + type: string + constant: + type: string + description: A constant literal. + function_arg_expression: + type: object + properties: + tag_introspection: + type: object + properties: + column_tag_value: + type: object + properties: + column_alias: + type: string + description: >- + The alias from MATCH COLUMNS that + identifies the column. + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - column_alias + - tag_key + description: >- + Extracts the value of a column-level + tag. + tag_value: + type: object + properties: + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - tag_key + description: >- + Extracts the value of a securable-level + tag. + description: >- + An expression that introspects tags at + query time. + description: >- + An expression evaluated at query time. Wraps + per-request expression variants (e.g., tag + introspection) so new variants can be added + without extending the FunctionArgument oneof. + description: >- + Optional list of column aliases or constant literals + to be passed as arguments to the row filter + function. The type of each column should match the + positional argument of the row filter function. + required: + - function_name + description: >- + Options for row filter policies. Valid only if + ``policy_type`` is ``POLICY_TYPE_ROW_FILTER``. Required + on create and optional on update. When specified on + update, the new options will replace the existing + options as a whole. + updated_at: + type: integer + description: >- + Time at which the policy was last modified, in epoch + milliseconds. Output only. + updated_by: + type: string + description: >- + Username of the user who last modified the policy. + Output only. + use_session_identity: + type: boolean + description: >- + Temporary for migrating customers to session identity. + Customers not currently using ABAC will not be able to + set this field to false and all new policies will have + this field default to true. Existing customers will have + this field default to false, but can set it to true to + opt in to session identity. after a grace period, this + field will be removed and all policies will use session + identity. Only for row filter and column mask policies. + Not applicable to deny policies. + when_condition: + type: string + description: Optional condition when the policy should take effect. + required: + - to_principals + - for_securable_type + - policy_type + description: Required. The policy to create. required: - - table + - policy_info responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/OnlineTable' + $ref: '#/components/schemas/PolicyInfo' default: description: Error response content: @@ -5811,38 +11005,53 @@ paths: type: string message: type: string - x-stackql-sdk-source: OnlineTablesAPI - /api/2.0/online-tables/{name}: + x-stackql-sdk-source: PoliciesAPI + /api/2.1/unity-catalog/policies/{on_securable_type}/{on_securable_fullname}/{name}: delete: - operationId: online_tables_delete - summary: >- - Delete an online table. Warning: This will delete all the data in the - online table. If the source + operationId: policies_delete_policy + summary: Delete an ABAC policy defined on a securable. tags: - catalog - - online_tables - description: >- - Delete an online table. Warning: This will delete all the data in the - online table. If the source - - Delta table was deleted or modified since this Online Table was created, - this will lose the data - - forever! - + - policies + description: |- + Delete an ABAC policy defined on a securable. + :param on_securable_type: str + Required. The type of the securable to delete the policy from. + :param on_securable_fullname: str + Required. The fully qualified name of the securable to delete the policy from. :param name: str - Full three-part (catalog, schema, table) name of the table. + Required. The name of the policy to delete + + :returns: :class:`DeletePolicyResponse` parameters: + - name: on_securable_type + in: path + required: true + schema: + type: string + description: Required. The type of the securable to delete the policy from. + - name: on_securable_fullname + in: path + required: true + schema: + type: string + description: >- + Required. The fully qualified name of the securable to delete the + policy from. - name: name in: path required: true schema: type: string - description: Full three-part (catalog, schema, table) name of the table. + description: Required. The name of the policy to delete responses: '200': description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DeletePolicyResponse' default: description: Error response content: @@ -5854,34 +11063,52 @@ paths: type: string message: type: string - x-stackql-sdk-source: OnlineTablesAPI + x-stackql-sdk-source: PoliciesAPI get: - operationId: online_tables_get - summary: Get information about an existing online table and its status. + operationId: policies_get_policy + summary: Get the policy definition on a securable tags: - catalog - - online_tables + - policies description: |- - Get information about an existing online table and its status. + Get the policy definition on a securable + :param on_securable_type: str + Required. The type of the securable to retrieve the policy for. + :param on_securable_fullname: str + Required. The fully qualified name of securable to retrieve policy for. :param name: str - Full three-part (catalog, schema, table) name of the table. + Required. The name of the policy to retrieve. - :returns: :class:`OnlineTable` + :returns: :class:`PolicyInfo` parameters: + - name: on_securable_type + in: path + required: true + schema: + type: string + description: Required. The type of the securable to retrieve the policy for. + - name: on_securable_fullname + in: path + required: true + schema: + type: string + description: >- + Required. The fully qualified name of securable to retrieve policy + for. - name: name in: path required: true schema: type: string - description: Full three-part (catalog, schema, table) name of the table. + description: Required. The name of the policy to retrieve. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/OnlineTable' + $ref: '#/components/schemas/PolicyInfo' default: description: Error response content: @@ -5893,25 +11120,65 @@ paths: type: string message: type: string - x-stackql-sdk-source: OnlineTablesAPI - /api/2.1/unity-catalog/policies: - post: - operationId: policies_create_policy - summary: >- - Creates a new policy on a securable. The new policy applies to the - securable and all its descendants. + x-stackql-sdk-source: PoliciesAPI + patch: + operationId: policies_update_policy + summary: Update an ABAC policy on a securable. tags: - catalog - policies - description: >- - Creates a new policy on a securable. The new policy applies to the - securable and all its descendants. - + description: |- + Update an ABAC policy on a securable. + :param on_securable_type: str + Required. The type of the securable to update the policy for. + :param on_securable_fullname: str + Required. The fully qualified name of the securable to update the policy for. + :param name: str + Required. The name of the policy to update. :param policy_info: :class:`PolicyInfo` - Required. The policy to create. + Optional fields to update. This is the request body for updating a policy. Use ``update_mask`` field + to specify which fields in the request is to be updated. + + - If ``update_mask`` is empty or "*", all specified fields will be updated. + - If ``update_mask`` is specified, only the fields specified in the ``update_mask`` will be updated. + If a field is specified in ``update_mask`` and not set in the request, the field will be cleared. + Users can use the update mask to explicitly unset optional fields such as ``exception_principals`` + and ``when_condition``. + :param update_mask: str (optional) + Optional. The update mask field for specifying user intentions on which fields to update in the + request. :returns: :class:`PolicyInfo` + parameters: + - name: on_securable_type + in: path + required: true + schema: + type: string + description: Required. The type of the securable to update the policy for. + - name: on_securable_fullname + in: path + required: true + schema: + type: string + description: >- + Required. The fully qualified name of the securable to update the + policy for. + - name: name + in: path + required: true + schema: + type: string + description: Required. The name of the policy to update. + - name: update_mask + in: query + required: false + schema: + type: string + description: >- + Optional. The update mask field for specifying user intentions on + which fields to update in the request. requestBody: content: application/json: @@ -5947,12 +11214,14 @@ paths: - VOLUME description: >- Type of securables that the policy should take effect - on. Only `TABLE` is supported at this moment. Required + on. Only ``TABLE`` is supported at this moment. Required on create and optional on update. policy_type: type: string x-enum: - POLICY_TYPE_COLUMN_MASK + - POLICY_TYPE_DENY + - POLICY_TYPE_GRANT - POLICY_TYPE_ROW_FILTER description: Type of the policy. Required on create. column_mask: @@ -5977,6 +11246,52 @@ paths: constant: type: string description: A constant literal. + function_arg_expression: + type: object + properties: + tag_introspection: + type: object + properties: + column_tag_value: + type: object + properties: + column_alias: + type: string + description: >- + The alias from MATCH COLUMNS that + identifies the column. + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - column_alias + - tag_key + description: >- + Extracts the value of a column-level + tag. + tag_value: + type: object + properties: + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - tag_key + description: >- + Extracts the value of a securable-level + tag. + description: >- + An expression that introspects tags at + query time. + description: >- + An expression evaluated at query time. Wraps + per-request expression variants (e.g., tag + introspection) so new variants can be added + without extending the FunctionArgument oneof. description: >- Optional list of column aliases or constant literals to be passed as additional arguments to the column @@ -5987,10 +11302,10 @@ paths: - on_column description: >- Options for column mask policies. Valid only if - `policy_type` is `POLICY_TYPE_COLUMN_MASK`. Required on - create and optional on update. When specified on update, - the new options will replace the existing options as a - whole. + ``policy_type`` is ``POLICY_TYPE_COLUMN_MASK``. Required + on create and optional on update. When specified on + update, the new options will replace the existing + options as a whole. comment: type: string description: Optional description of the policy. @@ -6004,6 +11319,20 @@ paths: description: >- Username of the user who created the policy. Output only. + deny: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges + description: >- + Options for deny policies. Valid only if ``policy_type`` + is ``POLICY_TYPE_DENY``. Required on create and optional + on update. When specified on update, the new options + will replace the existing options as a whole. except_principals: type: array items: @@ -6011,6 +11340,21 @@ paths: description: >- Optional list of user or group names that should be excluded from the policy. + grant: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges + description: >- + Options for grant policies. Valid only if + ``policy_type`` is ``POLICY_TYPE_GRANT``. Required on + create and optional on update. When specified on update, + the new options will replace the existing options as a + whole. id: type: string description: >- @@ -6030,15 +11374,15 @@ paths: column. description: >- Optional list of condition expressions used to match - table columns. Only valid when `for_securable_type` is - `TABLE`. When specified, the policy only applies to + table columns. Only valid when ``for_securable_type`` is + ``TABLE``. When specified, the policy only applies to tables whose columns satisfy all match conditions. name: type: string description: >- Name of the policy. Required on create and optional on - update. To rename the policy, set `name` to a different - value on update. + update. To rename the policy, set ``name`` to a + different value on update. on_securable_fullname: type: string description: >- @@ -6066,8 +11410,8 @@ paths: - VOLUME description: >- Type of the securable on which the policy is defined. - Only `CATALOG`, `SCHEMA` and `TABLE` are supported at - this moment. Required on create. + Only ``CATALOG``, ``SCHEMA`` and ``TABLE`` are supported + at this moment. Required on create. row_filter: type: object properties: @@ -6083,6 +11427,52 @@ paths: constant: type: string description: A constant literal. + function_arg_expression: + type: object + properties: + tag_introspection: + type: object + properties: + column_tag_value: + type: object + properties: + column_alias: + type: string + description: >- + The alias from MATCH COLUMNS that + identifies the column. + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - column_alias + - tag_key + description: >- + Extracts the value of a column-level + tag. + tag_value: + type: object + properties: + tag_key: + type: string + description: >- + 1024 matches the max_length on + FunctionArgument.constant above. + required: + - tag_key + description: >- + Extracts the value of a securable-level + tag. + description: >- + An expression that introspects tags at + query time. + description: >- + An expression evaluated at query time. Wraps + per-request expression variants (e.g., tag + introspection) so new variants can be added + without extending the FunctionArgument oneof. description: >- Optional list of column aliases or constant literals to be passed as arguments to the row filter @@ -6092,10 +11482,10 @@ paths: - function_name description: >- Options for row filter policies. Valid only if - `policy_type` is `POLICY_TYPE_ROW_FILTER`. Required on - create and optional on update. When specified on update, - the new options will replace the existing options as a - whole. + ``policy_type`` is ``POLICY_TYPE_ROW_FILTER``. Required + on create and optional on update. When specified on + update, the new options will replace the existing + options as a whole. updated_at: type: integer description: >- @@ -6106,6 +11496,18 @@ paths: description: >- Username of the user who last modified the policy. Output only. + use_session_identity: + type: boolean + description: >- + Temporary for migrating customers to session identity. + Customers not currently using ABAC will not be able to + set this field to false and all new policies will have + this field default to true. Existing customers will have + this field default to false, but can set it to true to + opt in to session identity. after a grace period, this + field will be removed and all policies will use session + identity. Only for row filter and column mask policies. + Not applicable to deny policies. when_condition: type: string description: Optional condition when the policy should take effect. @@ -6113,7 +11515,17 @@ paths: - to_principals - for_securable_type - policy_type - description: Required. The policy to create. + description: >- + Optional fields to update. This is the request body for + updating a policy. Use ``update_mask`` field to specify + which fields in the request is to be updated. - If + ``update_mask`` is empty or "*", all specified fields will + be updated. - If ``update_mask`` is specified, only the + fields specified in the ``update_mask`` will be updated. If + a field is specified in ``update_mask`` and not set in the + request, the field will be cleared. Users can use the update + mask to explicitly unset optional fields such as + ``exception_principals`` and ``when_condition``. required: - policy_info responses: @@ -6135,52 +11547,97 @@ paths: message: type: string x-stackql-sdk-source: PoliciesAPI - /api/2.1/unity-catalog/policies/{on_securable_type}/{on_securable_fullname}/{name}: - delete: - operationId: policies_delete_policy - summary: Delete an ABAC policy defined on a securable. + /api/2.1/unity-catalog/policies/{on_securable_type}/{on_securable_fullname}: + get: + operationId: policies_list_policies + summary: >- + List all policies defined on a securable. Optionally, the list can + include inherited policies defined tags: - catalog - policies - description: |- - Delete an ABAC policy defined on a securable. + description: >- + List all policies defined on a securable. Optionally, the list can + include inherited policies defined + + on the securable's parent schema or catalog. + + + PAGINATION BEHAVIOR: The API is by default paginated, a page may contain + zero results while still + + providing a next_page_token. Clients must continue reading pages until + next_page_token is absent, + + which is the only indication that the end of results has been reached. + :param on_securable_type: str - Required. The type of the securable to delete the policy from. + Required. The type of the securable to list policies for. :param on_securable_fullname: str - Required. The fully qualified name of the securable to delete the policy from. - :param name: str - Required. The name of the policy to delete + Required. The fully qualified name of securable to list policies for. + :param include_inherited: bool (optional) + Optional. Whether to include policies defined on parent securables. By default, the inherited + policies are not included. + :param max_results: int (optional) + Optional. Maximum number of policies to return on a single page (page length). - :returns: :class:`DeletePolicyResponse` + - When not set or set to 0, the page length is set to a server configured value (recommended); + - When set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + :param page_token: str (optional) + Optional. Opaque pagination token to go to next page based on previous query. + + :returns: Iterator over :class:`PolicyInfo` parameters: - name: on_securable_type in: path required: true schema: type: string - description: Required. The type of the securable to delete the policy from. + description: Required. The type of the securable to list policies for. - name: on_securable_fullname in: path required: true schema: type: string description: >- - Required. The fully qualified name of the securable to delete the - policy from. - - name: name - in: path - required: true + Required. The fully qualified name of securable to list policies + for. + - name: include_inherited + in: query + required: false + schema: + type: boolean + description: >- + Optional. Whether to include policies defined on parent securables. + By default, the inherited policies are not included. + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Optional. Maximum number of policies to return on a single page + (page length). - When not set or set to 0, the page length is set to + a server configured value (recommended); - When set to a value + greater than 0, the page length is the minimum of this value and a + server configured value; + - name: page_token + in: query + required: false schema: type: string - description: Required. The name of the policy to delete + description: >- + Optional. Opaque pagination token to go to next page based on + previous query. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/DeletePolicyResponse' + $ref: '#/components/schemas/ListPoliciesResponse' default: description: Error response content: @@ -6193,51 +11650,43 @@ paths: message: type: string x-stackql-sdk-source: PoliciesAPI - get: - operationId: policies_get_policy - summary: Get the policy definition on a securable + /api/2.1/unity-catalog/tables/{table_name}/monitor/refreshes/{refresh_id}/cancel: + post: + operationId: quality_monitors_cancel_refresh + summary: >- + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Cancels an tags: - catalog - - policies - description: |- - Get the policy definition on a securable + - quality_monitors + description: >- + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Cancels an - :param on_securable_type: str - Required. The type of the securable to retrieve the policy for. - :param on_securable_fullname: str - Required. The fully qualified name of securable to retrieve policy for. - :param name: str - Required. The name of the policy to retrieve. + already-initiated refresh job. - :returns: :class:`PolicyInfo` + + :param table_name: str + UC table name in format ``catalog.schema.table_name``. table_name is case insensitive and spaces are + disallowed. + :param refresh_id: int parameters: - - name: on_securable_type - in: path - required: true - schema: - type: string - description: Required. The type of the securable to retrieve the policy for. - - name: on_securable_fullname + - name: table_name in: path required: true schema: type: string description: >- - Required. The fully qualified name of securable to retrieve policy - for. - - name: name + UC table name in format ``catalog.schema.table_name``. table_name is + case insensitive and spaces are disallowed. + - name: refresh_id in: path required: true schema: - type: string - description: Required. The name of the policy to retrieve. + type: integer responses: '200': description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/PolicyInfo' default: description: Error response content: @@ -6249,284 +11698,385 @@ paths: type: string message: type: string - x-stackql-sdk-source: PoliciesAPI - patch: - operationId: policies_update_policy - summary: Update an ABAC policy on a securable. + x-stackql-sdk-source: QualityMonitorsAPI + /api/2.1/unity-catalog/tables/{table_name}/monitor: + post: + operationId: quality_monitors_create + summary: >- + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Creates a new tags: - catalog - - policies - description: |- - Update an ABAC policy on a securable. + - quality_monitors + description: >- + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Creates a new - :param on_securable_type: str - Required. The type of the securable to update the policy for. - :param on_securable_fullname: str - Required. The fully qualified name of the securable to update the policy for. - :param name: str - Required. The name of the policy to update. - :param policy_info: :class:`PolicyInfo` - Optional fields to update. This is the request body for updating a policy. Use `update_mask` field - to specify which fields in the request is to be updated. - If `update_mask` is empty or "*", all - specified fields will be updated. - If `update_mask` is specified, only the fields specified in the - `update_mask` will be updated. If a field is specified in `update_mask` and not set in the request, - the field will be cleared. Users can use the update mask to explicitly unset optional fields such as - `exception_principals` and `when_condition`. - :param update_mask: str (optional) - Optional. The update mask field for specifying user intentions on which fields to update in the - request. + monitor for the specified table. - :returns: :class:`PolicyInfo` + + The caller must either: + + + 1. be an owner of the table's parent catalog, have **USE_SCHEMA** on the + table's parent schema, and + have **SELECT** access on the table + 2. have **USE_CATALOG** on the table's parent catalog, be an owner of + the table's parent schema, and + have **SELECT** access on the table. + 3. have the following permissions: + + + - **USE_CATALOG** on the table's parent catalog + + - **USE_SCHEMA** on the table's parent schema + + - be an owner of the table. + + + Workspace assets, such as the dashboard, will be created in the + workspace where this call was made. + + + :param table_name: str + UC table name in format ``catalog.schema.table_name``. This field corresponds to the + {full_table_name_arg} arg in the endpoint path. + :param output_schema_name: str + [Create:REQ Update:REQ] Schema where output tables are created. Needs to be in 2-level format + {catalog}.{schema} + :param assets_dir: str + [Create:REQ Update:IGN] Field for specifying the absolute path to a custom directory to store + data-monitoring assets. Normally prepopulated to a default user location via UI and Python APIs. + :param baseline_table_name: str (optional) + [Create:OPT Update:OPT] Baseline table name. Baseline data is used to compute drift from the data in + the monitored ``table_name``. The baseline table and the monitored table shall have the same schema. + :param custom_metrics: List[:class:`MonitorMetric`] (optional) + [Create:OPT Update:OPT] Custom metrics. + :param data_classification_config: + :class:`MonitorDataClassificationConfig` (optional) + [Create:OPT Update:OPT] Data classification related config. + :param inference_log: :class:`MonitorInferenceLog` (optional) + + :param latest_monitor_failure_msg: str (optional) + [Create:ERR Update:IGN] The latest error message for a monitor failure. + :param notifications: :class:`MonitorNotifications` (optional) + [Create:OPT Update:OPT] Field for specifying notification settings. + :param schedule: :class:`MonitorCronSchedule` (optional) + [Create:OPT Update:OPT] The monitor schedule. + :param skip_builtin_dashboard: bool (optional) + Whether to skip creating a default dashboard summarizing data quality metrics. + :param slicing_exprs: List[str] (optional) + [Create:OPT Update:OPT] List of column expressions to slice data with for targeted analysis. The + data is grouped by each expression independently, resulting in a separate slice for each predicate + and its complements. For example ``slicing_exprs=[“col_1”, “col_2 > 10”]`` will generate the + following slices: two slices for ``col_2 > 10`` (True and False), and one slice per unique value in + ``col1``. For high-cardinality columns, only the top 100 unique values by frequency will generate + slices. + :param snapshot: :class:`MonitorSnapshot` (optional) + Configuration for monitoring snapshot tables. + :param time_series: :class:`MonitorTimeSeries` (optional) + Configuration for monitoring time series tables. + :param warehouse_id: str (optional) + Optional argument to specify the warehouse for dashboard creation. If not specified, the first + running warehouse will be used. + + :returns: :class:`MonitorInfo` parameters: - - name: on_securable_type - in: path - required: true - schema: - type: string - description: Required. The type of the securable to update the policy for. - - name: on_securable_fullname - in: path - required: true - schema: - type: string - description: >- - Required. The fully qualified name of the securable to update the - policy for. - - name: name + - name: table_name in: path required: true - schema: - type: string - description: Required. The name of the policy to update. - - name: update_mask - in: query - required: false schema: type: string description: >- - Optional. The update mask field for specifying user intentions on - which fields to update in the request. + UC table name in format ``catalog.schema.table_name``. This field + corresponds to the {full_table_name_arg} arg in the endpoint path. requestBody: content: application/json: schema: type: object properties: - policy_info: + output_schema_name: + type: string + description: >- + [Create:REQ Update:REQ] Schema where output tables are + created. Needs to be in 2-level format {catalog}.{schema} + assets_dir: + type: string + description: >- + [Create:REQ Update:IGN] Field for specifying the absolute + path to a custom directory to store data-monitoring assets. + Normally prepopulated to a default user location via UI and + Python APIs. + baseline_table_name: + type: string + description: >- + [Create:OPT Update:OPT] Baseline table name. Baseline data + is used to compute drift from the data in the monitored + ``table_name``. The baseline table and the monitored table + shall have the same schema. + custom_metrics: + type: array + items: + type: object + properties: + name: + type: string + description: Name of the metric in the output tables. + definition: + type: string + description: >- + Jinja template for a SQL expression that specifies how + to compute the metric. See `create metric definition + `__. + input_columns: + type: array + items: + type: string + description: >- + A list of column names in the input table the metric + should be computed for. Can use ``":table"`` to + indicate that the metric needs information from + multiple columns. + output_data_type: + type: string + description: The output type of the custom metric. + type: + type: string + x-enum: + - CUSTOM_METRIC_TYPE_AGGREGATE + - CUSTOM_METRIC_TYPE_DERIVED + - CUSTOM_METRIC_TYPE_DRIFT + description: >- + Can only be one of ``"CUSTOM_METRIC_TYPE_AGGREGATE"``, + ``"CUSTOM_METRIC_TYPE_DERIVED"``, or + ``"CUSTOM_METRIC_TYPE_DRIFT"``. The + ``"CUSTOM_METRIC_TYPE_AGGREGATE"`` and + ``"CUSTOM_METRIC_TYPE_DERIVED"`` metrics are computed + on a single table, whereas the + ``"CUSTOM_METRIC_TYPE_DRIFT"`` compare metrics across + baseline and input table, or across the two + consecutive time windows. - + CUSTOM_METRIC_TYPE_AGGREGATE: only depend on the + existing columns in your table - + CUSTOM_METRIC_TYPE_DERIVED: depend on previously + computed aggregate metrics - CUSTOM_METRIC_TYPE_DRIFT: + depend on previously computed aggregate or derived + metrics + required: + - name + - definition + - input_columns + - output_data_type + - type + description: Custom metric definition. + description: '[Create:OPT Update:OPT] Custom metrics.' + data_classification_config: type: object properties: - to_principals: - type: array - items: - type: string - for_securable_type: - type: string - x-enum: - - CATALOG - - CLEAN_ROOM - - CONNECTION - - CREDENTIAL - - EXTERNAL_LOCATION - - EXTERNAL_METADATA - - FUNCTION - - METASTORE - - PIPELINE - - PROVIDER - - RECIPIENT - - SCHEMA - - SHARE - - STAGING_TABLE - - STORAGE_CREDENTIAL - - TABLE - - VOLUME - description: >- - Type of securables that the policy should take effect - on. Only `TABLE` is supported at this moment. Required - on create and optional on update. - policy_type: + enabled: + type: boolean + description: Whether to enable data classification. + description: '[Create:OPT Update:OPT] Data classification related config.' + inference_log: + type: object + properties: + problem_type: type: string x-enum: - - POLICY_TYPE_COLUMN_MASK - - POLICY_TYPE_ROW_FILTER - description: Type of the policy. Required on create. - column_mask: - type: object - properties: - function_name: - type: string - on_column: - type: string - description: >- - The alias of the column to be masked. The alias must - refer to one of matched columns. The values of the - column is passed to the column mask function as the - first argument. Required on create and update. - using: - type: array - items: - type: object - properties: - alias: - type: string - constant: - type: string - description: A constant literal. - description: >- - Optional list of column aliases or constant literals - to be passed as additional arguments to the column - mask function. The type of each column should match - the positional argument of the column mask function. - required: - - function_name - - on_column - description: >- - Options for column mask policies. Valid only if - `policy_type` is `POLICY_TYPE_COLUMN_MASK`. Required on - create and optional on update. When specified on update, - the new options will replace the existing options as a - whole. - comment: - type: string - description: Optional description of the policy. - created_at: - type: integer + - PROBLEM_TYPE_CLASSIFICATION + - PROBLEM_TYPE_REGRESSION description: >- - Time at which the policy was created, in epoch - milliseconds. Output only. - created_by: + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + timestamp_col: type: string - description: >- - Username of the user who created the policy. Output - only. - except_principals: + description: Column for the timestamp. + granularities: type: array items: type: string description: >- - Optional list of user or group names that should be - excluded from the policy. - id: + List of granularities to use when aggregating data into + time windows based on their timestamp. + prediction_col: type: string - description: >- - Unique identifier of the policy. This field is output - only and is generated by the system. - match_columns: - type: array - items: - type: object - properties: - alias: - type: string - condition: - type: string - description: >- - The condition expression used to match a table - column. - description: >- - Optional list of condition expressions used to match - table columns. Only valid when `for_securable_type` is - `TABLE`. When specified, the policy only applies to - tables whose columns satisfy all match conditions. - name: + description: Column for the prediction. + model_id_col: type: string - description: >- - Name of the policy. Required on create and optional on - update. To rename the policy, set `name` to a different - value on update. - on_securable_fullname: + description: Column for the model identifier. + label_col: type: string - description: >- - Full name of the securable on which the policy is - defined. Required on create. - on_securable_type: + description: Column for the label. + prediction_proba_col: type: string - x-enum: - - CATALOG - - CLEAN_ROOM - - CONNECTION - - CREDENTIAL - - EXTERNAL_LOCATION - - EXTERNAL_METADATA - - FUNCTION - - METASTORE - - PIPELINE - - PROVIDER - - RECIPIENT - - SCHEMA - - SHARE - - STAGING_TABLE - - STORAGE_CREDENTIAL - - TABLE - - VOLUME - description: >- - Type of the securable on which the policy is defined. - Only `CATALOG`, `SCHEMA` and `TABLE` are supported at - this moment. Required on create. - row_filter: + description: Column for prediction probabilities + required: + - problem_type + - timestamp_col + - granularities + - prediction_col + - model_id_col + latest_monitor_failure_msg: + type: string + description: >- + [Create:ERR Update:IGN] The latest error message for a + monitor failure. + notifications: + type: object + properties: + on_failure: type: object properties: - function_name: - type: string - using: + email_addresses: type: array items: - type: object - properties: - alias: - type: string - constant: - type: string - description: A constant literal. - description: >- - Optional list of column aliases or constant literals - to be passed as arguments to the row filter - function. The type of each column should match the - positional argument of the row filter function. - required: - - function_name + type: string + on_new_classification_tag_detected: + type: object + properties: + email_addresses: + type: array + items: + type: string description: >- - Options for row filter policies. Valid only if - `policy_type` is `POLICY_TYPE_ROW_FILTER`. Required on - create and optional on update. When specified on update, - the new options will replace the existing options as a - whole. - updated_at: - type: integer + Destinations to send notifications on new classification + tag detected. + description: >- + [Create:OPT Update:OPT] Field for specifying notification + settings. + schedule: + type: object + properties: + quartz_cron_expression: + type: string + timezone_id: + type: string description: >- - Time at which the policy was last modified, in epoch - milliseconds. Output only. - updated_by: + The timezone id (e.g., ``PST``) in which to evaluate the + quartz expression. + pause_status: type: string + x-enum: + - PAUSED + - UNPAUSED + - UNSPECIFIED description: >- - Username of the user who last modified the policy. - Output only. - when_condition: + Read only field that indicates whether a schedule is + paused or not. + required: + - quartz_cron_expression + - timezone_id + description: '[Create:OPT Update:OPT] The monitor schedule.' + skip_builtin_dashboard: + type: boolean + description: >- + Whether to skip creating a default dashboard summarizing + data quality metrics. + slicing_exprs: + type: array + items: + type: string + description: >- + [Create:OPT Update:OPT] List of column expressions to slice + data with for targeted analysis. The data is grouped by each + expression independently, resulting in a separate slice for + each predicate and its complements. For example + ``slicing_exprs=[“col_1”, “col_2 > 10”]`` will generate the + following slices: two slices for ``col_2 > 10`` (True and + False), and one slice per unique value in ``col1``. For + high-cardinality columns, only the top 100 unique values by + frequency will generate slices. + snapshot: + type: object + properties: {} + description: Configuration for monitoring snapshot tables. + time_series: + type: object + properties: + timestamp_col: type: string - description: Optional condition when the policy should take effect. + description: Column for the timestamp. + granularities: + type: array + items: + type: string + description: >- + Granularities for aggregating data into time windows + based on their timestamp. Currently the following static + granularities are supported: {``\"5 minutes\"``, ``\"30 + minutes\"``, ``\"1 hour\"``, ``\"1 day\"``, + ``\"\u003cn\u003e week(s)\"``, ``\"1 month\"``, ``\"1 + year\"``}. required: - - to_principals - - for_securable_type - - policy_type + - timestamp_col + - granularities + description: Configuration for monitoring time series tables. + warehouse_id: + type: string description: >- - Optional fields to update. This is the request body for - updating a policy. Use `update_mask` field to specify which - fields in the request is to be updated. - If `update_mask` - is empty or "*", all specified fields will be updated. - If - `update_mask` is specified, only the fields specified in the - `update_mask` will be updated. If a field is specified in - `update_mask` and not set in the request, the field will be - cleared. Users can use the update mask to explicitly unset - optional fields such as `exception_principals` and - `when_condition`. + Optional argument to specify the warehouse for dashboard + creation. If not specified, the first running warehouse will + be used. required: - - policy_info + - output_schema_name + - assets_dir responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/PolicyInfo' + $ref: '#/components/schemas/MonitorInfo' default: description: Error response content: @@ -6535,99 +12085,74 @@ paths: type: object properties: error: - type: string - message: - type: string - x-stackql-sdk-source: PoliciesAPI - /api/2.1/unity-catalog/policies/{on_securable_type}/{on_securable_fullname}: - get: - operationId: policies_list_policies + type: string + message: + type: string + x-stackql-sdk-source: QualityMonitorsAPI + delete: + operationId: quality_monitors_delete summary: >- - List all policies defined on a securable. Optionally, the list can - include inherited policies defined + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Deletes a monitor tags: - catalog - - policies + - quality_monitors description: >- - List all policies defined on a securable. Optionally, the list can - include inherited policies defined + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Deletes a monitor - on the securable's parent schema or catalog. + for the specified table. - PAGINATION BEHAVIOR: The API is by default paginated, a page may contain - zero results while still + The caller must either: - providing a next_page_token. Clients must continue reading pages until - next_page_token is absent, - which is the only indication that the end of results has been reached. + 1. be an owner of the table's parent catalog + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema - :param on_securable_type: str - Required. The type of the securable to list policies for. - :param on_securable_fullname: str - Required. The fully qualified name of securable to list policies for. - :param include_inherited: bool (optional) - Optional. Whether to include policies defined on parent securables. By default, the inherited - policies are not included. - :param max_results: int (optional) - Optional. Maximum number of policies to return on a single page (page length). - When not set or set - to 0, the page length is set to a server configured value (recommended); - When set to a value - greater than 0, the page length is the minimum of this value and a server configured value; - :param page_token: str (optional) - Optional. Opaque pagination token to go to next page based on previous query. + 3. have the following permissions: - :returns: Iterator over :class:`PolicyInfo` + + - **USE_CATALOG** on the table's parent catalog + + - **USE_SCHEMA** on the table's parent schema + + - be an owner of the table. + + + Additionally, the call must be made from the workspace where the monitor + was created. + + + Note that the metric tables and dashboard will not be deleted as part of + this call; those assets must + + be manually cleaned up (if desired). + + + :param table_name: str + UC table name in format ``catalog.schema.table_name``. This field corresponds to the + {full_table_name_arg} arg in the endpoint path. + + :returns: :class:`DeleteMonitorResponse` parameters: - - name: on_securable_type - in: path - required: true - schema: - type: string - description: Required. The type of the securable to list policies for. - - name: on_securable_fullname + - name: table_name in: path required: true schema: type: string description: >- - Required. The fully qualified name of securable to list policies - for. - - name: include_inherited - in: query - required: false - schema: - type: boolean - description: >- - Optional. Whether to include policies defined on parent securables. - By default, the inherited policies are not included. - - name: max_results - in: query - required: false - schema: - type: integer - description: >- - Optional. Maximum number of policies to return on a single page - (page length). - When not set or set to 0, the page length is set to - a server configured value (recommended); - When set to a value - greater than 0, the page length is the minimum of this value and a - server configured value; - - name: page_token - in: query - required: false - schema: - type: string - description: >- - Optional. Opaque pagination token to go to next page based on - previous query. + UC table name in format ``catalog.schema.table_name``. This field + corresponds to the {full_table_name_arg} arg in the endpoint path. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListPoliciesResponse' + $ref: '#/components/schemas/DeleteMonitorResponse' default: description: Error response content: @@ -6639,27 +12164,54 @@ paths: type: string message: type: string - x-stackql-sdk-source: PoliciesAPI - /api/2.1/unity-catalog/tables/{table_name}/monitor/refreshes/{refresh_id}/cancel: - post: - operationId: quality_monitors_cancel_refresh + x-stackql-sdk-source: QualityMonitorsAPI + get: + operationId: quality_monitors_get summary: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Cancels an + (/api/data-quality/v1/monitors). Gets a monitor for tags: - catalog - quality_monitors description: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Cancels an + (/api/data-quality/v1/monitors). Gets a monitor for - already-initiated refresh job. + the specified table. + + + The caller must either: + + + 1. be an owner of the table's parent catalog + + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema. + + 3. have the following permissions: + + + - **USE_CATALOG** on the table's parent catalog + + - **USE_SCHEMA** on the table's parent schema + + - **SELECT** privilege on the table. + + + The returned information includes configuration values, as well as + information on assets created by + + the monitor. Some information (e.g., dashboard) may be filtered out if + the caller is in a different + + workspace than where the monitor was created. :param table_name: str - UC table name in format `catalog.schema.table_name`. table_name is case insensitive and spaces are - disallowed. - :param refresh_id: int + UC table name in format ``catalog.schema.table_name``. This field corresponds to the + {full_table_name_arg} arg in the endpoint path. + + :returns: :class:`MonitorInfo` parameters: - name: table_name in: path @@ -6667,16 +12219,15 @@ paths: schema: type: string description: >- - UC table name in format `catalog.schema.table_name`. table_name is - case insensitive and spaces are disallowed. - - name: refresh_id - in: path - required: true - schema: - type: integer + UC table name in format ``catalog.schema.table_name``. This field + corresponds to the {full_table_name_arg} arg in the endpoint path. responses: '200': description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/MonitorInfo' default: description: Error response content: @@ -6689,55 +12240,63 @@ paths: message: type: string x-stackql-sdk-source: QualityMonitorsAPI - /api/2.1/unity-catalog/tables/{table_name}/monitor: - post: - operationId: quality_monitors_create + put: + operationId: quality_monitors_update summary: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Creates a new + (/api/data-quality/v1/monitors). Updates a monitor tags: - catalog - quality_monitors description: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Creates a new + (/api/data-quality/v1/monitors). Updates a monitor - monitor for the specified table. + for the specified table. - The caller must either: 1. be an owner of the table's parent catalog, - have **USE_SCHEMA** on the + The caller must either: - table's parent schema, and have **SELECT** access on the table 2. have - **USE_CATALOG** on the table's - parent catalog, be an owner of the table's parent schema, and have - **SELECT** access on the table. 3. + 1. be an owner of the table's parent catalog - have the following permissions: - **USE_CATALOG** on the table's parent - catalog - **USE_SCHEMA** on + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema - the table's parent schema - be an owner of the table. + 3. have the following permissions: - Workspace assets, such as the dashboard, will be created in the - workspace where this call was made. + - **USE_CATALOG** on the table's parent catalog + + - **USE_SCHEMA** on the table's parent schema + + - be an owner of the table. + + + Additionally, the call must be made from the workspace where the monitor + was created, and the caller + + must be the original creator of the monitor. + + + Certain configuration fields, such as output asset identifiers, cannot + be updated. :param table_name: str - UC table name in format `catalog.schema.table_name`. This field corresponds to the + UC table name in format ``catalog.schema.table_name``. This field corresponds to the {full_table_name_arg} arg in the endpoint path. :param output_schema_name: str [Create:REQ Update:REQ] Schema where output tables are created. Needs to be in 2-level format {catalog}.{schema} - :param assets_dir: str - [Create:REQ Update:IGN] Field for specifying the absolute path to a custom directory to store - data-monitoring assets. Normally prepopulated to a default user location via UI and Python APIs. :param baseline_table_name: str (optional) [Create:OPT Update:OPT] Baseline table name. Baseline data is used to compute drift from the data in - the monitored `table_name`. The baseline table and the monitored table shall have the same schema. + the monitored ``table_name``. The baseline table and the monitored table shall have the same schema. :param custom_metrics: List[:class:`MonitorMetric`] (optional) [Create:OPT Update:OPT] Custom metrics. + :param dashboard_id: str (optional) + [Create:ERR Update:OPT] Id of dashboard that visualizes the computed metrics. This can be empty if + the monitor is in PENDING state. :param data_classification_config: :class:`MonitorDataClassificationConfig` (optional) [Create:OPT Update:OPT] Data classification related config. @@ -6749,22 +12308,17 @@ paths: [Create:OPT Update:OPT] Field for specifying notification settings. :param schedule: :class:`MonitorCronSchedule` (optional) [Create:OPT Update:OPT] The monitor schedule. - :param skip_builtin_dashboard: bool (optional) - Whether to skip creating a default dashboard summarizing data quality metrics. :param slicing_exprs: List[str] (optional) [Create:OPT Update:OPT] List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate - and its complements. For example `slicing_exprs=[“col_1”, “col_2 > 10”]` will generate the - following slices: two slices for `col_2 > 10` (True and False), and one slice per unique value in - `col1`. For high-cardinality columns, only the top 100 unique values by frequency will generate + and its complements. For example ``slicing_exprs=[“col_1”, “col_2 > 10”]`` will generate the + following slices: two slices for ``col_2 > 10`` (True and False), and one slice per unique value in + ``col1``. For high-cardinality columns, only the top 100 unique values by frequency will generate slices. :param snapshot: :class:`MonitorSnapshot` (optional) Configuration for monitoring snapshot tables. :param time_series: :class:`MonitorTimeSeries` (optional) Configuration for monitoring time series tables. - :param warehouse_id: str (optional) - Optional argument to specify the warehouse for dashboard creation. If not specified, the first - running warehouse will be used. :returns: :class:`MonitorInfo` parameters: @@ -6774,7 +12328,7 @@ paths: schema: type: string description: >- - UC table name in format `catalog.schema.table_name`. This field + UC table name in format ``catalog.schema.table_name``. This field corresponds to the {full_table_name_arg} arg in the endpoint path. requestBody: content: @@ -6787,19 +12341,12 @@ paths: description: >- [Create:REQ Update:REQ] Schema where output tables are created. Needs to be in 2-level format {catalog}.{schema} - assets_dir: - type: string - description: >- - [Create:REQ Update:IGN] Field for specifying the absolute - path to a custom directory to store data-monitoring assets. - Normally prepopulated to a default user location via UI and - Python APIs. baseline_table_name: type: string description: >- [Create:OPT Update:OPT] Baseline table name. Baseline data is used to compute drift from the data in the monitored - `table_name`. The baseline table and the monitored table + ``table_name``. The baseline table and the monitored table shall have the same schema. custom_metrics: type: array @@ -6813,9 +12360,8 @@ paths: type: string description: >- Jinja template for a SQL expression that specifies how - to compute the metric. See [create metric definition]. - [create metric definition]: - https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition + to compute the metric. See `create metric definition + `__. input_columns: type: array items: @@ -6858,6 +12404,12 @@ paths: - type description: Custom metric definition. description: '[Create:OPT Update:OPT] Custom metrics.' + dashboard_id: + type: string + description: >- + [Create:ERR Update:OPT] Id of dashboard that visualizes the + computed metrics. This can be empty if the monitor is in + PENDING state. data_classification_config: type: object properties: @@ -7004,11 +12556,6 @@ paths: - quartz_cron_expression - timezone_id description: '[Create:OPT Update:OPT] The monitor schedule.' - skip_builtin_dashboard: - type: boolean - description: >- - Whether to skip creating a default dashboard summarizing - data quality metrics. slicing_exprs: type: array items: @@ -7018,9 +12565,9 @@ paths: data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example - `slicing_exprs=[“col_1”, “col_2 > 10”]` will generate the - following slices: two slices for `col_2 > 10` (True and - False), and one slice per unique value in `col1`. For + ``slicing_exprs=[“col_1”, “col_2 > 10”]`` will generate the + following slices: two slices for ``col_2 > 10`` (True and + False), and one slice per unique value in ``col1``. For high-cardinality columns, only the top 100 unique values by frequency will generate slices. snapshot: @@ -7048,22 +12595,91 @@ paths: - timestamp_col - granularities description: Configuration for monitoring time series tables. - warehouse_id: - type: string - description: >- - Optional argument to specify the warehouse for dashboard - creation. If not specified, the first running warehouse will - be used. required: - output_schema_name - - assets_dir responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/MonitorInfo' + $ref: '#/components/schemas/MonitorInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: QualityMonitorsAPI + /api/2.1/unity-catalog/tables/{table_name}/monitor/refreshes/{refresh_id}: + get: + operationId: quality_monitors_get_refresh + summary: >- + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Gets info about a + tags: + - catalog + - quality_monitors + description: >- + Deprecated: Use Data Quality Monitors API instead + (/api/data-quality/v1/monitors). Gets info about a + + specific monitor refresh using the given refresh ID. + + + The caller must either: + + + 1. be an owner of the table's parent catalog + + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema + + 3. have the following permissions: + + + - **USE_CATALOG** on the table's parent catalog + + - **USE_SCHEMA** on the table's parent schema + + - **SELECT** privilege on the table. + + + Additionally, the call must be made from the workspace where the monitor + was created. + + + :param table_name: str + Full name of the table. + :param refresh_id: int + ID of the refresh. + + :returns: :class:`MonitorRefreshInfo` + parameters: + - name: table_name + in: path + required: true + schema: + type: string + description: Full name of the table. + - name: refresh_id + in: path + required: true + schema: + type: integer + description: ID of the refresh. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/MonitorRefreshInfo' default: description: Error response content: @@ -7076,48 +12692,50 @@ paths: message: type: string x-stackql-sdk-source: QualityMonitorsAPI - delete: - operationId: quality_monitors_delete + /api/2.1/unity-catalog/tables/{table_name}/monitor/refreshes: + get: + operationId: quality_monitors_list_refreshes summary: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Deletes a monitor + (/api/data-quality/v1/monitors). Gets an array tags: - catalog - quality_monitors description: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Deletes a monitor + (/api/data-quality/v1/monitors). Gets an array - for the specified table. + containing the history of the most recent refreshes (up to 25) for this + table. - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the + The caller must either: - table's parent catalog and be an owner of the table's parent schema 3. - have the following permissions: - - **USE_CATALOG** on the table's parent catalog - **USE_SCHEMA** on the - table's parent schema - be an + 1. be an owner of the table's parent catalog - owner of the table. + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema + 3. have the following permissions: - Additionally, the call must be made from the workspace where the monitor - was created. + - **USE_CATALOG** on the table's parent catalog - Note that the metric tables and dashboard will not be deleted as part of - this call; those assets must + - **USE_SCHEMA** on the table's parent schema - be manually cleaned up (if desired). + - **SELECT** privilege on the table. + + + Additionally, the call must be made from the workspace where the monitor + was created. :param table_name: str - UC table name in format `catalog.schema.table_name`. This field corresponds to the - {full_table_name_arg} arg in the endpoint path. + UC table name in format ``catalog.schema.table_name``. table_name is case insensitive and spaces are + disallowed. - :returns: :class:`DeleteMonitorResponse` + :returns: :class:`MonitorRefreshListResponse` parameters: - name: table_name in: path @@ -7125,15 +12743,15 @@ paths: schema: type: string description: >- - UC table name in format `catalog.schema.table_name`. This field - corresponds to the {full_table_name_arg} arg in the endpoint path. + UC table name in format ``catalog.schema.table_name``. table_name is + case insensitive and spaces are disallowed. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/DeleteMonitorResponse' + $ref: '#/components/schemas/MonitorRefreshListResponse' default: description: Error response content: @@ -7146,47 +12764,49 @@ paths: message: type: string x-stackql-sdk-source: QualityMonitorsAPI - get: - operationId: quality_monitors_get + post: + operationId: quality_monitors_run_refresh summary: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Gets a monitor for + (/api/data-quality/v1/monitors). Queues a metric tags: - catalog - quality_monitors description: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Gets a monitor for + (/api/data-quality/v1/monitors). Queues a metric - the specified table. + refresh on the monitor for the specified table. The refresh will execute + in the background. - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the + The caller must either: - table's parent catalog and be an owner of the table's parent schema. 3. - have the following - permissions: - **USE_CATALOG** on the table's parent catalog - - **USE_SCHEMA** on the table's parent + 1. be an owner of the table's parent catalog - schema - **SELECT** privilege on the table. + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema + 3. have the following permissions: - The returned information includes configuration values, as well as - information on assets created by - the monitor. Some information (e.g., dashboard) may be filtered out if - the caller is in a different + - **USE_CATALOG** on the table's parent catalog - workspace than where the monitor was created. + - **USE_SCHEMA** on the table's parent schema + + - be an owner of the table + + + Additionally, the call must be made from the workspace where the monitor + was created. :param table_name: str - UC table name in format `catalog.schema.table_name`. This field corresponds to the - {full_table_name_arg} arg in the endpoint path. + UC table name in format ``catalog.schema.table_name``. table_name is case insensitive and spaces are + disallowed. - :returns: :class:`MonitorInfo` + :returns: :class:`MonitorRefreshInfo` parameters: - name: table_name in: path @@ -7194,15 +12814,15 @@ paths: schema: type: string description: >- - UC table name in format `catalog.schema.table_name`. This field - corresponds to the {full_table_name_arg} arg in the endpoint path. + UC table name in format ``catalog.schema.table_name``. table_name is + case insensitive and spaces are disallowed. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/MonitorInfo' + $ref: '#/components/schemas/MonitorRefreshInfo' default: description: Error response content: @@ -7215,81 +12835,55 @@ paths: message: type: string x-stackql-sdk-source: QualityMonitorsAPI - put: - operationId: quality_monitors_update + /api/2.1/quality-monitoring/tables/{table_name}/monitor/dashboard: + post: + operationId: quality_monitors_regenerate_dashboard summary: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Updates a monitor + (/api/data-quality/v1/monitors). Regenerates the tags: - catalog - quality_monitors description: >- Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Updates a monitor + (/api/data-quality/v1/monitors). Regenerates the - for the specified table. + monitoring dashboard for the specified table. - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the + The caller must either: - table's parent catalog and be an owner of the table's parent schema 3. - have the following permissions: - - **USE_CATALOG** on the table's parent catalog - **USE_SCHEMA** on the - table's parent schema - be an + 1. be an owner of the table's parent catalog - owner of the table. + 2. have **USE_CATALOG** on the table's parent catalog and be an owner of + the table's parent schema + 3. have the following permissions: - Additionally, the call must be made from the workspace where the monitor - was created, and the caller - must be the original creator of the monitor. + - **USE_CATALOG** on the table's parent catalog + - **USE_SCHEMA** on the table's parent schema - Certain configuration fields, such as output asset identifiers, cannot - be updated. + - be an owner of the table + + + The call must be made from the workspace where the monitor was created. + The dashboard will be + + regenerated in the assets directory that was specified when the monitor + was created. :param table_name: str - UC table name in format `catalog.schema.table_name`. This field corresponds to the + UC table name in format ``catalog.schema.table_name``. This field corresponds to the {full_table_name_arg} arg in the endpoint path. - :param output_schema_name: str - [Create:REQ Update:REQ] Schema where output tables are created. Needs to be in 2-level format - {catalog}.{schema} - :param baseline_table_name: str (optional) - [Create:OPT Update:OPT] Baseline table name. Baseline data is used to compute drift from the data in - the monitored `table_name`. The baseline table and the monitored table shall have the same schema. - :param custom_metrics: List[:class:`MonitorMetric`] (optional) - [Create:OPT Update:OPT] Custom metrics. - :param dashboard_id: str (optional) - [Create:ERR Update:OPT] Id of dashboard that visualizes the computed metrics. This can be empty if - the monitor is in PENDING state. - :param data_classification_config: - :class:`MonitorDataClassificationConfig` (optional) - [Create:OPT Update:OPT] Data classification related config. - :param inference_log: :class:`MonitorInferenceLog` (optional) - - :param latest_monitor_failure_msg: str (optional) - [Create:ERR Update:IGN] The latest error message for a monitor failure. - :param notifications: :class:`MonitorNotifications` (optional) - [Create:OPT Update:OPT] Field for specifying notification settings. - :param schedule: :class:`MonitorCronSchedule` (optional) - [Create:OPT Update:OPT] The monitor schedule. - :param slicing_exprs: List[str] (optional) - [Create:OPT Update:OPT] List of column expressions to slice data with for targeted analysis. The - data is grouped by each expression independently, resulting in a separate slice for each predicate - and its complements. For example `slicing_exprs=[“col_1”, “col_2 > 10”]` will generate the - following slices: two slices for `col_2 > 10` (True and False), and one slice per unique value in - `col1`. For high-cardinality columns, only the top 100 unique values by frequency will generate - slices. - :param snapshot: :class:`MonitorSnapshot` (optional) - Configuration for monitoring snapshot tables. - :param time_series: :class:`MonitorTimeSeries` (optional) - Configuration for monitoring time series tables. + :param warehouse_id: str (optional) + Optional argument to specify the warehouse for dashboard regeneration. If not specified, the first + running warehouse will be used. - :returns: :class:`MonitorInfo` + :returns: :class:`RegenerateDashboardResponse` parameters: - name: table_name in: path @@ -7297,283 +12891,196 @@ paths: schema: type: string description: >- - UC table name in format `catalog.schema.table_name`. This field + UC table name in format ``catalog.schema.table_name``. This field corresponds to the {full_table_name_arg} arg in the endpoint path. - requestBody: - content: - application/json: - schema: - type: object - properties: - output_schema_name: - type: string - description: >- - [Create:REQ Update:REQ] Schema where output tables are - created. Needs to be in 2-level format {catalog}.{schema} - baseline_table_name: - type: string - description: >- - [Create:OPT Update:OPT] Baseline table name. Baseline data - is used to compute drift from the data in the monitored - `table_name`. The baseline table and the monitored table - shall have the same schema. - custom_metrics: - type: array - items: - type: object - properties: - name: - type: string - description: Name of the metric in the output tables. - definition: - type: string - description: >- - Jinja template for a SQL expression that specifies how - to compute the metric. See [create metric definition]. - [create metric definition]: - https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition - input_columns: - type: array - items: - type: string - description: >- - A list of column names in the input table the metric - should be computed for. Can use ``":table"`` to - indicate that the metric needs information from - multiple columns. - output_data_type: - type: string - description: The output type of the custom metric. - type: - type: string - x-enum: - - CUSTOM_METRIC_TYPE_AGGREGATE - - CUSTOM_METRIC_TYPE_DERIVED - - CUSTOM_METRIC_TYPE_DRIFT - description: >- - Can only be one of ``"CUSTOM_METRIC_TYPE_AGGREGATE"``, - ``"CUSTOM_METRIC_TYPE_DERIVED"``, or - ``"CUSTOM_METRIC_TYPE_DRIFT"``. The - ``"CUSTOM_METRIC_TYPE_AGGREGATE"`` and - ``"CUSTOM_METRIC_TYPE_DERIVED"`` metrics are computed - on a single table, whereas the - ``"CUSTOM_METRIC_TYPE_DRIFT"`` compare metrics across - baseline and input table, or across the two - consecutive time windows. - - CUSTOM_METRIC_TYPE_AGGREGATE: only depend on the - existing columns in your table - - CUSTOM_METRIC_TYPE_DERIVED: depend on previously - computed aggregate metrics - CUSTOM_METRIC_TYPE_DRIFT: - depend on previously computed aggregate or derived - metrics - required: - - name - - definition - - input_columns - - output_data_type - - type - description: Custom metric definition. - description: '[Create:OPT Update:OPT] Custom metrics.' - dashboard_id: - type: string - description: >- - [Create:ERR Update:OPT] Id of dashboard that visualizes the - computed metrics. This can be empty if the monitor is in - PENDING state. - data_classification_config: - type: object - properties: - enabled: - type: boolean - description: Whether to enable data classification. - description: '[Create:OPT Update:OPT] Data classification related config.' - inference_log: - type: object - properties: - problem_type: - type: string - x-enum: - - PROBLEM_TYPE_CLASSIFICATION - - PROBLEM_TYPE_REGRESSION - description: >- - Create a collection of name/value pairs. - - - Example enumeration: - - - >>> class Color(Enum): - - ... RED = 1 - - ... BLUE = 2 - - ... GREEN = 3 - - - Access them by: - - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - + requestBody: + content: + application/json: + schema: + type: object + properties: + warehouse_id: + type: string + description: >- + Optional argument to specify the warehouse for dashboard + regeneration. If not specified, the first running warehouse + will be used. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/RegenerateDashboardResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: QualityMonitorsAPI + /api/2.1/unity-catalog/models: + post: + operationId: registered_models_create + summary: Creates a new registered model in Unity Catalog. + tags: + - catalog + - registered_models + description: >- + Creates a new registered model in Unity Catalog. - Enumerations can be iterated over, and know how many - members they have: + File storage for model versions in the registered model will be located + in the default location which - >>> len(Color) + is specified by the parent schema, or the parent catalog, or the + Metastore. - 3 + For registered model creation to succeed, the user must satisfy the + following conditions: - >>> list(Color) - [, , ] + - The caller must be a metastore admin, or be the owner of the parent + catalog and schema, or have the + **USE_CATALOG** privilege on the parent catalog and the **USE_SCHEMA** privilege on the parent + schema. + - The caller must have the **CREATE MODEL** or **CREATE FUNCTION** + privilege on the parent schema. - Methods can be added to enumerations, and members can - have their own + :param aliases: List[:class:`RegisteredModelAlias`] (optional) + List of aliases associated with the registered model + :param browse_only: bool (optional) + Indicates whether the principal is limited to retrieving metadata for the associated object through + the BROWSE privilege when include_browse is enabled in the request. + :param catalog_name: str (optional) + The name of the catalog where the schema and the registered model reside + :param comment: str (optional) + The comment attached to the registered model + :param created_at: int (optional) + Creation timestamp of the registered model in milliseconds since the Unix epoch + :param created_by: str (optional) + The identifier of the user who created the registered model + :param full_name: str (optional) + The three-level (fully qualified) name of the registered model + :param metastore_id: str (optional) + The unique identifier of the metastore + :param name: str (optional) + The name of the registered model + :param owner: str (optional) + The identifier of the user who owns the registered model + :param schema_name: str (optional) + The name of the schema where the registered model resides + :param storage_location: str (optional) + The storage location on the cloud under which model version data files are stored + :param updated_at: int (optional) + Last-update timestamp of the registered model in milliseconds since the Unix epoch + :param updated_by: str (optional) + The identifier of the user who updated the registered model last time - attributes -- see the documentation for details. - timestamp_col: - type: string - description: Column for the timestamp. - granularities: - type: array - items: + :returns: :class:`RegisteredModelInfo` + requestBody: + content: + application/json: + schema: + type: object + properties: + aliases: + type: array + items: + type: object + properties: + alias_name: type: string - description: >- - List of granularities to use when aggregating data into - time windows based on their timestamp. - prediction_col: - type: string - description: Column for the prediction. - model_id_col: - type: string - description: Column for the model identifier. - label_col: - type: string - description: Column for the label. - prediction_proba_col: - type: string - description: Column for prediction probabilities - required: - - problem_type - - timestamp_col - - granularities - - prediction_col - - model_id_col - latest_monitor_failure_msg: + catalog_name: + type: string + description: The name of the catalog containing the model version + id: + type: string + description: The unique identifier of the alias + model_name: + type: string + description: >- + The name of the parent registered model of the model + version, relative to parent schema + schema_name: + type: string + description: >- + The name of the schema containing the model version, + relative to parent catalog + version_num: + type: integer + description: >- + Integer version number of the model version to which + this alias points. + description: List of aliases associated with the registered model + browse_only: + type: boolean + description: >- + Indicates whether the principal is limited to retrieving + metadata for the associated object through the BROWSE + privilege when include_browse is enabled in the request. + catalog_name: type: string description: >- - [Create:ERR Update:IGN] The latest error message for a - monitor failure. - notifications: - type: object - properties: - on_failure: - type: object - properties: - email_addresses: - type: array - items: - type: string - on_new_classification_tag_detected: - type: object - properties: - email_addresses: - type: array - items: - type: string - description: >- - Destinations to send notifications on new classification - tag detected. + The name of the catalog where the schema and the registered + model reside + comment: + type: string + description: The comment attached to the registered model + created_at: + type: integer description: >- - [Create:OPT Update:OPT] Field for specifying notification - settings. - schedule: - type: object - properties: - quartz_cron_expression: - type: string - timezone_id: - type: string - description: >- - The timezone id (e.g., ``PST``) in which to evaluate the - quartz expression. - pause_status: - type: string - x-enum: - - PAUSED - - UNPAUSED - - UNSPECIFIED - description: >- - Read only field that indicates whether a schedule is - paused or not. - required: - - quartz_cron_expression - - timezone_id - description: '[Create:OPT Update:OPT] The monitor schedule.' - slicing_exprs: - type: array - items: - type: string + Creation timestamp of the registered model in milliseconds + since the Unix epoch + created_by: + type: string + description: The identifier of the user who created the registered model + full_name: + type: string description: >- - [Create:OPT Update:OPT] List of column expressions to slice - data with for targeted analysis. The data is grouped by each - expression independently, resulting in a separate slice for - each predicate and its complements. For example - `slicing_exprs=[“col_1”, “col_2 > 10”]` will generate the - following slices: two slices for `col_2 > 10` (True and - False), and one slice per unique value in `col1`. For - high-cardinality columns, only the top 100 unique values by - frequency will generate slices. - snapshot: - type: object - properties: {} - description: Configuration for monitoring snapshot tables. - time_series: - type: object - properties: - timestamp_col: - type: string - description: Column for the timestamp. - granularities: - type: array - items: - type: string - description: >- - Granularities for aggregating data into time windows - based on their timestamp. Currently the following static - granularities are supported: {``\"5 minutes\"``, ``\"30 - minutes\"``, ``\"1 hour\"``, ``\"1 day\"``, - ``\"\u003cn\u003e week(s)\"``, ``\"1 month\"``, ``\"1 - year\"``}. - required: - - timestamp_col - - granularities - description: Configuration for monitoring time series tables. - required: - - output_schema_name + The three-level (fully qualified) name of the registered + model + metastore_id: + type: string + description: The unique identifier of the metastore + name: + type: string + description: The name of the registered model + owner: + type: string + description: The identifier of the user who owns the registered model + schema_name: + type: string + description: The name of the schema where the registered model resides + storage_location: + type: string + description: >- + The storage location on the cloud under which model version + data files are stored + updated_at: + type: integer + description: >- + Last-update timestamp of the registered model in + milliseconds since the Unix epoch + updated_by: + type: string + description: >- + The identifier of the user who updated the registered model + last time responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/MonitorInfo' + $ref: '#/components/schemas/RegisteredModelInfo' default: description: Error response content: @@ -7585,131 +13092,143 @@ paths: type: string message: type: string - x-stackql-sdk-source: QualityMonitorsAPI - /api/2.1/unity-catalog/tables/{table_name}/monitor/refreshes/{refresh_id}: + x-stackql-sdk-source: RegisteredModelsAPI get: - operationId: quality_monitors_get_refresh + operationId: registered_models_list summary: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Gets info about a + List registered models. You can list registered models under a + particular schema, or list all tags: - catalog - - quality_monitors + - registered_models description: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Gets info about a - - specific monitor refresh using the given refresh ID. - + List registered models. You can list registered models under a + particular schema, or list all - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the + registered models in the current metastore. - table's parent catalog and be an owner of the table's parent schema 3. - have the following permissions: - - **USE_CATALOG** on the table's parent catalog - **USE_SCHEMA** on the - table's parent schema - + The returned models are filtered based on the privileges of the calling + user. For example, the - **SELECT** privilege on the table. + metastore admin is able to list all the registered models. A regular + user needs to be the owner or + have the **EXECUTE** privilege on the registered model to receive the + registered models in the - Additionally, the call must be made from the workspace where the monitor - was created. + response. For the latter case, the caller must also be the owner or have + the **USE_CATALOG** privilege + on the parent catalog and the **USE_SCHEMA** privilege on the parent + schema. - :param table_name: str - Full name of the table. - :param refresh_id: int - ID of the refresh. - :returns: :class:`MonitorRefreshInfo` - parameters: - - name: table_name - in: path - required: true - schema: - type: string - description: Full name of the table. - - name: refresh_id - in: path - required: true - schema: - type: integer - description: ID of the refresh. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/MonitorRefreshInfo' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: QualityMonitorsAPI - /api/2.1/unity-catalog/tables/{table_name}/monitor/refreshes: - get: - operationId: quality_monitors_list_refreshes - summary: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Gets an array - tags: - - catalog - - quality_monitors - description: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Gets an array + There is no guarantee of a specific ordering of the elements in the + response. - containing the history of the most recent refreshes (up to 25) for this - table. + PAGINATION BEHAVIOR: The API is by default paginated, a page may contain + zero results while still - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the + providing a next_page_token. Clients must continue reading pages until + next_page_token is absent, - table's parent catalog and be an owner of the table's parent schema 3. - have the following permissions: + which is the only indication that the end of results has been reached. - - **USE_CATALOG** on the table's parent catalog - **USE_SCHEMA** on the - table's parent schema - - **SELECT** privilege on the table. + :param catalog_name: str (optional) + The identifier of the catalog under which to list registered models. If specified, schema_name must + be specified. + :param include_browse: bool (optional) + Whether to include registered models in the response for which the principal can only access + selective metadata for + :param max_results: int (optional) + Max number of registered models to return. + If both catalog and schema are specified: - Additionally, the call must be made from the workspace where the monitor - was created. + - when max_results is not specified, the page length is set to a server configured value (10000, as + of 4/2/2024). + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value (10000, as of 4/2/2024); + - when set to 0, the page length is set to a server configured value (10000, as of 4/2/2024); + - when set to a value less than 0, an invalid parameter error is returned; + If neither schema nor catalog is specified: - :param table_name: str - UC table name in format `catalog.schema.table_name`. table_name is case insensitive and spaces are - disallowed. + - when max_results is not specified, the page length is set to a server configured value (100, as of + 4/2/2024). + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value (1000, as of 4/2/2024); + - when set to 0, the page length is set to a server configured value (100, as of 4/2/2024); + - when set to a value less than 0, an invalid parameter error is returned; + :param page_token: str (optional) + Opaque token to send for the next page of results (pagination). + :param schema_name: str (optional) + The identifier of the schema under which to list registered models. If specified, catalog_name must + be specified. - :returns: :class:`MonitorRefreshListResponse` + :returns: Iterator over :class:`RegisteredModelInfo` parameters: - - name: table_name - in: path - required: true + - name: catalog_name + in: query + required: false schema: type: string description: >- - UC table name in format `catalog.schema.table_name`. table_name is - case insensitive and spaces are disallowed. + The identifier of the catalog under which to list registered models. + If specified, schema_name must be specified. + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include registered models in the response for which the + principal can only access selective metadata for + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + Max number of registered models to return. If both catalog and + schema are specified: - when max_results is not specified, the page + length is set to a server configured value (10000, as of 4/2/2024). + - when set to a value greater than 0, the page length is the minimum + of this value and a server configured value (10000, as of 4/2/2024); + - when set to 0, the page length is set to a server configured value + (10000, as of 4/2/2024); - when set to a value less than 0, an + invalid parameter error is returned; If neither schema nor catalog + is specified: - when max_results is not specified, the page length + is set to a server configured value (100, as of 4/2/2024). - when + set to a value greater than 0, the page length is the minimum of + this value and a server configured value (1000, as of 4/2/2024); - + when set to 0, the page length is set to a server configured value + (100, as of 4/2/2024); - when set to a value less than 0, an invalid + parameter error is returned; + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque token to send for the next page of results (pagination). + - name: schema_name + in: query + required: false + schema: + type: string + description: >- + The identifier of the schema under which to list registered models. + If specified, catalog_name must be specified. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/MonitorRefreshListResponse' + $ref: '#/components/schemas/ListRegisteredModelsResponse' default: description: Error response content: @@ -7721,60 +13240,42 @@ paths: type: string message: type: string - x-stackql-sdk-source: QualityMonitorsAPI - post: - operationId: quality_monitors_run_refresh + x-stackql-sdk-source: RegisteredModelsAPI + /api/2.1/unity-catalog/models/{full_name}: + delete: + operationId: registered_models_delete summary: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Queues a metric + Deletes a registered model and all its model versions from the specified + parent catalog and schema. tags: - catalog - - quality_monitors + - registered_models description: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Queues a metric - - refresh on the monitor for the specified table. The refresh will execute - in the background. - - - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the - - table's parent catalog and be an owner of the table's parent schema 3. - have the following permissions: - - - **USE_CATALOG** on the table's parent catalog - **USE_SCHEMA** on the - table's parent schema - be an + Deletes a registered model and all its model versions from the specified + parent catalog and schema. - owner of the table + The caller must be a metastore admin or an owner of the registered + model. For the latter case, the - Additionally, the call must be made from the workspace where the monitor - was created. + caller must also be the owner or have the **USE_CATALOG** privilege on + the parent catalog and the + **USE_SCHEMA** privilege on the parent schema. - :param table_name: str - UC table name in format `catalog.schema.table_name`. table_name is case insensitive and spaces are - disallowed. - :returns: :class:`MonitorRefreshInfo` + :param full_name: str + The three-level (fully qualified) name of the registered model parameters: - - name: table_name + - name: full_name in: path required: true schema: type: string - description: >- - UC table name in format `catalog.schema.table_name`. table_name is - case insensitive and spaces are disallowed. + description: The three-level (fully qualified) name of the registered model responses: '200': description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/MonitorRefreshInfo' default: description: Error response content: @@ -7786,78 +13287,64 @@ paths: type: string message: type: string - x-stackql-sdk-source: QualityMonitorsAPI - /api/2.1/quality-monitoring/tables/{table_name}/monitor/dashboard: - post: - operationId: quality_monitors_regenerate_dashboard - summary: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Regenerates the - tags: - - catalog - - quality_monitors - description: >- - Deprecated: Use Data Quality Monitors API instead - (/api/data-quality/v1/monitors). Regenerates the - - monitoring dashboard for the specified table. - - - The caller must either: 1. be an owner of the table's parent catalog 2. - have **USE_CATALOG** on the - - table's parent catalog and be an owner of the table's parent schema 3. - have the following permissions: - - - **USE_CATALOG** on the table's parent catalog - **USE_SCHEMA** on the - table's parent schema - be an + x-stackql-sdk-source: RegisteredModelsAPI + get: + operationId: registered_models_get + summary: Get a registered model. + tags: + - catalog + - registered_models + description: >- + Get a registered model. - owner of the table + The caller must be a metastore admin or an owner of (or have the + **EXECUTE** privilege on) the - The call must be made from the workspace where the monitor was created. - The dashboard will be + registered model. For the latter case, the caller must also be the owner + or have the **USE_CATALOG** - regenerated in the assets directory that was specified when the monitor - was created. + privilege on the parent catalog and the **USE_SCHEMA** privilege on the + parent schema. - :param table_name: str - UC table name in format `catalog.schema.table_name`. This field corresponds to the - {full_table_name_arg} arg in the endpoint path. - :param warehouse_id: str (optional) - Optional argument to specify the warehouse for dashboard regeneration. If not specified, the first - running warehouse will be used. + :param full_name: str + The three-level (fully qualified) name of the registered model + :param include_aliases: bool (optional) + Whether to include registered model aliases in the response + :param include_browse: bool (optional) + Whether to include registered models in the response for which the principal can only access + selective metadata for - :returns: :class:`RegenerateDashboardResponse` + :returns: :class:`RegisteredModelInfo` parameters: - - name: table_name + - name: full_name in: path required: true schema: type: string + description: The three-level (fully qualified) name of the registered model + - name: include_aliases + in: query + required: false + schema: + type: boolean + description: Whether to include registered model aliases in the response + - name: include_browse + in: query + required: false + schema: + type: boolean description: >- - UC table name in format `catalog.schema.table_name`. This field - corresponds to the {full_table_name_arg} arg in the endpoint path. - requestBody: - content: - application/json: - schema: - type: object - properties: - warehouse_id: - type: string - description: >- - Optional argument to specify the warehouse for dashboard - regeneration. If not specified, the first running warehouse - will be used. + Whether to include registered models in the response for which the + principal can only access selective metadata for responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/RegenerateDashboardResponse' + $ref: '#/components/schemas/RegisteredModelInfo' default: description: Error response content: @@ -7869,38 +13356,32 @@ paths: type: string message: type: string - x-stackql-sdk-source: QualityMonitorsAPI - /api/2.1/unity-catalog/models: - post: - operationId: registered_models_create - summary: Creates a new registered model in Unity Catalog. + x-stackql-sdk-source: RegisteredModelsAPI + patch: + operationId: registered_models_update + summary: Updates the specified registered model. tags: - catalog - registered_models description: >- - Creates a new registered model in Unity Catalog. - - - File storage for model versions in the registered model will be located - in the default location which + Updates the specified registered model. - is specified by the parent schema, or the parent catalog, or the - Metastore. + The caller must be a metastore admin or an owner of the registered + model. For the latter case, the - For registered model creation to succeed, the user must satisfy the - following conditions: - The caller + caller must also be the owner or have the **USE_CATALOG** privilege on + the parent catalog and the - must be a metastore admin, or be the owner of the parent catalog and - schema, or have the + **USE_SCHEMA** privilege on the parent schema. - **USE_CATALOG** privilege on the parent catalog and the **USE_SCHEMA** - privilege on the parent schema. - - The caller must have the **CREATE MODEL** or **CREATE FUNCTION** - privilege on the parent schema. + Currently only the name, the owner or the comment of the registered + model can be updated. + :param full_name: str + The three-level (fully qualified) name of the registered model :param aliases: List[:class:`RegisteredModelAlias`] (optional) List of aliases associated with the registered model :param browse_only: bool (optional) @@ -7914,12 +13395,12 @@ paths: Creation timestamp of the registered model in milliseconds since the Unix epoch :param created_by: str (optional) The identifier of the user who created the registered model - :param full_name: str (optional) - The three-level (fully qualified) name of the registered model :param metastore_id: str (optional) The unique identifier of the metastore :param name: str (optional) The name of the registered model + :param new_name: str (optional) + New name for the registered model. :param owner: str (optional) The identifier of the user who owns the registered model :param schema_name: str (optional) @@ -7932,6 +13413,13 @@ paths: The identifier of the user who updated the registered model last time :returns: :class:`RegisteredModelInfo` + parameters: + - name: full_name + in: path + required: true + schema: + type: string + description: The three-level (fully qualified) name of the registered model requestBody: content: application/json: @@ -7989,17 +13477,15 @@ paths: created_by: type: string description: The identifier of the user who created the registered model - full_name: - type: string - description: >- - The three-level (fully qualified) name of the registered - model metastore_id: type: string description: The unique identifier of the metastore name: type: string description: The name of the registered model + new_name: + type: string + description: New name for the registered model. owner: type: string description: The identifier of the user who owns the registered model @@ -8040,136 +13526,323 @@ paths: message: type: string x-stackql-sdk-source: RegisteredModelsAPI + /api/2.1/unity-catalog/resource-quotas/{parent_securable_type}/{parent_full_name}/{quota_name}: get: - operationId: registered_models_list + operationId: resource_quotas_get_quota summary: >- - List registered models. You can list registered models under a - particular schema, or list all + The GetQuota API returns usage information for a single resource quota, + defined as a child-parent tags: - catalog - - registered_models + - resource_quotas description: >- - List registered models. You can list registered models under a - particular schema, or list all + The GetQuota API returns usage information for a single resource quota, + defined as a child-parent + + pair. This API also refreshes the quota count if it is out of date. + Refreshes are triggered + + asynchronously. The updated count might not be returned in the first + call. + + + :param parent_securable_type: str + Securable type of the quota parent. + :param parent_full_name: str + Full name of the parent resource. Provide the metastore ID if the parent is a metastore. + :param quota_name: str + Name of the quota. Follows the pattern of the quota type, with "-quota" added as a suffix. + + :returns: :class:`GetQuotaResponse` + parameters: + - name: parent_securable_type + in: path + required: true + schema: + type: string + description: Securable type of the quota parent. + - name: parent_full_name + in: path + required: true + schema: + type: string + description: >- + Full name of the parent resource. Provide the metastore ID if the + parent is a metastore. + - name: quota_name + in: path + required: true + schema: + type: string + description: >- + Name of the quota. Follows the pattern of the quota type, with + "-quota" added as a suffix. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GetQuotaResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ResourceQuotasAPI + /api/2.1/unity-catalog/resource-quotas/all-resource-quotas: + get: + operationId: resource_quotas_list_quotas + summary: >- + ListQuotas returns all quota values under the metastore. There are no + SLAs on the freshness of the + tags: + - catalog + - resource_quotas + description: >- + ListQuotas returns all quota values under the metastore. There are no + SLAs on the freshness of the + + counts returned. This API does not trigger a refresh of quota counts. + + + PAGINATION BEHAVIOR: The API is by default paginated, a page may contain + zero results while still + + providing a next_page_token. Clients must continue reading pages until + next_page_token is absent, + + which is the only indication that the end of results has been reached. + + + :param max_results: int (optional) + The number of quotas to return. + :param page_token: str (optional) + Opaque token for the next page of results. + + :returns: Iterator over :class:`QuotaInfo` + parameters: + - name: max_results + in: query + required: false + schema: + type: integer + description: The number of quotas to return. + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque token for the next page of results. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListQuotasResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ResourceQuotasAPI + /api/3.0/rfa/requests: + post: + operationId: rfa_batch_create_access_requests + summary: >- + Creates access requests for Unity Catalog permissions for a specified + principal on a securable object. + tags: + - catalog + - rfa + description: >- + Creates access requests for Unity Catalog permissions for a specified + principal on a securable object. - registered models in the current metastore. + This Batch API can take in multiple principals, securable objects, and + permissions as the input and + returns the access request destinations for each. Principals must be + unique across the API call. - The returned models are filtered based on the privileges of the calling - user. For example, the - metastore admin is able to list all the registered models. A regular - user needs to be the owner or + The supported securable types are: "metastore", "catalog", "schema", + "table", "external_location", - have the **EXECUTE** privilege on the registered model to recieve the - registered models in the + "connection", "credential", "function", "registered_model", and + "volume". - response. For the latter case, the caller must also be the owner or have - the **USE_CATALOG** privilege - on the parent catalog and the **USE_SCHEMA** privilege on the parent - schema. + :param requests: List[:class:`CreateAccessRequest`] (optional) + A list of individual access requests, where each request corresponds to a set of permissions being + requested on a list of securables for a specified principal. + At most 30 requests per API call. - There is no guarantee of a specific ordering of the elements in the - response. + :returns: :class:`BatchCreateAccessRequestsResponse` + requestBody: + content: + application/json: + schema: + type: object + properties: + requests: + type: array + items: + type: object + properties: + behalf_of: + type: object + properties: + id: + type: string + principal_type: + type: string + x-enum: + - GROUP_PRINCIPAL + - SERVICE_PRINCIPAL + - USER_PRINCIPAL + description: >- + Create a collection of name/value pairs. - PAGINATION BEHAVIOR: The API is by default paginated, a page may contain - zero results while still + Example enumeration: - providing a next_page_token. Clients must continue reading pages until - next_page_token is absent, - which is the only indication that the end of results has been reached. + >>> class Color(Enum): + ... RED = 1 - :param catalog_name: str (optional) - The identifier of the catalog under which to list registered models. If specified, schema_name must - be specified. - :param include_browse: bool (optional) - Whether to include registered models in the response for which the principal can only access - selective metadata for - :param max_results: int (optional) - Max number of registered models to return. + ... BLUE = 2 - If both catalog and schema are specified: - when max_results is not specified, the page length is - set to a server configured value (10000, as of 4/2/2024). - when set to a value greater than 0, the - page length is the minimum of this value and a server configured value (10000, as of 4/2/2024); - - when set to 0, the page length is set to a server configured value (10000, as of 4/2/2024); - when - set to a value less than 0, an invalid parameter error is returned; - - If neither schema nor catalog is specified: - when max_results is not specified, the page length is - set to a server configured value (100, as of 4/2/2024). - when set to a value greater than 0, the - page length is the minimum of this value and a server configured value (1000, as of 4/2/2024); - - when set to 0, the page length is set to a server configured value (100, as of 4/2/2024); - when set - to a value less than 0, an invalid parameter error is returned; - :param page_token: str (optional) - Opaque token to send for the next page of results (pagination). - :param schema_name: str (optional) - The identifier of the schema under which to list registered models. If specified, catalog_name must - be specified. + ... GREEN = 3 - :returns: Iterator over :class:`RegisteredModelInfo` - parameters: - - name: catalog_name - in: query - required: false - schema: - type: string - description: >- - The identifier of the catalog under which to list registered models. - If specified, schema_name must be specified. - - name: include_browse - in: query - required: false - schema: - type: boolean - description: >- - Whether to include registered models in the response for which the - principal can only access selective metadata for - - name: max_results - in: query - required: false - schema: - type: integer - description: >- - Max number of registered models to return. If both catalog and - schema are specified: - when max_results is not specified, the page - length is set to a server configured value (10000, as of 4/2/2024). - - when set to a value greater than 0, the page length is the minimum - of this value and a server configured value (10000, as of 4/2/2024); - - when set to 0, the page length is set to a server configured value - (10000, as of 4/2/2024); - when set to a value less than 0, an - invalid parameter error is returned; If neither schema nor catalog - is specified: - when max_results is not specified, the page length - is set to a server configured value (100, as of 4/2/2024). - when - set to a value greater than 0, the page length is the minimum of - this value and a server configured value (1000, as of 4/2/2024); - - when set to 0, the page length is set to a server configured value - (100, as of 4/2/2024); - when set to a value less than 0, an invalid - parameter error is returned; - - name: page_token - in: query - required: false - schema: - type: string - description: Opaque token to send for the next page of results (pagination). - - name: schema_name - in: query - required: false - schema: - type: string - description: >- - The identifier of the schema under which to list registered models. - If specified, catalog_name must be specified. + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how + many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + comment: + type: string + description: >- + Optional. Comment associated with the request. At most + 200 characters, can only contain lowercase/uppercase + letters (a-z, A-Z), numbers (0-9), punctuation, and + spaces. + securable_permissions: + type: array + items: + type: object + properties: + permissions: + type: array + items: + type: string + securable: + type: object + properties: + full_name: + type: string + description: >- + Required. The full name of the + catalog/schema/table. Optional if + resource_name is present. + provider_share: + type: string + description: >- + Optional. The name of the Share object that + contains the securable when the securable is + getting shared in D2D Delta Sharing. + type: + type: string + x-enum: + - CATALOG + - CLEAN_ROOM + - CONNECTION + - CREDENTIAL + - EXTERNAL_LOCATION + - EXTERNAL_METADATA + - FUNCTION + - METASTORE + - PIPELINE + - PROVIDER + - RECIPIENT + - SCHEMA + - SHARE + - STAGING_TABLE + - STORAGE_CREDENTIAL + - TABLE + - VOLUME + description: >- + Required. The type of securable + (catalog/schema/table). Optional if + resource_name is present. + description: >- + The securable for which the access request + destinations are being requested. + description: >- + List of securables and their corresponding requested + UC privileges. At most 30 securables can be requested + for a principal per batched call. Each securable can + only be requested once per principal. + description: >- + A list of individual access requests, where each request + corresponds to a set of permissions being requested on a + list of securables for a specified principal. At most 30 + requests per API call. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListRegisteredModelsResponse' + $ref: '#/components/schemas/BatchCreateAccessRequestsResponse' default: description: Error response content: @@ -8181,42 +13854,59 @@ paths: type: string message: type: string - x-stackql-sdk-source: RegisteredModelsAPI - /api/2.1/unity-catalog/models/{full_name}: - delete: - operationId: registered_models_delete + x-stackql-sdk-source: RfaAPI + /api/3.0/rfa/destinations/{securable_type}/{full_name}: + get: + operationId: rfa_get_access_request_destinations summary: >- - Deletes a registered model and all its model versions from the specified - parent catalog and schema. + Gets an array of access request destinations for the specified + securable. Any caller can see URL tags: - catalog - - registered_models + - rfa description: >- - Deletes a registered model and all its model versions from the specified - parent catalog and schema. + Gets an array of access request destinations for the specified + securable. Any caller can see URL + + destinations or the destinations on the metastore. Otherwise, only those + with **BROWSE** permissions + on the securable can see destinations. - The caller must be a metastore admin or an owner of the registered - model. For the latter case, the - caller must also be the owner or have the **USE_CATALOG** privilege on - the parent catalog and the + The supported securable types are: "metastore", "catalog", "schema", + "table", "external_location", - **USE_SCHEMA** privilege on the parent schema. + "connection", "credential", "function", "registered_model", and + "volume". + :param securable_type: str + The type of the securable. :param full_name: str - The three-level (fully qualified) name of the registered model + The full name of the securable. + + :returns: :class:`AccessRequestDestinations` parameters: + - name: securable_type + in: path + required: true + schema: + type: string + description: The type of the securable. - name: full_name in: path required: true schema: type: string - description: The three-level (fully qualified) name of the registered model + description: The full name of the securable. responses: '200': description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/AccessRequestDestinations' default: description: Error response content: @@ -8228,64 +13918,240 @@ paths: type: string message: type: string - x-stackql-sdk-source: RegisteredModelsAPI - get: - operationId: registered_models_get - summary: Get a registered model. + x-stackql-sdk-source: RfaAPI + /api/3.0/rfa/destinations: + patch: + operationId: rfa_update_access_request_destinations + summary: >- + Updates the access request destinations for the given securable. The + caller must be a metastore admin, tags: - catalog - - registered_models + - rfa description: >- - Get a registered model. + Updates the access request destinations for the given securable. The + caller must be a metastore admin, + the owner of the securable, or a user that has the **MANAGE** privilege + on the securable in order to - The caller must be a metastore admin or an owner of (or have the - **EXECUTE** privilege on) the + assign destinations. A maximum of 5 emails and 5 external notification + destinations (Slack, Microsoft - registered model. For the latter case, the caller must also be the owner - or have the **USE_CATALOG** + Teams, and Generic Webhook destinations) can be assigned to a securable. + If a URL destination is - privilege on the parent catalog and the **USE_SCHEMA** privilege on the - parent schema. + assigned, no other destinations can be set. - :param full_name: str - The three-level (fully qualified) name of the registered model - :param include_aliases: bool (optional) - Whether to include registered model aliases in the response - :param include_browse: bool (optional) - Whether to include registered models in the response for which the principal can only access - selective metadata for + The supported securable types are: "metastore", "catalog", "schema", + "table", "external_location", - :returns: :class:`RegisteredModelInfo` + "connection", "credential", "function", "registered_model", and + "volume". + + + :param access_request_destinations: :class:`AccessRequestDestinations` + The access request destinations to assign to the securable. For each destination, a + **destination_id** and **destination_type** must be defined. + :param update_mask: str + The field mask must be a single string, with multiple fields separated by commas (no spaces). The + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only + the entire collection field can be specified. Field names must exactly match the resource field + names. + + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. + + :returns: :class:`AccessRequestDestinations` parameters: - - name: full_name - in: path + - name: update_mask + in: query required: true schema: type: string - description: The three-level (fully qualified) name of the registered model - - name: include_aliases - in: query - required: false - schema: - type: boolean - description: Whether to include registered model aliases in the response - - name: include_browse - in: query - required: false - schema: - type: boolean description: >- - Whether to include registered models in the response for which the - principal can only access selective metadata for + The field mask must be a single string, with multiple fields + separated by commas (no spaces). The field path is relative to the + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map + fields is not allowed, as only the entire collection field can be + specified. Field names must exactly match the resource field names. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. + requestBody: + content: + application/json: + schema: + type: object + properties: + access_request_destinations: + type: object + properties: + securable: + type: object + properties: + full_name: + type: string + description: >- + Required. The full name of the catalog/schema/table. + Optional if resource_name is present. + provider_share: + type: string + description: >- + Optional. The name of the Share object that contains + the securable when the securable is getting shared + in D2D Delta Sharing. + type: + type: string + x-enum: + - CATALOG + - CLEAN_ROOM + - CONNECTION + - CREDENTIAL + - EXTERNAL_LOCATION + - EXTERNAL_METADATA + - FUNCTION + - METASTORE + - PIPELINE + - PROVIDER + - RECIPIENT + - SCHEMA + - SHARE + - STAGING_TABLE + - STORAGE_CREDENTIAL + - TABLE + - VOLUME + description: >- + Required. The type of securable + (catalog/schema/table). Optional if resource_name is + present. + description: >- + Generic definition of a securable, which is uniquely + defined in a metastore by its type and full + name. + are_any_destinations_hidden: + type: boolean + description: >- + Indicates whether any destinations are hidden from the + caller due to a lack of permissions. This value is true + if the caller does not have permission to see all + destinations. + destination_source_securable: + type: object + properties: + full_name: + type: string + description: >- + Required. The full name of the catalog/schema/table. + Optional if resource_name is present. + provider_share: + type: string + description: >- + Optional. The name of the Share object that contains + the securable when the securable is getting shared + in D2D Delta Sharing. + type: + type: string + x-enum: + - CATALOG + - CLEAN_ROOM + - CONNECTION + - CREDENTIAL + - EXTERNAL_LOCATION + - EXTERNAL_METADATA + - FUNCTION + - METASTORE + - PIPELINE + - PROVIDER + - RECIPIENT + - SCHEMA + - SHARE + - STAGING_TABLE + - STORAGE_CREDENTIAL + - TABLE + - VOLUME + description: >- + Required. The type of securable + (catalog/schema/table). Optional if resource_name is + present. + description: >- + The source securable from which the destinations are + inherited. Either the same value as securable (if + destination is set directly on the securable) or the + nearest parent securable with destinations set. + destinations: + type: array + items: + type: object + properties: + destination_id: + type: string + destination_type: + type: string + x-enum: + - EMAIL + - GENERIC_WEBHOOK + - MICROSOFT_TEAMS + - SLACK + - URL + description: The type of the destination. + special_destination: + type: string + x-enum: + - SPECIAL_DESTINATION_CATALOG_OWNER + - SPECIAL_DESTINATION_CONNECTION_OWNER + - SPECIAL_DESTINATION_CREDENTIAL_OWNER + - SPECIAL_DESTINATION_EXTERNAL_LOCATION_OWNER + - SPECIAL_DESTINATION_FUNCTION_OWNER + - SPECIAL_DESTINATION_METASTORE_OWNER + - SPECIAL_DESTINATION_REGISTERED_MODEL_OWNER + - SPECIAL_DESTINATION_SCHEMA_OWNER + - SPECIAL_DESTINATION_TABLE_OWNER + - SPECIAL_DESTINATION_VOLUME_OWNER + description: >- + This field is used to denote whether the + destination is the email of the owner of the + securable object. The special destination cannot + be assigned to a securable and only represents the + default destination of the securable. The + securable types that support default special + destinations are: "catalog", "external_location", + "connection", "credential", and "metastore". The + **destination_type** of a **special_destination** + is always EMAIL. + description: The access request destinations for the securable. + full_name: + type: string + description: >- + The full name of the securable. Redundant with the name + in the securable object, but necessary for Terraform + integration + securable_type: + type: string + description: >- + The type of the securable. Redundant with the type in + the securable object, but necessary for Terraform + integration + required: + - securable + description: >- + The access request destinations to assign to the securable. + For each destination, a **destination_id** and + **destination_type** must be defined. + required: + - access_request_destinations responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/RegisteredModelInfo' + $ref: '#/components/schemas/AccessRequestDestinations' default: description: Error response content: @@ -8297,164 +14163,71 @@ paths: type: string message: type: string - x-stackql-sdk-source: RegisteredModelsAPI - patch: - operationId: registered_models_update - summary: Updates the specified registered model. + x-stackql-sdk-source: RfaAPI + /api/2.1/unity-catalog/schemas: + post: + operationId: schemas_create + summary: >- + Creates a new schema for catalog in the Metastore. The caller must be a + metastore admin, or have the tags: - catalog - - registered_models + - schemas description: >- - Updates the specified registered model. - - - The caller must be a metastore admin or an owner of the registered - model. For the latter case, the - - caller must also be the owner or have the **USE_CATALOG** privilege on - the parent catalog and the - - **USE_SCHEMA** privilege on the parent schema. - + Creates a new schema for catalog in the Metastore. The caller must be a + metastore admin, or have the - Currently only the name, the owner or the comment of the registered - model can be updated. + **CREATE_SCHEMA** privilege in the parent catalog. - :param full_name: str - The three-level (fully qualified) name of the registered model - :param aliases: List[:class:`RegisteredModelAlias`] (optional) - List of aliases associated with the registered model - :param browse_only: bool (optional) - Indicates whether the principal is limited to retrieving metadata for the associated object through - the BROWSE privilege when include_browse is enabled in the request. - :param catalog_name: str (optional) - The name of the catalog where the schema and the registered model reside + :param name: str + Name of schema, relative to parent catalog. + :param catalog_name: str + Name of parent catalog. :param comment: str (optional) - The comment attached to the registered model - :param created_at: int (optional) - Creation timestamp of the registered model in milliseconds since the Unix epoch - :param created_by: str (optional) - The identifier of the user who created the registered model - :param metastore_id: str (optional) - The unique identifier of the metastore - :param name: str (optional) - The name of the registered model - :param new_name: str (optional) - New name for the registered model. - :param owner: str (optional) - The identifier of the user who owns the registered model - :param schema_name: str (optional) - The name of the schema where the registered model resides - :param storage_location: str (optional) - The storage location on the cloud under which model version data files are stored - :param updated_at: int (optional) - Last-update timestamp of the registered model in milliseconds since the Unix epoch - :param updated_by: str (optional) - The identifier of the user who updated the registered model last time + User-provided free-form text description. + :param custom_max_retention_hours: int (optional) + Custom maximum retention period in hours for the schema. + :param properties: Dict[str,str] (optional) + A map of key-value properties attached to the securable. + :param storage_root: str (optional) + Storage root URL for managed tables within schema. - :returns: :class:`RegisteredModelInfo` - parameters: - - name: full_name - in: path - required: true - schema: - type: string - description: The three-level (fully qualified) name of the registered model + :returns: :class:`SchemaInfo` requestBody: content: application/json: schema: type: object properties: - aliases: - type: array - items: - type: object - properties: - alias_name: - type: string - catalog_name: - type: string - description: The name of the catalog containing the model version - id: - type: string - description: The unique identifier of the alias - model_name: - type: string - description: >- - The name of the parent registered model of the model - version, relative to parent schema - schema_name: - type: string - description: >- - The name of the schema containing the model version, - relative to parent catalog - version_num: - type: integer - description: >- - Integer version number of the model version to which - this alias points. - description: List of aliases associated with the registered model - browse_only: - type: boolean - description: >- - Indicates whether the principal is limited to retrieving - metadata for the associated object through the BROWSE - privilege when include_browse is enabled in the request. - catalog_name: - type: string - description: >- - The name of the catalog where the schema and the registered - model reside - comment: - type: string - description: The comment attached to the registered model - created_at: - type: integer - description: >- - Creation timestamp of the registered model in milliseconds - since the Unix epoch - created_by: - type: string - description: The identifier of the user who created the registered model - metastore_id: - type: string - description: The unique identifier of the metastore name: type: string - description: The name of the registered model - new_name: - type: string - description: New name for the registered model. - owner: - type: string - description: The identifier of the user who owns the registered model - schema_name: + description: Name of schema, relative to parent catalog. + catalog_name: type: string - description: The name of the schema where the registered model resides - storage_location: + description: Name of parent catalog. + comment: type: string - description: >- - The storage location on the cloud under which model version - data files are stored - updated_at: + description: User-provided free-form text description. + custom_max_retention_hours: type: integer - description: >- - Last-update timestamp of the registered model in - milliseconds since the Unix epoch - updated_by: + description: Custom maximum retention period in hours for the schema. + properties: + type: object + description: A map of key-value properties attached to the securable. + storage_root: type: string - description: >- - The identifier of the user who updated the registered model - last time + description: Storage root URL for managed tables within schema. + required: + - name + - catalog_name responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/RegisteredModelInfo' + $ref: '#/components/schemas/SchemaInfo' default: description: Error response content: @@ -8466,128 +14239,101 @@ paths: type: string message: type: string - x-stackql-sdk-source: RegisteredModelsAPI - /api/2.1/unity-catalog/resource-quotas/{parent_securable_type}/{parent_full_name}/{quota_name}: + x-stackql-sdk-source: SchemasAPI get: - operationId: resource_quotas_get_quota + operationId: schemas_list summary: >- - The GetQuota API returns usage information for a single resource quota, - defined as a child-parent + Gets an array of schemas for a catalog in the metastore. If the caller + is the metastore admin or the tags: - catalog - - resource_quotas + - schemas description: >- - The GetQuota API returns usage information for a single resource quota, - defined as a child-parent + Gets an array of schemas for a catalog in the metastore. If the caller + is the metastore admin or the - pair. This API also refreshes the quota count if it is out of date. - Refreshes are triggered + owner of the parent catalog, all schemas for the catalog will be + retrieved. Otherwise, only schemas - asynchronously. The updated count might not be returned in the first - call. + owned by the caller (or for which the caller has the **USE_SCHEMA** + privilege) will be retrieved. + There is no guarantee of a specific ordering of the elements in the + array. - :param parent_securable_type: str - Securable type of the quota parent. - :param parent_full_name: str - Full name of the parent resource. Provide the metastore ID if the parent is a metastore. - :param quota_name: str - Name of the quota. Follows the pattern of the quota type, with "-quota" added as a suffix. - :returns: :class:`GetQuotaResponse` - parameters: - - name: parent_securable_type - in: path - required: true - schema: - type: string - description: Securable type of the quota parent. - - name: parent_full_name - in: path - required: true - schema: - type: string - description: >- - Full name of the parent resource. Provide the metastore ID if the - parent is a metastore. - - name: quota_name - in: path - required: true - schema: - type: string - description: >- - Name of the quota. Follows the pattern of the quota type, with - "-quota" added as a suffix. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/GetQuotaResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: ResourceQuotasAPI - /api/2.1/unity-catalog/resource-quotas/all-resource-quotas: - get: - operationId: resource_quotas_list_quotas - summary: >- - ListQuotas returns all quota values under the metastore. There are no - SLAs on the freshness of the - tags: - - catalog - - resource_quotas - description: >- - ListQuotas returns all quota values under the metastore. There are no - SLAs on the freshness of the + NOTE: we recommend using max_results=0 to use the paginated version of + this API. Unpaginated calls - counts returned. This API does not trigger a refresh of quota counts. + will be deprecated soon. - PAGINATION BEHAVIOR: The API is by default paginated, a page may contain - zero results while still + PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page + may contain zero results while - providing a next_page_token. Clients must continue reading pages until - next_page_token is absent, + still providing a next_page_token. Clients must continue reading pages + until next_page_token is - which is the only indication that the end of results has been reached. + absent, which is the only indication that the end of results has been + reached. + :param catalog_name: str + Parent catalog for schemas of interest. + :param include_browse: bool (optional) + Whether to include schemas in the response for which the principal can only access selective + metadata for :param max_results: int (optional) - The number of quotas to return. + Maximum number of schemas to return. If not set, all the schemas are returned (not recommended). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; :param page_token: str (optional) - Opaque token for the next page of results. + Opaque pagination token to go to next page based on previous query. - :returns: Iterator over :class:`QuotaInfo` + :returns: Iterator over :class:`SchemaInfo` parameters: + - name: catalog_name + in: query + required: true + schema: + type: string + description: Parent catalog for schemas of interest. + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include schemas in the response for which the principal + can only access selective metadata for - name: max_results in: query required: false schema: type: integer - description: The number of quotas to return. + description: >- + Maximum number of schemas to return. If not set, all the schemas are + returned (not recommended). - when set to a value greater than 0, + the page length is the minimum of this value and a server configured + value; - when set to 0, the page length is set to a server + configured value (recommended); - when set to a value less than 0, + an invalid parameter error is returned; - name: page_token in: query required: false schema: type: string - description: Opaque token for the next page of results. + description: Opaque pagination token to go to next page based on previous query. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListQuotasResponse' + $ref: '#/components/schemas/ListSchemasResponse' default: description: Error response content: @@ -8599,191 +14345,43 @@ paths: type: string message: type: string - x-stackql-sdk-source: ResourceQuotasAPI - /api/3.0/rfa/requests: - post: - operationId: rfa_batch_create_access_requests + x-stackql-sdk-source: SchemasAPI + /api/2.1/unity-catalog/schemas/{full_name}: + delete: + operationId: schemas_delete summary: >- - Creates access requests for Unity Catalog permissions for a specified - principal on a securable object. + Deletes the specified schema from the parent catalog. The caller must be + the owner of the schema or an tags: - catalog - - rfa + - schemas description: >- - Creates access requests for Unity Catalog permissions for a specified - principal on a securable object. - - This Batch API can take in multiple principals, securable objects, and - permissions as the input and - - returns the access request destinations for each. Principals must be - unique across the API call. - - - The supported securable types are: "metastore", "catalog", "schema", - "table", "external_location", - - "connection", "credential", "function", "registered_model", and - "volume". - - - :param requests: List[:class:`CreateAccessRequest`] (optional) - A list of individual access requests, where each request corresponds to a set of permissions being - requested on a list of securables for a specified principal. - - At most 30 requests per API call. - - :returns: :class:`BatchCreateAccessRequestsResponse` - requestBody: - content: - application/json: - schema: - type: object - properties: - requests: - type: array - items: - type: object - properties: - behalf_of: - type: object - properties: - id: - type: string - principal_type: - type: string - x-enum: - - GROUP_PRINCIPAL - - SERVICE_PRINCIPAL - - USER_PRINCIPAL - description: >- - Create a collection of name/value pairs. - - - Example enumeration: - - - >>> class Color(Enum): - - ... RED = 1 - - ... BLUE = 2 - - ... GREEN = 3 - - - Access them by: - - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how - many members they have: - - - >>> len(Color) - - 3 - - - >>> list(Color) - - [, , ] + Deletes the specified schema from the parent catalog. The caller must be + the owner of the schema or an + owner of the parent catalog. - Methods can be added to enumerations, and members - can have their own - attributes -- see the documentation for details. - comment: - type: string - description: >- - Optional. Comment associated with the request. At most - 200 characters, can only contain lowercase/uppercase - letters (a-z, A-Z), numbers (0-9), punctuation, and - spaces. - securable_permissions: - type: array - items: - type: object - properties: - permissions: - type: array - items: - type: string - securable: - type: object - properties: - full_name: - type: string - description: >- - Required. The full name of the - catalog/schema/table. Optional if - resource_name is present. - provider_share: - type: string - description: >- - Optional. The name of the Share object that - contains the securable when the securable is - getting shared in D2D Delta Sharing. - type: - type: string - x-enum: - - CATALOG - - CLEAN_ROOM - - CONNECTION - - CREDENTIAL - - EXTERNAL_LOCATION - - EXTERNAL_METADATA - - FUNCTION - - METASTORE - - PIPELINE - - PROVIDER - - RECIPIENT - - SCHEMA - - SHARE - - STAGING_TABLE - - STORAGE_CREDENTIAL - - TABLE - - VOLUME - description: >- - Required. The type of securable - (catalog/schema/table). Optional if - resource_name is present. - description: >- - The securable for which the access request - destinations are being requested. - description: >- - List of securables and their corresponding requested - UC privileges. At most 30 securables can be requested - for a principal per batched call. Each securable can - only be requested once per principal. - description: >- - A list of individual access requests, where each request - corresponds to a set of permissions being requested on a - list of securables for a specified principal. At most 30 - requests per API call. + :param full_name: str + Full name of the schema. + :param force: bool (optional) + Force deletion even if the schema is not empty. + parameters: + - name: full_name + in: path + required: true + schema: + type: string + description: Full name of the schema. + - name: force + in: query + required: false + schema: + type: boolean + description: Force deletion even if the schema is not empty. responses: '200': description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/BatchCreateAccessRequestsResponse' default: description: Error response content: @@ -8794,60 +14392,52 @@ paths: error: type: string message: - type: string - x-stackql-sdk-source: RfaAPI - /api/3.0/rfa/destinations/{securable_type}/{full_name}: + type: string + x-stackql-sdk-source: SchemasAPI get: - operationId: rfa_get_access_request_destinations + operationId: schemas_get summary: >- - Gets an array of access request destinations for the specified - securable. Any caller can see URL + Gets the specified schema within the metastore. The caller must be a + metastore admin, the owner of the tags: - catalog - - rfa + - schemas description: >- - Gets an array of access request destinations for the specified - securable. Any caller can see URL - - destinations or the destinations on the metastore. Otherwise, only those - with **BROWSE** permissions - - on the securable can see destinations. - - - The supported securable types are: "metastore", "catalog", "schema", - "table", "external_location", + Gets the specified schema within the metastore. The caller must be a + metastore admin, the owner of the - "connection", "credential", "function", "registered_model", and - "volume". + schema, or a user that has the **USE_SCHEMA** privilege on the schema. - :param securable_type: str - The type of the securable. :param full_name: str - The full name of the securable. + Full name of the schema. + :param include_browse: bool (optional) + Whether to include schemas in the response for which the principal can only access selective + metadata for - :returns: :class:`AccessRequestDestinations` + :returns: :class:`SchemaInfo` parameters: - - name: securable_type - in: path - required: true - schema: - type: string - description: The type of the securable. - name: full_name in: path required: true schema: type: string - description: The full name of the securable. + description: Full name of the schema. + - name: include_browse + in: query + required: false + schema: + type: boolean + description: >- + Whether to include schemas in the response for which the principal + can only access selective metadata for responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/AccessRequestDestinations' + $ref: '#/components/schemas/SchemaInfo' default: description: Error response content: @@ -8859,241 +14449,89 @@ paths: type: string message: type: string - x-stackql-sdk-source: RfaAPI - /api/3.0/rfa/destinations: + x-stackql-sdk-source: SchemasAPI patch: - operationId: rfa_update_access_request_destinations + operationId: schemas_update summary: >- - Updates the access request destinations for the given securable. The - caller must be a metastore admin, + Updates a schema for a catalog. The caller must be the owner of the + schema or a metastore admin. If tags: - catalog - - rfa + - schemas description: >- - Updates the access request destinations for the given securable. The - caller must be a metastore admin, - - the owner of the securable, or a user that has the **MANAGE** privilege - on the securable in order to - - assign destinations. Destinations cannot be updated for securables - underneath schemas (tables, - - volumes, functions, and models). For these securable types, destinations - are inherited from the parent - - securable. A maximum of 5 emails and 5 external notification - destinations (Slack, Microsoft Teams, and - - Generic Webhook destinations) can be assigned to a securable. If a URL - destination is assigned, no - - other destinations can be set. - - - The supported securable types are: "metastore", "catalog", "schema", - "table", "external_location", - - "connection", "credential", "function", "registered_model", and - "volume". + Updates a schema for a catalog. The caller must be the owner of the + schema or a metastore admin. If + the caller is a metastore admin, only the **owner** field can be changed + in the update. If the - :param access_request_destinations: :class:`AccessRequestDestinations` - The access request destinations to assign to the securable. For each destination, a - **destination_id** and **destination_type** must be defined. - :param update_mask: str - The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only - the entire collection field can be specified. Field names must exactly match the resource field - names. + **name** field must be updated, the caller must be a metastore admin or + have the **CREATE_SCHEMA** - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + privilege on the parent catalog. - :returns: :class:`AccessRequestDestinations` - parameters: - - name: update_mask - in: query - required: true - schema: - type: string - description: >- - The field mask must be a single string, with multiple fields - separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map - fields is not allowed, as only the entire collection field can be - specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. - requestBody: - content: - application/json: - schema: - type: object - properties: - access_request_destinations: - type: object - properties: - securable: - type: object - properties: - full_name: - type: string - description: >- - Required. The full name of the catalog/schema/table. - Optional if resource_name is present. - provider_share: - type: string - description: >- - Optional. The name of the Share object that contains - the securable when the securable is getting shared - in D2D Delta Sharing. - type: - type: string - x-enum: - - CATALOG - - CLEAN_ROOM - - CONNECTION - - CREDENTIAL - - EXTERNAL_LOCATION - - EXTERNAL_METADATA - - FUNCTION - - METASTORE - - PIPELINE - - PROVIDER - - RECIPIENT - - SCHEMA - - SHARE - - STAGING_TABLE - - STORAGE_CREDENTIAL - - TABLE - - VOLUME - description: >- - Required. The type of securable - (catalog/schema/table). Optional if resource_name is - present. - description: >- - Generic definition of a securable, which is uniquely - defined in a metastore by its type and full - name. - are_any_destinations_hidden: - type: boolean - description: >- - Indicates whether any destinations are hidden from the - caller due to a lack of permissions. This value is true - if the caller does not have permission to see all - destinations. - destination_source_securable: - type: object - properties: - full_name: - type: string - description: >- - Required. The full name of the catalog/schema/table. - Optional if resource_name is present. - provider_share: - type: string - description: >- - Optional. The name of the Share object that contains - the securable when the securable is getting shared - in D2D Delta Sharing. - type: - type: string - x-enum: - - CATALOG - - CLEAN_ROOM - - CONNECTION - - CREDENTIAL - - EXTERNAL_LOCATION - - EXTERNAL_METADATA - - FUNCTION - - METASTORE - - PIPELINE - - PROVIDER - - RECIPIENT - - SCHEMA - - SHARE - - STAGING_TABLE - - STORAGE_CREDENTIAL - - TABLE - - VOLUME - description: >- - Required. The type of securable - (catalog/schema/table). Optional if resource_name is - present. - description: >- - The source securable from which the destinations are - inherited. Either the same value as securable (if - destination is set directly on the securable) or the - nearest parent securable with destinations set. - destinations: - type: array - items: - type: object - properties: - destination_id: - type: string - destination_type: - type: string - x-enum: - - EMAIL - - GENERIC_WEBHOOK - - MICROSOFT_TEAMS - - SLACK - - URL - description: The type of the destination. - special_destination: - type: string - x-enum: - - SPECIAL_DESTINATION_CATALOG_OWNER - - SPECIAL_DESTINATION_CONNECTION_OWNER - - SPECIAL_DESTINATION_CREDENTIAL_OWNER - - SPECIAL_DESTINATION_EXTERNAL_LOCATION_OWNER - - SPECIAL_DESTINATION_METASTORE_OWNER - description: >- - This field is used to denote whether the - destination is the email of the owner of the - securable object. The special destination cannot - be assigned to a securable and only represents the - default destination of the securable. The - securable types that support default special - destinations are: "catalog", "external_location", - "connection", "credential", and "metastore". The - **destination_type** of a **special_destination** - is always EMAIL. - description: The access request destinations for the securable. - full_name: - type: string - description: >- - The full name of the securable. Redundant with the name - in the securable object, but necessary for Terraform - integration - securable_type: - type: string - description: >- - The type of the securable. Redundant with the type in - the securable object, but necessary for Terraform - integration - required: - - securable + + :param full_name: str + Full name of the schema. + :param comment: str (optional) + User-provided free-form text description. + :param custom_max_retention_hours: int (optional) + Custom maximum retention period in hours for the schema. + :param enable_predictive_optimization: + :class:`EnablePredictiveOptimization` (optional) + Whether predictive optimization should be enabled for this object and objects under it. + :param new_name: str (optional) + New name for the schema. + :param owner: str (optional) + Username of current owner of schema. + :param properties: Dict[str,str] (optional) + A map of key-value properties attached to the securable. + + :returns: :class:`SchemaInfo` + parameters: + - name: full_name + in: path + required: true + schema: + type: string + description: Full name of the schema. + requestBody: + content: + application/json: + schema: + type: object + properties: + comment: + type: string + description: User-provided free-form text description. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the schema. + enable_predictive_optimization: + type: string + x-enum: + - DISABLE + - ENABLE + - INHERIT description: >- - The access request destinations to assign to the securable. - For each destination, a **destination_id** and - **destination_type** must be defined. - required: - - access_request_destinations + Whether predictive optimization should be enabled for this + object and objects under it. + new_name: + type: string + description: New name for the schema. + owner: + type: string + description: Username of current owner of schema. + properties: + type: object + description: A map of key-value properties attached to the securable. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/AccessRequestDestinations' + $ref: '#/components/schemas/SchemaInfo' default: description: Error response content: @@ -9105,66 +14543,134 @@ paths: type: string message: type: string - x-stackql-sdk-source: RfaAPI - /api/2.1/unity-catalog/schemas: + x-stackql-sdk-source: SchemasAPI + /api/2.1/unity-catalog/secrets: post: - operationId: schemas_create - summary: >- - Creates a new schema for catalog in the Metastore. The caller must be a - metastore admin, or have the + operationId: secrets_uc_create_secret + summary: Creates a new secret in Unity Catalog. tags: - catalog - - schemas + - secrets_uc description: >- - Creates a new schema for catalog in the Metastore. The caller must be a - metastore admin, or have the + Creates a new secret in Unity Catalog. - **CREATE_SCHEMA** privilege in the parent catalog. + You must be the owner of the parent schema or have the **CREATE_SECRET** + and **USE SCHEMA** privileges - :param name: str - Name of schema, relative to parent catalog. - :param catalog_name: str - Name of parent catalog. - :param comment: str (optional) - User-provided free-form text description. - :param properties: Dict[str,str] (optional) - A map of key-value properties attached to the securable. - :param storage_root: str (optional) - Storage root URL for managed tables within schema. + on the parent schema and **USE CATALOG** on the parent catalog. - :returns: :class:`SchemaInfo` + + The secret is stored in the specified catalog and schema, and the + **value** field contains the + + sensitive data to be securely stored. + + + :param secret: :class:`Secret` + The secret object to create. The **name**, **catalog_name**, **schema_name**, and **value** fields + are required. + + :returns: :class:`Secret` requestBody: content: application/json: schema: type: object properties: - name: - type: string - description: Name of schema, relative to parent catalog. - catalog_name: - type: string - description: Name of parent catalog. - comment: - type: string - description: User-provided free-form text description. - properties: + secret: type: object - description: A map of key-value properties attached to the securable. - storage_root: - type: string - description: Storage root URL for managed tables within schema. + properties: + name: + type: string + description: The name of the secret, relative to its parent schema. + catalog_name: + type: string + description: >- + The name of the catalog where the schema and the secret + reside. + schema_name: + type: string + description: The name of the schema where the secret resides. + value: + type: string + description: >- + The secret value to store. This field is input-only and + is not returned in responses — use the + **effective_value** field (via GetSecret with + **include_value** set to true) to read the secret value. + The maximum size is 60 KiB (pre-encryption). Accepted + content includes passwords, tokens, keys, and other + sensitive credential data. + comment: + type: string + description: User-provided free-form text description of the secret. + create_time: + type: string + format: date-time + description: The time at which this secret was created. + created_by: + type: string + description: The principal that created the secret. + effective_owner: + type: string + description: >- + The effective owner of the secret, which may differ from + the directly-set **owner** due to inheritance. + effective_value: + type: string + description: >- + The secret value. Only populated in responses when you + have the **READ_SECRET** privilege and **include_value** + is set to true in the request. The maximum size is 60 + KiB. + expire_time: + type: string + format: date-time + description: >- + User-provided expiration time of the secret. This field + indicates when the secret should no longer be used and + may be displayed as a warning in the UI. It is purely + informational and does not trigger any automatic actions + or affect the secret's lifecycle. + full_name: + type: string + description: >- + The three-level (fully qualified) name of the secret, in + the form of **catalog_name.schema_name.secret_name**. + metastore_id: + type: string + description: Unique identifier of the metastore hosting the secret. + owner: + type: string + description: >- + The owner of the secret. Defaults to the creating + principal on creation. Can be updated to transfer + ownership of the secret to another principal. + update_time: + type: string + format: date-time + description: The time at which this secret was last updated. + updated_by: + type: string + description: The principal that last updated the secret. + required: + - name + - catalog_name + - schema_name + - value + description: >- + The secret object to create. The **name**, **catalog_name**, + **schema_name**, and **value** fields are required. required: - - name - - catalog_name + - secret responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/SchemaInfo' + $ref: '#/components/schemas/Secret' default: description: Error response content: @@ -9176,99 +14682,94 @@ paths: type: string message: type: string - x-stackql-sdk-source: SchemasAPI + x-stackql-sdk-source: SecretsUcAPI get: - operationId: schemas_list - summary: >- - Gets an array of schemas for a catalog in the metastore. If the caller - is the metastore admin or the + operationId: secrets_uc_list_secrets + summary: Lists secrets in Unity Catalog. tags: - catalog - - schemas + - secrets_uc description: >- - Gets an array of schemas for a catalog in the metastore. If the caller - is the metastore admin or the - - owner of the parent catalog, all schemas for the catalog will be - retrieved. Otherwise, only schemas - - owned by the caller (or for which the caller has the **USE_SCHEMA** - privilege) will be retrieved. + Lists secrets in Unity Catalog. - There is no guarantee of a specific ordering of the elements in the - array. + You must be a metastore admin, the owner of the secret, or have the + **MANAGE** privilege on the - NOTE: we recommend using max_results=0 to use the paginated version of - this API. Unpaginated calls + secret. - will be deprecated soon. + Both **catalog_name** and **schema_name** must be specified together to + filter secrets within a - PAGINATION BEHAVIOR: When using pagination (max_results >= 0), a page - may contain zero results while + specific schema. Results are paginated; use the **page_token** field + from the response to retrieve - still providing a next_page_token. Clients must continue reading pages - until next_page_token is + subsequent pages. - absent, which is the only indication that the end of results has been - reached. + :param catalog_name: str (optional) + The name of the catalog under which to list secrets. Both **catalog_name** and **schema_name** must + be specified together. + :param page_size: int (optional) + Maximum number of secrets to return. - :param catalog_name: str - Parent catalog for schemas of interest. - :param include_browse: bool (optional) - Whether to include schemas in the response for which the principal can only access selective - metadata for - :param max_results: int (optional) - Maximum number of schemas to return. If not set, all the schemas are returned (not recommended). - - when set to a value greater than 0, the page length is the minimum of this value and a server - configured value; - when set to 0, the page length is set to a server configured value - (recommended); - when set to a value less than 0, an invalid parameter error is returned; + - If not specified, at most 1000 secrets are returned. + - If set to a value greater than 0, the page length is the minimum of this value and 1000. + - If set to 0, the page length is set to 1000. + - If set to a value less than 0, an invalid parameter error is returned. :param page_token: str (optional) - Opaque pagination token to go to next page based on previous query. + Opaque pagination token to go to the next page based on previous query. The maximum page length is + determined by a server configured value. + :param schema_name: str (optional) + The name of the schema under which to list secrets. Both **catalog_name** and **schema_name** must + be specified together. - :returns: Iterator over :class:`SchemaInfo` + :returns: Iterator over :class:`Secret` parameters: - name: catalog_name in: query - required: true + required: false schema: type: string - description: Parent catalog for schemas of interest. - - name: include_browse + description: >- + The name of the catalog under which to list secrets. Both + **catalog_name** and **schema_name** must be specified together. + - name: page_size in: query required: false schema: - type: boolean + type: integer description: >- - Whether to include schemas in the response for which the principal - can only access selective metadata for - - name: max_results + Maximum number of secrets to return. - If not specified, at most + 1000 secrets are returned. - If set to a value greater than 0, the + page length is the minimum of this value and 1000. - If set to 0, + the page length is set to 1000. - If set to a value less than 0, an + invalid parameter error is returned. + - name: page_token in: query required: false schema: - type: integer + type: string description: >- - Maximum number of schemas to return. If not set, all the schemas are - returned (not recommended). - when set to a value greater than 0, - the page length is the minimum of this value and a server configured - value; - when set to 0, the page length is set to a server - configured value (recommended); - when set to a value less than 0, - an invalid parameter error is returned; - - name: page_token + Opaque pagination token to go to the next page based on previous + query. The maximum page length is determined by a server configured + value. + - name: schema_name in: query required: false schema: type: string - description: Opaque pagination token to go to next page based on previous query. + description: >- + The name of the schema under which to list secrets. Both + **catalog_name** and **schema_name** must be specified together. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListSchemasResponse' + $ref: '#/components/schemas/ListSecretsResponse' default: description: Error response content: @@ -9280,40 +14781,31 @@ paths: type: string message: type: string - x-stackql-sdk-source: SchemasAPI - /api/2.1/unity-catalog/schemas/{full_name}: + x-stackql-sdk-source: SecretsUcAPI + /api/2.1/unity-catalog/secrets/{full_name}: delete: - operationId: schemas_delete - summary: >- - Deletes the specified schema from the parent catalog. The caller must be - the owner of the schema or an + operationId: secrets_uc_delete_secret + summary: Deletes a secret by its three-level (fully qualified) name. tags: - catalog - - schemas - description: >- - Deletes the specified schema from the parent catalog. The caller must be - the owner of the schema or an - - owner of the parent catalog. + - secrets_uc + description: |- + Deletes a secret by its three-level (fully qualified) name. + You must be the owner of the secret or a metastore admin. :param full_name: str - Full name of the schema. - :param force: bool (optional) - Force deletion even if the schema is not empty. + The three-level (fully qualified) name of the secret (for example, + **catalog_name.schema_name.secret_name**). parameters: - name: full_name in: path required: true schema: type: string - description: Full name of the schema. - - name: force - in: query - required: false - schema: - type: boolean - description: Force deletion even if the schema is not empty. + description: >- + The three-level (fully qualified) name of the secret (for example, + **catalog_name.schema_name.secret_name**). responses: '200': description: Success @@ -9328,51 +14820,50 @@ paths: type: string message: type: string - x-stackql-sdk-source: SchemasAPI + x-stackql-sdk-source: SecretsUcAPI get: - operationId: schemas_get - summary: >- - Gets the specified schema within the metastore. The caller must be a - metastore admin, the owner of the + operationId: secrets_uc_get_secret + summary: Gets a secret by its three-level (fully qualified) name. tags: - catalog - - schemas + - secrets_uc description: >- - Gets the specified schema within the metastore. The caller must be a - metastore admin, the owner of the + Gets a secret by its three-level (fully qualified) name. + + + You must be a metastore admin, the owner of the secret, or have the + **MANAGE** privilege on the + + secret. - schema, or a user that has the **USE_SCHEMA** privilege on the schema. + + The secret value isn't returned by default. To retrieve it, you must + also have the **READ_SECRET** + + privilege and set **include_value** to true in the request. :param full_name: str - Full name of the schema. - :param include_browse: bool (optional) - Whether to include schemas in the response for which the principal can only access selective - metadata for + The three-level (fully qualified) name of the secret (for example, + **catalog_name.schema_name.secret_name**). - :returns: :class:`SchemaInfo` + :returns: :class:`Secret` parameters: - - name: full_name - in: path - required: true - schema: - type: string - description: Full name of the schema. - - name: include_browse - in: query - required: false + - name: full_name + in: path + required: true schema: - type: boolean + type: string description: >- - Whether to include schemas in the response for which the principal - can only access selective metadata for + The three-level (fully qualified) name of the secret (for example, + **catalog_name.schema_name.secret_name**). responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/SchemaInfo' + $ref: '#/components/schemas/Secret' default: description: Error response content: @@ -9384,84 +14875,165 @@ paths: type: string message: type: string - x-stackql-sdk-source: SchemasAPI + x-stackql-sdk-source: SecretsUcAPI patch: - operationId: schemas_update - summary: >- - Updates a schema for a catalog. The caller must be the owner of the - schema or a metastore admin. If + operationId: secrets_uc_update_secret + summary: Updates an existing secret in Unity Catalog. tags: - catalog - - schemas + - secrets_uc description: >- - Updates a schema for a catalog. The caller must be the owner of the - schema or a metastore admin. If + Updates an existing secret in Unity Catalog. - the caller is a metastore admin, only the __owner__ field can be changed - in the update. If the - __name__ field must be updated, the caller must be a metastore admin or - have the **CREATE_SCHEMA** + You must be the owner of the secret or a metastore admin. If you are a + metastore admin, only the - privilege on the parent catalog. + **owner** field can be changed. - :param full_name: str - Full name of the schema. - :param comment: str (optional) - User-provided free-form text description. - :param enable_predictive_optimization: - :class:`EnablePredictiveOptimization` (optional) - Whether predictive optimization should be enabled for this object and objects under it. - :param new_name: str (optional) - New name for the schema. - :param owner: str (optional) - Username of current owner of schema. - :param properties: Dict[str,str] (optional) - A map of key-value properties attached to the securable. + Use the **update_mask** field to specify which fields to update. + Supported updatable fields include - :returns: :class:`SchemaInfo` + **value**, **comment**, **owner**, and **expire_time**. + + + :param full_name: str + The three-level (fully qualified) name of the secret (for example, + **catalog_name.schema_name.secret_name**). + :param secret: :class:`Secret` + The secret object containing the fields to update. Only fields specified in **update_mask** will be + updated. + :param update_mask: FieldMask + The field mask specifying which fields of the secret to update. + + - If **update_mask** is **"*"**, all fields specified in **secret** are updated. + - If **update_mask** specifies one or more fields, only those fields are updated. Each specified + field must be set in **secret**. Supported fields: **value**, **comment**, **owner**, + **expire_time**. To change the secret name, delete and recreate the secret. + + :returns: :class:`Secret` parameters: - name: full_name in: path required: true schema: type: string - description: Full name of the schema. + description: >- + The three-level (fully qualified) name of the secret (for example, + **catalog_name.schema_name.secret_name**). + - name: update_mask + in: query + required: true + schema: + type: object + description: >- + The field mask specifying which fields of the secret to update. - If + **update_mask** is **"*"**, all fields specified in **secret** are + updated. - If **update_mask** specifies one or more fields, only + those fields are updated. Each specified field must be set in + **secret**. Supported fields: **value**, **comment**, **owner**, + **expire_time**. To change the secret name, delete and recreate the + secret. requestBody: content: application/json: schema: type: object properties: - comment: - type: string - description: User-provided free-form text description. - enable_predictive_optimization: - type: string - x-enum: - - DISABLE - - ENABLE - - INHERIT - description: >- - Whether predictive optimization should be enabled for this - object and objects under it. - new_name: - type: string - description: New name for the schema. - owner: - type: string - description: Username of current owner of schema. - properties: + secret: type: object - description: A map of key-value properties attached to the securable. + properties: + name: + type: string + description: The name of the secret, relative to its parent schema. + catalog_name: + type: string + description: >- + The name of the catalog where the schema and the secret + reside. + schema_name: + type: string + description: The name of the schema where the secret resides. + value: + type: string + description: >- + The secret value to store. This field is input-only and + is not returned in responses — use the + **effective_value** field (via GetSecret with + **include_value** set to true) to read the secret value. + The maximum size is 60 KiB (pre-encryption). Accepted + content includes passwords, tokens, keys, and other + sensitive credential data. + comment: + type: string + description: User-provided free-form text description of the secret. + create_time: + type: string + format: date-time + description: The time at which this secret was created. + created_by: + type: string + description: The principal that created the secret. + effective_owner: + type: string + description: >- + The effective owner of the secret, which may differ from + the directly-set **owner** due to inheritance. + effective_value: + type: string + description: >- + The secret value. Only populated in responses when you + have the **READ_SECRET** privilege and **include_value** + is set to true in the request. The maximum size is 60 + KiB. + expire_time: + type: string + format: date-time + description: >- + User-provided expiration time of the secret. This field + indicates when the secret should no longer be used and + may be displayed as a warning in the UI. It is purely + informational and does not trigger any automatic actions + or affect the secret's lifecycle. + full_name: + type: string + description: >- + The three-level (fully qualified) name of the secret, in + the form of **catalog_name.schema_name.secret_name**. + metastore_id: + type: string + description: Unique identifier of the metastore hosting the secret. + owner: + type: string + description: >- + The owner of the secret. Defaults to the creating + principal on creation. Can be updated to transfer + ownership of the secret to another principal. + update_time: + type: string + format: date-time + description: The time at which this secret was last updated. + updated_by: + type: string + description: The principal that last updated the secret. + required: + - name + - catalog_name + - schema_name + - value + description: >- + The secret object containing the fields to update. Only + fields specified in **update_mask** will be updated. + required: + - secret responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/SchemaInfo' + $ref: '#/components/schemas/Secret' default: description: Error response content: @@ -9473,7 +15045,7 @@ paths: type: string message: type: string - x-stackql-sdk-source: SchemasAPI + x-stackql-sdk-source: SecretsUcAPI /api/2.1/unity-catalog/storage-credentials: post: operationId: storage_credentials_create @@ -9546,13 +15118,13 @@ paths: description: >- The Azure resource ID of the Azure Databricks Access Connector. Use the format - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. managed_identity_id: type: string description: >- The Azure resource ID of the managed identity. Use the format, - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not @@ -9641,13 +15213,13 @@ paths: get: operationId: storage_credentials_list summary: >- - Gets an array of storage credentials (as __StorageCredentialInfo__ + Gets an array of storage credentials (as **StorageCredentialInfo** objects). The array is limited to tags: - catalog - storage_credentials description: >- - Gets an array of storage credentials (as __StorageCredentialInfo__ + Gets an array of storage credentials (as **StorageCredentialInfo** objects). The array is limited to only those storage credentials the caller has permission to access. If @@ -9680,10 +15252,12 @@ paths: to update the credential–workspace binding. :param max_results: int (optional) Maximum number of storage credentials to return. If not set, all the storage credentials are - returned (not recommended). - when set to a value greater than 0, the page length is the minimum of - this value and a server configured value; - when set to 0, the page length is set to a server - configured value (recommended); - when set to a value less than 0, an invalid parameter error is - returned; + returned (not recommended). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -9912,7 +15486,7 @@ paths: description: >- The Azure resource ID of the Azure Databricks Access Connector. Use the format - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: >- @@ -9923,7 +15497,7 @@ paths: description: >- The Azure resource ID of the managed identity. Use the format, - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not @@ -9988,6 +15562,7 @@ paths: x-enum: - ISOLATION_MODE_ISOLATED - ISOLATION_MODE_OPEN + - ISOLATION_MODE_OPEN_IN_ACCOUNT description: >- Whether the current securable is accessible from all workspaces or a specific set of workspaces. @@ -10031,24 +15606,24 @@ paths: operationId: storage_credentials_validate summary: >- Validates a storage credential. At least one of - __external_location_name__ and __url__ need to be + **external_location_name** and **url** need to be tags: - catalog - storage_credentials description: >- Validates a storage credential. At least one of - __external_location_name__ and __url__ need to be + **external_location_name** and **url** need to be provided. If only one of them is provided, it will be used for validation. And if both are provided, - the __url__ will be used for validation, and __external_location_name__ + the **url** will be used for validation, and **external_location_name** will be ignored when checking overlapping urls. - Either the __storage_credential_name__ or the cloud-specific credential + Either the **storage_credential_name** or the cloud-specific credential must be provided. @@ -10107,13 +15682,13 @@ paths: description: >- The Azure resource ID of the Azure Databricks Access Connector. Use the format - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. managed_identity_id: type: string description: >- The Azure resource ID of the managed identity. Use the format, - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not @@ -10341,10 +15916,13 @@ paths: :param metastore_id: str The ID for the metastore in which the system schema resides. :param max_results: int (optional) - Maximum number of schemas to return. - When set to 0, the page length is set to a server configured - value (recommended); - When set to a value greater than 0, the page length is the minimum of this - value and a server configured value; - When set to a value less than 0, an invalid parameter error - is returned; - If not set, all the schemas are returned (not recommended). + Maximum number of schemas to return. + + - When set to 0, the page length is set to a server configured value (recommended); + - When set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - When set to a value less than 0, an invalid parameter error is returned; + - If not set, all the schemas are returned (not recommended). :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -10405,22 +15983,16 @@ paths: For the table constraint creation to succeed, the user must satisfy both - of these conditions: - the - - user must have the **USE_CATALOG** privilege on the table's parent - catalog, the **USE_SCHEMA** + of these conditions: - privilege on the table's parent schema, and be the owner of the table. - - if the new constraint is a - - __ForeignKeyConstraint__, the user must have the **USE_CATALOG** - privilege on the referenced parent - - table's catalog, the **USE_SCHEMA** privilege on the referenced parent - table's schema, and be the - - owner of the referenced parent table. + - the user must have the **USE_CATALOG** privilege on the table's parent + catalog, the **USE_SCHEMA** + privilege on the table's parent schema, and be the owner of the table. + - if the new constraint is a **ForeignKeyConstraint**, the user must + have the **USE_CATALOG** + privilege on the referenced parent table's catalog, the **USE_SCHEMA** privilege on the referenced + parent table's schema, and be the owner of the referenced parent table. :param full_name_arg: str The full name of the table referenced by the constraint. @@ -10501,7 +16073,7 @@ paths: description: >- A table constraint, as defined by *one* of the following fields being set: - __primary_key_constraint__, __foreign_key_constraint__, __named_table_constraint__. + **primary_key_constraint**, **foreign_key_constraint**, **named_table_constraint**. required: - full_name_arg - constraint @@ -10536,22 +16108,16 @@ paths: For the table constraint deletion to succeed, the user must satisfy both - of these conditions: - the - - user must have the **USE_CATALOG** privilege on the table's parent - catalog, the **USE_SCHEMA** - - privilege on the table's parent schema, and be the owner of the table. - - if __cascade__ argument is + of these conditions: - **true**, the user must have the following permissions on all of the - child tables: the **USE_CATALOG** - - privilege on the table's catalog, the **USE_SCHEMA** privilege on the - table's schema, and be the owner - - of the table. + - the user must have the **USE_CATALOG** privilege on the table's parent + catalog, the **USE_SCHEMA** + privilege on the table's parent schema, and be the owner of the table. + - if **cascade** argument is **true**, the user must have the following + permissions on all of the + child tables: the **USE_CATALOG** privilege on the table's catalog, the **USE_SCHEMA** privilege on + the table's schema, and be the owner of the table. :param full_name: str Full name of the table referenced by the constraint. @@ -10664,7 +16230,7 @@ paths: :param storage_location: str Storage root URL for table (for **MANAGED**, **EXTERNAL** tables). :param columns: List[:class:`ColumnInfo`] (optional) - The array of __ColumnInfo__ definitions of the table's columns. + The array of **ColumnInfo** definitions of the table's columns. :param properties: Dict[str,str] (optional) A map of key-value properties attached to the securable. @@ -10861,6 +16427,7 @@ paths: - DATE - DECIMAL - DOUBLE + - FILE - FLOAT - GEOGRAPHY - GEOMETRY @@ -10872,7 +16439,9 @@ paths: - SHORT - STRING - STRUCT + - TABLEREF_TYPE - TABLE_TYPE + - TIME - TIMESTAMP - TIMESTAMP_NTZ - USER_DEFINED_TYPE @@ -10941,7 +16510,7 @@ paths: Full data type specification as SQL/catalogString text. description: >- - The array of __ColumnInfo__ definitions of the table's + The array of **ColumnInfo** definitions of the table's columns. properties: type: object @@ -11026,10 +16595,12 @@ paths: :param include_manifest_capabilities: bool (optional) Whether to include a manifest containing table capabilities in the response. :param max_results: int (optional) - Maximum number of tables to return. If not set, all the tables are returned (not recommended). - - when set to a value greater than 0, the page length is the minimum of this value and a server - configured value; - when set to 0, the page length is set to a server configured value - (recommended); - when set to a value less than 0, an invalid parameter error is returned; + Maximum number of tables to return. If not set, all the tables are returned (not recommended). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value less than 0, an invalid parameter error is returned; :param omit_columns: bool (optional) Whether to omit the columns of the table from the response or not. :param omit_properties: bool (optional) @@ -11187,18 +16758,19 @@ paths: Gets a table from the metastore for a specific catalog and schema. The caller must satisfy one of the - following requirements: * Be a metastore admin * Be the owner of the - parent catalog * Be the owner of + following requirements: - the parent schema and have the **USE_CATALOG** privilege on the parent - catalog * Have the - **USE_CATALOG** privilege on the parent catalog and the **USE_SCHEMA** - privilege on the parent schema, + - Be a metastore admin - and either be the table owner or have the **SELECT** privilege on the - table. + - Be the owner of the parent catalog + - Be the owner of the parent schema and have the **USE_CATALOG** + privilege on the parent catalog + + - Have the **USE_CATALOG** privilege on the parent catalog and the + **USE_SCHEMA** privilege on the + parent schema, and either be the table owner or have the **SELECT** privilege on the table. :param full_name: str Full name of the table. @@ -11328,20 +16900,22 @@ paths: Gets if a table exists in the metastore for a specific catalog and schema. The caller must satisfy one - of the following requirements: * Be a metastore admin * Be the owner of - the parent catalog * Be the + of the following requirements: + - owner of the parent schema and have the **USE_CATALOG** privilege on the - parent catalog * Have the + - Be a metastore admin - **USE_CATALOG** privilege on the parent catalog and the **USE_SCHEMA** - privilege on the parent schema, + - Be the owner of the parent catalog - and either be the table owner or have the **SELECT** privilege on the - table. * Have **BROWSE** + - Be the owner of the parent schema and have the **USE_CATALOG** + privilege on the parent catalog - privilege on the parent catalog * Have **BROWSE** privilege on the - parent schema + - Have the **USE_CATALOG** privilege on the parent catalog and the + **USE_SCHEMA** privilege on the + parent schema, and either be the table owner or have the **SELECT** privilege on the table. + - Have **BROWSE** privilege on the parent catalog + + - Have **BROWSE** privilege on the parent schema :param full_name: str @@ -11390,20 +16964,14 @@ paths: summaries returned are either: - * summaries for tables (within the current metastore and parent catalog + - summaries for tables (within the current metastore and parent catalog and schema), when the user is - - a metastore admin, or: * summaries for tables and schemas (within the - current metastore and parent - - catalog) for which the user has ownership or the **SELECT** privilege on - the table and ownership or - - **USE_SCHEMA** privilege on the schema, provided that the user also has - ownership or the - - **USE_CATALOG** privilege on the parent catalog. - + a metastore admin, or: + - summaries for tables and schemas (within the current metastore and + parent catalog) for which the + user has ownership or the **SELECT** privilege on the table and ownership or **USE_SCHEMA** + privilege on the schema, provided that the user also has ownership or the **USE_CATALOG** privilege + on the parent catalog. There is no guarantee of a specific ordering of the elements in the array. @@ -11424,10 +16992,13 @@ paths: Whether to include a manifest containing table capabilities in the response. :param max_results: int (optional) Maximum number of summaries for tables to return. If not set, the page length is set to a server - configured value (10000, as of 1/5/2024). - when set to a value greater than 0, the page length is - the minimum of this value and a server configured value (10000, as of 1/5/2024); - when set to 0, - the page length is set to a server configured value (10000, as of 1/5/2024) (recommended); - when - set to a value less than 0, an invalid parameter error is returned; + configured value (10000, as of 1/5/2024). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value (10000, as of 1/5/2024); + - when set to 0, the page length is set to a server configured value (10000, as of 1/5/2024) + (recommended); + - when set to a value less than 0, an invalid parameter error is returned; :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. :param schema_name_pattern: str (optional) @@ -11661,6 +17232,73 @@ paths: message: type: string x-stackql-sdk-source: TemporaryTableCredentialsAPI + /api/2.0/unity-catalog/temporary-volume-credentials: + post: + operationId: temporary_volume_credentials_generate_temporary_volume_credentials + summary: >- + Get a short-lived credential for directly accessing the volume data on + cloud storage. The metastore + tags: + - catalog + - temporary_volume_credentials + description: >- + Get a short-lived credential for directly accessing the volume data on + cloud storage. The metastore + + must have **external_access_enabled** flag set to true (default false). + The caller must have the + + **EXTERNAL_USE_SCHEMA** privilege on the parent schema and this + privilege can only be granted by + + catalog owners. + + + :param operation: :class:`VolumeOperation` (optional) + The operation performed against the volume data, either READ_VOLUME or WRITE_VOLUME. If WRITE_VOLUME + is specified, the credentials returned will have write permissions, otherwise, it will be read only. + :param volume_id: str (optional) + Id of the volume to read or write. + + :returns: :class:`GenerateTemporaryVolumeCredentialResponse` + requestBody: + content: + application/json: + schema: + type: object + properties: + operation: + type: string + x-enum: + - READ_VOLUME + - WRITE_VOLUME + description: >- + The operation performed against the volume data, either + READ_VOLUME or WRITE_VOLUME. If WRITE_VOLUME is specified, + the credentials returned will have write permissions, + otherwise, it will be read only. + volume_id: + type: string + description: Id of the volume to read or write. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GenerateTemporaryVolumeCredentialResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: TemporaryVolumeCredentialsAPI /api/2.1/unity-catalog/volumes: post: operationId: volumes_create @@ -11683,28 +17321,28 @@ paths: For the volume creation to succeed, the user must satisfy following - conditions: - The caller must be a - - metastore admin, or be the owner of the parent catalog and schema, or - have the **USE_CATALOG** + conditions: - privilege on the parent catalog and the **USE_SCHEMA** privilege on the - parent schema. - The caller - must have **CREATE VOLUME** privilege on the parent schema. + - The caller must be a metastore admin, or be the owner of the parent + catalog and schema, or have the + **USE_CATALOG** privilege on the parent catalog and the **USE_SCHEMA** privilege on the parent + schema. + - The caller must have **CREATE VOLUME** privilege on the parent schema. - For an external volume, following conditions also need to satisfy - The - caller must have **CREATE + For an external volume, following conditions also need to satisfy - EXTERNAL VOLUME** privilege on the external location. - There are no - other tables, nor volumes - existing in the specified storage location. - The specified storage - location is not under the location + - The caller must have **CREATE EXTERNAL VOLUME** privilege on the + external location. - of other tables, nor volumes, or catalogs or schemas. + - There are no other tables, nor volumes existing in the specified + storage location. + - The specified storage location is not under the location of other + tables, nor volumes, or catalogs + or schemas. :param catalog_name: str The name of the catalog where the schema and the volume are @@ -11715,9 +17353,8 @@ paths: :param volume_type: :class:`VolumeType` The type of the volume. An external volume is located in the specified external location. A managed volume is located in the default location which is specified by the parent schema, or the parent - catalog, or the Metastore. [Learn more] - - [Learn more]: https://docs.databricks.com/aws/en/volumes/managed-vs-external + catalog, or the Metastore. `Learn more + `__ :param comment: str (optional) The comment attached to the volume :param storage_location: str (optional) @@ -11748,9 +17385,8 @@ paths: The type of the volume. An external volume is located in the specified external location. A managed volume is located in the default location which is specified by the parent - schema, or the parent catalog, or the Metastore. [Learn - more] [Learn more]: - https://docs.databricks.com/aws/en/volumes/managed-vs-external + schema, or the parent catalog, or the Metastore. `Learn more + `__ comment: type: string description: The comment attached to the volume @@ -11832,11 +17468,13 @@ paths: :param max_results: int (optional) Maximum number of volumes to return (page length). - If not set, the page length is set to a server configured value (10000, as of 1/29/2024). - when set - to a value greater than 0, the page length is the minimum of this value and a server configured - value (10000, as of 1/29/2024); - when set to 0, the page length is set to a server configured value - (10000, as of 1/29/2024) (recommended); - when set to a value less than 0, an invalid parameter - error is returned; + If not set, the page length is set to a server configured value (10000, as of 1/29/2024). + + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value (10000, as of 1/29/2024); + - when set to 0, the page length is set to a server configured value (10000, as of 1/29/2024) + (recommended); + - when set to a value less than 0, an invalid parameter error is returned; Note: this parameter controls only the maximum number of volumes to return. The actual number of volumes returned in a page may be smaller than this value, including 0, even if there are more @@ -12241,10 +17879,13 @@ paths: :param securable_name: str The name of the securable. :param max_results: int (optional) - Maximum number of workspace bindings to return. - When set to 0, the page length is set to a server - configured value (recommended); - When set to a value greater than 0, the page length is the minimum - of this value and a server configured value; - When set to a value less than 0, an invalid parameter - error is returned; - If not set, all the workspace bindings are returned (not recommended). + Maximum number of workspace bindings to return. + + - When set to 0, the page length is set to a server configured value (recommended); + - When set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - When set to a value less than 0, an invalid parameter error is returned; + - If not set, all the workspace bindings are returned (not recommended). :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -12477,40 +18118,172 @@ components: AccountsListMetastoresResponse: type: object properties: - metastores: + metastores: + type: array + items: + $ref: '#/components/schemas/MetastoreInfo' + description: An array of metastore information objects. + description: Metastores were returned successfully. + AccountsMetastoreAssignment: + type: object + properties: + metastore_assignment: + $ref: '#/components/schemas/MetastoreAssignment' + description: The workspace metastore assignment was successfully returned. + AccountsStorageCredentialInfo: + type: object + properties: + credential_info: + $ref: '#/components/schemas/StorageCredentialInfo' + description: The storage credential was successfully retrieved. + AccountsUpdateMetastoreAssignmentResponse: + type: object + properties: {} + description: The metastore assignment was successfully updated. + AccountsUpdateMetastoreResponse: + type: object + properties: + metastore_info: + $ref: '#/components/schemas/MetastoreInfo' + description: The metastore update request succeeded. + AccountsUpdateStorageCredentialResponse: + type: object + properties: + credential_info: + $ref: '#/components/schemas/StorageCredentialInfo' + description: The storage credential was successfully updated. + AgentService: + type: object + properties: + agent_service_type: + $ref: '#/components/schemas/AgentServiceAgentServiceType' + description: >- + Type of agent service. Required on create. Immutable after creation. + For future internal / genie / app kinds, determines how the backing + agent surface is provisioned. + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through the BROWSE + privilege. + comment: + type: string + description: User-provided description. + config: + $ref: '#/components/schemas/AgentServiceConfig' + description: >- + Operational configuration: connection, base_path, system_prompt, + model_service, optional inference table, optional rate limits. + Required on CreateAgentService; on UpdateAgentService it is required + only when ``config`` (or a ``config.*`` subpath) appears in + ``update_mask``. + create_time: + type: string + format: date-time + description: When the agent service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: >- + The resolved owner of the agent service. Falls back to the caller's + identity when ``owner`` is not explicitly set on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated from the + entity's state and returned on every read. To use it as an if-match + precondition on a mutation, echo the last-read value back via the + dedicated ``etag`` field on the Update / Delete request; the server + rejects the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the agent service. + name: + type: string + description: >- + Resource name of the agent service. Format: + ``agent-services/{catalog}.{schema}.{agent_service}``. Each + ``{...}`` component is capped at 255 characters individually. + Server-derived on Create from ``parent`` + ``agent_service_id``; + required and immutable on Update/Get/Delete. + owner: + type: string + description: >- + The owner of the agent service. Write-only; read owner via + effective_owner. + update_time: + type: string + format: date-time + description: When the agent service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: >- + An agent service represents a governed registration of an external AI + agent (Bedrock Agents, + Snowflake Cortex Agents, Foundry, etc) in Unity Catalog. It acts as a schema-scoped securable + that references a UC Connection holding the agent platform's auth + endpoint, and exposes the + agent for discovery, authorization, and audit logging. Initial kind: AGENT_SERVICE_STANDARD. + Future kinds (INTERNAL, GENIE, APP) can be added as separate enum values. + AgentServiceConfig: + type: object + properties: + base_path: + type: string + description: >- + Vendor base path appended to the connection url at invocation time. + Bedrock example: ``/agents/AGENT123/agentAliases/ALIAS1/sessions``. + Empty when the agent platform does not require a per-agent suffix. + inference_table: + $ref: '#/components/schemas/InferenceTableConfig' + description: >- + Inference table config for invocation payload logging. Reuses the + shared InferenceTableConfig (same as + ModelServiceConfig.inference_table). + model_service: + type: string + description: >- + Resource name of the model service backing this agent. Empty when + the agent platform manages its own model selection (e.g. Bedrock + Agents). Format: + ``model-services/{catalog}.{schema}.{model_service}``. + rate_limits: type: array items: - $ref: '#/components/schemas/MetastoreInfo' - description: An array of metastore information objects. - description: Metastores were returned successfully. - AccountsMetastoreAssignment: - type: object - properties: - metastore_assignment: - $ref: '#/components/schemas/MetastoreAssignment' - description: The workspace metastore assignment was successfully returned. - AccountsStorageCredentialInfo: - type: object - properties: - credential_info: - $ref: '#/components/schemas/StorageCredentialInfo' - description: The storage credential was successfully retrieved. - AccountsUpdateMetastoreAssignmentResponse: - type: object - properties: {} - description: The metastore assignment was successfully updated. - AccountsUpdateMetastoreResponse: - type: object - properties: - metastore_info: - $ref: '#/components/schemas/MetastoreInfo' - description: The metastore update request succeeded. - AccountsUpdateStorageCredentialResponse: + $ref: '#/components/schemas/RateLimit' + description: >- + Per-principal rate limits applied to invocations routed through this + agent service. Repeated to support per-USER / USER_GROUP / + SERVICE_PRINCIPAL / SERVICE / USER_DEFAULT scopes simultaneously, + mirroring McpServiceConfig.rate_limits. Empty when no rate limit is + configured. + source_connection: + $ref: '#/components/schemas/AgentServiceConfigSourceConnection' + description: External agent referenced via a UC Connection. + system_prompt: + type: string + description: System prompt prepended to invocations. Max 8 KB. + description: >- + Operational configuration for an agent service. Groups the source + connection, base path, system + prompt, model service reference, optional inference table, and optional rate limits. The + ``source_connection.name`` is supplied on create. + AgentServiceConfigSourceConnection: type: object properties: - credential_info: - $ref: '#/components/schemas/StorageCredentialInfo' - description: The storage credential was successfully updated. + name: + type: string + is_deleted: + type: boolean + required: + - name + description: >- + UC Connection for the agent platform, such as Bedrock Agents, Snowflake + Cortex Agents, or + Foundry. On create, provide ``name`` in the schema-scoped form + ``connections/{catalog}.{schema}.{connection}``. On read, the service returns ``name``. ArtifactAllowlistInfo: type: object properties: @@ -12642,6 +18415,17 @@ components: Service Principal or Managed Identity. Read more at https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token + AzureEncryptionSettings: + type: object + properties: + azure_tenant_id: + type: string + azure_cmk_access_connector_id: + type: string + azure_cmk_managed_identity_id: + type: string + required: + - azure_tenant_id AzureManagedIdentity: type: object properties: @@ -12650,7 +18434,7 @@ components: description: >- The Azure resource ID of the Azure Databricks Access Connector. Use the format - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. @@ -12658,7 +18442,7 @@ components: type: string description: >- The Azure resource ID of the managed identity. Use the format, - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume @@ -12674,12 +18458,12 @@ components: description: >- The Azure resource ID of the Azure Databricks Access Connector. Use the format - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. managed_identity_id: type: string description: >- The Azure resource ID of the managed identity. Use the format, - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume @@ -12695,7 +18479,7 @@ components: description: >- The Azure resource ID of the Azure Databricks Access Connector. Use the format - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}`. + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.Databricks/accessConnectors/{connector-name}``. credential_id: type: string description: The Databricks internal ID that represents this managed identity. @@ -12703,7 +18487,7 @@ components: type: string description: >- The Azure resource ID of the managed identity. Use the format, - `/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}` + ``/subscriptions/{guid}/resourceGroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identity-name}`` This is only available for user-assgined identities. For system-assigned identities, the access_connector_id is used to identify the identity. If this field is not provided, then we assume @@ -12778,11 +18562,21 @@ components: CancelRefreshResponse: type: object properties: {} + CatalogFederationConfig: + type: object + properties: + schema_filter_type: + $ref: '#/components/schemas/SchemaFilterType' + description: Controls how the catalog's schema filter items are treated. + description: Federation-only configuration for a FOREIGN catalog. CatalogInfo: type: object properties: browse_only: type: boolean + catalog_federation_config: + $ref: '#/components/schemas/CatalogFederationConfig' + description: Federation-only configuration, present only for FOREIGN catalogs. catalog_type: $ref: '#/components/schemas/CatalogType' comment: @@ -12791,447 +18585,1023 @@ components: connection_name: type: string description: The name of the connection to an external data source. + conversion_info: + $ref: '#/components/schemas/ConversionInfo' + description: Status of conversion of FOREIGN catalog to UC Native catalog. created_at: type: integer description: Time at which this catalog was created, in epoch milliseconds. created_by: type: string description: Username of catalog creator. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the catalog + dr_replication_info: + $ref: '#/components/schemas/DrReplicationInfo' + description: Disaster Recovery replication state snapshot. effective_predictive_optimization_flag: $ref: '#/components/schemas/EffectivePredictiveOptimizationFlag' enable_predictive_optimization: $ref: '#/components/schemas/EnablePredictiveOptimization' description: >- - Whether predictive optimization should be enabled for this object - and objects under it. + Whether predictive optimization should be enabled for this object + and objects under it. + full_name: + type: string + description: The full name of the catalog. Corresponds with the name field. + isolation_mode: + $ref: '#/components/schemas/CatalogIsolationMode' + description: >- + Whether the current securable is accessible from all workspaces or a + specific set of workspaces. + managed_encryption_settings: + $ref: '#/components/schemas/EncryptionSettings' + description: Control CMK encryption for managed catalog data + metastore_id: + type: string + description: Unique identifier of parent metastore. + name: + type: string + description: Name of catalog. + options: + type: object + description: A map of key-value properties attached to the securable. + owner: + type: string + description: Username of current owner of catalog. + properties: + type: object + description: A map of key-value properties attached to the securable. + provider_name: + type: string + description: >- + The name of delta sharing provider. A Delta Sharing catalog is a + catalog that is based on a Delta share on a remote sharing server. + provisioning_info: + $ref: '#/components/schemas/ProvisioningInfo' + securable_type: + $ref: '#/components/schemas/SecurableType' + share_name: + type: string + description: The name of the share under the share provider. + storage_location: + type: string + description: Storage Location URL (full path) for managed tables within catalog. + storage_root: + type: string + description: Storage root URL for managed tables within catalog. + updated_at: + type: integer + description: Time at which this catalog was last modified, in epoch milliseconds. + updated_by: + type: string + description: Username of user who last modified catalog. + CloudflareApiToken: + type: object + properties: + access_key_id: + type: string + description: The access key ID associated with the API token. + secret_access_key: + type: string + description: The secret access token generated for the above access key ID. + account_id: + type: string + description: The ID of the account associated with the API token. + required: + - access_key_id + - secret_access_key + description: |- + The Cloudflare API token configuration. Read more at + https://developers.cloudflare.com/r2/api/s3/tokens/ + ColumnInfo: + type: object + properties: + comment: + type: string + mask: + $ref: '#/components/schemas/ColumnMask' + name: + type: string + description: Name of Column. + nullable: + type: boolean + description: 'Whether field may be Null (default: true).' + partition_index: + type: integer + description: Partition index for column. + position: + type: integer + description: Ordinal position of column (starting at position 0). + type_interval_type: + type: string + description: Format of IntervalType. + type_json: + type: string + description: Full data type specification, JSON-serialized. + type_name: + $ref: '#/components/schemas/ColumnTypeName' + type_precision: + type: integer + description: Digits of precision; required for DecimalTypes. + type_scale: + type: integer + description: Digits to right of decimal; Required for DecimalTypes. + type_text: + type: string + description: Full data type specification as SQL/catalogString text. + ColumnMask: + type: object + properties: + function_name: + type: string + using_arguments: + type: array + items: + $ref: '#/components/schemas/PolicyFunctionArgument' + description: >- + The list of additional table columns or literals to be passed as + additional arguments to a column mask function. This is the + replacement of the deprecated using_column_names field and carries + information about the types (alias or constant) of the arguments to + the mask function. + using_column_names: + type: array + items: + type: string + description: >- + The list of additional table columns to be passed as input to the + column mask function. The first arg of the mask function should be + of the type of the column being masked and the types of the rest of + the args should match the types of columns in 'using_column_names'. + ColumnMaskOptions: + type: object + properties: + function_name: + type: string + on_column: + type: string + description: >- + The alias of the column to be masked. The alias must refer to one of + matched columns. The values of the column is passed to the column + mask function as the first argument. Required on create and update. + using: + type: array + items: + $ref: '#/components/schemas/FunctionArgument' + description: >- + Optional list of column aliases or constant literals to be passed as + additional arguments to the column mask function. The type of each + column should match the positional argument of the column mask + function. + required: + - function_name + - on_column + ColumnRelationship: + type: object + properties: + source: + type: string + target: + type: string + ColumnTagValueExtraction: + type: object + properties: + column_alias: + type: string + description: The alias from MATCH COLUMNS that identifies the column. + tag_key: + type: string + description: 1024 matches the max_length on FunctionArgument.constant above. + required: + - column_alias + - tag_key + description: >- + Extracts the value of a column-level tag: get_column_tag_value(col, + "tagKey"). + ConditionalDisplay: + type: object + properties: + depends_on_option: + type: string + description: >- + The name of the option whose value determines visibility of this + option. + hidden_when_values: + type: array + items: + type: string + description: >- + The values of the depends_on_option that will hide this option. If + empty or not set, this option follows default visibility (shown + unless is_hidden is true). If depends_on_option has any of these + values, this option is hidden. + description: >- + Defines when an option should be hidden based on another option's value. + For example, for + pre-created OAuth connections, some options are conditionally hidden. This field works in + conjunction with OptionSpec.is_hidden: + + - If OptionSpec.is_hidden is true, the option is always hidden regardless of ConditionalDisplay. + - If OptionSpec.is_hidden is false (or unset), ConditionalDisplay determines visibility: + - If depends_on_option matches any value in hidden_when_values, hide this option. + - Otherwise, show this option. + ConnectionDependency: + type: object + properties: + connection_name: + type: string + description: >- + Full name of the dependent connection, in the form of + **connection_name**. + description: A connection that is dependent on a SQL object. + ConnectionInfo: + type: object + properties: + comment: + type: string + connection_id: + type: string + description: Unique identifier of the Connection. + connection_type: + $ref: '#/components/schemas/ConnectionType' + description: The type of connection. + created_at: + type: integer + description: Time at which this connection was created, in epoch milliseconds. + created_by: + type: string + description: Username of connection creator. + credential_type: + $ref: '#/components/schemas/CredentialType' + description: The type of credential. + environment_settings: + $ref: '#/components/schemas/EnvironmentSettings' + description: >- + [Create,Update:OPT] Connection environment settings as + EnvironmentSettings object. full_name: type: string - description: The full name of the catalog. Corresponds with the name field. - isolation_mode: - $ref: '#/components/schemas/CatalogIsolationMode' - description: >- - Whether the current securable is accessible from all workspaces or a - specific set of workspaces. + description: Full name of connection. metastore_id: type: string description: Unique identifier of parent metastore. name: type: string - description: Name of catalog. + description: Name of the connection. options: type: object description: A map of key-value properties attached to the securable. owner: type: string - description: Username of current owner of catalog. + description: Username of current owner of the connection. properties: type: object description: A map of key-value properties attached to the securable. - provider_name: - type: string - description: >- - The name of delta sharing provider. A Delta Sharing catalog is a - catalog that is based on a Delta share on a remote sharing server. provisioning_info: $ref: '#/components/schemas/ProvisioningInfo' + read_only: + type: boolean + description: If the connection is read only. + secrets: + type: object + description: >- + A map of option names to UC Secret references. Keys are connection + option names (same as in OptionsKVPairs) and values are UC Secret + fully qualified names. securable_type: $ref: '#/components/schemas/SecurableType' - share_name: - type: string - description: The name of the share under the share provider. - storage_location: - type: string - description: Storage Location URL (full path) for managed tables within catalog. - storage_root: - type: string - description: Storage root URL for managed tables within catalog. updated_at: type: integer - description: Time at which this catalog was last modified, in epoch milliseconds. + description: Time at which this connection was updated, in epoch milliseconds. updated_by: type: string - description: Username of user who last modified catalog. - CloudflareApiToken: + description: Username of user who last modified connection. + url: + type: string + description: URL of the remote data source, extracted from options. + ContinuousUpdateStatus: type: object properties: - access_key_id: - type: string - description: The access key ID associated with the API token. - secret_access_key: - type: string - description: The secret access token generated for the above access key ID. - account_id: + initial_pipeline_sync_progress: + $ref: '#/components/schemas/PipelineProgress' + description: Progress of the initial data synchronization. + last_processed_commit_version: + type: integer + description: >- + The last source table Delta version that was synced to the online + table. Note that this Delta version may not be completely synced to + the online table yet. + timestamp: type: string - description: The ID of the account associated with the API token. - required: - - access_key_id - - secret_access_key - description: |- - The Cloudflare API token configuration. Read more at - https://developers.cloudflare.com/r2/api/s3/tokens/ - ColumnInfo: + description: >- + The timestamp of the last time any data was synchronized from the + source table to the online table. + description: >- + Detailed status of an online table. Shown if the online table is in the + ONLINE_CONTINUOUS_UPDATE + or the ONLINE_UPDATING_PIPELINE_RESOURCES state. + ConversionInfo: + type: object + properties: + state: + $ref: '#/components/schemas/ConversionInfoState' + description: The conversion state of the resource. + description: Status of conversion of FOREIGN entity into UC Native entity. + CreateAccessRequest: type: object properties: + behalf_of: + $ref: '#/components/schemas/Principal' comment: type: string - mask: - $ref: '#/components/schemas/ColumnMask' + description: >- + Optional. Comment associated with the request. At most 200 + characters, can only contain lowercase/uppercase letters (a-z, A-Z), + numbers (0-9), punctuation, and spaces. + securable_permissions: + type: array + items: + $ref: '#/components/schemas/SecurablePermissions' + description: >- + List of securables and their corresponding requested UC privileges. + At most 30 securables can be requested for a principal per batched + call. Each securable can only be requested once per principal. + CreateAccessRequestResponse: + type: object + properties: + behalf_of: + $ref: '#/components/schemas/Principal' + request_destinations: + type: array + items: + $ref: '#/components/schemas/AccessRequestDestinations' + description: >- + The access request destinations for all the securables the principal + requested. + CreateAccountsMetastore: + type: object + properties: name: type: string - description: Name of Column. - nullable: + external_access_enabled: type: boolean - description: 'Whether field may be Null (default: true).' - partition_index: - type: integer - description: Partition index for column. - position: - type: integer - description: Ordinal position of column (starting at position 0). - type_interval_type: + description: >- + Whether to allow non-DBR clients to directly access entities under + the metastore. + region: type: string - description: Format of IntervalType. - type_json: + description: >- + Cloud region which the metastore serves (e.g., ``us-west-2``, + ``westus``). + storage_root: type: string - description: Full data type specification, JSON-serialized. - type_name: - $ref: '#/components/schemas/ColumnTypeName' - type_precision: - type: integer - description: Digits of precision; required for DecimalTypes. - type_scale: - type: integer - description: Digits to right of decimal; Required for DecimalTypes. - type_text: + description: The storage root URL for metastore + required: + - name + CreateAccountsStorageCredential: + type: object + properties: + name: type: string - description: Full data type specification as SQL/catalogString text. - ColumnMask: + aws_iam_role: + $ref: '#/components/schemas/AwsIamRoleRequest' + description: The AWS IAM role configuration. + azure_managed_identity: + $ref: '#/components/schemas/AzureManagedIdentityRequest' + description: The Azure managed identity configuration. + azure_service_principal: + $ref: '#/components/schemas/AzureServicePrincipal' + description: The Azure service principal configuration. + cloudflare_api_token: + $ref: '#/components/schemas/CloudflareApiToken' + description: The Cloudflare API token configuration. + comment: + type: string + description: Comment associated with the credential. + databricks_gcp_service_account: + $ref: '#/components/schemas/DatabricksGcpServiceAccountRequest' + description: The Databricks managed GCP service account configuration. + read_only: + type: boolean + description: >- + Whether the credential is usable only for read operations. Only + applicable when purpose is **STORAGE**. + required: + - name + CreateFunction: type: object properties: - function_name: + name: type: string - using_arguments: - type: array - items: - $ref: '#/components/schemas/PolicyFunctionArgument' + catalog_name: + type: string + description: Name of parent Catalog. + schema_name: + type: string + description: Name of parent Schema relative to its parent Catalog. + input_params: + $ref: '#/components/schemas/FunctionParameterInfos' + description: Function input parameters. + data_type: + $ref: '#/components/schemas/ColumnTypeName' + description: Scalar function return data type. + full_data_type: + type: string + description: Pretty printed function data type. + routine_body: + $ref: '#/components/schemas/CreateFunctionRoutineBody' description: >- - The list of additional table columns or literals to be passed as - additional arguments to a column mask function. This is the - replacement of the deprecated using_column_names field and carries - information about the types (alias or constant) of the arguments to - the mask function. - using_column_names: - type: array - items: - type: string + Function language. When **EXTERNAL** is used, the language of the + routine function should be specified in the **external_language** + field, and the **return_params** of the function cannot be used (as + **TABLE** return type is not supported), and the **sql_data_access** + field must be **NO_SQL**. + routine_definition: + type: string + description: Function body. + parameter_style: + $ref: '#/components/schemas/CreateFunctionParameterStyle' + description: Function parameter style. **S** is the value for SQL. + is_deterministic: + type: boolean + description: Whether the function is deterministic. + sql_data_access: + $ref: '#/components/schemas/CreateFunctionSqlDataAccess' + description: Function SQL data access. + is_null_call: + type: boolean + description: Function null call. + security_type: + $ref: '#/components/schemas/CreateFunctionSecurityType' + description: Function security type. + specific_name: + type: string + description: Specific name of the function; Reserved for future use. + comment: + type: string + description: User-provided free-form text description. + external_language: + type: string + description: External function language. + external_name: + type: string + description: External function name. + properties: + type: string + description: JSON-serialized key-value pair map, encoded (escaped) as a string. + return_params: + $ref: '#/components/schemas/FunctionParameterInfos' + description: Table function return parameters. + routine_dependencies: + $ref: '#/components/schemas/DependencyList' + description: function dependencies. + sql_path: + type: string description: >- - The list of additional table columns to be passed as input to the - column mask function. The first arg of the mask function should be - of the type of the column being masked and the types of the rest of - the args should match the types of columns in 'using_column_names'. - ColumnMaskOptions: + List of schemes whose objects can be referenced without + qualification. + required: + - name + - catalog_name + - schema_name + - input_params + - data_type + - full_data_type + - routine_body + - routine_definition + - parameter_style + - is_deterministic + - sql_data_access + - is_null_call + - security_type + - specific_name + CreateMetastoreAssignment: type: object properties: - function_name: + workspace_id: + type: integer + metastore_id: type: string - on_column: + description: The unique ID of the metastore. + default_catalog_name: type: string description: >- - The alias of the column to be masked. The alias must refer to one of - matched columns. The values of the column is passed to the column - mask function as the first argument. Required on create and update. - using: - type: array - items: - $ref: '#/components/schemas/FunctionArgument' - description: >- - Optional list of column aliases or constant literals to be passed as - additional arguments to the column mask function. The type of each - column should match the positional argument of the column mask - function. + The name of the default catalog in the metastore. This field is + deprecated. Please use "Default Namespace API" to configure the + default catalog for a Databricks workspace. required: - - function_name - - on_column - ColumnRelationship: + - workspace_id + - metastore_id + - default_catalog_name + CreateRequestExternalLineage: type: object properties: source: - type: string + $ref: '#/components/schemas/ExternalLineageObject' target: + $ref: '#/components/schemas/ExternalLineageObject' + description: Target object of the external lineage relationship. + columns: + type: array + items: + $ref: '#/components/schemas/ColumnRelationship' + description: List of column relationships between source and target objects. + id: type: string - ConnectionDependency: + description: Unique identifier of the external lineage relationship. + properties: + type: object + description: >- + Key-value properties associated with the external lineage + relationship. + required: + - source + - target + CredentialDependency: type: object properties: - connection_name: + credential_name: type: string description: >- - Full name of the dependent connection, in the form of - __connection_name__. - description: A connection that is dependent on a SQL object. - ConnectionInfo: + Full name of the dependent credential, in the form of + **credential_name**. + description: A credential that is dependent on a SQL object. + CredentialInfo: type: object properties: + aws_iam_role: + $ref: '#/components/schemas/AwsIamRole' + azure_managed_identity: + $ref: '#/components/schemas/AzureManagedIdentity' + description: The Azure managed identity configuration. + azure_service_principal: + $ref: '#/components/schemas/AzureServicePrincipal' + description: The Azure service principal configuration. comment: type: string - description: User-provided free-form text description. - connection_id: - type: string - description: Unique identifier of the Connection. - connection_type: - $ref: '#/components/schemas/ConnectionType' - description: The type of connection. + description: Comment associated with the credential. created_at: type: integer - description: Time at which this connection was created, in epoch milliseconds. + description: Time at which this credential was created, in epoch milliseconds. created_by: type: string - description: Username of connection creator. - credential_type: - $ref: '#/components/schemas/CredentialType' - description: The type of credential. + description: Username of credential creator. + databricks_gcp_service_account: + $ref: '#/components/schemas/DatabricksGcpServiceAccount' + description: The Databricks managed GCP service account configuration. full_name: type: string - description: Full name of connection. + description: The full name of the credential. + id: + type: string + description: The unique identifier of the credential. + isolation_mode: + $ref: '#/components/schemas/IsolationMode' + description: >- + Whether the current securable is accessible from all workspaces or a + specific set of workspaces. metastore_id: type: string - description: Unique identifier of parent metastore. + description: Unique identifier of the parent metastore. name: type: string - description: Name of the connection. - options: - type: object - description: A map of key-value properties attached to the securable. + description: >- + The credential name. The name must be unique among storage and + service credentials within the metastore. owner: type: string - description: Username of current owner of the connection. - properties: - type: object - description: A map of key-value properties attached to the securable. - provisioning_info: - $ref: '#/components/schemas/ProvisioningInfo' + description: Username of current owner of credential. + purpose: + $ref: '#/components/schemas/CredentialPurpose' + description: Indicates the purpose of the credential. read_only: type: boolean - description: If the connection is read only. - securable_type: - $ref: '#/components/schemas/SecurableType' + description: >- + Whether the credential is usable only for read operations. Only + applicable when purpose is **STORAGE**. updated_at: type: integer - description: Time at which this connection was updated, in epoch milliseconds. + description: >- + Time at which this credential was last modified, in epoch + milliseconds. updated_by: type: string - description: Username of user who last modified connection. - url: + description: Username of user who last modified the credential. + used_for_managed_storage: + type: boolean + description: >- + Whether this credential is the current metastore's root storage + credential. Only applicable when purpose is **STORAGE**. + CredentialValidationResult: + type: object + properties: + message: type: string - description: URL of the remote data source, extracted from options. - description: 'Next ID: 24' - ContinuousUpdateStatus: + result: + $ref: '#/components/schemas/ValidateCredentialResult' + description: The results of the tested operation. + DatabricksGcpServiceAccount: + type: object + properties: + credential_id: + type: string + description: The Databricks internal ID that represents this managed identity. + email: + type: string + description: The email of the service account. + private_key_id: + type: string + description: The ID that represents the private key for this Service Account + description: >- + GCP long-lived credential. Databricks-created Google Cloud Storage + service account. + DatabricksGcpServiceAccountRequest: + type: object + properties: {} + description: >- + GCP long-lived credential. Databricks-created Google Cloud Storage + service account. + DatabricksGcpServiceAccountResponse: + type: object + properties: + credential_id: + type: string + description: The Databricks internal ID that represents this managed identity. + email: + type: string + description: The email of the service account. + description: >- + GCP long-lived credential. Databricks-created Google Cloud Storage + service account. + DeleteCredentialResponse: + type: object + properties: {} + DeleteMonitorResponse: + type: object + properties: {} + DeletePolicyResponse: + type: object + properties: {} + DeleteRequestExternalLineage: + type: object + properties: + source: + $ref: '#/components/schemas/ExternalLineageObject' + target: + $ref: '#/components/schemas/ExternalLineageObject' + description: Target object of the external lineage relationship. + id: + type: string + description: Unique identifier of the external lineage relationship. + required: + - source + - target + DeleteResponse: + type: object + properties: {} + DeleteTableConstraintResponse: + type: object + properties: {} + DeltaRuntimePropertiesKvPairs: + type: object + properties: + delta_runtime_properties: + type: object + description: A map of key-value properties attached to the securable. + required: + - delta_runtime_properties + description: >- + Properties pertaining to the current state of the delta table as given + by the commit server. + This does not contain **delta.*** (input) properties in **TableInfo.properties**. + DenyOptions: + type: object + properties: + privileges: + type: array + items: + type: string + required: + - privileges + Dependency: + type: object + properties: + connection: + $ref: '#/components/schemas/ConnectionDependency' + credential: + $ref: '#/components/schemas/CredentialDependency' + function: + $ref: '#/components/schemas/FunctionDependency' + metastore_name: + type: string + description: >- + Metastore context for the dependency. Global UC server responses + currently populate this with the metastore UUID string so clients + can disambiguate dependencies with identical names across + metastores. + secret: + $ref: '#/components/schemas/SecretDependency' + description: A dependency on a Unity Catalog secret. + table: + $ref: '#/components/schemas/TableDependency' + volume: + $ref: '#/components/schemas/VolumeDependency' + description: A dependency on a Unity Catalog volume. + description: >- + A dependency of a SQL object. One of the following fields must be + defined: **table**, + **function**, **connection**, **credential**, **volume**, or **secret**. + DependencyList: + type: object + properties: + dependencies: + type: array + items: + $ref: '#/components/schemas/Dependency' + description: Array of dependencies. + description: A list of dependencies. + DisableResponse: + type: object + properties: {} + DrReplicationInfo: type: object properties: - initial_pipeline_sync_progress: - $ref: '#/components/schemas/PipelineProgress' - description: Progress of the initial data synchronization. - last_processed_commit_version: + last_failover_time_ms: type: integer description: >- - The last source table Delta version that was synced to the online - table. Note that this Delta version may not be completely synced to - the online table yet. - timestamp: + Wall-clock epoch milliseconds when this catalog was last promoted to + primary via failover or failback. Set by DR Manager. Used by + Predictive Optimization to suppress operations until sufficient + workload history accumulates. + replicated_entities: type: string description: >- - The timestamp of the last time any data was synchronized from the - source table to the online table. - description: >- - Detailed status of an online table. Shown if the online table is in the - ONLINE_CONTINUOUS_UPDATE - or the ONLINE_UPDATING_PIPELINE_RESOURCES state. - CreateAccessRequest: + See + https://docs.google.com/document/d/1X0A_3hMhzuS2V1E3zB0x5wxPsFx70bVYK5rHep2AjW8. + status: + $ref: '#/components/schemas/DrReplicationStatus' + description: Metadata related to Disaster Recovery. + EffectivePermissionsList: type: object properties: - behalf_of: - $ref: '#/components/schemas/Principal' - comment: + next_page_token: type: string - description: >- - Optional. Comment associated with the request. At most 200 - characters, can only contain lowercase/uppercase letters (a-z, A-Z), - numbers (0-9), punctuation, and spaces. - securable_permissions: + privilege_assignments: type: array items: - $ref: '#/components/schemas/SecurablePermissions' + $ref: '#/components/schemas/EffectivePrivilegeAssignment' description: >- - List of securables and their corresponding requested UC privileges. - At most 30 securables can be requested for a principal per batched - call. Each securable can only be requested once per principal. - CreateAccessRequestResponse: + The privileges conveyed to each principal (either directly or via + inheritance) + EffectivePredictiveOptimizationFlag: type: object properties: - behalf_of: - $ref: '#/components/schemas/Principal' - request_destinations: - type: array - items: - $ref: '#/components/schemas/AccessRequestDestinations' + value: + $ref: '#/components/schemas/EnablePredictiveOptimization' + inherited_from_name: + type: string description: >- - The access request destinations for all the securables the principal - requested. - CreateAccountsMetastore: + The name of the object from which the flag was inherited. If there + was no inheritance, this field is left blank. + inherited_from_type: + $ref: >- + #/components/schemas/EffectivePredictiveOptimizationFlagInheritedFromType + description: >- + The type of the object from which the flag was inherited. If there + was no inheritance, this field is left blank. + required: + - value + EffectivePrivilege: type: object properties: - name: + inherited_from_name: type: string - external_access_enabled: - type: boolean + inherited_from_type: + $ref: '#/components/schemas/SecurableType' description: >- - Whether to allow non-DBR clients to directly access entities under - the metastore. - region: + The type of the object that conveys this privilege via inheritance. + This field is omitted when privilege is not inherited (it's assigned + to the securable itself). + privilege: + $ref: '#/components/schemas/Privilege' + description: The privilege assigned to the principal. + EffectivePrivilegeAssignment: + type: object + properties: + principal: type: string + privileges: + type: array + items: + $ref: '#/components/schemas/EffectivePrivilege' description: >- - Cloud region which the metastore serves (e.g., `us-west-2`, - `westus`). - storage_root: + The privileges conveyed to the principal (either directly or via + inheritance). + EnableResponse: + type: object + properties: {} + EncryptionDetails: + type: object + properties: + sse_encryption_details: + $ref: '#/components/schemas/SseEncryptionDetails' + description: >- + Server-Side Encryption properties for clients communicating with AWS + s3. + description: Encryption options that apply to clients connecting to cloud storage. + EncryptionSettings: + type: object + properties: + azure_encryption_settings: + $ref: '#/components/schemas/AzureEncryptionSettings' + description: optional Azure settings - only required if an Azure CMK is used. + azure_key_vault_key_id: type: string - description: The storage root URL for metastore - required: - - name - CreateAccountsStorageCredential: + description: the AKV URL in Azure, null otherwise. + customer_managed_key_id: + type: string + description: the CMK uuid in AWS and GCP, null otherwise. + description: >- + Encryption Settings are used to carry metadata for securable encryption + at rest. Currently used + for catalogs, we can use the information supplied here to interact with a CMK. + EntityTagAssignment: type: object properties: - name: + entity_name: type: string - aws_iam_role: - $ref: '#/components/schemas/AwsIamRoleRequest' - description: The AWS IAM role configuration. - azure_managed_identity: - $ref: '#/components/schemas/AzureManagedIdentityRequest' - description: The Azure managed identity configuration. - azure_service_principal: - $ref: '#/components/schemas/AzureServicePrincipal' - description: The Azure service principal configuration. - cloudflare_api_token: - $ref: '#/components/schemas/CloudflareApiToken' - description: The Cloudflare API token configuration. - comment: + description: The fully qualified name of the entity to which the tag is assigned + tag_key: type: string - description: Comment associated with the credential. - databricks_gcp_service_account: - $ref: '#/components/schemas/DatabricksGcpServiceAccountRequest' - description: The Databricks managed GCP service account configuration. - read_only: + description: The key of the tag + entity_type: + type: string + description: The type of the entity to which the tag is assigned. + inherited: type: boolean + description: Boolean which indicates whether this tag is inherited. + source_type: + $ref: '#/components/schemas/TagAssignmentSourceType' description: >- - Whether the credential is usable only for read operations. Only - applicable when purpose is **STORAGE**. + The source type of the tag assignment, e.g., user-assigned or + system-assigned + tag_value: + type: string + description: The value of the tag + update_time: + type: string + format: date-time + description: The timestamp when the tag assignment was last updated + updated_by: + type: string + description: The user or principal who updated the tag assignment required: - - name - CreateFunction: + - entity_name + - tag_key + - entity_type + description: Represents a tag assignment to an entity + EnvironmentSettings: + type: object + properties: + environment_version: + type: string + java_dependencies: + type: array + items: + type: string + ExternalLineageExternalMetadata: type: object properties: name: type: string - catalog_name: + ExternalLineageExternalMetadataInfo: + type: object + properties: + entity_type: type: string - description: Name of parent Catalog. - schema_name: + description: Type of entity represented by the external metadata object. + event_time: type: string - description: Name of parent Schema relative to its parent Catalog. - input_params: - $ref: '#/components/schemas/FunctionParameterInfos' - description: Function input parameters. - data_type: - $ref: '#/components/schemas/ColumnTypeName' - description: Scalar function return data type. - full_data_type: + description: Timestamp of the lineage event. + name: type: string - description: Pretty printed function data type. - routine_body: - $ref: '#/components/schemas/CreateFunctionRoutineBody' - description: >- - Function language. When **EXTERNAL** is used, the language of the - routine function should be specified in the **external_language** - field, and the **return_params** of the function cannot be used (as - **TABLE** return type is not supported), and the **sql_data_access** - field must be **NO_SQL**. - routine_definition: + description: Name of the external metadata object. + system_type: + $ref: '#/components/schemas/SystemType' + description: Type of external system. + description: Represents the external metadata object in the lineage event. + ExternalLineageFileInfo: + type: object + properties: + event_time: type: string - description: Function body. - parameter_style: - $ref: '#/components/schemas/CreateFunctionParameterStyle' - description: Function parameter style. **S** is the value for SQL. - is_deterministic: - type: boolean - description: Whether the function is deterministic. - sql_data_access: - $ref: '#/components/schemas/CreateFunctionSqlDataAccess' - description: Function SQL data access. - is_null_call: - type: boolean - description: Function null call. - security_type: - $ref: '#/components/schemas/CreateFunctionSecurityType' - description: Function security type. - specific_name: + description: Timestamp of the lineage event. + path: type: string - description: Specific name of the function; Reserved for future use. - comment: + description: URL of the path. + securable_name: type: string - description: User-provided free-form text description. - external_language: + description: The full name of the securable on the path. + securable_type: + type: string + description: The securable type of the securable on the path. + storage_location: + type: string + description: The storage location associated with securable on the path. + description: Represents the path information in the lineage event. + ExternalLineageInfo: + type: object + properties: + external_lineage_info: + $ref: '#/components/schemas/ExternalLineageRelationshipInfo' + description: >- + Information about the edge metadata of the external lineage + relationship. + external_metadata_info: + $ref: '#/components/schemas/ExternalLineageExternalMetadataInfo' + description: >- + Information about external metadata involved in the lineage + relationship. + file_info: + $ref: '#/components/schemas/ExternalLineageFileInfo' + description: Information about the file involved in the lineage relationship. + model_info: + $ref: '#/components/schemas/ExternalLineageModelVersionInfo' + description: >- + Information about the model version involved in the lineage + relationship. + table_info: + $ref: '#/components/schemas/ExternalLineageTableInfo' + description: Information about the table involved in the lineage relationship. + volume_info: + $ref: '#/components/schemas/ExternalLineageVolumeInfo' + description: Information about the volume involved in the lineage relationship. + description: Lineage response containing lineage information of a data asset. + ExternalLineageModelVersion: + type: object + properties: + name: type: string - description: External function language. - external_name: + version: type: string - description: External function name. - properties: + ExternalLineageModelVersionInfo: + type: object + properties: + event_time: type: string - description: JSON-serialized key-value pair map, encoded (escaped) as a string. - return_params: - $ref: '#/components/schemas/FunctionParameterInfos' - description: Table function return parameters. - routine_dependencies: - $ref: '#/components/schemas/DependencyList' - description: function dependencies. - sql_path: + description: Timestamp of the lineage event. + model_name: type: string - description: >- - List of schemes whose objects can be referenced without - qualification. - required: - - name - - catalog_name - - schema_name - - input_params - - data_type - - full_data_type - - routine_body - - routine_definition - - parameter_style - - is_deterministic - - sql_data_access - - is_null_call - - security_type - - specific_name - CreateMetastoreAssignment: + description: Name of the model. + version: + type: integer + description: Version number of the model. + description: Represents the model version information in the lineage event. + ExternalLineageObject: type: object properties: - workspace_id: - type: integer - metastore_id: + external_metadata: + $ref: '#/components/schemas/ExternalLineageExternalMetadata' + model_version: + $ref: '#/components/schemas/ExternalLineageModelVersion' + path: + $ref: '#/components/schemas/ExternalLineagePath' + table: + $ref: '#/components/schemas/ExternalLineageTable' + volume: + $ref: '#/components/schemas/ExternalLineageVolume' + ExternalLineagePath: + type: object + properties: + url: type: string - description: The unique ID of the metastore. - default_catalog_name: + ExternalLineageRelationship: + type: object + properties: + source: + $ref: '#/components/schemas/ExternalLineageObject' + target: + $ref: '#/components/schemas/ExternalLineageObject' + description: Target object of the external lineage relationship. + columns: + type: array + items: + $ref: '#/components/schemas/ColumnRelationship' + description: List of column relationships between source and target objects. + id: type: string + description: Unique identifier of the external lineage relationship. + properties: + type: object description: >- - The name of the default catalog in the metastore. This field is - deprecated. Please use "Default Namespace API" to configure the - default catalog for a Databricks workspace. + Key-value properties associated with the external lineage + relationship. required: - - workspace_id - - metastore_id - - default_catalog_name - CreateRequestExternalLineage: + - source + - target + ExternalLineageRelationshipInfo: type: object properties: source: @@ -13255,916 +19625,1161 @@ components: required: - source - target - CredentialDependency: + ExternalLineageTable: type: object properties: - credential_name: + name: type: string - description: >- - Full name of the dependent credential, in the form of - __credential_name__. - description: A credential that is dependent on a SQL object. - CredentialInfo: + ExternalLineageTableInfo: type: object properties: - aws_iam_role: - $ref: '#/components/schemas/AwsIamRole' - azure_managed_identity: - $ref: '#/components/schemas/AzureManagedIdentity' - description: The Azure managed identity configuration. - azure_service_principal: - $ref: '#/components/schemas/AzureServicePrincipal' - description: The Azure service principal configuration. + catalog_name: + type: string + description: Name of Catalog. + event_time: + type: string + description: Timestamp of the lineage event. + name: + type: string + description: Name of Table. + schema_name: + type: string + description: Name of Schema. + description: Represents the table information in the lineage event. + ExternalLineageVolume: + type: object + properties: + name: + type: string + ExternalLineageVolumeInfo: + type: object + properties: + catalog_name: + type: string + event_time: + type: string + format: date-time + description: Timestamp of the lineage event. + name: + type: string + description: Name of the volume. + schema_name: + type: string + description: Name of the schema. + ExternalLocationInfo: + type: object + properties: + browse_only: + type: boolean comment: type: string - description: Comment associated with the credential. + description: User-provided free-form text description. created_at: type: integer - description: Time at which this credential was created, in epoch milliseconds. + description: >- + Time at which this external location was created, in epoch + milliseconds. created_by: type: string - description: Username of credential creator. - databricks_gcp_service_account: - $ref: '#/components/schemas/DatabricksGcpServiceAccount' - description: The Databricks managed GCP service account configuration. - full_name: + description: Username of external location creator. + credential_id: type: string - description: The full name of the credential. - id: + description: Unique ID of the location's storage credential. + credential_name: type: string - description: The unique identifier of the credential. + description: Name of the storage credential used with this location. + effective_enable_file_events: + type: boolean + description: >- + The effective value of ``enable_file_events`` after applying + server-side defaults. + effective_file_event_queue: + $ref: '#/components/schemas/FileEventQueue' + description: >- + The effective file event queue configuration after applying + server-side defaults. Always populated when a queue is provisioned, + regardless of whether the user explicitly set + ``enable_file_events``. Use this field instead of + ``file_event_queue`` for reading the actual queue state. + enable_file_events: + type: boolean + description: >- + Whether to enable file events on this external location. Default to + ``true``. Set to ``false`` to disable file events. The actual + applied value may differ due to server-side defaults; check + ``effective_enable_file_events`` for the effective state. + encryption_details: + $ref: '#/components/schemas/EncryptionDetails' + fallback: + type: boolean + description: >- + Indicates whether fallback mode is enabled for this external + location. When fallback mode is enabled, the access to the location + falls back to cluster credentials if UC credentials are not + sufficient. + file_event_queue: + $ref: '#/components/schemas/FileEventQueue' + description: >- + File event queue settings. If ``enable_file_events`` is not + ``false``, must be defined and have exactly one of the documented + properties. isolation_mode: $ref: '#/components/schemas/IsolationMode' - description: >- - Whether the current securable is accessible from all workspaces or a - specific set of workspaces. metastore_id: type: string - description: Unique identifier of the parent metastore. + description: Unique identifier of metastore hosting the external location. name: type: string - description: >- - The credential name. The name must be unique among storage and - service credentials within the metastore. + description: Name of the external location. owner: type: string - description: Username of current owner of credential. - purpose: - $ref: '#/components/schemas/CredentialPurpose' - description: Indicates the purpose of the credential. + description: The owner of the external location. read_only: type: boolean - description: >- - Whether the credential is usable only for read operations. Only - applicable when purpose is **STORAGE**. + description: Indicates whether the external location is read-only. updated_at: type: integer description: >- - Time at which this credential was last modified, in epoch + Time at which external location this was last modified, in epoch milliseconds. updated_by: type: string - description: Username of user who last modified the credential. - used_for_managed_storage: - type: boolean - description: >- - Whether this credential is the current metastore's root storage - credential. Only applicable when purpose is **STORAGE**. - CredentialValidationResult: - type: object - properties: - message: + description: Username of user who last modified the external location. + url: type: string - result: - $ref: '#/components/schemas/ValidateCredentialResult' - description: The results of the tested operation. - DatabricksGcpServiceAccount: + description: Path URL of the external location. + ExternalMetadata: type: object properties: - credential_id: + name: + type: string + system_type: + $ref: '#/components/schemas/SystemType' + description: Type of external system. + entity_type: + type: string + description: Type of entity within the external system. + columns: + type: array + items: + type: string + description: List of columns associated with the external metadata object. + create_time: + type: string + description: Time at which this external metadata object was created. + created_by: + type: string + description: Username of external metadata object creator. + description: + type: string + description: User-provided free-form text description. + id: type: string - description: The Databricks internal ID that represents this managed identity. - email: + description: Unique identifier of the external metadata object. + metastore_id: type: string - description: The email of the service account. - private_key_id: + description: Unique identifier of parent metastore. + owner: type: string - description: The ID that represents the private key for this Service Account - description: >- - GCP long-lived credential. Databricks-created Google Cloud Storage - service account. - DatabricksGcpServiceAccountRequest: - type: object - properties: {} - description: >- - GCP long-lived credential. Databricks-created Google Cloud Storage - service account. - DatabricksGcpServiceAccountResponse: - type: object - properties: - credential_id: + description: Owner of the external metadata object. + properties: + type: object + description: >- + A map of key-value properties attached to the external metadata + object. + update_time: type: string - description: The Databricks internal ID that represents this managed identity. - email: + description: Time at which this external metadata object was last modified. + updated_by: type: string - description: The email of the service account. - description: >- - GCP long-lived credential. Databricks-created Google Cloud Storage - service account. - DeleteCredentialResponse: - type: object - properties: {} - DeleteMonitorResponse: - type: object - properties: {} - DeletePolicyResponse: - type: object - properties: {} - DeleteRequestExternalLineage: - type: object - properties: - source: - $ref: '#/components/schemas/ExternalLineageObject' - target: - $ref: '#/components/schemas/ExternalLineageObject' - description: Target object of the external lineage relationship. - id: + description: Username of user who last modified external metadata object. + url: type: string - description: Unique identifier of the external lineage relationship. + description: URL associated with the external metadata object. required: - - source - - target - DeleteResponse: - type: object - properties: {} - DeleteTableConstraintResponse: - type: object - properties: {} - DeltaRuntimePropertiesKvPairs: + - name + - system_type + - entity_type + FailedStatus: type: object properties: - delta_runtime_properties: - type: object - description: A map of key-value properties attached to the securable. - required: - - delta_runtime_properties + last_processed_commit_version: + type: integer + description: >- + The last source table Delta version that was synced to the online + table. Note that this Delta version may only be partially synced to + the online table. Only populated if the table is still online and + available for serving. + timestamp: + type: string + description: >- + The timestamp of the last time any data was synchronized from the + source table to the online table. Only populated if the table is + still online and available for serving. description: >- - Properties pertaining to the current state of the delta table as given - by the commit server. - This does not contain **delta.*** (input) properties in __TableInfo.properties__. - Dependency: + Detailed status of an online table. Shown if the online table is in the + OFFLINE_FAILED or the + ONLINE_PIPELINE_FAILED state. + FileEventQueue: type: object properties: - connection: - $ref: '#/components/schemas/ConnectionDependency' - credential: - $ref: '#/components/schemas/CredentialDependency' - function: - $ref: '#/components/schemas/FunctionDependency' - table: - $ref: '#/components/schemas/TableDependency' - description: >- - A dependency of a SQL object. One of the following fields must be - defined: __table__, - __function__, __connection__, or __credential__. - DependencyList: + managed_aqs: + $ref: '#/components/schemas/AzureQueueStorage' + managed_onelake: + $ref: '#/components/schemas/OneLakeEventQueue' + managed_pubsub: + $ref: '#/components/schemas/GcpPubsub' + managed_sqs: + $ref: '#/components/schemas/AwsSqsQueue' + provided_aqs: + $ref: '#/components/schemas/AzureQueueStorage' + provided_onelake: + $ref: '#/components/schemas/OneLakeEventQueue' + provided_pubsub: + $ref: '#/components/schemas/GcpPubsub' + provided_sqs: + $ref: '#/components/schemas/AwsSqsQueue' + ForeignKeyConstraint: type: object properties: - dependencies: + name: + type: string + child_columns: type: array items: - $ref: '#/components/schemas/Dependency' - description: Array of dependencies. - description: A list of dependencies. - DisableResponse: - type: object - properties: {} - EffectivePermissionsList: - type: object - properties: - next_page_token: + type: string + description: Column names for this constraint. + parent_table: type: string - privilege_assignments: + description: The full name of the parent constraint. + parent_columns: type: array items: - $ref: '#/components/schemas/EffectivePrivilegeAssignment' - description: >- - The privileges conveyed to each principal (either directly or via - inheritance) - EffectivePredictiveOptimizationFlag: + type: string + description: Column names for this constraint. + rely: + type: boolean + description: True if the constraint is RELY, false or unset if NORELY. + required: + - name + - child_columns + - parent_table + - parent_columns + FunctionArgExpression: type: object properties: - value: - $ref: '#/components/schemas/EnablePredictiveOptimization' - inherited_from_name: - type: string - description: >- - The name of the object from which the flag was inherited. If there - was no inheritance, this field is left blank. - inherited_from_type: - $ref: >- - #/components/schemas/EffectivePredictiveOptimizationFlagInheritedFromType - description: >- - The type of the object from which the flag was inherited. If there - was no inheritance, this field is left blank. - required: - - value - EffectivePrivilege: + tag_introspection: + $ref: '#/components/schemas/TagIntrospectionExpression' + description: An expression that introspects tags at query time. + description: >- + An expression that is evaluated at query time against per-request + context. New variants (e.g., + identity attributes) are added as additional oneof cases. + FunctionArgument: type: object properties: - inherited_from_name: + alias: type: string - inherited_from_type: - $ref: '#/components/schemas/SecurableType' + constant: + type: string + description: A constant literal. + function_arg_expression: + $ref: '#/components/schemas/FunctionArgExpression' description: >- - The type of the object that conveys this privilege via inheritance. - This field is omitted when privilege is not inherited (it's assigned - to the securable itself). - privilege: - $ref: '#/components/schemas/Privilege' - description: The privilege assigned to the principal. - EffectivePrivilegeAssignment: + An expression evaluated at query time. Wraps per-request expression + variants (e.g., tag introspection) so new variants can be added + without extending the FunctionArgument oneof. + FunctionDependency: type: object properties: - principal: + function_full_name: type: string - privileges: - type: array - items: - $ref: '#/components/schemas/EffectivePrivilege' description: >- - The privileges conveyed to the principal (either directly or via - inheritance). - EnableResponse: - type: object - properties: {} - EncryptionDetails: + Full name of the dependent function, in the form of + **catalog_name**.**schema_name**.**function_name**. + required: + - function_full_name + description: A function that is dependent on a SQL object. + FunctionInfo: type: object properties: - sse_encryption_details: - $ref: '#/components/schemas/SseEncryptionDetails' + browse_only: + type: boolean + catalog_name: + type: string + description: Name of parent Catalog. + comment: + type: string + description: User-provided free-form text description. + created_at: + type: integer + description: Time at which this function was created, in epoch milliseconds. + created_by: + type: string + description: Username of function creator. + data_type: + $ref: '#/components/schemas/ColumnTypeName' + description: Scalar function return data type. + external_language: + type: string + description: External function language. + external_name: + type: string + description: External function name. + full_data_type: + type: string + description: Pretty printed function data type. + full_name: + type: string description: >- - Server-Side Encryption properties for clients communicating with AWS - s3. - description: Encryption options that apply to clients connecting to cloud storage. - EntityTagAssignment: - type: object - properties: - entity_name: + Full name of Function, in form of + **catalog_name**.**schema_name**.**function_name** + function_id: + type: string + description: Id of Function, relative to parent schema. + input_params: + $ref: '#/components/schemas/FunctionParameterInfos' + description: Function input parameters. + is_deterministic: + type: boolean + description: Whether the function is deterministic. + is_null_call: + type: boolean + description: Function null call. + metastore_id: type: string - description: The fully qualified name of the entity to which the tag is assigned - tag_key: + description: Unique identifier of parent metastore. + name: type: string - description: The key of the tag - entity_type: + description: Name of function, relative to parent schema. + owner: type: string + description: Username of current owner of the function. + parameter_style: + $ref: '#/components/schemas/FunctionInfoParameterStyle' + description: Function parameter style. **S** is the value for SQL. + properties: + type: string + description: JSON-serialized key-value pair map, encoded (escaped) as a string. + return_params: + $ref: '#/components/schemas/FunctionParameterInfos' + description: Table function return parameters. + routine_body: + $ref: '#/components/schemas/FunctionInfoRoutineBody' description: >- - The type of the entity to which the tag is assigned. Allowed values - are: catalogs, schemas, tables, columns, volumes. - source_type: - $ref: '#/components/schemas/TagAssignmentSourceType' - description: >- - The source type of the tag assignment, e.g., user-assigned or - system-assigned - tag_value: + Function language. When **EXTERNAL** is used, the language of the + routine function should be specified in the **external_language** + field, and the **return_params** of the function cannot be used (as + **TABLE** return type is not supported), and the **sql_data_access** + field must be **NO_SQL**. + routine_definition: type: string - description: The value of the tag - update_time: + description: Function body. + routine_dependencies: + $ref: '#/components/schemas/DependencyList' + description: function dependencies. + schema_name: type: string - format: date-time - description: The timestamp when the tag assignment was last updated - updated_by: + description: Name of parent Schema relative to its parent Catalog. + security_type: + $ref: '#/components/schemas/FunctionInfoSecurityType' + description: Function security type. + specific_name: type: string - description: The user or principal who updated the tag assignment - required: - - entity_name - - tag_key - - entity_type - description: Represents a tag assignment to an entity - ExternalLineageExternalMetadata: - type: object - properties: - name: + description: Specific name of the function; Reserved for future use. + sql_data_access: + $ref: '#/components/schemas/FunctionInfoSqlDataAccess' + description: Function SQL data access. + sql_path: type: string - ExternalLineageExternalMetadataInfo: + description: >- + List of schemes whose objects can be referenced without + qualification. + updated_at: + type: integer + description: >- + Time at which this function was last modified, in epoch + milliseconds. + updated_by: + type: string + description: Username of user who last modified the function. + FunctionParameterInfo: type: object properties: - entity_type: - type: string - description: Type of entity represented by the external metadata object. - event_time: - type: string - description: Timestamp of the lineage event. name: type: string - description: Name of the external metadata object. - system_type: - $ref: '#/components/schemas/SystemType' - description: Type of external system. - description: Represents the external metadata object in the lineage event. - ExternalLineageFileInfo: - type: object - properties: - event_time: + type_text: type: string - description: Timestamp of the lineage event. - path: + description: Full data type spec, SQL/catalogString text. + type_name: + $ref: '#/components/schemas/ColumnTypeName' + description: Name of type (INT, STRUCT, MAP, etc.) + position: + type: integer + description: Ordinal position of column (starting at position 0). + comment: type: string - description: URL of the path. - securable_name: + description: User-provided free-form text description. + parameter_default: type: string - description: The full name of the securable on the path. - securable_type: + description: Default value of the parameter. + parameter_mode: + $ref: '#/components/schemas/FunctionParameterMode' + description: Function parameter mode. + parameter_type: + $ref: '#/components/schemas/FunctionParameterType' + description: Function parameter type. + type_interval_type: type: string - description: The securable type of the securable on the path. - storage_location: + description: Format of IntervalType. + type_json: type: string - description: The storage location associated with securable on the path. - description: Represents the path information in the lineage event. - ExternalLineageInfo: + description: Full data type spec, JSON-serialized. + type_precision: + type: integer + description: Digits of precision; required on Create for DecimalTypes. + type_scale: + type: integer + description: Digits to right of decimal; Required on Create for DecimalTypes. + required: + - name + - type_text + - type_name + - position + FunctionParameterInfos: type: object properties: - external_lineage_info: - $ref: '#/components/schemas/ExternalLineageRelationshipInfo' - description: >- - Information about the edge metadata of the external lineage - relationship. - external_metadata_info: - $ref: '#/components/schemas/ExternalLineageExternalMetadataInfo' - description: >- - Information about external metadata involved in the lineage - relationship. - file_info: - $ref: '#/components/schemas/ExternalLineageFileInfo' - description: Information about the file involved in the lineage relationship. - model_info: - $ref: '#/components/schemas/ExternalLineageModelVersionInfo' - description: >- - Information about the model version involved in the lineage - relationship. - table_info: - $ref: '#/components/schemas/ExternalLineageTableInfo' - description: Information about the table involved in the lineage relationship. - description: Lineage response containing lineage information of a data asset. - ExternalLineageModelVersion: + parameters: + type: array + items: + $ref: '#/components/schemas/FunctionParameterInfo' + GcpOauthToken: type: object properties: - name: - type: string - version: + oauth_token: type: string - ExternalLineageModelVersionInfo: + description: |- + GCP temporary credentials for API authentication. Read more at + https://developers.google.com/identity/protocols/oauth2/service-account + GcpPubsub: type: object properties: - event_time: + managed_resource_id: type: string - description: Timestamp of the lineage event. - model_name: + subscription_name: type: string - description: Name of the model. - version: - type: integer - description: Version number of the model. - description: Represents the model version information in the lineage event. - ExternalLineageObject: - type: object - properties: - external_metadata: - $ref: '#/components/schemas/ExternalLineageExternalMetadata' - model_version: - $ref: '#/components/schemas/ExternalLineageModelVersion' - path: - $ref: '#/components/schemas/ExternalLineagePath' - table: - $ref: '#/components/schemas/ExternalLineageTable' - ExternalLineagePath: + description: >- + The Pub/Sub subscription name in the format + projects/{project}/subscriptions/{subscription name}. Only required + for provided_pubsub. + GenerateTemporaryPathCredentialResponse: type: object properties: + aws_temp_credentials: + $ref: '#/components/schemas/AwsCredentials' + azure_aad: + $ref: '#/components/schemas/AzureActiveDirectoryToken' + azure_user_delegation_sas: + $ref: '#/components/schemas/AzureUserDelegationSas' + expiration_time: + type: integer + description: >- + Server time when the credential will expire, in epoch milliseconds. + The API client is advised to cache the credential given this + expiration time. + gcp_oauth_token: + $ref: '#/components/schemas/GcpOauthToken' + r2_temp_credentials: + $ref: '#/components/schemas/R2Credentials' + uc_encrypted_token: + $ref: '#/components/schemas/UcEncryptedToken' url: type: string - ExternalLineageRelationship: + description: The URL of the storage path accessible by the temporary credential. + GenerateTemporaryServiceCredentialAzureOptions: type: object properties: - source: - $ref: '#/components/schemas/ExternalLineageObject' - target: - $ref: '#/components/schemas/ExternalLineageObject' - description: Target object of the external lineage relationship. - columns: + resources: type: array items: - $ref: '#/components/schemas/ColumnRelationship' - description: List of column relationships between source and target objects. - id: - type: string - description: Unique identifier of the external lineage relationship. - properties: - type: object + type: string description: >- - Key-value properties associated with the external lineage - relationship. - required: - - source - - target - ExternalLineageRelationshipInfo: - type: object - properties: - source: - $ref: '#/components/schemas/ExternalLineageObject' - target: - $ref: '#/components/schemas/ExternalLineageObject' - description: Target object of the external lineage relationship. - columns: + The resources to which the temporary Azure credential should apply. + These resources are the scopes that are passed to the token provider + (see + https://learn.microsoft.com/python/api/azure-core/azure.core.credentials.tokencredential?view=azure-python) + description: The Azure cloud options to customize the requested temporary credential + GenerateTemporaryServiceCredentialGcpOptions: + type: object + properties: + scopes: type: array items: - $ref: '#/components/schemas/ColumnRelationship' - description: List of column relationships between source and target objects. - id: - type: string - description: Unique identifier of the external lineage relationship. - properties: - type: object + type: string description: >- - Key-value properties associated with the external lineage - relationship. - required: - - source - - target - ExternalLineageTable: + The scopes to which the temporary GCP credential should apply. These + resources are the scopes that are passed to the token provider (see + https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.credentials.Credentials) + description: The GCP cloud options to customize the requested temporary credential + GenerateTemporaryTableCredentialResponse: type: object properties: - name: + aws_temp_credentials: + $ref: '#/components/schemas/AwsCredentials' + azure_aad: + $ref: '#/components/schemas/AzureActiveDirectoryToken' + azure_user_delegation_sas: + $ref: '#/components/schemas/AzureUserDelegationSas' + expiration_time: + type: integer + description: >- + Server time when the credential will expire, in epoch milliseconds. + The API client is advised to cache the credential given this + expiration time. + gcp_oauth_token: + $ref: '#/components/schemas/GcpOauthToken' + r2_temp_credentials: + $ref: '#/components/schemas/R2Credentials' + uc_encrypted_token: + $ref: '#/components/schemas/UcEncryptedToken' + url: type: string - ExternalLineageTableInfo: + description: The URL of the storage path accessible by the temporary credential. + GenerateTemporaryVolumeCredentialResponse: type: object properties: - catalog_name: - type: string - description: Name of Catalog. - event_time: - type: string - description: Timestamp of the lineage event. - name: - type: string - description: Name of Table. - schema_name: + aws_temp_credentials: + $ref: '#/components/schemas/AwsCredentials' + azure_aad: + $ref: '#/components/schemas/AzureActiveDirectoryToken' + azure_user_delegation_sas: + $ref: '#/components/schemas/AzureUserDelegationSas' + expiration_time: + type: integer + description: >- + Server time when the credential will expire, in epoch milliseconds. + The API client is advised to cache the credential given this + expiration time. + gcp_oauth_token: + $ref: '#/components/schemas/GcpOauthToken' + r2_temp_credentials: + $ref: '#/components/schemas/R2Credentials' + uc_encrypted_token: + $ref: '#/components/schemas/UcEncryptedToken' + url: type: string - description: Name of Schema. - description: Represents the table information in the lineage event. - ExternalLocationInfo: + description: The URL of the storage path accessible by the temporary credential. + GetCatalogWorkspaceBindingsResponse: type: object properties: - browse_only: - type: boolean - comment: + workspaces: + type: array + items: + type: integer + GetMetastoreSummaryResponse: + type: object + properties: + cloud: type: string - description: User-provided free-form text description. created_at: type: integer - description: >- - Time at which this external location was created, in epoch - milliseconds. + description: Time at which this metastore was created, in epoch milliseconds. created_by: type: string - description: Username of external location creator. - credential_id: - type: string - description: Unique ID of the location's storage credential. - credential_name: + description: Username of metastore creator. + default_data_access_config_id: type: string - description: Name of the storage credential used with this location. - effective_enable_file_events: - type: boolean description: >- - The effective value of `enable_file_events` after applying - server-side defaults. - enable_file_events: - type: boolean + Unique identifier of the metastore's (Default) Data Access + Configuration. + delta_sharing_organization_name: + type: string description: >- - Whether to enable file events on this external location. Default to - `true`. Set to `false` to disable file events. The actual applied - value may differ due to server-side defaults; check - `effective_enable_file_events` for the effective state. - encryption_details: - $ref: '#/components/schemas/EncryptionDetails' - fallback: + The organization name of a Delta Sharing entity, to be used in + Databricks-to-Databricks Delta Sharing as the official name. + delta_sharing_recipient_token_lifetime_in_seconds: + type: integer + description: The lifetime of delta sharing recipient token in seconds. + delta_sharing_scope: + $ref: '#/components/schemas/DeltaSharingScopeEnum' + description: The scope of Delta Sharing enabled for the metastore. + external_access_enabled: type: boolean description: >- - Indicates whether fallback mode is enabled for this external - location. When fallback mode is enabled, the access to the location - falls back to cluster credentials if UC credentials are not - sufficient. - file_event_queue: - $ref: '#/components/schemas/FileEventQueue' + Whether to allow non-DBR clients to directly access entities under + the metastore. + global_metastore_id: + type: string description: >- - File event queue settings. If `enable_file_events` is not `false`, - must be defined and have exactly one of the documented properties. - isolation_mode: - $ref: '#/components/schemas/IsolationMode' + Globally unique metastore ID across clouds and regions, of the form + ``cloud:region:metastore_id``. metastore_id: type: string - description: Unique identifier of metastore hosting the external location. + description: Unique identifier of metastore. name: type: string - description: Name of the external location. + description: The user-specified name of the metastore. owner: type: string - description: The owner of the external location. - read_only: - type: boolean - description: Indicates whether the external location is read-only. + description: The owner of the metastore. + privilege_model_version: + type: string + description: >- + Privilege model version of the metastore, of the form + ``major.minor`` (e.g., ``1.0``). + region: + type: string + description: >- + Cloud region which the metastore serves (e.g., ``us-west-2``, + ``westus``). + storage_root: + type: string + description: The storage root URL for metastore + storage_root_credential_id: + type: string + description: UUID of storage credential to access the metastore storage_root. + storage_root_credential_name: + type: string + description: Name of the storage credential to access the metastore storage_root. updated_at: type: integer description: >- - Time at which external location this was last modified, in epoch + Time at which the metastore was last modified, in epoch milliseconds. updated_by: type: string - description: Username of user who last modified the external location. - url: - type: string - description: Path URL of the external location. - ExternalMetadata: + description: Username of user who last modified the metastore. + GetPermissionsResponse: type: object properties: - name: + next_page_token: type: string - system_type: - $ref: '#/components/schemas/SystemType' - description: Type of external system. - entity_type: + privilege_assignments: + type: array + items: + $ref: '#/components/schemas/PrivilegeAssignment' + description: The privileges assigned to each principal + GetQuotaResponse: + type: object + properties: + quota_info: + $ref: '#/components/schemas/QuotaInfo' + GetWorkspaceBindingsResponse: + type: object + properties: + bindings: + type: array + items: + $ref: '#/components/schemas/WorkspaceBinding' + next_page_token: type: string - description: Type of entity within the external system. - columns: + description: >- + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + GrantOptions: + type: object + properties: + privileges: type: array items: type: string - description: List of columns associated with the external metadata object. - create_time: - type: string - description: Time at which this external metadata object was created. - created_by: - type: string - description: Username of external metadata object creator. - description: - type: string - description: User-provided free-form text description. - id: - type: string - description: Unique identifier of the external metadata object. - metastore_id: - type: string - description: Unique identifier of parent metastore. - owner: + required: + - privileges + InferenceTableConfig: + type: object + properties: + parent: type: string - description: Owner of the external metadata object. - properties: - type: object description: >- - A map of key-value properties attached to the external metadata - object. - update_time: + Parent UC schema where the inference table is created. Format: + ``schemas/{catalog}.{schema}``. Set at create time and immutable + thereafter; changing it on an existing service is rejected. + disabled: + type: boolean + description: >- + Indicates whether payload logging is disabled (opt-out). Unset means + that payload logging is active (the on-by-default state coincides + with the proto zero-value, so the server never fills this field for + a client that leaves it unset). Set ``disabled = true`` to pause + runtime logging while keeping the sub-message attached (preserving + ``parent`` and ``table_name_prefix`` for a later flip back to + active). ``parent`` remains required either way. + is_deleted: + type: boolean + description: >- + True when the bound inference TABLE has been deleted but the parent + service still references it. The dangling reference is surfaced (not + silently dropped) so callers can see the broken dependency. AI + Gateway payload logging fails closed in this state. + table: type: string - description: Time at which this external metadata object was last modified. - updated_by: + description: >- + Resolved UC table for payload logs. Format: + ``tables/{catalog}.{schema}.{table}``. + table_name_prefix: type: string - description: Username of user who last modified external metadata object. - url: + description: >- + Prefix for the inference-table's UC-registered name. The actual leaf + name UC stores is ``_payload``; the ``_payload`` + suffix is appended automatically. To find the actual UC table after + Create, read the ``table`` field on the response. Defaults to + ``_payload`` when unset. Set at create time and + immutable thereafter; changing it on an existing service is + rejected. + required: + - parent + description: |- + Inference table configuration for payload logging on a model service. + + ``parent`` is always REQUIRED when the sub-message is set; the destination UC schema is needed + to construct or rebind the payload TABLE regardless of whether payload logging is currently + active. Payload logging is active by default; set ``disabled = true`` to pause runtime logging + without dropping the table or the binding. + ListAccountMetastoreAssignmentsResponse: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + description: The metastore assignments were successfully returned. + ListAccountStorageCredentialsResponse: + type: object + properties: + storage_credentials: + type: array + items: + $ref: '#/components/schemas/StorageCredentialInfo' + description: An array of metastore storage credentials. + description: The metastore storage credentials were successfully returned. + ListAgentServicesResponse: + type: object + properties: + agent_services: + type: array + items: + $ref: '#/components/schemas/AgentService' + description: The list of agent services. + next_page_token: type: string - description: URL associated with the external metadata object. - required: - - name - - system_type - - entity_type - FailedStatus: + description: Pagination token for retrieving the next page of results. + description: Response for listing agent services. + ListCatalogsResponse: type: object properties: - last_processed_commit_version: - type: integer - description: >- - The last source table Delta version that was synced to the online - table. Note that this Delta version may only be partially synced to - the online table. Only populated if the table is still online and - available for serving. - timestamp: + catalogs: + type: array + items: + $ref: '#/components/schemas/CatalogInfo' + next_page_token: type: string description: >- - The timestamp of the last time any data was synchronized from the - source table to the online table. Only populated if the table is - still online and available for serving. - description: >- - Detailed status of an online table. Shown if the online table is in the - OFFLINE_FAILED or the - ONLINE_PIPELINE_FAILED state. - FileEventQueue: + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListConnectionsResponse: type: object properties: - managed_aqs: - $ref: '#/components/schemas/AzureQueueStorage' - managed_pubsub: - $ref: '#/components/schemas/GcpPubsub' - managed_sqs: - $ref: '#/components/schemas/AwsSqsQueue' - provided_aqs: - $ref: '#/components/schemas/AzureQueueStorage' - provided_pubsub: - $ref: '#/components/schemas/GcpPubsub' - provided_sqs: - $ref: '#/components/schemas/AwsSqsQueue' - ForeignKeyConstraint: + connections: + type: array + items: + $ref: '#/components/schemas/ConnectionInfo' + next_page_token: + type: string + description: >- + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListCredentialsResponse: type: object properties: - name: - type: string - child_columns: + credentials: type: array items: - type: string - description: Column names for this constraint. - parent_table: + $ref: '#/components/schemas/CredentialInfo' + next_page_token: type: string - description: The full name of the parent constraint. - parent_columns: + description: >- + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListEffectivePrivilegeAssignmentsResponse: + type: object + properties: + effective_privilege_assignments: type: array items: - type: string - description: Column names for this constraint. - rely: - type: boolean - description: True if the constraint is RELY, false or unset if NORELY. - required: - - name - - child_columns - - parent_table - - parent_columns - FunctionArgument: + $ref: '#/components/schemas/EffectivePrivilegeAssignment' + next_page_token: + type: string + description: >- + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListEntityTagAssignmentsResponse: type: object properties: - alias: + next_page_token: type: string - constant: + tag_assignments: + type: array + items: + $ref: '#/components/schemas/EntityTagAssignment' + description: The list of tag assignments + ListExternalLineageRelationshipsResponse: + type: object + properties: + external_lineage_relationships: + type: array + items: + $ref: '#/components/schemas/ExternalLineageInfo' + next_page_token: type: string - description: A constant literal. - FunctionDependency: + ListExternalLocationsResponse: type: object properties: - function_full_name: + external_locations: + type: array + items: + $ref: '#/components/schemas/ExternalLocationInfo' + next_page_token: type: string description: >- - Full name of the dependent function, in the form of - __catalog_name__.__schema_name__.__function_name__. - required: - - function_full_name - description: A function that is dependent on a SQL object. - FunctionInfo: + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListExternalMetadataResponse: type: object properties: - browse_only: - type: boolean - catalog_name: - type: string - description: Name of parent Catalog. - comment: - type: string - description: User-provided free-form text description. - created_at: - type: integer - description: Time at which this function was created, in epoch milliseconds. - created_by: - type: string - description: Username of function creator. - data_type: - $ref: '#/components/schemas/ColumnTypeName' - description: Scalar function return data type. - external_language: - type: string - description: External function language. - external_name: - type: string - description: External function name. - full_data_type: + external_metadata: + type: array + items: + $ref: '#/components/schemas/ExternalMetadata' + next_page_token: type: string - description: Pretty printed function data type. - full_name: + ListFunctionsResponse: + type: object + properties: + functions: + type: array + items: + $ref: '#/components/schemas/FunctionInfo' + next_page_token: type: string description: >- - Full name of Function, in form of - **catalog_name**.**schema_name**.**function_name** - function_id: - type: string - description: Id of Function, relative to parent schema. - input_params: - $ref: '#/components/schemas/FunctionParameterInfos' - description: Function input parameters. - is_deterministic: - type: boolean - description: Whether the function is deterministic. - is_null_call: - type: boolean - description: Function null call. - metastore_id: - type: string - description: Unique identifier of parent metastore. - name: - type: string - description: Name of function, relative to parent schema. - owner: + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListMcpServicesResponse: + type: object + properties: + mcp_services: + type: array + items: + $ref: '#/components/schemas/McpService' + description: The list of MCP services. + next_page_token: type: string - description: Username of current owner of the function. - parameter_style: - $ref: '#/components/schemas/FunctionInfoParameterStyle' - description: Function parameter style. **S** is the value for SQL. - properties: + description: Pagination token for retrieving the next page of results. + description: Response for listing MCP services. + ListMetastoresResponse: + type: object + properties: + metastores: + type: array + items: + $ref: '#/components/schemas/MetastoreInfo' + next_page_token: type: string - description: JSON-serialized key-value pair map, encoded (escaped) as a string. - return_params: - $ref: '#/components/schemas/FunctionParameterInfos' - description: Table function return parameters. - routine_body: - $ref: '#/components/schemas/FunctionInfoRoutineBody' description: >- - Function language. When **EXTERNAL** is used, the language of the - routine function should be specified in the **external_language** - field, and the **return_params** of the function cannot be used (as - **TABLE** return type is not supported), and the **sql_data_access** - field must be **NO_SQL**. - routine_definition: - type: string - description: Function body. - routine_dependencies: - $ref: '#/components/schemas/DependencyList' - description: function dependencies. - schema_name: - type: string - description: Name of parent Schema relative to its parent Catalog. - security_type: - $ref: '#/components/schemas/FunctionInfoSecurityType' - description: Function security type. - specific_name: - type: string - description: Specific name of the function; Reserved for future use. - sql_data_access: - $ref: '#/components/schemas/FunctionInfoSqlDataAccess' - description: Function SQL data access. - sql_path: + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListModelProviderServicesResponse: + type: object + properties: + model_provider_services: + type: array + items: + $ref: '#/components/schemas/ModelProviderService' + description: The list of model provider services. + next_page_token: type: string - description: >- - List of schemes whose objects can be referenced without - qualification. - updated_at: - type: integer - description: >- - Time at which this function was last modified, in epoch - milliseconds. - updated_by: + description: Pagination token for retrieving the next page of results. + description: Response for listing model provider services. + ListModelServicesResponse: + type: object + properties: + model_services: + type: array + items: + $ref: '#/components/schemas/ModelService' + description: The list of model services. + next_page_token: type: string - description: Username of user who last modified the function. - FunctionParameterInfo: + description: Pagination token for retrieving the next page of results. + description: Response for listing model services. + ListModelVersionsResponse: type: object properties: - name: + model_versions: + type: array + items: + $ref: '#/components/schemas/ModelVersionInfo' + next_page_token: type: string - type_text: + description: >- + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request (for the next page of results). + ListPoliciesResponse: + type: object + properties: + next_page_token: type: string - description: Full data type spec, SQL/catalogString text. - type_name: - $ref: '#/components/schemas/ColumnTypeName' - description: Name of type (INT, STRUCT, MAP, etc.) - position: - type: integer - description: Ordinal position of column (starting at position 0). - comment: + policies: + type: array + items: + $ref: '#/components/schemas/PolicyInfo' + description: The list of retrieved policies. + ListPrivilegeAssignmentsResponse: + type: object + properties: + next_page_token: type: string - description: User-provided free-form text description. - parameter_default: + privilege_assignments: + type: array + items: + $ref: '#/components/schemas/PrivilegeAssignment' + ListQuotasResponse: + type: object + properties: + next_page_token: type: string - description: Default value of the parameter. - parameter_mode: - $ref: '#/components/schemas/FunctionParameterMode' - description: Function parameter mode. - parameter_type: - $ref: '#/components/schemas/FunctionParameterType' - description: Function parameter type. - type_interval_type: + quotas: + type: array + items: + $ref: '#/components/schemas/QuotaInfo' + description: An array of returned QuotaInfos. + ListRegisteredModelsResponse: + type: object + properties: + next_page_token: type: string - description: Format of IntervalType. - type_json: + registered_models: + type: array + items: + $ref: '#/components/schemas/RegisteredModelInfo' + ListSchemasResponse: + type: object + properties: + next_page_token: type: string - description: Full data type spec, JSON-serialized. - type_precision: - type: integer - description: Digits of precision; required on Create for DecimalTypes. - type_scale: - type: integer - description: Digits to right of decimal; Required on Create for DecimalTypes. - required: - - name - - type_text - - type_name - - position - FunctionParameterInfos: + schemas: + type: array + items: + $ref: '#/components/schemas/SchemaInfo' + description: An array of schema information objects. + ListSecretsResponse: type: object properties: - parameters: + next_page_token: + type: string + description: >- + Opaque token to retrieve the next page of results. Absent if there + are no more pages. **page_token** should be set to this value for + the next request. + secrets: type: array items: - $ref: '#/components/schemas/FunctionParameterInfo' - GcpOauthToken: + $ref: '#/components/schemas/Secret' + description: An array of secret objects. + description: Response message for ListSecrets. + ListStorageCredentialsResponse: type: object properties: - oauth_token: + next_page_token: type: string - description: |- - GCP temporary credentials for API authentication. Read more at - https://developers.google.com/identity/protocols/oauth2/service-account - GcpPubsub: + storage_credentials: + type: array + items: + $ref: '#/components/schemas/StorageCredentialInfo' + ListSystemSchemasResponse: type: object properties: - managed_resource_id: + next_page_token: type: string - subscription_name: + schemas: + type: array + items: + $ref: '#/components/schemas/SystemSchemaInfo' + description: An array of system schema information objects. + ListTableSummariesResponse: + type: object + properties: + next_page_token: type: string - description: >- - The Pub/Sub subscription name in the format - projects/{project}/subscriptions/{subscription name}. Only required - for provided_pubsub. - GenerateTemporaryPathCredentialResponse: + tables: + type: array + items: + $ref: '#/components/schemas/TableSummary' + description: List of table summaries. + ListTablesResponse: type: object properties: - aws_temp_credentials: - $ref: '#/components/schemas/AwsCredentials' - azure_aad: - $ref: '#/components/schemas/AzureActiveDirectoryToken' - azure_user_delegation_sas: - $ref: '#/components/schemas/AzureUserDelegationSas' - expiration_time: - type: integer - description: >- - Server time when the credential will expire, in epoch milliseconds. - The API client is advised to cache the credential given this - expiration time. - gcp_oauth_token: - $ref: '#/components/schemas/GcpOauthToken' - r2_temp_credentials: - $ref: '#/components/schemas/R2Credentials' - url: + next_page_token: type: string - description: The URL of the storage path accessible by the temporary credential. - GenerateTemporaryServiceCredentialAzureOptions: + tables: + type: array + items: + $ref: '#/components/schemas/TableInfo' + description: An array of table information objects. + ListVolumesResponseContent: type: object properties: - resources: + next_page_token: + type: string + volumes: type: array items: - type: string + $ref: '#/components/schemas/VolumeInfo' + MatchColumn: + type: object + properties: + alias: + type: string + condition: + type: string + description: The condition expression used to match a table column. + McpService: + type: object + properties: + browse_only: + type: boolean description: >- - The resources to which the temporary Azure credential should apply. - These resources are the scopes that are passed to the token provider - (see - https://learn.microsoft.com/python/api/azure-core/azure.core.credentials.tokencredential?view=azure-python) - description: The Azure cloud options to customize the requested temporary credential - GenerateTemporaryServiceCredentialGcpOptions: + Whether the caller sees only metadata available through the BROWSE + privilege. + comment: + type: string + description: User-provided description. + config: + $ref: '#/components/schemas/McpServiceConfig' + description: >- + Operational configuration: connection, tool selectors, rate limit. + Required on CreateMcpService; on UpdateMcpService it is required + only when ``config`` (or a ``config.*`` subpath) appears in + ``update_mask``. + create_time: + type: string + format: date-time + description: When the MCP service was created. + created_by: + type: string + description: Creator identity. + effective_owner: + type: string + description: >- + The resolved owner of the MCP service. Falls back to the caller's + identity when ``owner`` is not explicitly set on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated from the + entity's state and returned on every read. To use it as an if-match + precondition on a mutation, echo the last-read value back via the + dedicated ``etag`` field on the Update / Delete request; the server + rejects the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the MCP service. + name: + type: string + description: >- + Resource name of the MCP service. Format: + ``mcp-services/{catalog}.{schema}.{mcp_service}``. Each ``{...}`` + component is capped at 255 characters individually. Server-derived + on Create from ``parent`` + ``mcp_service_id``; required and + immutable on Update/Get/Delete. + owner: + type: string + description: >- + The owner of the MCP service. Write-only; read owner via + effective_owner. + update_time: + type: string + format: date-time + description: When the MCP service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: >- + A governed MCP server registration in Unity Catalog. Acts as a container + securable that + references an MCP server -- customer-external via a UC Connection, or Databricks-hosted via an + internal server -- and exposes its tools for discovery, authorization, and invocation. + McpServiceConfig: type: object properties: - scopes: + include_tool_selectors: type: array items: type: string description: >- - The scopes to which the temporary GCP credential should apply. These - resources are the scopes that are passed to the token provider (see - https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.credentials.Credentials) - description: The GCP cloud options to customize the requested temporary credential - GenerateTemporaryTableCredentialResponse: + Glob or exact-match patterns selecting which tools from the MCP + server to expose. Prefix match for patterns with ``*``, exact match + otherwise. An empty list means all tools are included. Per-element + max 256 chars. + rate_limits: + type: array + items: + $ref: '#/components/schemas/RateLimit' + description: >- + Per-principal rate limits applied to tool invocations routed through + this MCP service. Repeated to support per-USER / USER_GROUP / + SERVICE_PRINCIPAL / SERVICE / USER_DEFAULT scopes simultaneously, + mirroring the ``ModelServiceConfig.rate_limits`` shape. Empty when + no rate limit is configured. + source_connection: + $ref: '#/components/schemas/McpServiceConfigSourceConnection' + description: UC Connection referencing the MCP server. + description: >- + Operational configuration for an MCP service. Groups the source + reference, tool selectors, and + rate limit -- the fields that configure how the MCP service behaves at invocation time. + McpServiceConfigSourceConnection: type: object properties: - aws_temp_credentials: - $ref: '#/components/schemas/AwsCredentials' - azure_aad: - $ref: '#/components/schemas/AzureActiveDirectoryToken' - azure_user_delegation_sas: - $ref: '#/components/schemas/AzureUserDelegationSas' - expiration_time: - type: integer - description: >- - Server time when the credential will expire, in epoch milliseconds. - The API client is advised to cache the credential given this - expiration time. - gcp_oauth_token: - $ref: '#/components/schemas/GcpOauthToken' - r2_temp_credentials: - $ref: '#/components/schemas/R2Credentials' - url: + name: type: string - description: The URL of the storage path accessible by the temporary credential. - GetCatalogWorkspaceBindingsResponse: + is_deleted: + type: boolean + required: + - name + description: >- + UC Connection that hosts the MCP server. On create, provide ``name`` in + the schema-scoped form + ``connections/{catalog}.{schema}.{connection}``. On read, the service populates the resolved + connection metadata and preserves a dangling source so callers can diagnose a deleted backing + connection. + MetastoreAssignment: type: object properties: - workspaces: - type: array - items: - type: integer - GetMetastoreSummaryResponse: + workspace_id: + type: integer + metastore_id: + type: string + description: The unique ID of the metastore. + default_catalog_name: + type: string + description: >- + The name of the default catalog in the metastore. This field is + deprecated. Please use "Default Namespace API" to configure the + default catalog for a Databricks workspace. + required: + - workspace_id + - metastore_id + MetastoreInfo: type: object properties: cloud: @@ -14200,7 +20815,7 @@ components: type: string description: >- Globally unique metastore ID across clouds and regions, of the form - `cloud:region:metastore_id`. + ``cloud:region:metastore_id``. metastore_id: type: string description: Unique identifier of metastore. @@ -14213,13 +20828,13 @@ components: privilege_model_version: type: string description: >- - Privilege model version of the metastore, of the form `major.minor` - (e.g., `1.0`). + Privilege model version of the metastore, of the form + ``major.minor`` (e.g., ``1.0``). region: type: string description: >- - Cloud region which the metastore serves (e.g., `us-west-2`, - `westus`). + Cloud region which the metastore serves (e.g., ``us-west-2``, + ``westus``). storage_root: type: string description: The storage root URL for metastore @@ -14237,355 +20852,850 @@ components: updated_by: type: string description: Username of user who last modified the metastore. - GetPermissionsResponse: + ModelProviderService: type: object properties: - next_page_token: + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through the BROWSE + privilege. + comment: type: string - privilege_assignments: - type: array - items: - $ref: '#/components/schemas/PrivilegeAssignment' - description: The privileges assigned to each principal - GetQuotaResponse: - type: object - properties: - quota_info: - $ref: '#/components/schemas/QuotaInfo' - GetWorkspaceBindingsResponse: - type: object - properties: - bindings: - type: array - items: - $ref: '#/components/schemas/WorkspaceBinding' - next_page_token: + description: User-provided description. + config: + $ref: '#/components/schemas/ModelProviderServiceConfig' + description: >- + Behavioral configuration: provider connection, model catalog, and + passthrough policy. See ``ModelProviderServiceConfig`` for the + per-field contract. Required on CreateModelProviderService; on + Update it is required only when ``config`` (or a ``config.*`` + subpath) appears in ``update_mask``. + create_time: + type: string + format: date-time + description: When the provider service was created. + created_by: + type: string + description: Creator identity. + effective_owner: type: string description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListAccountMetastoreAssignmentsResponse: + The resolved owner of the model provider service. Falls back to the + caller's identity when ``owner`` is not explicitly set on creation. + etag: + type: string + description: >- + Optimistic concurrency control token. Server-generated from the + entity's state and returned on every read. To use it as an if-match + precondition on a mutation, echo the last-read value back via the + dedicated ``etag`` field on the Update / Delete request; the server + rejects the mutation if the stored etag differs. + metastore_id: + type: string + description: Metastore hosting the provider service. + name: + type: string + description: >- + Resource name of the provider service. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at 255 characters individually. + Server-derived on Create from ``parent`` + + ``model_provider_service_id``; required and immutable on + Update/Get/Delete. + owner: + type: string + description: >- + The owner of the model provider service. Write-only; read owner via + effective_owner. + update_time: + type: string + format: date-time + description: When the provider service was last modified. + updated_by: + type: string + description: Identity of the last updater. + description: >- + A governed external model-provider connection stored in Unity Catalog + (e.g. an OpenAI API + account, an Azure OpenAI deployment, an Amazon Bedrock account). Owns the provider type and the + auth/configuration the platform needs to invoke that provider, and is referenced from + ``ExternalModelConfig.model_provider_service`` on a ModelService. + + One ModelProviderService can back many ModelServices (e.g. an ``openai_prod`` provider serving + multiple models); a single ModelService can fan out across multiple ModelProviderServices for + traffic split or failover. + ModelProviderServiceConfig: type: object properties: - workspace_ids: + allow_all_targets: + type: boolean + description: >- + When true, accepts any model exposed by the upstream provider; + ``targets`` is not required and does not restrict routability. When + false, only models listed in ``targets`` are routable. + amazon_bedrock: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAmazonBedrockProviderConfig + anthropic: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAnthropicProviderConfig + azure_openai: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAzureOpenAiProviderConfig + custom: + $ref: '#/components/schemas/ModelProviderServiceConfigCustomProviderConfig' + forward_headers: + type: boolean + description: >- + Whether to forward incoming request headers to the upstream + provider. Applies to managed (multi-model) requests as well as + passthrough requests served by this provider service. + Governance-level decision by the provider service owner; not + selectable per inference call. + forward_query_parameters: + type: boolean + description: >- + Whether to forward incoming request query parameters to the upstream + provider. Same trust-boundary semantics as ``forward_headers``. + forward_unmanaged_paths: + type: boolean + description: >- + Whether to forward request paths that fall outside this service's + managed API set to the upstream provider as opaque passthrough. When + true, requests addressed to subpaths not recognized by the managed + API surface are proxied to the upstream provider over the same + provider connection. When false, only managed-API paths are served. + Governance-level decision by the provider service owner; expanding + this expands the trust boundary that the ModelProviderService + exposes. + gemini_enterprise: + $ref: >- + #/components/schemas/ModelProviderServiceConfigGeminiEnterpriseProviderConfig + inference_table: + $ref: '#/components/schemas/InferenceTableConfig' + description: >- + Inference table configuration for payload logging when this provider + service is invoked directly. When it is invoked through a model + service, the model service's own inference table captures the + invocation instead. Mirrors ``ModelServiceConfig.inference_table`` / + ``AgentServiceConfig.inference_table``. + microsoft_foundry: + $ref: >- + #/components/schemas/ModelProviderServiceConfigMicrosoftFoundryProviderConfig + openai: + $ref: '#/components/schemas/ModelProviderServiceConfigOpenAiProviderConfig' + provider_type: + $ref: >- + #/components/schemas/ModelProviderServiceConfigExternalModelProviderType + description: >- + Provider type discriminator. Required at create time; immutable + after. Determines which variant of the ``provider`` oneof must be + set. May not be changed via Update; attempts to include + ``config.provider_type`` in + ``UpdateModelProviderServiceRequest.update_mask`` are rejected. + Required on CreateModelProviderService and immutable thereafter. + rate_limits: type: array items: - type: integer - description: The metastore assignments were successfully returned. - ListAccountStorageCredentialsResponse: - type: object - properties: - storage_credentials: + $ref: '#/components/schemas/RateLimit' + description: >- + Rate limits applied when this provider service is invoked directly. + When it is invoked through a model service, the model service's own + ``rate_limits`` apply instead. Mirrors + ``ModelServiceConfig.rate_limits`` / + ``McpServiceConfig.rate_limits``. + targets: type: array items: - $ref: '#/components/schemas/StorageCredentialInfo' - description: An array of metastore storage credentials. - description: The metastore storage credentials were successfully returned. - ListCatalogsResponse: + $ref: '#/components/schemas/ModelProviderServiceConfigModelTargetConfig' + description: >- + Routing targets this provider service exposes (provider-side model + identifier + unified API types per entry). Required (>=1) when + ``allow_all_targets = false``; optional and additive when + ``allow_all_targets = true``. References from + ``ExternalModelConfig.target`` must match an entry here unless + ``allow_all_targets = true``. + description: >- + Behavioral configuration for a ModelProviderService: provider connection + (auth + + provider-specific fields), the catalog of models this provider service can route to, and the + passthrough policy that governs how request headers, query parameters, and unmanaged subpaths + cross the trust boundary to the upstream provider. + ModelProviderServiceConfigAmazonBedrockProviderConfig: type: object properties: - catalogs: - type: array - items: - $ref: '#/components/schemas/CatalogInfo' - next_page_token: - type: string - description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListConnectionsResponse: + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAmazonBedrockProviderDirectConfig + description: Amazon Bedrock provider configuration. + ModelProviderServiceConfigAmazonBedrockProviderDirectConfig: type: object properties: - connections: - type: array - items: - $ref: '#/components/schemas/ConnectionInfo' - next_page_token: + aws_access_key: + $ref: '#/components/schemas/ModelProviderServiceConfigAwsAccessKey' + description: >- + AWS access-key-pair auth. Mutually exclusive with + ``service_credential``. Supersedes the flat ``aws_access_key_id`` / + ``aws_secret_access_key`` fields. + aws_access_key_id: + type: string + description: >- + Deprecated flat AWS access key ID. Superseded by + ``aws_access_key.access_key_id``. Kept for one migration cycle; the + handler mirrors it to/from ``aws_access_key``. Treated as + username-equivalent (not a secret value): round-trips on reads and + is scrubbed from audit logs. + aws_secret_access_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Deprecated flat AWS secret access key. Superseded by + ``aws_access_key.secret_access_key``. Kept for one migration cycle; + the handler mirrors it to/from ``aws_access_key``. Supplied as + inline plaintext via ``ProviderSecret.plaintext``. + region: type: string description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListCredentialsResponse: + AWS region where the Bedrock endpoint is hosted (e.g., + ``us-east-1``). Required on Create. + service_credential: + $ref: '#/components/schemas/ModelProviderServiceConfigServiceCredential' + description: >- + Reference to a UC service credential authorizing Bedrock requests. + On Create the caller supplies ``service_credential.name`` in the + AIP-122 resource-name form ``credentials/{name}``. Required on + Create when using UC-service-credential auth; mutually exclusive + with ``aws_access_key``. The credential is referenced by name; its + value is not carried here. On read the resolved ``id`` and + ``is_deleted`` are also populated. Only supported on AWS-hosted + workspaces; Create requests from other clouds are rejected with + INVALID_PARAMETER_VALUE. + description: |- + Direct form of Amazon Bedrock provider config. + + Authentication is one of two mutually exclusive modes, exactly one of which must be supplied on + Create: + + - Access keys: set ``aws_access_key``, leave ``service_credential`` unset. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``aws_access_key`` unset. The credential value lives in UC and + is referenced by name, not held on this message. Setting more than one mode is rejected. + ModelProviderServiceConfigAnthropicProviderConfig: type: object properties: - credentials: - type: array - items: - $ref: '#/components/schemas/CredentialInfo' - next_page_token: - type: string + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAnthropicProviderDirectConfig description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListEntityTagAssignmentsResponse: + Direct (inline-credentials) form: caller supplies the API key in the + request body. Required on Create unless ``relayed`` is set. + relayed: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAnthropicProviderRelayedConfig + description: >- + Relayed (credential-less) form: no Anthropic credential is stored. + Each inference request instead carries the caller's own OAuth token, + which the platform forwards to Anthropic on outbound requests. + Mutually exclusive with ``direct``; no ``api_key`` is required or + persisted. + description: >- + Anthropic provider configuration. Exactly one of ``direct`` or + ``relayed`` must be set on + Create; the two are mutually exclusive. + ModelProviderServiceConfigAnthropicProviderDirectConfig: type: object properties: - next_page_token: - type: string - tag_assignments: - type: array - items: - $ref: '#/components/schemas/EntityTagAssignment' - description: The list of tag assignments - ListExternalLineageRelationshipsResponse: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Anthropic API key. Required on Create. Sent as the ``x-api-key`` + header on outbound requests. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + description: Direct form of Anthropic provider config. + ModelProviderServiceConfigAnthropicProviderRelayedConfig: type: object properties: - external_lineage_relationships: - type: array - items: - $ref: '#/components/schemas/ExternalLineageInfo' - next_page_token: - type: string - ListExternalLocationsResponse: + plan_type: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAnthropicProviderRelayedConfigAnthropicRelayedPlanType + description: >- + Which Anthropic subscription tier the relayed token belongs to. + Optional; when unset the MPS gets the full governance surface (see + TEAM_ENTERPRISE). Immutable after Create, so the tier cannot be + flipped in place. + description: >- + Relayed form of Anthropic provider config: no credential is stored. + Authentication is the + caller's own OAuth token, forwarded to Anthropic on outbound requests, so there is no persisted + secret. Presence of this variant is the signal that the provider service uses relayed auth; + ``plan_type`` further distinguishes which Anthropic subscription tier the token belongs to. + ModelProviderServiceConfigAwsAccessKey: type: object properties: - external_locations: - type: array - items: - $ref: '#/components/schemas/ExternalLocationInfo' - next_page_token: + access_key_id: type: string description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListExternalMetadataResponse: + AWS access key ID. Required on Create when using access-key auth. + Treated as username-equivalent (not a secret value): round-trips on + reads and is scrubbed from audit logs. + secret_access_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + AWS secret access key paired with ``access_key_id``. Required on + Create when using access-key auth. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + description: 'AWS access-key-pair auth for Amazon Bedrock: a SigV4-signing key pair.' + ModelProviderServiceConfigAzureOpenAiProviderConfig: type: object properties: - external_metadata: - type: array - items: - $ref: '#/components/schemas/ExternalMetadata' - next_page_token: - type: string - ListFunctionsResponse: + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigAzureOpenAiProviderDirectConfig + description: Azure OpenAI provider configuration. + ModelProviderServiceConfigAzureOpenAiProviderDirectConfig: type: object properties: - functions: - type: array - items: - $ref: '#/components/schemas/FunctionInfo' - next_page_token: - type: string + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListMetastoresResponse: - type: object - properties: - metastores: - type: array - items: - $ref: '#/components/schemas/MetastoreInfo' - next_page_token: + Azure OpenAI API key. Mutually exclusive with the Entra and + service-credential modes. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + base_url: type: string description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListModelVersionsResponse: - type: object - properties: - model_versions: - type: array - items: - $ref: '#/components/schemas/ModelVersionInfo' - next_page_token: + Full Azure OpenAI endpoint base URL, e.g. + ``https://myresource.openai.azure.com``. Required on Create. + client_id: type: string description: >- - Opaque token to retrieve the next page of results. Absent if there - are no more pages. __page_token__ should be set to this value for - the next request (for the next page of results). - ListPoliciesResponse: + Deprecated flat Entra client ID. Superseded by + ``entra_service_principal.client_id``. Kept for one migration cycle; + the handler mirrors it to/from ``entra_service_principal``. + client_secret: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Deprecated flat Entra client secret. Superseded by + ``entra_service_principal.client_secret``. Kept for one migration + cycle; the handler mirrors it to/from ``entra_service_principal``. + Supplied as inline plaintext via ``ProviderSecret.plaintext``. + entra_service_principal: + $ref: '#/components/schemas/ModelProviderServiceConfigEntraServicePrincipal' + description: >- + Entra ID (service principal) auth. Mutually exclusive with + ``api_key`` and ``service_credential``. Supersedes the flat + ``tenant_id`` / ``client_id`` / ``client_secret`` fields. + service_credential: + $ref: '#/components/schemas/ModelProviderServiceConfigServiceCredential' + description: >- + Reference to a UC service credential authorizing Azure OpenAI + requests. On Create the caller supplies ``service_credential.name`` + in the AIP-122 resource-name form ``credentials/{name}``. Required + on Create when using UC-service-credential auth; mutually exclusive + with ``api_key`` and ``entra_service_principal``. The credential is + referenced by name; its value is not carried here. On read the + resolved ``id`` and ``is_deleted`` are also populated. Only + supported on Azure-hosted workspaces; Create requests from other + clouds are rejected with INVALID_PARAMETER_VALUE. + tenant_id: + type: string + description: >- + Deprecated flat Entra tenant ID. Superseded by + ``entra_service_principal.tenant_id``. Kept for one migration cycle; + the handler mirrors it to/from ``entra_service_principal``. + description: >- + Direct form of Azure OpenAI provider config. Exactly one of three + mutually-exclusive auth modes + must be supplied on Create: + + - API key: set ``api_key``, leave ``entra_service_principal`` and ``service_credential`` unset. + - Entra ID (service principal): set ``entra_service_principal``, leave ``api_key`` and + ``service_credential`` unset. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``api_key`` and ``entra_service_principal`` unset. The + credential value lives in UC and is referenced by name, not held on this message. Only + supported on Azure-hosted workspaces. Setting more than one mode is rejected. + ModelProviderServiceConfigCustomProviderConfig: type: object properties: - next_page_token: - type: string - policies: - type: array - items: - $ref: '#/components/schemas/PolicyInfo' - description: The list of retrieved policies. - ListQuotasResponse: + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigCustomProviderDirectConfig + description: >- + Custom provider configuration: arbitrary HTTP endpoint with bearer-token + auth. + ModelProviderServiceConfigCustomProviderDirectConfig: type: object properties: - next_page_token: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Bearer token forwarded as the ``Authorization: Bearer ...`` header + on outbound requests. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. Set this for bearer-token auth. + base_url: type: string - quotas: - type: array - items: - $ref: '#/components/schemas/QuotaInfo' - description: An array of returned QuotaInfos. - ListRegisteredModelsResponse: + description: >- + Endpoint URL of the OpenAI-compatible service (e.g., + ``https://api.example.com/v1``). Required on Create. + description: |- + Direct form of custom provider config. + + Authentication is one of two mutually exclusive modes, exactly one of which must be supplied on + Create: + + - Bearer: set ``api_key``, leave ``header_auth`` unset. The secret is forwarded as + ``Authorization: Bearer ``. + - Header: set ``header_auth``, leave ``api_key`` unset. The secret is forwarded as + ``: ``. Setting both modes or neither mode is rejected. + ModelProviderServiceConfigEntraServicePrincipal: type: object properties: - next_page_token: + client_id: type: string - registered_models: - type: array - items: - $ref: '#/components/schemas/RegisteredModelInfo' - ListSchemasResponse: + description: Entra ID client (application) ID. Required on Create. + client_secret: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Entra ID client secret. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + tenant_id: + type: string + description: Entra ID (Azure AD) tenant ID. Required on Create. + description: >- + Entra ID (Azure AD) service-principal auth: AI Gateway exchanges the + ``tenant_id`` + + ``client_id`` identify the service principal, and the ``credential`` oneof proves that identity, + exchanged for an Entra bearer token on outbound requests via the OAuth2 client-credentials + grant. Shared by the Azure OpenAI and Microsoft Foundry provider configs. + ModelProviderServiceConfigGeminiEnterpriseProviderConfig: type: object properties: - next_page_token: - type: string - schemas: - type: array - items: - $ref: '#/components/schemas/SchemaInfo' - description: An array of schema information objects. - ListStorageCredentialsResponse: + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigGeminiEnterpriseProviderDirectConfig + description: Gemini Enterprise provider configuration. + ModelProviderServiceConfigGeminiEnterpriseProviderDirectConfig: type: object properties: - next_page_token: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Google Gemini Enterprise API key. Required on Create. Supplied as + inline plaintext via ``ProviderSecret.plaintext``. + project_id: type: string - storage_credentials: - type: array - items: - $ref: '#/components/schemas/StorageCredentialInfo' - ListSystemSchemasResponse: + description: >- + GCP project ID hosting the Gemini Enterprise endpoint. Required on + Create. + region: + type: string + description: >- + GCP region of the Gemini Enterprise endpoint (e.g., + ``us-central1``). Required on Create. + description: Direct form of Gemini Enterprise provider config. + ModelProviderServiceConfigMicrosoftFoundryProviderConfig: type: object properties: - next_page_token: - type: string - schemas: - type: array - items: - $ref: '#/components/schemas/SystemSchemaInfo' - description: An array of system schema information objects. - ListTableSummariesResponse: + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigMicrosoftFoundryProviderDirectConfig + description: Microsoft Foundry provider configuration. + ModelProviderServiceConfigMicrosoftFoundryProviderDirectConfig: type: object properties: - next_page_token: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Microsoft AI Foundry API key. Mutually exclusive with the Entra and + service-credential modes. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + base_url: type: string - tables: - type: array - items: - $ref: '#/components/schemas/TableSummary' - description: List of table summaries. - ListTablesResponse: + description: Microsoft AI Foundry endpoint URL. Required on Create. + client_id: + type: string + description: >- + Deprecated flat Entra client ID. Superseded by + ``entra_service_principal.client_id``. Kept for one migration cycle; + the handler mirrors it to/from ``entra_service_principal``. + client_secret: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + Deprecated flat Entra client secret. Superseded by + ``entra_service_principal.client_secret``. Kept for one migration + cycle; the handler mirrors it to/from ``entra_service_principal``. + Supplied as inline plaintext via ``ProviderSecret.plaintext``. + entra_service_principal: + $ref: '#/components/schemas/ModelProviderServiceConfigEntraServicePrincipal' + description: >- + Entra ID (service principal) auth. Mutually exclusive with + ``api_key`` and ``service_credential``. Supersedes the flat + ``tenant_id`` / ``client_id`` / ``client_secret`` fields. + service_credential: + $ref: '#/components/schemas/ModelProviderServiceConfigServiceCredential' + description: >- + Reference to a UC service credential authorizing Microsoft Foundry + requests. On Create the caller supplies ``service_credential.name`` + in the AIP-122 resource-name form ``credentials/{name}``. Required + on Create when using UC-service-credential auth; mutually exclusive + with ``api_key`` and ``entra_service_principal``. The credential is + referenced by name; its value is not carried here. On read the + resolved ``id`` and ``is_deleted`` are also populated. Only + supported on Azure-hosted workspaces; Create requests from other + clouds are rejected with INVALID_PARAMETER_VALUE. + tenant_id: + type: string + description: >- + Deprecated flat Entra tenant ID. Superseded by + ``entra_service_principal.tenant_id``. Kept for one migration cycle; + the handler mirrors it to/from ``entra_service_principal``. + description: |- + Direct form of Microsoft Foundry provider config. + + Authentication is one of three mutually exclusive modes, exactly one of which must be supplied + on Create: + + - API key: set ``api_key``, leave ``entra_service_principal`` and ``service_credential`` unset. + - Entra ID (service principal): set ``entra_service_principal``, leave ``api_key`` and + ``service_credential`` unset. AI Gateway exchanges these for an Entra bearer token on outbound + requests via the OAuth2 client-credentials grant. + - UC service credential: set ``service_credential.name`` to the AIP-122 resource-name form + ``credentials/{name}``, leave ``api_key`` and ``entra_service_principal`` unset. The + credential value lives in UC and is referenced by name, not held on this message. Only + supported on Azure-hosted workspaces. Setting more than one mode is rejected. + ModelProviderServiceConfigModelTargetConfig: type: object properties: - next_page_token: + model: type: string - tables: + description: >- + Provider-side model identifier (e.g. "gpt-5", "claude-opus-4-7"). + This is a string on the LLM provider's side, not a UC entity. The UC + governance hook for external destinations is the + ModelProviderService referenced by + ``ExternalModelConfig.model_provider_service``, not the model + itself. + native_api_types: type: array items: - $ref: '#/components/schemas/TableInfo' - description: An array of table information objects. - ListVolumesResponseContent: + type: string + description: >- + Provider-native API types the model supports (e.g. + "openai/v1/chat/completions"). Used by the platform for + request/response translation from the unified API type. At most 64 + entries of at most 256 characters each; the list is persisted into + the destination binding's bounded storage envelope. + required: + - model + description: Model target configuration for an external model destination. + ModelProviderServiceConfigOpenAiProviderConfig: type: object properties: - next_page_token: - type: string - volumes: - type: array - items: - $ref: '#/components/schemas/VolumeInfo' - MatchColumn: + direct: + $ref: >- + #/components/schemas/ModelProviderServiceConfigOpenAiProviderDirectConfig + description: OpenAI provider configuration. + ModelProviderServiceConfigOpenAiProviderDirectConfig: type: object properties: - alias: + api_key: + $ref: '#/components/schemas/ModelProviderServiceConfigProviderSecret' + description: >- + OpenAI API key. Required on Create. Supplied as inline plaintext via + ``ProviderSecret.plaintext``. + base_url: type: string - condition: + description: >- + Optional custom base URL. Defaults to ``https://api.openai.com/v1``. + Use for OpenAI-API-compatible third-party endpoints or in-network + proxies. + organization: type: string - description: The condition expression used to match a table column. - MetastoreAssignment: + description: >- + Optional OpenAI organization ID. When set, the platform forwards it + as the ``OpenAI-Organization`` header. + description: Direct (inline-credentials) form of the OpenAI provider config. + ModelProviderServiceConfigProviderSecret: type: object properties: - workspace_id: - type: integer - metastore_id: + plaintext: type: string - description: The unique ID of the metastore. - default_catalog_name: + description: >- + Inline plaintext credential. INPUT_ONLY: the value never round-trips + on reads. Get and List responses omit ``plaintext``; the field's + presence in the read shape only indicates that a secret is + configured. + description: >- + A secret value supplied as part of an inline provider config. The caller + supplies the value as + inline ``plaintext`` on writes; the platform stores it encrypted. The ``plaintext`` field is + ``INPUT_ONLY`` and never round-trips on reads. + ModelProviderServiceConfigServiceCredential: + type: object + properties: + name: type: string description: >- - The name of the default catalog in the metastore. This field is - deprecated. Please use "Default Namespace API" to configure the - default catalog for a Databricks workspace. + Resource name of the bound UC service credential, in the AIP-122 + form ``credentials/{name}`` (a metastore-level single-part + credential name). On create the caller supplies the name here. On + read it reflects the credential's current name at read time. required: - - workspace_id - - metastore_id - MetastoreInfo: + - name + description: >- + ---- Provider configuration (nested; see the ``provider`` oneof below) + ---- The customer-owned + UC service credential a ModelProviderService uses to authenticate to its provider, referenced by + name. + ModelService: type: object properties: - cloud: + browse_only: + type: boolean + description: >- + Whether the caller sees only metadata available through the BROWSE + privilege. + comment: type: string - created_at: - type: integer - description: Time at which this metastore was created, in epoch milliseconds. - created_by: + description: User-provided description. + config: + $ref: '#/components/schemas/ModelServiceConfig' + description: >- + Operational configuration: destinations, routing, rate limits, + inference table. Required on CreateModelService; on + UpdateModelService it is required only when ``config`` (or a + ``config.*`` subpath) appears in ``update_mask``. + create_time: type: string - description: Username of metastore creator. - default_data_access_config_id: + format: date-time + description: When the model service was created. + created_by: type: string - description: >- - Unique identifier of the metastore's (Default) Data Access - Configuration. - delta_sharing_organization_name: + description: Creator identity. + effective_owner: type: string description: >- - The organization name of a Delta Sharing entity, to be used in - Databricks-to-Databricks Delta Sharing as the official name. - delta_sharing_recipient_token_lifetime_in_seconds: - type: integer - description: The lifetime of delta sharing recipient token in seconds. - delta_sharing_scope: - $ref: '#/components/schemas/DeltaSharingScopeEnum' - description: The scope of Delta Sharing enabled for the metastore. - external_access_enabled: - type: boolean - description: >- - Whether to allow non-DBR clients to directly access entities under - the metastore. - global_metastore_id: + The resolved owner of the ModelService. Falls back to the caller's + identity when ``owner`` is not explicitly set on creation. + etag: type: string description: >- - Globally unique metastore ID across clouds and regions, of the form - `cloud:region:metastore_id`. + Optimistic concurrency control token. Server-generated from the + entity's state and returned on every read. To use it as an if-match + precondition on a mutation, echo the last-read value back via the + dedicated ``etag`` field on the Update / Delete request; the server + rejects the mutation if the stored etag differs. metastore_id: type: string - description: Unique identifier of metastore. + description: Metastore hosting the model service. name: type: string - description: The user-specified name of the metastore. + description: >- + Resource name of the model service. Format: + ``model-services/{catalog}.{schema}.{model_service}``. Each + ``{...}`` component is capped at 255 characters individually. + Server-derived on Create from ``parent`` + ``model_service_id``; + required and immutable on Update/Get/Delete. owner: - type: string - description: The owner of the metastore. - privilege_model_version: type: string description: >- - Privilege model version of the metastore, of the form `major.minor` - (e.g., `1.0`). - region: - type: string + The owner of the model service. Write-only; read owner via + effective_owner. + supported_api_types: + type: array + items: + type: string description: >- - Cloud region which the metastore serves (e.g., `us-west-2`, - `westus`). - storage_root: + Unified API types this endpoint supports (e.g. "chat", "embeddings", + "completions"). Derived from the destinations' backing models / + providers at read time. + update_time: type: string - description: The storage root URL for metastore - storage_root_credential_id: + format: date-time + description: When the model service was last modified. + updated_by: type: string - description: UUID of storage credential to access the metastore storage_root. - storage_root_credential_name: + description: Identity of the last updater. + description: >- + A governed AI Gateway endpoint in Unity Catalog that routes inference + requests to one or more + model destinations (for example a foundation model or an external LLM reached through a + ModelProviderService). Applies centralized access control, rate limits, guardrails, and auditing + to the traffic it serves. + ModelServiceConfig: + type: object + properties: + inference_table: + $ref: '#/components/schemas/InferenceTableConfig' + description: Inference table config for payload logging. + rate_limits: + type: array + items: + $ref: '#/components/schemas/RateLimit' + description: Rate limits applied to requests routed through this model service. + routing: + $ref: '#/components/schemas/ModelServiceConfigRoutingConfig' + description: 'Routing configuration: destinations, routing strategy, and fallback.' + description: Operational configuration wrapped around the ModelService resource. + ModelServiceConfigDestinationConfig: + type: object + properties: + name: type: string - description: Name of the storage credential to access the metastore storage_root. - updated_at: + description: User-facing label for this destination, used in routing references. + destination_type: + $ref: >- + #/components/schemas/ModelServiceConfigDestinationConfigDestinationType + description: Backing-model category. Determines which oneof variant is populated. + external_model_config: + $ref: '#/components/schemas/ModelServiceConfigExternalModelConfig' + is_deleted: + type: boolean + description: >- + True when the destination's backing UC entity (MODEL for + foundation-model destinations, MODEL_PROVIDER_SERVICE for external + destinations) has been deleted but the destination row still + references it. The dangling destination is surfaced (not silently + dropped) so callers can see the broken routing. Inference traffic + through this destination fails closed (BAD_REQUEST / + FAILED_PRECONDITION). + pay_per_token_config: + $ref: '#/components/schemas/ModelServiceConfigPayPerTokenConfig' + provisioned_throughput_config: + $ref: '#/components/schemas/ModelServiceConfigProvisionedThroughputConfig' + traffic_percentage: type: integer description: >- - Time at which the metastore was last modified, in epoch - milliseconds. - updated_by: + Share of traffic sent to this destination, 0-100. Optional on + fallback destinations; see FallbackConfig. + required: + - name + - destination_type + description: >- + A destination the model service can route traffic to. Exactly one of the + per-type configs inside + ``type_config`` must be set, and it must match ``destination_type``. + ModelServiceConfigExternalModelConfig: + type: object + properties: + model_provider_service: type: string - description: Username of user who last modified the metastore. + description: >- + Resource name of the governed ModelProviderService that owns + provider auth and provider-specific configuration. The referenced + ModelProviderService also carries the provider type, so this message + does not surface it directly. Format: + ``model-provider-services/{catalog}.{schema}.{model_provider_service}``. + Each ``{...}`` component is capped at 255 characters individually. + target: + $ref: '#/components/schemas/ModelProviderServiceConfigModelTargetConfig' + description: >- + Routing target for the destination: the provider-side model selected + from the referenced ModelProviderService's ``targets`` catalog, plus + the unified API types the platform should translate to/from at + request time. + required: + - model_provider_service + - target + description: >- + Configuration for an external-foundation-model destination. Provider + auth and provider-specific + cloud configuration are owned by a separate, governed ModelProviderService entity referenced via + ``model_provider_service``; the platform resolves the provider at invocation time. + ModelServiceConfigFallbackConfig: + type: object + properties: + destinations: + type: array + items: + $ref: '#/components/schemas/ModelServiceConfigDestinationConfig' + description: >- + Ordered list of fallback destinations. Traversal is in list order; + the attempt count is the length of the list. At most 5 are allowed. + description: >- + Fallback routing, applied after the primary destination returns a + retryable error. Traversal is + in list order; the attempt count is the length of the list. + ModelServiceConfigPayPerTokenConfig: + type: object + properties: + model: + type: string + description: >- + Resource name of the UC model. Format: + ``models/{catalog}.{schema}.{model}``. + required: + - model + description: >- + Configuration for a pay-per-token foundation-model destination. + Identifies the foundation model + by its UC resource name; the platform resolves it to a Model Serving endpoint at request time. + ModelServiceConfigProvisionedThroughputConfig: + type: object + properties: + model_serving_endpoint: + type: string + description: >- + Name of the backing Model Serving endpoint serving the provisioned- + throughput foundation model, as the AIP-122 typed resource name + ``serving-endpoints/{name}``. The same UC model can be served on + multiple Model Serving endpoints (different throughput / region / + config); the caller picks which one this destination routes to. The + endpoint must exist at create time. + model: + type: string + description: >- + UC model FQN of the model served by the backing endpoint (e.g., + ``system.ai.databricks-claude-opus-4-6``). Resolved from Model + Serving at Create/Update time. + required: + - model_serving_endpoint + description: >- + Configuration for a provisioned-throughput foundation-model destination. + References a + pre-existing Model Serving endpoint that serves the model; sizing (provisioned throughput, burst + scaling, model version) is owned by the Model Serving endpoint itself, not by this message. + ModelServiceConfigRoutingConfig: + type: object + properties: + destinations: + type: array + items: + $ref: '#/components/schemas/ModelServiceConfigDestinationConfig' + description: >- + Primary routing destinations. At most 10 are allowed. At least one + is required on CreateModelService; on UpdateModelService it is + required only when ``config.routing`` (or a ``config.routing.*`` + subpath) appears in ``update_mask``. + fallback: + $ref: '#/components/schemas/ModelServiceConfigFallbackConfig' + description: Fallback routing config, applied after primary destinations fail. + first_token_timeout: + type: string + description: >- + Timeout for the first token of a streaming response. If a + destination does not return its first token within this duration, AI + Gateway aborts the attempt and fails over to the next destination. + Applies to streaming requests only. Leave unset for no first-token + timeout. + traffic_splitting: + $ref: '#/components/schemas/ModelServiceConfigRoutingConfigTrafficSplitting' + description: >- + Marker message selecting request-based traffic splitting. Traffic is + distributed according to each destination's traffic_percentage + value; no configuration lives on this message itself. + description: >- + Routing configuration for a model service, nesting destinations, routing + strategy, and fallback + under a single sub-message. + ModelServiceConfigRoutingConfigTrafficSplitting: + type: object + properties: {} + description: >- + Marker message selecting request-based traffic splitting across primary + destinations. Split + weights are read from each DestinationConfig.traffic_percentage. ModelVersionInfo: type: object properties: @@ -14734,7 +21844,7 @@ components: type: string description: >- [Create:ERR Update:IGN] UC table to monitor. Format: - `catalog.schema.table_name` + ``catalog.schema.table_name`` status: $ref: '#/components/schemas/MonitorInfoStatus' description: '[Create:ERR Update:IGN] The monitor status.' @@ -14742,12 +21852,12 @@ components: type: string description: >- [Create:ERR Update:IGN] Table that stores profile metrics data. - Format: `catalog.schema.table_name`. + Format: ``catalog.schema.table_name``. drift_metrics_table_name: type: string description: >- [Create:ERR Update:IGN] Table that stores drift metrics data. - Format: `catalog.schema.table_name`. + Format: ``catalog.schema.table_name``. monitor_version: type: integer description: >- @@ -14765,7 +21875,7 @@ components: type: string description: >- [Create:OPT Update:OPT] Baseline table name. Baseline data is used - to compute drift from the data in the monitored `table_name`. The + to compute drift from the data in the monitored ``table_name``. The baseline table and the monitored table shall have the same schema. custom_metrics: type: array @@ -14801,10 +21911,10 @@ components: [Create:OPT Update:OPT] List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and - its complements. For example `slicing_exprs=[“col_1”, “col_2 > 10”]` - will generate the following slices: two slices for `col_2 > 10` - (True and False), and one slice per unique value in `col1`. For - high-cardinality columns, only the top 100 unique values by + its complements. For example ``slicing_exprs=[“col_1”, “col_2 > + 10”]`` will generate the following slices: two slices for ``col_2 > + 10`` (True and False), and one slice per unique value in ``col1``. + For high-cardinality columns, only the top 100 unique values by frequency will generate slices. snapshot: $ref: '#/components/schemas/MonitorSnapshot' @@ -14829,9 +21939,8 @@ components: type: string description: >- Jinja template for a SQL expression that specifies how to compute - the metric. See [create metric definition]. [create metric - definition]: - https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition + the metric. See `create metric definition + `__. input_columns: type: array items: @@ -14959,6 +22068,43 @@ components: destinations are: "catalog", "external_location", "connection", "credential", and "metastore". The **destination_type** of a **special_destination** is always EMAIL. + OneLakeEventQueue: + type: object + properties: + consumer_group: + type: string + description: >- + Event Hubs consumer group used to consume file events. Defaults to + "$Default" when unset. Recommended for provided_onelake: create a + dedicated consumer group on the Event Hub for file events to avoid + contending with the customer's other consumers. + event_hub_name: + type: string + description: >- + Event Hub entity name within the namespace. Only required for + provided_onelake. + event_hub_url: + type: string + description: >- + The Event Hub URL in the format + https://{namespace}.servicebus.windows.net/{event_hub_name}. + Deprecated: use fully_qualified_namespace + event_hub_name instead. + fully_qualified_namespace: + type: string + description: >- + The fully qualified domain name of the Event Hubs namespace, e.g. + {yournamespace}.servicebus.windows.net. Only required for + provided_onelake. + managed_resource_id: + type: string + description: >- + Unique identifier included in the name of the file events managed + resources. + description: >- + File event queue for OneLake (Microsoft Fabric) locations. Events flow + through Fabric + Eventstream in both arms; CSMS consumes from a user-provided Azure Event Hub (provided_onelake) + or from a Fabric Eventstream that CSMS provisions in the user's workspace (managed_onelake). OnlineTable: type: object properties: @@ -15061,6 +22207,11 @@ components: For drop down / radio button selections, UI will want to know the possible input values, it can also be used by other option types to limit input selections. + conditional_display: + $ref: '#/components/schemas/ConditionalDisplay' + description: >- + Conditional display configuration. Specifies when this option should + be hidden based on another option's value. default_value: type: string description: >- @@ -15136,6 +22287,15 @@ components: The principal whose privileges we are changing. Only one of principal or principal_id should be specified, never both at the same time. + principal_id: + type: integer + description: >- + An opaque internal ID that identifies the principal whose privileges + should be removed. This field is intended for removing privileges + associated with a deleted user. When set, only the entries specified + in the remove field are processed; any entries in the add field will + be rejected. Only one of principal or principal_id should be + specified, never both at the same time. remove: type: array items: @@ -15191,16 +22351,16 @@ components: $ref: '#/components/schemas/SecurableType' description: >- Type of securables that the policy should take effect on. Only - `TABLE` is supported at this moment. Required on create and optional - on update. + ``TABLE`` is supported at this moment. Required on create and + optional on update. policy_type: $ref: '#/components/schemas/PolicyType' description: Type of the policy. Required on create. column_mask: $ref: '#/components/schemas/ColumnMaskOptions' description: >- - Options for column mask policies. Valid only if `policy_type` is - `POLICY_TYPE_COLUMN_MASK`. Required on create and optional on + Options for column mask policies. Valid only if ``policy_type`` is + ``POLICY_TYPE_COLUMN_MASK``. Required on create and optional on update. When specified on update, the new options will replace the existing options as a whole. comment: @@ -15214,6 +22374,13 @@ components: created_by: type: string description: Username of the user who created the policy. Output only. + deny: + $ref: '#/components/schemas/DenyOptions' + description: >- + Options for deny policies. Valid only if ``policy_type`` is + ``POLICY_TYPE_DENY``. Required on create and optional on update. + When specified on update, the new options will replace the existing + options as a whole. except_principals: type: array items: @@ -15221,6 +22388,13 @@ components: description: >- Optional list of user or group names that should be excluded from the policy. + grant: + $ref: '#/components/schemas/GrantOptions' + description: >- + Options for grant policies. Valid only if ``policy_type`` is + ``POLICY_TYPE_GRANT``. Required on create and optional on update. + When specified on update, the new options will replace the existing + options as a whole. id: type: string description: >- @@ -15232,14 +22406,14 @@ components: $ref: '#/components/schemas/MatchColumn' description: >- Optional list of condition expressions used to match table columns. - Only valid when `for_securable_type` is `TABLE`. When specified, the - policy only applies to tables whose columns satisfy all match + Only valid when ``for_securable_type`` is ``TABLE``. When specified, + the policy only applies to tables whose columns satisfy all match conditions. name: type: string description: >- Name of the policy. Required on create and optional on update. To - rename the policy, set `name` to a different value on update. + rename the policy, set ``name`` to a different value on update. on_securable_fullname: type: string description: >- @@ -15249,15 +22423,15 @@ components: $ref: '#/components/schemas/SecurableType' description: >- Type of the securable on which the policy is defined. Only - `CATALOG`, `SCHEMA` and `TABLE` are supported at this moment. + ``CATALOG``, ``SCHEMA`` and ``TABLE`` are supported at this moment. Required on create. row_filter: $ref: '#/components/schemas/RowFilterOptions' description: >- - Options for row filter policies. Valid only if `policy_type` is - `POLICY_TYPE_ROW_FILTER`. Required on create and optional on update. - When specified on update, the new options will replace the existing - options as a whole. + Options for row filter policies. Valid only if ``policy_type`` is + ``POLICY_TYPE_ROW_FILTER``. Required on create and optional on + update. When specified on update, the new options will replace the + existing options as a whole. updated_at: type: integer description: >- @@ -15266,6 +22440,17 @@ components: updated_by: type: string description: Username of the user who last modified the policy. Output only. + use_session_identity: + type: boolean + description: >- + Temporary for migrating customers to session identity. Customers not + currently using ABAC will not be able to set this field to false and + all new policies will have this field default to true. Existing + customers will have this field default to false, but can set it to + true to opt in to session identity. after a grace period, this field + will be removed and all policies will use session identity. Only for + row filter and column mask policies. Not applicable to deny + policies. when_condition: type: string description: Optional condition when the policy should take effect. @@ -15306,6 +22491,11 @@ components: properties: principal: type: string + principal_id: + type: integer + description: >- + Unique identifier of the principal. For active principals, both + ``principal`` and ``principal_id`` are present. privileges: type: array items: @@ -15368,6 +22558,51 @@ components: description: |- R2 temporary credentials for API authentication. Read more at https://developers.cloudflare.com/r2/api/s3/tokens/. + RateLimit: + type: object + properties: + key: + $ref: '#/components/schemas/RateLimitRateLimitKey' + description: Scope key. Determines whether ``principal`` is required. + renewal_period: + $ref: '#/components/schemas/RateLimitRateLimitRenewalPeriod' + description: Renewal period. + principal: + type: string + description: >- + Principal this limit applies to: user email, group name, or service + principal application ID. Required unless ``key`` is + ``RATE_LIMIT_KEY_SERVICE``, ``RATE_LIMIT_KEY_USER_DEFAULT``, or + ``RATE_LIMIT_KEY_REQUEST_TAG`` (which must not set a principal). + request_tag_key: + type: string + description: >- + Request tag key this limit applies to. Required when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``, forbidden otherwise. + request_tag_value: + type: string + description: >- + Request tag value this limit applies to. Only valid when ``key`` is + ``RATE_LIMIT_KEY_REQUEST_TAG``. Leave unset to apply the limit to + every value of ``request_tag_key`` (an any-value default); a set + value is a specific override for that value. + requests: + type: integer + description: >- + Max requests allowed within a renewal period. Leave unset for no + request limit. + tokens: + type: integer + description: >- + Max tokens allowed within a renewal period. Leave unset for no token + limit. + required: + - key + - renewal_period + description: >- + A rate limit applied to service requests. Leave ``requests`` or + ``tokens`` unset to impose no + limit on that dimension; set a value to cap that dimension within the renewal period. RegenerateDashboardResponse: type: object properties: @@ -15481,10 +22716,6 @@ components: properties: browse_only: type: boolean - description: >- - Indicates whether the principal is limited to retrieving metadata - for the associated object through the BROWSE privilege when - include_browse is enabled in the request. catalog_name: type: string description: Name of parent catalog. @@ -15500,6 +22731,9 @@ components: created_by: type: string description: Username of schema creator. + custom_max_retention_hours: + type: integer + description: Custom maximum retention period in hours for the schema. effective_predictive_optimization_flag: $ref: '#/components/schemas/EffectivePredictiveOptimizationFlag' enable_predictive_optimization: @@ -15509,7 +22743,7 @@ components: and objects under it. full_name: type: string - description: Full name of schema, in form of __catalog_name__.__schema_name__. + description: Full name of schema, in form of **catalog_name**.**schema_name**. metastore_id: type: string description: Unique identifier of parent metastore. @@ -15518,26 +22752,117 @@ components: description: Name of schema, relative to parent catalog. owner: type: string - description: Username of current owner of schema. - properties: - type: object - description: A map of key-value properties attached to the securable. - schema_id: + description: Username of current owner of schema. + properties: + type: object + description: A map of key-value properties attached to the securable. + schema_id: + type: string + description: The unique identifier of the schema. + storage_location: + type: string + description: Storage location for managed tables within schema. + storage_root: + type: string + description: Storage root URL for managed tables within schema. + updated_at: + type: integer + description: Time at which this schema was created, in epoch milliseconds. + updated_by: + type: string + description: Username of user who last modified schema. + Secret: + type: object + properties: + name: + type: string + description: The name of the secret, relative to its parent schema. + catalog_name: + type: string + description: The name of the catalog where the schema and the secret reside. + schema_name: + type: string + description: The name of the schema where the secret resides. + value: + type: string + description: >- + The secret value to store. This field is input-only and is not + returned in responses — use the **effective_value** field (via + GetSecret with **include_value** set to true) to read the secret + value. The maximum size is 60 KiB (pre-encryption). Accepted content + includes passwords, tokens, keys, and other sensitive credential + data. + comment: + type: string + description: User-provided free-form text description of the secret. + create_time: + type: string + format: date-time + description: The time at which this secret was created. + created_by: + type: string + description: The principal that created the secret. + effective_owner: + type: string + description: >- + The effective owner of the secret, which may differ from the + directly-set **owner** due to inheritance. + effective_value: + type: string + description: >- + The secret value. Only populated in responses when you have the + **READ_SECRET** privilege and **include_value** is set to true in + the request. The maximum size is 60 KiB. + expire_time: + type: string + format: date-time + description: >- + User-provided expiration time of the secret. This field indicates + when the secret should no longer be used and may be displayed as a + warning in the UI. It is purely informational and does not trigger + any automatic actions or affect the secret's lifecycle. + full_name: + type: string + description: >- + The three-level (fully qualified) name of the secret, in the form of + **catalog_name.schema_name.secret_name**. + metastore_id: type: string - description: The unique identifier of the schema. - storage_location: + description: Unique identifier of the metastore hosting the secret. + owner: type: string - description: Storage location for managed tables within schema. - storage_root: + description: >- + The owner of the secret. Defaults to the creating principal on + creation. Can be updated to transfer ownership of the secret to + another principal. + update_time: type: string - description: Storage root URL for managed tables within schema. - updated_at: - type: integer - description: Time at which this schema was created, in epoch milliseconds. + format: date-time + description: The time at which this secret was last updated. updated_by: type: string - description: Username of user who last modified schema. - description: 'Next ID: 45' + description: The principal that last updated the secret. + required: + - name + - catalog_name + - schema_name + - value + description: >- + A secret stored in Unity Catalog. Secrets are three-level namespace + objects + (catalog.schema.secret) that securely store sensitive credential data such as passwords, tokens, + and keys. + SecretDependency: + type: object + properties: + secret_full_name: + type: string + description: >- + Full name of the dependent secret, in the form of + **catalog_name**.**schema_name**.**secret_name**. + required: + - secret_full_name + description: A secret that is dependent on a SQL object. Securable: type: object properties: @@ -15705,7 +23030,7 @@ components: description: >- A table constraint, as defined by *one* of the following fields being set: - __primary_key_constraint__, __foreign_key_constraint__, __named_table_constraint__. + **primary_key_constraint**, **foreign_key_constraint**, **named_table_constraint**. TableDependency: type: object properties: @@ -15713,7 +23038,7 @@ components: type: string description: >- Full name of the dependent table, in the form of - __catalog_name__.__schema_name__.__table_name__. + **catalog_name**.**schema_name**.**table_name**. required: - table_full_name description: A table that is dependent on a SQL object. @@ -15740,7 +23065,7 @@ components: type: array items: $ref: '#/components/schemas/ColumnInfo' - description: The array of __ColumnInfo__ definitions of the table's columns. + description: The array of **ColumnInfo** definitions of the table's columns. comment: type: string description: User-provided free-form text description. @@ -15775,7 +23100,7 @@ components: type: string description: >- Full name of table, in form of - __catalog_name__.__schema_name__.__table_name__ + **catalog_name**.**schema_name**.**table_name** metastore_id: type: string description: Unique identifier of parent metastore. @@ -15822,7 +23147,7 @@ components: $ref: '#/components/schemas/TableConstraint' description: >- List of table constraints. Note: this field is not set in the output - of the __listTables__ API. + of the **listTables** API. table_id: type: string description: The unique identifier of the table. @@ -15837,7 +23162,7 @@ components: view_definition: type: string description: >- - View definition SQL (when __table_type__ is **VIEW**, + View definition SQL (when **table_type** is **VIEW**, **MATERIALIZED_VIEW**, or **STREAMING_TABLE**) view_dependencies: $ref: '#/components/schemas/DependencyList' @@ -15848,7 +23173,7 @@ components: empty list, the dependency is provided but is empty; - when DependencyList is not an empty list, dependencies are provided and recorded. Note: this field is not set in the output of the - __listTables__ API. + **listTables** API. TableRowFilter: type: object properties: @@ -15887,6 +23212,16 @@ components: has. table_type: $ref: '#/components/schemas/TableType' + TagIntrospectionExpression: + type: object + properties: + column_tag_value: + $ref: '#/components/schemas/ColumnTagValueExtraction' + description: Extracts the value of a column-level tag. + tag_value: + $ref: '#/components/schemas/TagValueExtraction' + description: Extracts the value of a securable-level tag. + description: An expression that introspects tags at query time. TagKeyValue: type: object properties: @@ -15895,6 +23230,15 @@ components: value: type: string description: value of the tag associated with the key, could be optional + TagValueExtraction: + type: object + properties: + tag_key: + type: string + description: 1024 matches the max_length on FunctionArgument.constant above. + required: + - tag_key + description: 'Extracts the value of a securable-level tag: get_tag_value("tagKey").' TemporaryCredentials: type: object properties: @@ -15910,6 +23254,8 @@ components: expiration time. gcp_oauth_token: $ref: '#/components/schemas/GcpOauthToken' + r2_temp_credentials: + $ref: '#/components/schemas/R2Credentials' TriggeredUpdateStatus: type: object properties: @@ -15931,6 +23277,16 @@ components: Detailed status of an online table. Shown if the online table is in the ONLINE_TRIGGERED_UPDATE or the ONLINE_NO_PENDING_UPDATE state. + UcEncryptedToken: + type: object + properties: + encrypted_payload: + type: string + description: Stores encrypted ScopedCloudToken as a base64-encoded string + description: >- + Encrypted token used when we cannot downscope the cloud provider token + appropriately See: + https://docs.google.com/document/d/1hEKDnSckuU5PIS798CtfqBElrMR6OJuR2wgz_BjhMSY UnassignResponse: type: object properties: {} @@ -15956,8 +23312,8 @@ components: privilege_model_version: type: string description: >- - Privilege model version of the metastore, of the form `major.minor` - (e.g., `1.0`). + Privilege model version of the metastore, of the form + ``major.minor`` (e.g., ``1.0``). storage_root_credential_id: type: string description: UUID of storage credential to access the metastore storage_root. @@ -16094,6 +23450,17 @@ components: result: $ref: '#/components/schemas/ValidationResultResult' description: The results of the tested operation. + VolumeDependency: + type: object + properties: + volume_full_name: + type: string + description: >- + Full name of the dependent volume, in the form of + **catalog_name**.**schema_name**.**volume_name**. + required: + - volume_full_name + description: A volume that is dependent on a SQL object. VolumeInfo: type: object properties: @@ -16150,8 +23517,8 @@ components: The type of the volume. An external volume is located in the specified external location. A managed volume is located in the default location which is specified by the parent schema, or the - parent catalog, or the Metastore. [Learn more] [Learn more]: - https://docs.databricks.com/aws/en/volumes/managed-vs-external + parent catalog, or the Metastore. `Learn more + `__ WorkspaceBinding: type: object properties: @@ -16162,6 +23529,18 @@ components: description: One of READ_WRITE/READ_ONLY. Default is READ_WRITE. required: - workspace_id + AgentServiceAgentServiceType: + type: string + x-enum: + - AGENT_SERVICE_TYPE_EXTERNAL + description: >- + Type of agent service. Determines how the backing agent surface is + provisioned and how + + invocation routes to the runtime. Nested to scope enum values + (guardrail: + + avoid-generic-enum-names). ArtifactType: type: string x-enum: @@ -16174,6 +23553,7 @@ components: x-enum: - ISOLATED - OPEN + - OPEN_IN_ACCOUNT description: |- Create a collection of name/value pairs. @@ -16232,6 +23612,7 @@ components: - DATE - DECIMAL - DOUBLE + - FILE - FLOAT - GEOGRAPHY - GEOMETRY @@ -16243,7 +23624,9 @@ components: - SHORT - STRING - STRUCT + - TABLEREF_TYPE - TABLE_TYPE + - TIME - TIMESTAMP - TIMESTAMP_NTZ - USER_DEFINED_TYPE @@ -16288,27 +23671,173 @@ components: ConnectionType: type: string x-enum: + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AMPLITUDE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS + - BIGLAKE - BIGQUERY + - CERIDIAN_DAYFORCE + - CONFLUENCE - DATABRICKS + - DELIGHTED + - DYNAMICS365 + - EPIC_CLARITY + - FRESHSERVICE + - FRONT - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB - GLUE + - GMAIL + - GONG + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_CLOUD_LAKEHOUSE + - GOOGLE_WORKSPACE + - GURU + - HIBOB - HIVE_METASTORE - HTTP + - HUBSPOT + - ICEBERG_REST + - IRONCLAD + - JDBC + - KINESIS + - LINEAR + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MONDAY_COM - MYSQL + - NETSKOPE_LOGS + - NOTION - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PALANTIR + - PARTNERSTACK + - PENDO - POSTGRESQL - POWER_BI + - PUBSUB + - QUICKBOOKS - REDSHIFT - SALESFORCE - SALESFORCE_DATA_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID - SERVICENOW + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SMARTSHEET + - SNAPCHAT_ADS - SNOWFLAKE + - SPLUNK - SQLDW - SQLSERVER + - SQUARE - TERADATA - UNKNOWN_CONNECTION_TYPE + - VERKADA - WORKDAY_RAAS - description: 'Next Id: 72' + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + ConversionInfoState: + type: string + x-enum: + - COMPLETED + - IN_PROGRESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. CreateFunctionParameterStyle: type: string x-enum: @@ -16524,7 +24053,11 @@ components: - ANY_STATIC_CREDENTIAL - BEARER_TOKEN - EDGEGRID_AKAMAI + - GENERIC_TOKEN_EXCHANGE + - INLINE_YAML - OAUTH_ACCESS_TOKEN + - OAUTH_DCR + - OAUTH_GOOGLE_SERVICE_ACCOUNT - OAUTH_M2M - OAUTH_MTLS - OAUTH_REFRESH_TOKEN @@ -16537,7 +24070,43 @@ components: - SSWS_TOKEN - UNKNOWN_CREDENTIAL_TYPE - USERNAME_PASSWORD - description: 'Next Id: 17' + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. DataSourceFormat: type: string x-enum: @@ -16657,6 +24226,48 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + DrReplicationStatus: + type: string + x-enum: + - DR_REPLICATION_STATUS_PRIMARY + - DR_REPLICATION_STATUS_SECONDARY + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. EffectivePredictiveOptimizationFlagInheritedFromType: @@ -16999,6 +24610,7 @@ components: x-enum: - ISOLATION_MODE_ISOLATED - ISOLATION_MODE_OPEN + - ISOLATION_MODE_OPEN_IN_ACCOUNT description: |- Create a collection of name/value pairs. @@ -17078,11 +24690,84 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + ListMcpServicesRequestView: + type: string + x-enum: + - BASIC + - FULL + description: >- + Controls which fields are populated on each McpService in the response. + The server treats unset + + / VIEW_UNSPECIFIED as BASIC. Callers needing the full configuration must + request it explicitly + + with ``view = FULL``. + ListModelProviderServicesRequestView: + type: string + x-enum: + - BASIC + - FULL + description: >- + Controls which fields are populated on each ModelProviderService in the + response. The server + + treats unset / VIEW_UNSPECIFIED as BASIC. Callers needing the full + configuration must request it + + explicitly with ``view = FULL``. + ListModelServicesRequestView: + type: string + x-enum: + - BASIC + - FULL + description: >- + Controls which fields are populated on each ModelService in the + response. The server treats + + unset / VIEW_UNSPECIFIED as BASIC. Callers needing the full + configuration must request it + + explicitly with ``view = FULL``. MatchType: type: string x-enum: - PREFIX_MATCH description: The artifact pattern matching type + ModelProviderServiceConfigAnthropicProviderRelayedConfigAnthropicRelayedPlanType: + type: string + x-enum: + - ANTHROPIC_RELAYED_PLAN_TYPE_MAX + - ANTHROPIC_RELAYED_PLAN_TYPE_TEAM_ENTERPRISE + description: >- + Which Anthropic subscription tier the relayed OAuth token belongs to. + Immutable after Create + + (switching tiers changes which governance controls the platform + enforces). Only MAX and + + TEAM_ENTERPRISE differ in the governance surface the gateway can + enforce, not in how the token + + is relayed. + ModelProviderServiceConfigExternalModelProviderType: + type: string + x-enum: + - EXTERNAL_MODEL_PROVIDER_TYPE_AMAZON_BEDROCK + - EXTERNAL_MODEL_PROVIDER_TYPE_ANTHROPIC + - EXTERNAL_MODEL_PROVIDER_TYPE_AZURE_OPENAI + - EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM + - EXTERNAL_MODEL_PROVIDER_TYPE_GEMINI_ENTERPRISE + - EXTERNAL_MODEL_PROVIDER_TYPE_MICROSOFT_FOUNDRY + - EXTERNAL_MODEL_PROVIDER_TYPE_OPENAI + description: External LLM provider for an EXTERNAL_FOUNDATION_MODEL destination. + ModelServiceConfigDestinationConfigDestinationType: + type: string + x-enum: + - DESTINATION_TYPE_EXTERNAL_FOUNDATION_MODEL + - DESTINATION_TYPE_PAY_PER_TOKEN_FOUNDATION_MODEL + - DESTINATION_TYPE_PROVISIONED_THROUGHPUT_FOUNDATION_MODEL + description: Backing-model category for a model service destination. ModelVersionInfoStatus: type: string x-enum: @@ -17245,14 +24930,17 @@ components: ``"CUSTOM_METRIC_TYPE_DRIFT"`` compare metrics across baseline and input table, or across the - two consecutive time windows. - CUSTOM_METRIC_TYPE_AGGREGATE: only - depend on the existing + two consecutive time windows. + + + - CUSTOM_METRIC_TYPE_AGGREGATE: only depend on the existing columns in + your table - columns in your table - CUSTOM_METRIC_TYPE_DERIVED: depend on previously - computed aggregate + - CUSTOM_METRIC_TYPE_DERIVED: depend on previously computed aggregate + metrics - metrics - CUSTOM_METRIC_TYPE_DRIFT: depend on previously computed - aggregate or derived metrics + - CUSTOM_METRIC_TYPE_DRIFT: depend on previously computed aggregate or + derived metrics MonitorRefreshInfoState: type: string x-enum: @@ -17350,6 +25038,7 @@ components: - OPTION_MULTILINE_STRING - OPTION_NUMBER - OPTION_SERVICE_CREDENTIAL + - OPTION_STORAGE_CREDENTIAL - OPTION_STRING description: >- Type of the option, we purposely follow JavaScript types so that the UI @@ -17406,6 +25095,8 @@ components: type: string x-enum: - POLICY_TYPE_COLUMN_MASK + - POLICY_TYPE_DENY + - POLICY_TYPE_GRANT - POLICY_TYPE_ROW_FILTER description: |- Create a collection of name/value pairs. @@ -17495,40 +25186,70 @@ components: - APPLY_TAG - BROWSE - CREATE + - CREATE_CASE_COLLECTION - CREATE_CATALOG - CREATE_CLEAN_ROOM - CREATE_CONNECTION + - CREATE_DATASOURCE - CREATE_EXTERNAL_LOCATION - CREATE_EXTERNAL_TABLE - CREATE_EXTERNAL_VOLUME + - CREATE_FEATURE + - CREATE_FLOW - CREATE_FOREIGN_CATALOG - CREATE_FOREIGN_SECURABLE - CREATE_FUNCTION - CREATE_MANAGED_STORAGE - CREATE_MATERIALIZED_VIEW + - CREATE_MEMORY_STORE - CREATE_MODEL - CREATE_PROVIDER - CREATE_RECIPIENT + - CREATE_RULE - CREATE_SCHEMA + - CREATE_SEMANTIC_GRAPH + - CREATE_SERVICE - CREATE_SERVICE_CREDENTIAL - CREATE_SHARE + - CREATE_SKILL - CREATE_STORAGE_CREDENTIAL + - CREATE_STREAM - CREATE_TABLE - CREATE_VIEW - CREATE_VOLUME + - DELETE + - DELETE_EVENTS + - DELETE_SECURITY_DATA - EXECUTE - EXECUTE_CLEAN_ROOM_TASK - EXTERNAL_USE_SCHEMA + - INSERT + - INSERT_SECURITY_DATA - MANAGE + - MANAGE_ACCESS + - MANAGE_ACCESS_CONTROL - MANAGE_ALLOWLIST + - MANAGE_GRANTS - MODIFY - MODIFY_CLEAN_ROOM + - READ_EVENTS + - READ_FEATURE - READ_FILES + - READ_FLOW + - READ_MEMORY_STORE + - READ_METADATA - READ_PRIVATE_FILES + - READ_SECURITY_DATA + - READ_SEMANTIC_GRAPH + - READ_SKILL + - READ_STREAM - READ_VOLUME - REFRESH - SELECT - SET_SHARE_PERMISSION + - UPDATE + - UPDATE_EVENTS + - UPDATE_SECURITY_DATA - USAGE - USE_CATALOG - USE_CONNECTION @@ -17537,8 +25258,16 @@ components: - USE_RECIPIENT - USE_SCHEMA - USE_SHARE + - USE_VOLUME + - VIEW_ADMIN_METADATA + - VIEW_METADATA + - VIEW_OBJECT - WRITE_FILES + - WRITE_FLOW + - WRITE_MEMORY_STORE - WRITE_PRIVATE_FILES + - WRITE_SEMANTIC_GRAPH + - WRITE_SKILL - WRITE_VOLUME description: |- Create a collection of name/value pairs. @@ -17623,9 +25352,49 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + RateLimitRateLimitKey: + type: string + x-enum: + - RATE_LIMIT_KEY_REQUEST_TAG + - RATE_LIMIT_KEY_SERVICE + - RATE_LIMIT_KEY_SERVICE_PRINCIPAL + - RATE_LIMIT_KEY_USER + - RATE_LIMIT_KEY_USER_DEFAULT + - RATE_LIMIT_KEY_USER_GROUP + description: Scope key for a rate limit. + RateLimitRateLimitRenewalPeriod: + type: string + x-enum: + - RATE_LIMIT_RENEWAL_PERIOD_HOUR + - RATE_LIMIT_RENEWAL_PERIOD_MINUTE + description: Renewal period for a rate limit. + SchemaFilterType: + type: string + x-enum: + - SCHEMA_FILTER_TYPE_EXCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE + - SCHEMA_FILTER_TYPE_INCLUDE_ALL + description: Determines how a FOREIGN catalog's schema filter items are interpreted. SecurableKind: type: string x-enum: + - CATALOG_FOREIGN_BIGLAKE + - CATALOG_FOREIGN_GOOGLE_CLOUD_LAKEHOUSE + - CATALOG_FOREIGN_ICEBERG_REST + - CONNECTION_BIGLAKE_SERVICE_ACCOUNT + - CONNECTION_COMMUNITY_OAUTH_M2M + - CONNECTION_COMMUNITY_OAUTH_U2M + - CONNECTION_COMMUNITY_OAUTH_U2M_MAPPING + - CONNECTION_GOOGLE_CLOUD_LAKEHOUSE_OAUTH_U2M + - CONNECTION_GOOGLE_CLOUD_LAKEHOUSE_SERVICE_ACCOUNT + - CONNECTION_ICEBERG_REST_BEARER_TOKEN + - CONNECTION_ICEBERG_REST_OAUTH_M2M + - RECIPIENT_EMAIL + - RECIPIENT_EMAIL_DATABRICKS + - RECIPIENT_EMAIL_TOKEN + - SCHEMA_FOREIGN_BIGLAKE + - SCHEMA_FOREIGN_GOOGLE_CLOUD_LAKEHOUSE + - SCHEMA_FOREIGN_ICEBERG_REST - TABLE_DB_STORAGE - TABLE_DELTA - TABLE_DELTASHARING @@ -17636,6 +25405,7 @@ components: - TABLE_DELTA_ICEBERG_MANAGED - TABLE_DELTA_UNIFORM_HUDI_EXTERNAL - TABLE_DELTA_UNIFORM_ICEBERG_EXTERNAL + - TABLE_DELTA_UNIFORM_ICEBERG_EXTERNAL_DELTASHARING - TABLE_DELTA_UNIFORM_ICEBERG_FOREIGN_DELTASHARING - TABLE_DELTA_UNIFORM_ICEBERG_FOREIGN_HIVE_METASTORE_EXTERNAL - TABLE_DELTA_UNIFORM_ICEBERG_FOREIGN_HIVE_METASTORE_MANAGED @@ -17643,9 +25413,12 @@ components: - TABLE_EXTERNAL - TABLE_FEATURE_STORE - TABLE_FEATURE_STORE_EXTERNAL + - TABLE_FOREIGN_BIGLAKE - TABLE_FOREIGN_BIGQUERY - TABLE_FOREIGN_DATABRICKS - TABLE_FOREIGN_DELTASHARING + - TABLE_FOREIGN_DELTA_DELTASHARING + - TABLE_FOREIGN_GOOGLE_CLOUD_LAKEHOUSE - TABLE_FOREIGN_HIVE_METASTORE - TABLE_FOREIGN_HIVE_METASTORE_DBFS_EXTERNAL - TABLE_FOREIGN_HIVE_METASTORE_DBFS_MANAGED @@ -17657,10 +25430,12 @@ components: - TABLE_FOREIGN_HIVE_METASTORE_SHALLOW_CLONE_EXTERNAL - TABLE_FOREIGN_HIVE_METASTORE_SHALLOW_CLONE_MANAGED - TABLE_FOREIGN_HIVE_METASTORE_VIEW + - TABLE_FOREIGN_ICEBERG_REST - TABLE_FOREIGN_MONGODB - TABLE_FOREIGN_MYSQL - TABLE_FOREIGN_NETSUITE - TABLE_FOREIGN_ORACLE + - TABLE_FOREIGN_PALANTIR - TABLE_FOREIGN_POSTGRESQL - TABLE_FOREIGN_REDSHIFT - TABLE_FOREIGN_SALESFORCE @@ -17689,7 +25464,43 @@ components: - TABLE_SYSTEM_DELTASHARING - TABLE_VIEW - TABLE_VIEW_DELTASHARING - description: 'Latest kind: CONNECTION_GOOGLE_DRIVE_SERVICE_ACCOUNT = 301; Next id: 302' + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. SecurableType: type: string x-enum: @@ -17718,7 +25529,12 @@ components: - SPECIAL_DESTINATION_CONNECTION_OWNER - SPECIAL_DESTINATION_CREDENTIAL_OWNER - SPECIAL_DESTINATION_EXTERNAL_LOCATION_OWNER + - SPECIAL_DESTINATION_FUNCTION_OWNER - SPECIAL_DESTINATION_METASTORE_OWNER + - SPECIAL_DESTINATION_REGISTERED_MODEL_OWNER + - SPECIAL_DESTINATION_SCHEMA_OWNER + - SPECIAL_DESTINATION_TABLE_OWNER + - SPECIAL_DESTINATION_VOLUME_OWNER description: |- Create a collection of name/value pairs. @@ -17984,6 +25800,48 @@ components: - PASS - SKIP description: A enum represents the result of the file operation + VolumeOperation: + type: string + x-enum: + - READ_VOLUME + - WRITE_VOLUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. VolumeType: type: string x-enum: @@ -18032,11 +25890,162 @@ components: - BINDING_TYPE_READ_ONLY - BINDING_TYPE_READ_WRITE description: >- - Using `BINDING_TYPE_` prefix here to avoid conflict with - `TableOperation` enum in + Using ``BINDING_TYPE_`` prefix here to avoid conflict with + ``TableOperation`` enum in - `credentials_common.proto`. + ``credentials_common.proto``. x-stackQL-resources: + ai_gateway_agent_services: + id: databricks_workspace.catalog.ai_gateway_agent_services + name: ai_gateway_agent_services + title: Ai Gateway Agent Services + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1agent-services/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1agent-services/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.agent_services + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_agent_services/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_agent_services/methods/create + update: [] + delete: [] + replace: [] + ai_gateway_mcp_services: + id: databricks_workspace.catalog.ai_gateway_mcp_services + name: ai_gateway_mcp_services + title: Ai Gateway Mcp Services + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1mcp-services/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1mcp-services/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.mcp_services + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_mcp_services/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_mcp_services/methods/create + update: [] + delete: [] + replace: [] + ai_gateway_model_provider_services: + id: databricks_workspace.catalog.ai_gateway_model_provider_services + name: ai_gateway_model_provider_services + title: Ai Gateway Model Provider Services + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1model-provider-services/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1model-provider-services/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.model_provider_services + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_provider_services/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_provider_services/methods/create + update: [] + delete: [] + replace: [] + ai_gateway_model_services: + id: databricks_workspace.catalog.ai_gateway_model_services + name: ai_gateway_model_services + title: Ai Gateway Model Services + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1model-services/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1model-services/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.model_services + delete: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_services/methods/get + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_services/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_services/methods/create + update: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_services/methods/update + delete: + - $ref: >- + #/components/x-stackQL-resources/ai_gateway_model_services/methods/delete + replace: [] artifact_allowlists: id: databricks_workspace.catalog.artifact_allowlists name: artifact_allowlists @@ -18495,6 +26504,48 @@ components: update: [] delete: [] replace: [] + privilege_assignments: + id: databricks_workspace.catalog.privilege_assignments + name: privilege_assignments + title: Privilege Assignments + methods: + list: + operation: + $ref: >- + #/paths/~1api~12.1~1unity-catalog~1privilege-assignments~1{securable_type}~1{full_name}/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.privilege_assignments + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/privilege_assignments/methods/list + insert: [] + update: [] + delete: [] + replace: [] + effective_privilege_assignments: + id: databricks_workspace.catalog.effective_privilege_assignments + name: effective_privilege_assignments + title: Effective Privilege Assignments + methods: + list: + operation: + $ref: >- + #/paths/~1api~12.1~1unity-catalog~1effective-privilege-assignments~1{securable_type}~1{full_name}/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.privilege_assignments + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/effective_privilege_assignments/methods/list + insert: [] + update: [] + delete: [] + replace: [] metastores: id: databricks_workspace.catalog.metastores name: metastores @@ -19061,6 +27112,64 @@ components: delete: - $ref: '#/components/x-stackQL-resources/schemas/methods/delete' replace: [] + secrets_uc: + id: databricks_workspace.catalog.secrets_uc + name: secrets_uc + title: Secrets Uc + methods: + secrets_uc_create_secret: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1secrets/post' + response: + mediaType: application/json + openAPIDocKey: '200' + secrets_uc_list_secrets: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1secrets/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.secrets + secrets_uc_delete_secret: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1secrets~1{full_name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + secrets_uc_get_secret: + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1secrets~1{full_name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + secrets_uc_update_secret: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1unity-catalog~1secrets~1{full_name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/secrets_uc/methods/secrets_uc_get_secret + - $ref: >- + #/components/x-stackQL-resources/secrets_uc/methods/secrets_uc_list_secrets + insert: + - $ref: >- + #/components/x-stackQL-resources/secrets_uc/methods/secrets_uc_create_secret + update: + - $ref: >- + #/components/x-stackQL-resources/secrets_uc/methods/secrets_uc_update_secret + delete: + - $ref: >- + #/components/x-stackQL-resources/secrets_uc/methods/secrets_uc_delete_secret + replace: [] storage_credentials: id: databricks_workspace.catalog.storage_credentials name: storage_credentials @@ -19319,6 +27428,29 @@ components: update: [] delete: [] replace: [] + temporary_volume_credentials: + id: databricks_workspace.catalog.temporary_volume_credentials + name: temporary_volume_credentials + title: Temporary Volume Credentials + methods: + temporary_volume_credentials_generate_temporary_volume_credentials: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1unity-catalog~1temporary-volume-credentials/post + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: + - $ref: >- + #/components/x-stackQL-resources/temporary_volume_credentials/methods/temporary_volume_credentials_generate_temporary_volume_credentials + update: [] + delete: [] + replace: [] volumes: id: databricks_workspace.catalog.volumes name: volumes diff --git a/providers/src/databricks_workspace/v00.00.00000/services/cleanrooms.yaml b/providers/src/databricks_workspace/v00.00.00000/services/cleanrooms.yaml index d19248b5..8f1f1b74 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/cleanrooms.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/cleanrooms.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks cleanrooms service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.cleanrooms servers: - url: https://{deployment_name}.cloud.databricks.com @@ -228,6 +228,8 @@ paths: type: string x-enum: - FOREIGN_TABLE + - GENIE_SPACE + - JAR_ANALYSIS - NOTEBOOK_FILE - TABLE - VIEW @@ -264,11 +266,110 @@ paths: Local details for a foreign that are only available to its owner. Present if and only if **asset_type** is **FOREIGN_TABLE** + genie_space: + type: object + properties: {} + description: >- + Genie space details. The space_id sub-field is only + returned to the owner; other fields (e.g. + prompts_limit_per_day) are visible to all collaborators. + Present if and only if **asset_type** is **GENIE_SPACE** + jar_analysis: + type: object + properties: + central_jar_file_paths: + type: array + items: + type: string + description: + type: string + description: >- + Optional description of the jar analysis shown to + all collaborators. + environment_version: + type: string + description: >- + The serverless environment version used to execute + the JAR analysis (e.g. "4"). Defaults to + "4-scala-preview" if not specified. + etag: + type: string + description: >- + Server generated etag that represents the jar + analysis version. + main_class_name: + type: string + description: >- + The full name of the class containing the main + method to be executed. This class must be contained + in a JAR provided as a library The code must use + ``SparkContext.getOrCreate`` to obtain a Spark + context; otherwise, runs of the job fail + review_state: + type: string + x-enum: + - APPROVED + - PENDING + - REJECTED + description: Top-level status derived from all reviews. + reviews: + type: array + items: + type: object + properties: + comment: + type: string + description: review comment + created_at_millis: + type: integer + description: timestamp of when the review was submitted + review_state: + type: string + x-enum: + - APPROVED + - PENDING + - REJECTED + description: review outcome + review_sub_reason: + type: string + x-enum: + - AUTO_APPROVED + description: >- + specified when the review was not explicitly + made by a user + reviewer_collaborator_alias: + type: string + description: collaborator alias of the reviewer + description: >- + This only applies to a JAR Analysis as a + first-class asset in the Clean Room, and not to + Volumes + description: All existing approvals or rejections. + runner_collaborator_aliases: + type: array + items: + type: string + description: Collaborators that can run the jar. + description: >- + Jar analysis details available to all collaborators of + the clean room. Present if and only if **asset_type** is + **JAR_ANALYSIS** notebook: type: object properties: notebook_content: type: string + description: + type: string + description: >- + Optional description of the notebook shown to all + collaborators. + environment_version: + type: string + description: >- + The serverless environment version used to execute + the notebook (e.g. "4"). Defaults to "2" if not + specified. etag: type: string description: >- @@ -475,6 +576,7 @@ paths: Asset type. Can either be NOTEBOOK_FILE or JAR_ANALYSIS. :param name: str Name of the asset + :param jar_analysis_review: :class:`JarAnalysisVersionReview` (optional) :param notebook_review: :class:`NotebookVersionReview` (optional) :returns: :class:`CreateCleanRoomAssetReviewResponse` @@ -506,11 +608,31 @@ paths: type: string x-enum: - FOREIGN_TABLE + - GENIE_SPACE + - JAR_ANALYSIS - NOTEBOOK_FILE - TABLE - VIEW - VOLUME description: Asset type. Can either be NOTEBOOK_FILE or JAR_ANALYSIS. + jar_analysis_review: + type: object + properties: + etag: + type: string + review_state: + type: string + x-enum: + - APPROVED + - PENDING + - REJECTED + description: Review outcome + comment: + type: string + description: Review comment + required: + - etag + - review_state notebook_review: type: object properties: @@ -698,8 +820,8 @@ paths: For notebooks, the name is the notebook file name. For jar analyses, the name is the jar analysis name. :param asset: :class:`CleanRoomAsset` - The asset to update. The asset's `name` and `asset_type` fields are used to identify the asset to - update. + The asset to update. The asset's ``name`` and ``asset_type`` fields are used to identify the asset + to update. :returns: :class:`CleanRoomAsset` parameters: @@ -736,6 +858,8 @@ paths: type: string x-enum: - FOREIGN_TABLE + - GENIE_SPACE + - JAR_ANALYSIS - NOTEBOOK_FILE - TABLE - VIEW @@ -758,6 +882,8 @@ paths: type: string x-enum: - FOREIGN_TABLE + - GENIE_SPACE + - JAR_ANALYSIS - NOTEBOOK_FILE - TABLE - VIEW @@ -794,11 +920,110 @@ paths: Local details for a foreign that are only available to its owner. Present if and only if **asset_type** is **FOREIGN_TABLE** + genie_space: + type: object + properties: {} + description: >- + Genie space details. The space_id sub-field is only + returned to the owner; other fields (e.g. + prompts_limit_per_day) are visible to all collaborators. + Present if and only if **asset_type** is **GENIE_SPACE** + jar_analysis: + type: object + properties: + central_jar_file_paths: + type: array + items: + type: string + description: + type: string + description: >- + Optional description of the jar analysis shown to + all collaborators. + environment_version: + type: string + description: >- + The serverless environment version used to execute + the JAR analysis (e.g. "4"). Defaults to + "4-scala-preview" if not specified. + etag: + type: string + description: >- + Server generated etag that represents the jar + analysis version. + main_class_name: + type: string + description: >- + The full name of the class containing the main + method to be executed. This class must be contained + in a JAR provided as a library The code must use + ``SparkContext.getOrCreate`` to obtain a Spark + context; otherwise, runs of the job fail + review_state: + type: string + x-enum: + - APPROVED + - PENDING + - REJECTED + description: Top-level status derived from all reviews. + reviews: + type: array + items: + type: object + properties: + comment: + type: string + description: review comment + created_at_millis: + type: integer + description: timestamp of when the review was submitted + review_state: + type: string + x-enum: + - APPROVED + - PENDING + - REJECTED + description: review outcome + review_sub_reason: + type: string + x-enum: + - AUTO_APPROVED + description: >- + specified when the review was not explicitly + made by a user + reviewer_collaborator_alias: + type: string + description: collaborator alias of the reviewer + description: >- + This only applies to a JAR Analysis as a + first-class asset in the Clean Room, and not to + Volumes + description: All existing approvals or rejections. + runner_collaborator_aliases: + type: array + items: + type: string + description: Collaborators that can run the jar. + description: >- + Jar analysis details available to all collaborators of + the clean room. Present if and only if **asset_type** is + **JAR_ANALYSIS** notebook: type: object properties: notebook_content: type: string + description: + type: string + description: >- + Optional description of the notebook shown to all + collaborators. + environment_version: + type: string + description: >- + The serverless environment version used to execute + the notebook (e.g. "4"). Defaults to "2" if not + specified. etag: type: string description: >- @@ -921,7 +1146,7 @@ paths: - name - asset_type description: >- - The asset to update. The asset's `name` and `asset_type` + The asset to update. The asset's ``name`` and ``asset_type`` fields are used to identify the asset to update. required: - asset_type @@ -984,8 +1209,8 @@ paths: - ANY_AUTHOR description: >- Scope of authors covered by the rule. Only one of - `author_collaborator_alias` and `author_scope` can be - set. + ``author_collaborator_alias`` and ``author_scope`` can + be set. clean_room_name: type: string description: >- @@ -1207,8 +1432,8 @@ paths: - ANY_AUTHOR description: >- Scope of authors covered by the rule. Only one of - `author_collaborator_alias` and `author_scope` can be - set. + ``author_collaborator_alias`` and ``author_scope`` can + be set. clean_room_name: type: string description: >- @@ -1318,6 +1543,78 @@ paths: message: type: string x-stackql-sdk-source: CleanRoomTaskRunsAPI + /api/2.0/clean-rooms/{clean_room_name}/task-runs: + get: + operationId: clean_room_task_runs_list_clean_room_task_runs_handler + summary: List all the historical task runs in a clean room. + tags: + - cleanrooms + - clean_room_task_runs + description: |- + List all the historical task runs in a clean room. + + :param clean_room_name: str + Name of the clean room. + :param name: str (optional) + Executable name. + :param page_size: int (optional) + The maximum number of task runs to return. Maximum value of 100. + :param page_token: str (optional) + Opaque pagination token to go to next page based on previous query. + :param task_type: :class:`CleanRoomTaskType` (optional) + Filter by the type of Clean Room task. + + :returns: Iterator over :class:`CleanRoomTaskRun` + parameters: + - name: clean_room_name + in: path + required: true + schema: + type: string + description: Name of the clean room. + - name: name + in: query + required: false + schema: + type: string + description: Executable name. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of task runs to return. Maximum value of 100. + - name: page_token + in: query + required: false + schema: + type: string + description: Opaque pagination token to go to next page based on previous query. + - name: task_type + in: query + required: false + schema: + $ref: '#/components/schemas/CleanRoomTaskType' + description: Filter by the type of Clean Room task. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListCleanRoomTaskRunsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: CleanRoomTaskRunsAPI /api/2.0/clean-rooms: post: operationId: clean_rooms_create @@ -1334,13 +1631,15 @@ paths: name field inside the clean_room field can be used to poll the clean room status, using the - :method:cleanrooms/get method. When this method returns, the clean room - will be in a PROVISIONING + `cleanrooms/get + `__ method. + When this method - state, with only name, owner, comment, created_at and status populated. - The clean room will be usable + returns, the clean room will be in a PROVISIONING state, with only name, + owner, comment, created_at - once it enters an ACTIVE state. + and status populated. The clean room will be usable once it enters an + ACTIVE state. The caller must be a metastore admin or have the **CREATE_CLEAN_ROOM** @@ -1424,6 +1723,13 @@ paths: created_at: type: integer description: When the clean room was created, in epoch milliseconds. + enable_shared_output: + type: boolean + description: >- + Whether allow task to write to shared output schema. + When enabled, clean room task runs triggered by the + current collaborator can write to the run-scoped shared + output schema which is accessible by all collaborators. local_collaborator_alias: type: string description: >- @@ -1432,10 +1738,9 @@ paths: name: type: string description: >- - The name of the clean room. It should follow [UC - securable naming requirements]. [UC securable naming - requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + The name of the clean room. It should follow `UC + securable naming requirements + `__. output_catalog: type: object properties: @@ -1532,10 +1837,9 @@ paths: of this clean room, and used to derive multiple values internally such as catalog alias and clean room name for single metastore - clean rooms. It should follow [UC securable - naming requirements]. [UC securable naming - requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + clean rooms. It should follow `UC securable + naming requirements + `__. display_name: type: string description: >- @@ -1612,10 +1916,9 @@ paths: of this clean room, and used to derive multiple values internally such as catalog alias and clean room name for single metastore clean - rooms. It should follow [UC securable naming - requirements]. [UC securable naming - requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + rooms. It should follow `UC securable naming + requirements + `__. display_name: type: string description: >- @@ -1661,6 +1964,22 @@ paths: description: >- Egress network policy to apply to the central clean room workspace. + enable_shared_output: + type: boolean + description: >- + Whether to enable shared output for the central + clean room. When enabled, clean room task runs can + write to the run-scoped shared output schema which + is accessible by all collaborators. + package_provider_collaborator_alias: + type: string + description: >- + Alias of the provider collaborator. If set, packaged + clean rooms mode is enabled. The consumer's + experience is restricted: they can view notebook + names and READMEs, add their own data assets, and + trigger runs, but cannot view notebook code, + provider data assets, or notebook run output. region: type: string description: Region of the central clean room. @@ -1669,6 +1988,38 @@ paths: to specify cloud_vendor, region, and collaborators.global_metastore_id. This field will not be filled in the ListCleanRooms call. + replication_config: + type: object + properties: + enabled: + type: boolean + description: >- + Whether this collaborator's input share is + replicated into the central clean-room region. + schedule: + type: object + properties: + interval: + type: object + properties: + duration: + type: string + description: >- + How often the collaborator's input-share + data should be replicated. When unset, the + service-default interval applies. + description: Fixed-interval replication schedule. + description: >- + Optional schedule describing when replication runs. + When unset, the service-default schedule (currently + every hour) applies. + description: >- + Replication setting for the calling collaborator's input + share. Set via UpdateCleanRoom before the collaborator + adds their first asset (immutable thereafter), and read + back on GetCleanRoom. When enabled, that collaborator's + input data is replicated into the central clean-room + region. status: type: string x-enum: @@ -1966,7 +2317,7 @@ paths: privilege, or be metastore admin. - When the caller is a metastore admin, only the __owner__ field can be + When the caller is a metastore admin, only the **owner** field can be updated. @@ -2054,6 +2405,13 @@ paths: created_at: type: integer description: When the clean room was created, in epoch milliseconds. + enable_shared_output: + type: boolean + description: >- + Whether allow task to write to shared output schema. + When enabled, clean room task runs triggered by the + current collaborator can write to the run-scoped shared + output schema which is accessible by all collaborators. local_collaborator_alias: type: string description: >- @@ -2062,10 +2420,9 @@ paths: name: type: string description: >- - The name of the clean room. It should follow [UC - securable naming requirements]. [UC securable naming - requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + The name of the clean room. It should follow `UC + securable naming requirements + `__. output_catalog: type: object properties: @@ -2162,10 +2519,9 @@ paths: of this clean room, and used to derive multiple values internally such as catalog alias and clean room name for single metastore - clean rooms. It should follow [UC securable - naming requirements]. [UC securable naming - requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + clean rooms. It should follow `UC securable + naming requirements + `__. display_name: type: string description: >- @@ -2242,10 +2598,9 @@ paths: of this clean room, and used to derive multiple values internally such as catalog alias and clean room name for single metastore clean - rooms. It should follow [UC securable naming - requirements]. [UC securable naming - requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + rooms. It should follow `UC securable naming + requirements + `__. display_name: type: string description: >- @@ -2291,6 +2646,22 @@ paths: description: >- Egress network policy to apply to the central clean room workspace. + enable_shared_output: + type: boolean + description: >- + Whether to enable shared output for the central + clean room. When enabled, clean room task runs can + write to the run-scoped shared output schema which + is accessible by all collaborators. + package_provider_collaborator_alias: + type: string + description: >- + Alias of the provider collaborator. If set, packaged + clean rooms mode is enabled. The consumer's + experience is restricted: they can view notebook + names and READMEs, add their own data assets, and + trigger runs, but cannot view notebook code, + provider data assets, or notebook run output. region: type: string description: Region of the central clean room. @@ -2299,6 +2670,38 @@ paths: to specify cloud_vendor, region, and collaborators.global_metastore_id. This field will not be filled in the ListCleanRooms call. + replication_config: + type: object + properties: + enabled: + type: boolean + description: >- + Whether this collaborator's input share is + replicated into the central clean-room region. + schedule: + type: object + properties: + interval: + type: object + properties: + duration: + type: string + description: >- + How often the collaborator's input-share + data should be replicated. When unset, the + service-default interval applies. + description: Fixed-interval replication schedule. + description: >- + Optional schedule describing when replication runs. + When unset, the service-default schedule (currently + every hour) applies. + description: >- + Replication setting for the calling collaborator's input + share. Set via UpdateCleanRoom before the collaborator + adds their first asset (immutable thereafter), and read + back on GetCleanRoom. When enabled, that collaborator's + input data is replicated into the central clean-room + region. status: type: string x-enum: @@ -2343,15 +2746,22 @@ components: created_at: type: integer description: When the clean room was created, in epoch milliseconds. + enable_shared_output: + type: boolean + description: >- + Whether allow task to write to shared output schema. When enabled, + clean room task runs triggered by the current collaborator can write + to the run-scoped shared output schema which is accessible by all + collaborators. local_collaborator_alias: type: string description: The alias of the collaborator tied to the local clean room. name: type: string description: >- - The name of the clean room. It should follow [UC securable naming - requirements]. [UC securable naming requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + The name of the clean room. It should follow `UC securable naming + requirements + `__. output_catalog: $ref: '#/components/schemas/CleanRoomOutputCatalog' description: >- @@ -2369,6 +2779,14 @@ components: Central clean room details. During creation, users need to specify cloud_vendor, region, and collaborators.global_metastore_id. This field will not be filled in the ListCleanRooms call. + replication_config: + $ref: '#/components/schemas/ReplicationConfig' + description: >- + Replication setting for the calling collaborator's input share. Set + via UpdateCleanRoom before the collaborator adds their first asset + (immutable thereafter), and read back on GetCleanRoom. When enabled, + that collaborator's input data is replicated into the central + clean-room region. status: $ref: '#/components/schemas/CleanRoomStatusEnum' description: Clean room status. @@ -2409,6 +2827,18 @@ components: description: >- Local details for a foreign that are only available to its owner. Present if and only if **asset_type** is **FOREIGN_TABLE** + genie_space: + $ref: '#/components/schemas/CleanRoomAssetGenieSpace' + description: >- + Genie space details. The space_id sub-field is only returned to the + owner; other fields (e.g. prompts_limit_per_day) are visible to all + collaborators. Present if and only if **asset_type** is + **GENIE_SPACE** + jar_analysis: + $ref: '#/components/schemas/CleanRoomAssetJarAnalysis' + description: >- + Jar analysis details available to all collaborators of the clean + room. Present if and only if **asset_type** is **JAR_ANALYSIS** notebook: $ref: '#/components/schemas/CleanRoomAssetNotebook' description: >- @@ -2461,11 +2891,61 @@ components: type: string required: - local_name + CleanRoomAssetGenieSpace: + type: object + properties: {} + CleanRoomAssetJarAnalysis: + type: object + properties: + central_jar_file_paths: + type: array + items: + type: string + description: + type: string + description: Optional description of the jar analysis shown to all collaborators. + environment_version: + type: string + description: >- + The serverless environment version used to execute the JAR analysis + (e.g. "4"). Defaults to "4-scala-preview" if not specified. + etag: + type: string + description: Server generated etag that represents the jar analysis version. + main_class_name: + type: string + description: >- + The full name of the class containing the main method to be + executed. This class must be contained in a JAR provided as a + library The code must use ``SparkContext.getOrCreate`` to obtain a + Spark context; otherwise, runs of the job fail + review_state: + $ref: >- + #/components/schemas/CleanRoomJarAnalysisReviewJarAnalysisReviewState + description: Top-level status derived from all reviews. + reviews: + type: array + items: + $ref: '#/components/schemas/CleanRoomJarAnalysisReview' + description: All existing approvals or rejections. + runner_collaborator_aliases: + type: array + items: + type: string + description: Collaborators that can run the jar. CleanRoomAssetNotebook: type: object properties: notebook_content: type: string + description: + type: string + description: Optional description of the notebook shown to all collaborators. + environment_version: + type: string + description: >- + The serverless environment version used to execute the notebook + (e.g. "4"). Defaults to "2" if not specified. etag: type: string description: Server generated etag that represents the notebook version. @@ -2527,7 +3007,7 @@ components: $ref: '#/components/schemas/CleanRoomAutoApprovalRuleAuthorScope' description: >- Scope of authors covered by the rule. Only one of - `author_collaborator_alias` and `author_scope` can be set. + ``author_collaborator_alias`` and ``author_scope`` can be set. clean_room_name: type: string description: The name of the clean room this auto-approval rule belongs to. @@ -2552,9 +3032,9 @@ components: Collaborator alias specified by the clean room creator. It is unique across all collaborators of this clean room, and used to derive multiple values internally such as catalog alias and clean room name - for single metastore clean rooms. It should follow [UC securable - naming requirements]. [UC securable naming requirements]: - https://docs.databricks.com/en/data-governance/unity-catalog/index.html#securable-object-naming-requirements + for single metastore clean rooms. It should follow `UC securable + naming requirements + `__. display_name: type: string description: >- @@ -2590,6 +3070,29 @@ components: required: - collaborator_alias description: Publicly visible clean room collaborator. + CleanRoomJarAnalysisReview: + type: object + properties: + comment: + type: string + description: review comment + created_at_millis: + type: integer + description: timestamp of when the review was submitted + review_state: + $ref: >- + #/components/schemas/CleanRoomJarAnalysisReviewJarAnalysisReviewState + description: review outcome + review_sub_reason: + $ref: >- + #/components/schemas/CleanRoomJarAnalysisReviewJarAnalysisReviewSubReason + description: specified when the review was not explicitly made by a user + reviewer_collaborator_alias: + type: string + description: collaborator alias of the reviewer + description: >- + This only applies to a JAR Analysis as a first-class asset in the Clean + Room, and not to Volumes CleanRoomNotebookReview: type: object properties: @@ -2645,6 +3148,17 @@ components: run_duration: type: integer description: Duration of the task run, in milliseconds. + shared_output_schema_expiration_time: + type: integer + description: >- + Expiration time of the shared output schema of the task run (if + any), in epoch milliseconds. + shared_output_schema_name: + type: string + description: >- + Name of the shared output schema associated with the clean rooms + notebook task run. This schema is accessible by all collaborators + when enable_shared_output is true. start_time: type: integer description: When the task run started, in epoch milliseconds. @@ -2682,10 +3196,78 @@ components: egress_network_policy: type: string description: Egress network policy to apply to the central clean room workspace. + enable_shared_output: + type: boolean + description: >- + Whether to enable shared output for the central clean room. When + enabled, clean room task runs can write to the run-scoped shared + output schema which is accessible by all collaborators. + package_provider_collaborator_alias: + type: string + description: >- + Alias of the provider collaborator. If set, packaged clean rooms + mode is enabled. The consumer's experience is restricted: they can + view notebook names and READMEs, add their own data assets, and + trigger runs, but cannot view notebook code, provider data assets, + or notebook run output. region: type: string description: Region of the central clean room. description: Publicly visible central clean room details. + CleanRoomTaskRun: + type: object + properties: + analysis_details: + $ref: '#/components/schemas/CleanRoomTaskRunCleanRoomTaskAnalysisDetails' + description: Information about the analysis run (etag, updated at) + collaborator_job_run_info: + $ref: '#/components/schemas/CollaboratorJobRunInfo' + description: >- + Job run info of the task in the runner's local workspace. This field + is only included in the LIST API if the task was run within the same + workspace the API is being called. If the task run was in a + different workspace under the same metastore, only the workspace_id + is included. + name: + type: string + description: Name of the executable. + output_info: + $ref: '#/components/schemas/CleanRoomTaskRunOutputInfo' + description: Information about run output + run_duration: + type: integer + description: Duration of the task run, in milliseconds. + shared_output_info: + $ref: '#/components/schemas/CleanRoomTaskRunOutputInfo' + description: >- + Information about shared output accessible by all collaborators. + This field is only populated when enable_shared_output is true. + start_time: + type: integer + description: When the task run started, in epoch milliseconds. + task_run_state: + type: string + description: State of the task run. + task_type: + $ref: '#/components/schemas/CleanRoomTaskType' + description: The type of Clean Room task. + description: Stores information about a single task run. + CleanRoomTaskRunCleanRoomTaskAnalysisDetails: + type: object + properties: + etag: + type: string + updated_at: + type: integer + description: The timestamp of when the asset was last updated. + CleanRoomTaskRunOutputInfo: + type: object + properties: + output_schema_expiration_time: + type: integer + output_schema_name: + type: string + description: Name of the output schema associated with the clean room task run. CollaboratorJobRunInfo: type: object properties: @@ -2720,8 +3302,17 @@ components: CreateCleanRoomAssetReviewResponse: type: object properties: + jar_analysis_review_state: + $ref: >- + #/components/schemas/CleanRoomJarAnalysisReviewJarAnalysisReviewState + jar_analysis_reviews: + type: array + items: + $ref: '#/components/schemas/CleanRoomJarAnalysisReview' + description: All existing jar analysis approvals or rejections notebook_review_state: $ref: '#/components/schemas/CleanRoomNotebookReviewNotebookReviewState' + description: Top-level status derived from all reviews notebook_reviews: type: array items: @@ -2741,6 +3332,21 @@ components: Response for delete clean room request. Using an empty message since the generic Empty proto does not externd UnshadedMessageMarker. + JarAnalysisVersionReview: + type: object + properties: + etag: + type: string + review_state: + $ref: >- + #/components/schemas/CleanRoomJarAnalysisReviewJarAnalysisReviewState + description: Review outcome + comment: + type: string + description: Review comment + required: + - etag + - review_state ListCleanRoomAssetRevisionsResponse: type: object properties: @@ -2782,6 +3388,16 @@ components: items: $ref: '#/components/schemas/CleanRoomNotebookTaskRun' description: Name of the clean room. + ListCleanRoomTaskRunsResponse: + type: object + properties: + next_page_token: + type: string + runs: + type: array + items: + $ref: '#/components/schemas/CleanRoomTaskRun' + description: Task runs in the clean room. ListCleanRoomsResponse: type: object properties: @@ -2809,6 +3425,52 @@ components: required: - etag - review_state + ReplicationConfig: + type: object + properties: + enabled: + type: boolean + description: >- + Whether this collaborator's input share is replicated into the + central clean-room region. + schedule: + $ref: '#/components/schemas/ReplicationSchedule' + description: >- + Optional schedule describing when replication runs. When unset, the + service-default schedule (currently every hour) applies. + description: >- + Replication setting for a collaborator's input share. When enabled, that + collaborator's input + data is replicated into the central clean-room region so the clean room can run locally against + replicated data instead of reaching across regions to the source. + + This is settable (via UpdateCleanRoom) until the collaborator adds their first asset, and is + immutable thereafter. Copied from (not imported) the managed-catalog share-level + ReplicationConfig. + ReplicationInterval: + type: object + properties: + duration: + type: string + description: >- + How often the collaborator's input-share data should be replicated. + When unset, the service-default interval applies. + description: |- + How often a collaborator's replicated input share is refreshed. + + Copied from (not imported) the managed-catalog share-level ReplicationInterval so the clean-room + API and the Delta Sharing API stay independently evolvable. + ReplicationSchedule: + type: object + properties: + interval: + $ref: '#/components/schemas/ReplicationInterval' + description: Fixed-interval replication schedule. + description: >- + Schedule describing when a collaborator's replicated input share is + refreshed. + + Copied from (not imported) the managed-catalog share-level ReplicationSchedule. CleanRoomAccessRestricted: type: string x-enum: @@ -2855,6 +3517,8 @@ components: type: string x-enum: - FOREIGN_TABLE + - GENIE_SPACE + - JAR_ANALYSIS - NOTEBOOK_FILE - TABLE - VIEW @@ -2978,6 +3642,90 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CleanRoomJarAnalysisReviewJarAnalysisReviewState: + type: string + x-enum: + - APPROVED + - PENDING + - REJECTED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CleanRoomJarAnalysisReviewJarAnalysisReviewSubReason: + type: string + x-enum: + - AUTO_APPROVED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. CleanRoomNotebookReviewNotebookReviewState: @@ -3150,6 +3898,48 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CleanRoomTaskType: + type: string + x-enum: + - JAR + - NOTEBOOK + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. x-stackQL-resources: @@ -3327,6 +4117,26 @@ components: update: [] delete: [] replace: [] + clean_room_task_runs: + id: databricks_workspace.cleanrooms.clean_room_task_runs + name: clean_room_task_runs + title: Clean Room Task Runs + methods: + clean_room_task_runs_list_clean_room_task_runs_handler: + operation: + $ref: '#/paths/~1api~12.0~1clean-rooms~1{clean_room_name}~1task-runs/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.runs + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/clean_room_task_runs/methods/clean_room_task_runs_list_clean_room_task_runs_handler + insert: [] + update: [] + delete: [] + replace: [] clean_rooms: id: databricks_workspace.cleanrooms.clean_rooms name: clean_rooms diff --git a/providers/src/databricks_workspace/v00.00.00000/services/compute.yaml b/providers/src/databricks_workspace/v00.00.00000/services/compute.yaml index 24b2d36b..5bda10d3 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/compute.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/compute.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks compute service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.compute servers: - url: https://{deployment_name}.cloud.databricks.com @@ -22,13 +22,29 @@ paths: tags: - compute - cluster_policies - description: |- + description: >- Creates a new policy with prescribed settings. - :param definition: str (optional) - Policy definition document expressed in [Databricks Cluster Policy Definition Language]. - [Databricks Cluster Policy Definition Language]: https://docs.databricks.com/administration-guide/clusters/policy-definition.html + :param auto_enforcement_config: :class:`PolicyAutoEnforcementConfig` + (optional) + If present, auto enforcement is enabled for the policy. After the policy is edited, a background + operation may be scheduled to scan and edit all clusters and jobs using this policy to be in + compliance with the policy. + + The background operation is created when a policy edit does one of the following: + + - Changes the policy definition, or + - Changes auto enforcement from disabled to enabled Additionally, changes to the policy definition + or auto enforcement configuration may cause an in-progress operation to be restarted. + + To cancel an in-progress operation, edit the policy to delete this auto enforcement configuration. + + The background operation status is reported via the ``background_enforcement`` field when reading + the policy. + :param definition: str (optional) + Policy definition document expressed in `Databricks Cluster Policy Definition Language + `__. :param description: str (optional) Additional human-readable description of the cluster policy. :param libraries: List[:class:`Library`] (optional) @@ -41,19 +57,18 @@ paths: Cluster Policy name requested by the user. This has to be unique. Length must be between 1 and 100 characters. :param policy_family_definition_overrides: str (optional) - Policy definition JSON document expressed in [Databricks Policy Definition Language]. The JSON + Policy definition JSON document expressed in `Databricks Policy Definition Language + `__. The JSON document must be passed as a string and cannot be embedded in the requests. You can use this to customize the policy definition inherited from the policy family. Policy rules specified here are merged into the inherited policy definition. - - [Databricks Policy Definition Language]: https://docs.databricks.com/administration-guide/clusters/policy-definition.html :param policy_family_id: str (optional) ID of the policy family. The cluster policy's policy definition inherits the policy family's policy definition. - Cannot be used with `definition`. Use `policy_family_definition_overrides` instead to customize the - policy definition. + Cannot be used with ``definition``. Use ``policy_family_definition_overrides`` instead to customize + the policy definition. :returns: :class:`CreatePolicyResponse` requestBody: @@ -62,13 +77,37 @@ paths: schema: type: object properties: + auto_enforcement_config: + type: object + properties: + enforce_mode: + type: string + x-enum: + - ENFORCE_IMMEDIATELY + - WAIT_FOR_TERMINATION + description: >- + For running clusters, whether to defer enforcement until + the cluster terminates or to restart it immediately. + description: >- + If present, auto enforcement is enabled for the policy. + After the policy is edited, a background operation may be + scheduled to scan and edit all clusters and jobs using this + policy to be in compliance with the policy. The background + operation is created when a policy edit does one of the + following: - Changes the policy definition, or - Changes + auto enforcement from disabled to enabled Additionally, + changes to the policy definition or auto enforcement + configuration may cause an in-progress operation to be + restarted. To cancel an in-progress operation, edit the + policy to delete this auto enforcement configuration. The + background operation status is reported via the + ``background_enforcement`` field when reading the policy. definition: type: string description: >- - Policy definition document expressed in [Databricks Cluster - Policy Definition Language]. [Databricks Cluster Policy - Definition Language]: - https://docs.databricks.com/administration-guide/clusters/policy-definition.html + Policy definition document expressed in `Databricks Cluster + Policy Definition Language + `__. description: type: string description: Additional human-readable description of the cluster policy. @@ -100,13 +139,13 @@ paths: description: >- URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "jar": - "/Workspace/path/to/library.jar" }`, `{ "jar" : - "/Volumes/path/to/library.jar" }` or `{ "jar": - "s3://my-bucket/library.jar" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "jar": + "/Workspace/path/to/library.jar" }``, ``{ "jar" : + "/Volumes/path/to/library.jar" }`` or ``{ "jar": + "s3://my-bucket/library.jar" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. maven: type: object properties: @@ -118,7 +157,7 @@ paths: type: string description: >- List of dependences to exclude. For example: - `["slf4j:slf4j", "*:hadoop-client"]`. Maven + ``["slf4j:slf4j", "*:hadoop-client"]``. Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html. repo: @@ -131,7 +170,8 @@ paths: - coordinates description: >- Specification of a maven library to be installed. For - example: `{ "coordinates": "org.jsoup:jsoup:1.7.2" }` + example: ``{ "coordinates": "org.jsoup:jsoup:1.7.2" + }`` pypi: type: object properties: @@ -146,28 +186,28 @@ paths: - package description: >- Specification of a PyPi library to be installed. For - example: `{ "package": "simplejson" }` + example: ``{ "package": "simplejson" }`` requirements: type: string description: >- URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are - supported. For example: `{ "requirements": - "/Workspace/path/to/requirements.txt" }` or `{ + supported. For example: ``{ "requirements": + "/Workspace/path/to/requirements.txt" }`` or ``{ "requirements" : "/Volumes/path/to/requirements.txt" - }` + }`` whl: type: string description: >- URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "whl": - "/Workspace/path/to/library.whl" }`, `{ "whl" : - "/Volumes/path/to/library.whl" }` or `{ "whl": - "s3://my-bucket/library.whl" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "whl": + "/Workspace/path/to/library.whl" }``, ``{ "whl" : + "/Volumes/path/to/library.whl" }`` or ``{ "whl": + "s3://my-bucket/library.whl" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. description: >- A list of libraries to be installed on the next cluster restart that uses this policy. The maximum number of @@ -185,21 +225,21 @@ paths: policy_family_definition_overrides: type: string description: >- - Policy definition JSON document expressed in [Databricks - Policy Definition Language]. The JSON document must be - passed as a string and cannot be embedded in the requests. - You can use this to customize the policy definition - inherited from the policy family. Policy rules specified - here are merged into the inherited policy definition. - [Databricks Policy Definition Language]: - https://docs.databricks.com/administration-guide/clusters/policy-definition.html + Policy definition JSON document expressed in `Databricks + Policy Definition Language + `__. + The JSON document must be passed as a string and cannot be + embedded in the requests. You can use this to customize the + policy definition inherited from the policy family. Policy + rules specified here are merged into the inherited policy + definition. policy_family_id: type: string description: >- ID of the policy family. The cluster policy's policy definition inherits the policy family's policy definition. - Cannot be used with `definition`. Use - `policy_family_definition_overrides` instead to customize + Cannot be used with ``definition``. Use + ``policy_family_definition_overrides`` instead to customize the policy definition. responses: '200': @@ -280,10 +320,25 @@ paths: :param policy_id: str The ID of the policy to update. - :param definition: str (optional) - Policy definition document expressed in [Databricks Cluster Policy Definition Language]. + :param auto_enforcement_config: :class:`PolicyAutoEnforcementConfig` + (optional) + If present, auto enforcement is enabled for the policy. After the policy is edited, a background + operation may be scheduled to scan and edit all clusters and jobs using this policy to be in + compliance with the policy. + + The background operation is created when a policy edit does one of the following: + + - Changes the policy definition, or + - Changes auto enforcement from disabled to enabled Additionally, changes to the policy definition + or auto enforcement configuration may cause an in-progress operation to be restarted. - [Databricks Cluster Policy Definition Language]: https://docs.databricks.com/administration-guide/clusters/policy-definition.html + To cancel an in-progress operation, edit the policy to delete this auto enforcement configuration. + + The background operation status is reported via the ``background_enforcement`` field when reading + the policy. + :param definition: str (optional) + Policy definition document expressed in `Databricks Cluster Policy Definition Language + `__. :param description: str (optional) Additional human-readable description of the cluster policy. :param libraries: List[:class:`Library`] (optional) @@ -296,19 +351,18 @@ paths: Cluster Policy name requested by the user. This has to be unique. Length must be between 1 and 100 characters. :param policy_family_definition_overrides: str (optional) - Policy definition JSON document expressed in [Databricks Policy Definition Language]. The JSON + Policy definition JSON document expressed in `Databricks Policy Definition Language + `__. The JSON document must be passed as a string and cannot be embedded in the requests. You can use this to customize the policy definition inherited from the policy family. Policy rules specified here are merged into the inherited policy definition. - - [Databricks Policy Definition Language]: https://docs.databricks.com/administration-guide/clusters/policy-definition.html :param policy_family_id: str (optional) ID of the policy family. The cluster policy's policy definition inherits the policy family's policy definition. - Cannot be used with `definition`. Use `policy_family_definition_overrides` instead to customize the - policy definition. + Cannot be used with ``definition``. Use ``policy_family_definition_overrides`` instead to customize + the policy definition. requestBody: content: application/json: @@ -318,13 +372,37 @@ paths: policy_id: type: string description: The ID of the policy to update. + auto_enforcement_config: + type: object + properties: + enforce_mode: + type: string + x-enum: + - ENFORCE_IMMEDIATELY + - WAIT_FOR_TERMINATION + description: >- + For running clusters, whether to defer enforcement until + the cluster terminates or to restart it immediately. + description: >- + If present, auto enforcement is enabled for the policy. + After the policy is edited, a background operation may be + scheduled to scan and edit all clusters and jobs using this + policy to be in compliance with the policy. The background + operation is created when a policy edit does one of the + following: - Changes the policy definition, or - Changes + auto enforcement from disabled to enabled Additionally, + changes to the policy definition or auto enforcement + configuration may cause an in-progress operation to be + restarted. To cancel an in-progress operation, edit the + policy to delete this auto enforcement configuration. The + background operation status is reported via the + ``background_enforcement`` field when reading the policy. definition: type: string description: >- - Policy definition document expressed in [Databricks Cluster - Policy Definition Language]. [Databricks Cluster Policy - Definition Language]: - https://docs.databricks.com/administration-guide/clusters/policy-definition.html + Policy definition document expressed in `Databricks Cluster + Policy Definition Language + `__. description: type: string description: Additional human-readable description of the cluster policy. @@ -356,13 +434,13 @@ paths: description: >- URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "jar": - "/Workspace/path/to/library.jar" }`, `{ "jar" : - "/Volumes/path/to/library.jar" }` or `{ "jar": - "s3://my-bucket/library.jar" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "jar": + "/Workspace/path/to/library.jar" }``, ``{ "jar" : + "/Volumes/path/to/library.jar" }`` or ``{ "jar": + "s3://my-bucket/library.jar" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. maven: type: object properties: @@ -374,7 +452,7 @@ paths: type: string description: >- List of dependences to exclude. For example: - `["slf4j:slf4j", "*:hadoop-client"]`. Maven + ``["slf4j:slf4j", "*:hadoop-client"]``. Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html. repo: @@ -387,7 +465,8 @@ paths: - coordinates description: >- Specification of a maven library to be installed. For - example: `{ "coordinates": "org.jsoup:jsoup:1.7.2" }` + example: ``{ "coordinates": "org.jsoup:jsoup:1.7.2" + }`` pypi: type: object properties: @@ -402,28 +481,28 @@ paths: - package description: >- Specification of a PyPi library to be installed. For - example: `{ "package": "simplejson" }` + example: ``{ "package": "simplejson" }`` requirements: type: string description: >- URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are - supported. For example: `{ "requirements": - "/Workspace/path/to/requirements.txt" }` or `{ + supported. For example: ``{ "requirements": + "/Workspace/path/to/requirements.txt" }`` or ``{ "requirements" : "/Volumes/path/to/requirements.txt" - }` + }`` whl: type: string description: >- URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "whl": - "/Workspace/path/to/library.whl" }`, `{ "whl" : - "/Volumes/path/to/library.whl" }` or `{ "whl": - "s3://my-bucket/library.whl" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "whl": + "/Workspace/path/to/library.whl" }``, ``{ "whl" : + "/Volumes/path/to/library.whl" }`` or ``{ "whl": + "s3://my-bucket/library.whl" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. description: >- A list of libraries to be installed on the next cluster restart that uses this policy. The maximum number of @@ -441,21 +520,21 @@ paths: policy_family_definition_overrides: type: string description: >- - Policy definition JSON document expressed in [Databricks - Policy Definition Language]. The JSON document must be - passed as a string and cannot be embedded in the requests. - You can use this to customize the policy definition - inherited from the policy family. Policy rules specified - here are merged into the inherited policy definition. - [Databricks Policy Definition Language]: - https://docs.databricks.com/administration-guide/clusters/policy-definition.html + Policy definition JSON document expressed in `Databricks + Policy Definition Language + `__. + The JSON document must be passed as a string and cannot be + embedded in the requests. You can use this to customize the + policy definition inherited from the policy family. Policy + rules specified here are merged into the inherited policy + definition. policy_family_id: type: string description: >- ID of the policy family. The cluster policy's policy definition inherits the policy family's policy definition. - Cannot be used with `definition`. Use - `policy_family_definition_overrides` instead to customize + Cannot be used with ``definition``. Use + ``policy_family_definition_overrides`` instead to customize the policy definition. required: - policy_id @@ -490,6 +569,8 @@ paths: :param policy_id: str Canonical unique identifier for the Cluster Policy. + :param policy_view: :class:`PolicyView` (optional) + Controls which fields are returned. :returns: :class:`Policy` parameters: @@ -499,6 +580,12 @@ paths: schema: type: string description: Canonical unique identifier for the Cluster Policy. + - name: policy_view + in: query + required: false + schema: + $ref: '#/components/schemas/PolicyView' + description: Controls which fields are returned. responses: '200': description: Success @@ -760,11 +847,15 @@ paths: Returns a list of policies accessible by the requesting user. :param sort_column: :class:`ListSortColumn` (optional) - The cluster policy attribute to sort by. * `POLICY_CREATION_TIME` - Sort result list by policy - creation time. * `POLICY_NAME` - Sort result list by policy name. + The cluster policy attribute to sort by. + + - ``POLICY_CREATION_TIME`` - Sort result list by policy creation time. + - ``POLICY_NAME`` - Sort result list by policy name. :param sort_order: :class:`ListSortOrder` (optional) - The order in which the policies get listed. * `DESC` - Sort result list in descending order. * `ASC` - - Sort result list in ascending order. + The order in which the policies get listed. + + - ``DESC`` - Sort result list in descending order. + - ``ASC`` - Sort result list in ascending order. :returns: Iterator over :class:`Policy` parameters: @@ -774,8 +865,8 @@ paths: schema: $ref: '#/components/schemas/ListSortColumn' description: >- - The cluster policy attribute to sort by. * `POLICY_CREATION_TIME` - - Sort result list by policy creation time. * `POLICY_NAME` - Sort + The cluster policy attribute to sort by. - ``POLICY_CREATION_TIME`` + - Sort result list by policy creation time. - ``POLICY_NAME`` - Sort result list by policy name. - name: sort_order in: query @@ -783,8 +874,8 @@ paths: schema: $ref: '#/components/schemas/ListSortOrder' description: >- - The order in which the policies get listed. * `DESC` - Sort result - list in descending order. * `ASC` - Sort result list in ascending + The order in which the policies get listed. - ``DESC`` - Sort result + list in descending order. - ``ASC`` - Sort result list in ascending order. responses: '200': @@ -821,7 +912,7 @@ paths: this operation. The service principal application ID can be supplied as an argument to - `owner_username`. + ``owner_username``. :param cluster_id: str @@ -894,16 +985,17 @@ paths: Rather than authoring the cluster's JSON definition from scratch, Databricks recommends filling out - the [create compute UI] and then copying the generated JSON definition - from the UI. + the `create compute UI + `__ and then copying + the - - [create compute UI]: https://docs.databricks.com/compute/configure.html + generated JSON definition from the UI. :param spark_version: str - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of available Spark versions can be - retrieved by using the :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list of available Spark versions can + be retrieved by using the `clusters/sparkVersions + `__ API call. :param apply_policy_default_values: bool (optional) When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied. @@ -926,21 +1018,22 @@ paths: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every - `5 mins`. The destination of driver logs is `$destination/$clusterId/driver`, while the destination - of executor logs is `$destination/$clusterId/executor`. + ``5 mins``. The destination of driver logs is ``$destination/$clusterId/driver``, while the + destination of executor logs is ``$destination/$clusterId/executor``. :param cluster_name: str (optional) Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs. :param custom_tags: Dict[str,str] (optional) Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS - instances and EBS volumes) with these tags in addition to `default_tags`. Notes: + instances and EBS volumes) with these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags :param data_security_mode: :class:`DataSecurityMode` (optional) + :param dependency_mode: :class:`DependencyMode` (optional) + Controls dependency configuration for the cluster. :param docker_image: :class:`DockerImage` (optional) Custom docker image BYOC :param driver_instance_pool_id: str (optional) @@ -951,7 +1044,7 @@ paths: Flexible node type configuration for the driver node. :param driver_node_type_id: str (optional) The node type of the Spark driver. Note that this field is optional; if unset, the driver node type - will be set as the same value as `node_type_id` defined above. + will be set as the same value as ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and @@ -966,30 +1059,31 @@ paths: creation, a set of default values will be used. :param init_scripts: List[:class:`InitScriptInfo`] (optional) The configuration for storing init scripts. Any number of destinations can be specified. The scripts - are executed sequentially in the order provided. If `cluster_log_conf` is specified, init script - logs are sent to `//init_scripts`. + are executed sequentially in the order provided. If ``cluster_log_conf`` is specified, init script + logs are sent to ``//init_scripts``. :param instance_pool_id: str (optional) The optional ID of the instance pool to which the cluster belongs. :param is_single_node: bool (optional) - This field can only be used when `kind = CLASSIC_PREVIEW`. + This field can only be used when ``kind = CLASSIC_PREVIEW``. - When set to true, Databricks will automatically set single node related `custom_tags`, `spark_conf`, - and `num_workers` + When set to true, Databricks will automatically set single node related ``custom_tags``, + ``spark_conf``, and ``num_workers`` :param kind: :class:`Kind` (optional) :param node_type_id: str (optional) This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes `__ API + call. :param num_workers: int (optional) Number of worker nodes that this cluster should have. A cluster has one Spark Driver and - `num_workers` Executors for a total of `num_workers` + 1 Spark nodes. + ``num_workers`` Executors for a total of ``num_workers`` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas - the workers listed in `spark_info` will gradually increase from 5 to 10 as the new nodes are + the workers listed in ``spark_info`` will gradually increase from 5 to 10 as the new nodes are provisioned. :param policy_id: str (optional) The ID of the cluster policy used to create the cluster if applicable. @@ -999,41 +1093,41 @@ paths: :param runtime_engine: :class:`RuntimeEngine` (optional) Determines the cluster's runtime engine, either standard or Photon. - This field is not compatible with legacy `spark_version` values that contain `-photon-`. Remove - `-photon-` from the `spark_version` and set `runtime_engine` to `PHOTON`. + This field is not compatible with legacy ``spark_version`` values that contain ``-photon-``. Remove + ``-photon-`` from the ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used. :param single_user_name: str (optional) - Single user name if data_security_mode is `SINGLE_USER` + Single user name if data_security_mode is ``SINGLE_USER`` :param spark_conf: Dict[str,str] (optional) An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via - `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` respectively. + ``spark.driver.extraJavaOptions`` and ``spark.executor.extraJavaOptions`` respectively. :param spark_env_vars: Dict[str,str] (optional) An object containing a set of optional, user-specified environment variable key-value pairs. Please - note that key-value pair of the form (X,Y) will be exported as is (i.e., `export X='Y'`) while + note that key-value pair of the form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. - In order to specify an additional set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This ensures that all default databricks + In order to specify an additional set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. - Example Spark environment variables: `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + Example Spark environment variables: ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` :param ssh_public_keys: List[str] (optional) SSH public key contents that will be added to each Spark node in this cluster. The corresponding - private keys can be used to login with the user name `ubuntu` on port `2200`. Up to 10 keys can be - specified. + private keys can be used to login with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can + be specified. :param total_initial_remote_disk_size: int (optional) If set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED disks. :param use_ml_runtime: bool (optional) - This field can only be used when `kind = CLASSIC_PREVIEW`. + This field can only be used when ``kind = CLASSIC_PREVIEW``. - `effective_spark_version` is determined by `spark_version` (DBR release), this field - `use_ml_runtime`, and whether `node_type_id` is gpu node or not. + ``effective_spark_version`` is determined by ``spark_version`` (DBR release), this field + ``use_ml_runtime``, and whether ``node_type_id`` is gpu node or not. :param worker_node_type_flexibility: :class:`NodeTypeFlexibility` (optional) Flexible node type configuration for worker nodes. @@ -1052,9 +1146,11 @@ paths: spark_version: type: string description: >- - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A - list of available Spark versions can be retrieved by using - the :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. + A list of available Spark versions can be retrieved by using + the `clusters/sparkVersions + `__ + API call. apply_policy_default_values: type: boolean description: >- @@ -1095,11 +1191,13 @@ paths: - SPOT_WITH_FALLBACK description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. + ``first_on_demand`` ones. - Note: If `first_on_demand` is zero, this availability - type will be used for the entire cluster. + Note: If ``first_on_demand`` is zero, this availability + type will be used for the entire + + cluster. ebs_volume_count: type: integer description: >- @@ -1109,17 +1207,17 @@ paths: cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail. These - EBS volumes will be mounted at `/ebs0`, `/ebs1`, and + EBS volumes will be mounted at ``/ebs0``, ``/ebs1``, and etc. Instance store volumes will be mounted at - `/local_disk0`, `/local_disk1`, and etc. If EBS volumes - are attached, Databricks will configure Spark to use - only the EBS volumes for scratch storage because + ``/local_disk0``, ``/local_disk1``, and etc. If EBS + volumes are attached, Databricks will configure Spark to + use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, Databricks will configure Spark to use instance store volumes. Please note that if EBS volumes are specified, then the Spark configuration - `spark.local.dir` will be overridden. + ``spark.local.dir`` will be overridden. ebs_volume_iops: type: integer description: >- @@ -1150,18 +1248,18 @@ paths: first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be - placed on on-demand instances. If this value is greater - than 0, the cluster driver node in particular will be - placed on an on-demand instance. If this value is - greater than or equal to the current cluster size, all - nodes will be placed on on-demand instances. If this + The first ``first_on_demand`` nodes of the cluster will + be placed on on-demand instances. If this value is + greater than 0, the cluster driver node in particular + will be placed on an on-demand instance. If this value + is greater than or equal to the current cluster size, + all nodes will be placed on on-demand instances. If this value is less than the current cluster size, - `first_on_demand` nodes will be placed on on-demand + ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not - affect cluster size and cannot currently be mutated over - the lifetime of a cluster. + ``availability`` instances. Note that this value does + not affect cluster size and cannot currently be mutated + over the lifetime of a cluster. instance_profile_arn: type: string description: >- @@ -1178,10 +1276,10 @@ paths: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster - needs a new `r3.xlarge` spot instance, then the bid - price is half of the price of on-demand `r3.xlarge` + needs a new ``r3.xlarge`` spot instance, then the bid + price is half of the price of on-demand ``r3.xlarge`` instances. Similarly, if this field is set to 200, the - bid price is twice the price of on-demand `r3.xlarge` + bid price is twice the price of on-demand ``r3.xlarge`` instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this @@ -1202,7 +1300,7 @@ paths: with high availability, and will retry placement in a different AZ if there is not enough capacity. The list of available zones as well as the default value can be - found by using the `List Zones` method. + found by using the ``List Zones`` method. description: >- Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of @@ -1218,24 +1316,43 @@ paths: - SPOT_WITH_FALLBACK_AZURE description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. Note: If `first_on_demand` is - zero, this availability type will be used for the entire - cluster. + ``first_on_demand`` ones. Note: If ``first_on_demand`` + is zero, this availability type will be used for the + entire cluster. + capacity_reservation_group: + type: string + description: >- + The Azure capacity reservation group resource ID to use + for launching VMs. When specified, VMs will be launched + using the provided capacity reservation. Capacity + reservations can only be specified when the workspace + uses injected vnet (i.e. customer defined vnet not + managed by databricks). Ensure the databricks-login-prod + Enterprise Application is granted the following four + permissions: 1. + Microsoft.Compute/capacityReservationGroups/read 2. + Microsoft.Compute/capacityReservationGroups/deploy/action + 3. + Microsoft.Compute/capacityReservationGroups/capacityReservations/read + 4. + Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action + Format: + ``/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}`` first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be - placed on on-demand instances. This value should be + The first ``first_on_demand`` nodes of the cluster will + be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, - `first_on_demand` nodes will be placed on on-demand + ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not - affect cluster size and cannot currently be mutated over - the lifetime of a cluster. + ``availability`` instances. Note that this value does + not affect cluster size and cannot currently be mutated + over the lifetime of a cluster. log_analytics_info: type: object properties: @@ -1278,19 +1395,19 @@ paths: properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: >- - destination needs to be provided. e.g. `{ "dbfs" : { - "destination" : "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs" : { + "destination" : "dbfs:/home/cluster_log" } }`` s3: type: object properties: destination: type: string description: >- - S3 destination, e.g. `s3://my-bucket/some-prefix` + S3 destination, e.g. ``s3://my-bucket/some-prefix`` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please @@ -1300,55 +1417,57 @@ paths: type: string description: >- (Optional) Set canned access control list for the - logs, e.g. `bucket-owner-full-control`. If - `canned_cal` is set, please make sure the cluster - iam role has `s3:PutObjectAcl` permission on the + logs, e.g. ``bucket-owner-full-control``. If + ``canned_cal`` is set, please make sure the cluster + iam role has ``s3:PutObjectAcl`` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set - `bucket-owner-full-control` to make bucket owner + ``bucket-owner-full-control`` to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- (Optional) Flag to enable server side encryption, - `false` by default. + ``false`` by default. encryption_type: type: string description: >- - (Optional) The encryption type, it could be `sse-s3` - or `sse-kms`. It will be used only when encryption - is enabled and the default type is `sse-s3`. + (Optional) The encryption type, it could be + ``sse-s3`` or ``sse-kms``. It will be used only when + encryption is enabled and the default type is + ``sse-s3``. endpoint: type: string description: >- S3 endpoint, e.g. - `https://s3-us-west-2.amazonaws.com`. Either region - or endpoint needs to be set. If both are set, + ``https://s3-us-west-2.amazonaws.com``. Either + region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: type: string description: >- (Optional) Kms key which will be used if encryption - is enabled and encryption type is set to `sse-kms`. + is enabled and encryption type is set to + ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or + S3 region, e.g. ``us-west-2``. Either region or endpoint needs to be set. If both are set, endpoint will be used. required: - destination description: >- destination and either the region or endpoint need to be - provided. e.g. `{ "s3": { "destination" : + provided. e.g. ``{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" - } }` Cluster iam role is used to access s3, please make - sure the cluster iam role in `instance_profile_arn` has - permission to write data to the s3 destination. + } }`` Cluster iam role is used to access s3, please make + sure the cluster iam role in ``instance_profile_arn`` + has permission to write data to the s3 destination. volumes: type: object properties: @@ -1356,24 +1475,24 @@ paths: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided, e.g. `{ "volumes": { + destination needs to be provided, e.g. ``{ "volumes": { "destination": - "/Volumes/catalog/schema/volume/cluster_log" } }` + "/Volumes/catalog/schema/volume/cluster_log" } }`` description: >- The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every - `5 mins`. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination of - executor logs is `$destination/$clusterId/executor`. + ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. cluster_name: type: string description: >- @@ -1386,7 +1505,7 @@ paths: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) - with these tags in addition to `default_tags`. Notes: - + with these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags @@ -1408,40 +1527,27 @@ paths: when accessing data from a cluster. - The following modes can only be used when `kind = - CLASSIC_PREVIEW`. * `DATA_SECURITY_MODE_AUTO`: - - Databricks will choose the most appropriate access mode - depending on your compute configuration. - - * `DATA_SECURITY_MODE_STANDARD`: Alias for `USER_ISOLATION`. - * `DATA_SECURITY_MODE_DEDICATED`: - - Alias for `SINGLE_USER`. + - ``DATA_SECURITY_MODE_AUTO``: Databricks will choose the + most appropriate access mode depending + on your compute configuration. + - ``DATA_SECURITY_MODE_STANDARD``: A secure cluster that can + be shared by multiple users. + Cluster users are fully isolated so that they cannot see each other’s data and credentials. + Most data governance features are supported in this mode. But programming languages and + cluster features might be limited. + - ``DATA_SECURITY_MODE_DEDICATED``: A secure cluster that + can only be exclusively used by a + single user specified in ``single_user_name``. Most programming languages, cluster features + and data governance features are available in this mode. + The following modes are legacy aliases for the above modes: - The following modes can be used regardless of `kind`. * - `NONE`: No security isolation for - multiple users sharing the cluster. Data governance features - are not available in this mode. * + - ``USER_ISOLATION``: Legacy alias for + ``DATA_SECURITY_MODE_STANDARD``. - `SINGLE_USER`: A secure cluster that can only be exclusively - used by a single user specified in - - `single_user_name`. Most programming languages, cluster - features and data governance features - - are available in this mode. * `USER_ISOLATION`: A secure - cluster that can be shared by multiple - - users. Cluster users are fully isolated so that they cannot - see each other's data and - - credentials. Most data governance features are supported in - this mode. But programming languages - - and cluster features might be limited. + - ``SINGLE_USER``: Legacy alias for + ``DATA_SECURITY_MODE_DEDICATED``. The following modes are deprecated starting with Databricks @@ -1450,19 +1556,25 @@ paths: future Databricks Runtime versions: - * `LEGACY_TABLE_ACL`: This mode is for users migrating from - legacy Table ACL clusters. * - - `LEGACY_PASSTHROUGH`: This mode is for users migrating from - legacy Passthrough on high - - concurrency clusters. * `LEGACY_SINGLE_USER`: This mode is - for users migrating from legacy - - Passthrough on standard clusters. * - `LEGACY_SINGLE_USER_STANDARD`: This mode provides a way that + - ``LEGACY_TABLE_ACL``: This mode is for users migrating + from legacy Table ACL clusters. - doesn’t have UC nor passthrough enabled. + - ``LEGACY_PASSTHROUGH``: This mode is for users migrating + from legacy Passthrough on high + concurrency clusters. + - ``LEGACY_SINGLE_USER``: This mode is for users migrating + from legacy Passthrough on standard + clusters. + - ``LEGACY_SINGLE_USER_STANDARD``: This mode provides a way + that doesn’t have UC nor + passthrough enabled. + dependency_mode: + type: string + x-enum: + - DEPENDENCY_MODE_AUTO + - DEPENDENCY_MODE_CLUSTER_LIBRARIES + - DEPENDENCY_MODE_ENVIRONMENTS + description: Controls dependency configuration for the cluster. docker_image: type: object properties: @@ -1501,7 +1613,7 @@ paths: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the - same value as `node_type_id` defined above. This field, + same value as ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, @@ -1532,21 +1644,31 @@ paths: boot_disk_size: type: integer description: Boot disk size in GB + confidential_compute_type: + type: string + x-enum: + - CONFIDENTIAL_COMPUTE_TYPE_NONE + - SEV_SNP + description: >- + The confidential computing technology for this cluster's + instances. Currently only SEV_SNP is supported, and only + on N2D instance types. When not set, no confidential + computing is applied. first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be - placed on on-demand instances. This value should be + The first ``first_on_demand`` nodes of the cluster will + be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, - `first_on_demand` nodes will be placed on on-demand + ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not - affect cluster size and cannot currently be mutated over - the lifetime of a cluster. + ``availability`` instances. Note that this value does + not affect cluster size and cannot currently be mutated + over the lifetime of a cluster. google_service_account: type: string description: >- @@ -1560,10 +1682,11 @@ paths: description: >- If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. - Each local SSD is 375GB in size. Refer to [GCP - documentation] for the supported number of local SSDs - for each instance type. [GCP documentation]: - https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds + Each local SSD is 375GB in size. Refer to `GCP + documentation + `__ + for the supported number of local SSDs for each instance + type. use_preemptible_executors: type: boolean description: >- @@ -1600,23 +1723,23 @@ paths: type: string description: >- abfss destination, e.g. - `abfss://@.dfs.core.windows.net/`. + ``abfss://@.dfs.core.windows.net/``. required: - destination description: >- destination needs to be provided, e.g. - `abfss://@.dfs.core.windows.net/` + ``abfss://@.dfs.core.windows.net/`` dbfs: type: object properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: >- - destination needs to be provided. e.g. `{ "dbfs": { - "destination" : "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs": { + "destination" : "dbfs:/home/cluster_log" } }`` file: type: object properties: @@ -1625,8 +1748,8 @@ paths: required: - destination description: >- - destination needs to be provided, e.g. `{ "file": { - "destination": "file:/my/local/file.sh" } }` + destination needs to be provided, e.g. ``{ "file": { + "destination": "file:/my/local/file.sh" } }`` gcs: type: object properties: @@ -1634,56 +1757,57 @@ paths: type: string description: >- GCS destination/URI, e.g. - `gs://my-bucket/some-prefix` + ``gs://my-bucket/some-prefix`` required: - destination description: >- - destination needs to be provided, e.g. `{ "gcs": { - "destination": "gs://my-bucket/file.sh" } }` + destination needs to be provided, e.g. ``{ "gcs": { + "destination": "gs://my-bucket/file.sh" } }`` s3: type: object properties: destination: type: string description: >- - S3 destination, e.g. `s3://my-bucket/some-prefix` - Note that logs will be delivered using cluster iam - role, please make sure you set cluster iam role - and the role has write access to the destination. - Please also note that you cannot use AWS keys to - deliver logs. + S3 destination, e.g. + ``s3://my-bucket/some-prefix`` Note that logs will + be delivered using cluster iam role, please make + sure you set cluster iam role and the role has + write access to the destination. Please also note + that you cannot use AWS keys to deliver logs. canned_acl: type: string description: >- (Optional) Set canned access control list for the - logs, e.g. `bucket-owner-full-control`. If - `canned_cal` is set, please make sure the cluster - iam role has `s3:PutObjectAcl` permission on the - destination bucket and prefix. The full list of - possible canned acl can be found at + logs, e.g. ``bucket-owner-full-control``. If + ``canned_cal`` is set, please make sure the + cluster iam role has ``s3:PutObjectAcl`` + permission on the destination bucket and prefix. + The full list of possible canned acl can be found + at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set - `bucket-owner-full-control` to make bucket owner + ``bucket-owner-full-control`` to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- (Optional) Flag to enable server side encryption, - `false` by default. + ``false`` by default. encryption_type: type: string description: >- (Optional) The encryption type, it could be - `sse-s3` or `sse-kms`. It will be used only when - encryption is enabled and the default type is - `sse-s3`. + ``sse-s3`` or ``sse-kms``. It will be used only + when encryption is enabled and the default type is + ``sse-s3``. endpoint: type: string description: >- S3 endpoint, e.g. - `https://s3-us-west-2.amazonaws.com`. Either + ``https://s3-us-west-2.amazonaws.com``. Either region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: @@ -1691,23 +1815,23 @@ paths: description: >- (Optional) Kms key which will be used if encryption is enabled and encryption type is set - to `sse-kms`. + to ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or + S3 region, e.g. ``us-west-2``. Either region or endpoint needs to be set. If both are set, endpoint will be used. required: - destination description: >- destination and either the region or endpoint need to - be provided. e.g. `{ \"s3\": { \"destination\": + be provided. e.g. ``{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": - \"us-west-2\" } }` Cluster iam role is used to access + \"us-west-2\" } }`` Cluster iam role is used to access s3, please make sure the cluster iam role in - `instance_profile_arn` has permission to write data to - the s3 destination. + ``instance_profile_arn`` has permission to write data + to the s3 destination. volumes: type: object properties: @@ -1715,14 +1839,14 @@ paths: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided. e.g. `{ \"volumes\" - : { \"destination\" : \"/Volumes/my-init.sh\" } }` + destination needs to be provided. e.g. ``{ \"volumes\" + : { \"destination\" : \"/Volumes/my-init.sh\" } }`` workspace: type: object properties: @@ -1730,20 +1854,20 @@ paths: type: string description: >- wsfs destination, e.g. - `workspace:/cluster-init-scripts/setup-datadog.sh` + ``workspace:/cluster-init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided, e.g. `{ "workspace": - { "destination": - "/cluster-init-scripts/setup-datadog.sh" } }` - description: 'Config for an individual init script Next ID: 11' + destination needs to be provided, e.g. ``{ + "workspace": { "destination": + "/cluster-init-scripts/setup-datadog.sh" } }`` + description: Config for an individual init script description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed - sequentially in the order provided. If `cluster_log_conf` is - specified, init script logs are sent to - `//init_scripts`. + sequentially in the order provided. If ``cluster_log_conf`` + is specified, init script logs are sent to + ``//init_scripts``. instance_pool_id: type: string description: >- @@ -1752,9 +1876,10 @@ paths: is_single_node: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. + This field can only be used when ``kind = CLASSIC_PREVIEW``. When set to true, Databricks will automatically set single - node related `custom_tags`, `spark_conf`, and `num_workers` + node related ``custom_tags``, ``spark_conf``, and + ``num_workers`` kind: type: string x-enum: @@ -1763,33 +1888,28 @@ paths: The kind of compute described by this compute specification. - Depending on `kind`, different validations and default + Depending on ``kind``, different validations and default values will be applied. - Clusters with `kind = CLASSIC_PREVIEW` support the following - fields, whereas clusters with no - - specified `kind` do not. * - [is_single_node](/api/workspace/clusters/create#is_single_node) - * + Clusters with ``kind = CLASSIC_PREVIEW`` support the + following fields, whereas clusters with no - [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - * + specified ``kind`` do not. - [data_security_mode](/api/workspace/clusters/create#data_security_mode) - set to - `DATA_SECURITY_MODE_AUTO`, `DATA_SECURITY_MODE_DEDICATED`, - or `DATA_SECURITY_MODE_STANDARD` + - + [is_single_node](/api/workspace/clusters/create#is_single_node) + - + [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - By using the [simple form], your clusters are automatically - using `kind = CLASSIC_PREVIEW`. + By using the `simple form + `__, + your - [simple form]: - https://docs.databricks.com/compute/simple-form.html + clusters are automatically using ``kind = CLASSIC_PREVIEW``. node_type_id: type: string description: >- @@ -1798,20 +1918,22 @@ paths: example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes + `__ + API call. num_workers: type: integer description: >- Number of worker nodes that this cluster should have. A - cluster has one Spark Driver and `num_workers` Executors for - a total of `num_workers` + 1 Spark nodes. Note: When reading - the properties of a cluster, this field reflects the desired - number of workers rather than the actual current number of - workers. For instance, if a cluster is resized from 5 to 10 - workers, this field will immediately be updated to reflect - the target size of 10 workers, whereas the workers listed in - `spark_info` will gradually increase from 5 to 10 as the new - nodes are provisioned. + cluster has one Spark Driver and ``num_workers`` Executors + for a total of ``num_workers`` + 1 Spark nodes. Note: When + reading the properties of a cluster, this field reflects the + desired number of workers rather than the actual current + number of workers. For instance, if a cluster is resized + from 5 to 10 workers, this field will immediately be updated + to reflect the target size of 10 workers, whereas the + workers listed in ``spark_info`` will gradually increase + from 5 to 10 as the new nodes are provisioned. policy_id: type: string description: >- @@ -1832,38 +1954,38 @@ paths: description: >- Determines the cluster's runtime engine, either standard or Photon. This field is not compatible with legacy - `spark_version` values that contain `-photon-`. Remove - `-photon-` from the `spark_version` and set `runtime_engine` - to `PHOTON`. If left unspecified, the runtime engine - defaults to standard unless the spark_version contains - -photon-, in which case Photon will be used. + ``spark_version`` values that contain ``-photon-``. Remove + ``-photon-`` from the ``spark_version`` and set + ``runtime_engine`` to ``PHOTON``. If left unspecified, the + runtime engine defaults to standard unless the spark_version + contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: Single user name if data_security_mode is ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors - via `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + via ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is - (i.e., `export X='Y'`) while launching the driver and + (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional set of - `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. + ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark - environment variables: `{"SPARK_WORKER_MEMORY": "28000m", - "SPARK_LOCAL_DIRS": "/local_disk0"}` or - `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + environment variables: ``{"SPARK_WORKER_MEMORY": "28000m", + "SPARK_LOCAL_DIRS": "/local_disk0"}`` or + ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` ssh_public_keys: type: array items: @@ -1871,8 +1993,8 @@ paths: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be - used to login with the user name `ubuntu` on port `2200`. Up - to 10 keys can be specified. + used to login with the user name ``ubuntu`` on port + ``2200``. Up to 10 keys can be specified. total_initial_remote_disk_size: type: integer description: >- @@ -1882,10 +2004,11 @@ paths: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. - `effective_spark_version` is determined by `spark_version` - (DBR release), this field `use_ml_runtime`, and whether - `node_type_id` is gpu node or not. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by + ``spark_version`` (DBR release), this field + ``use_ml_runtime``, and whether ``node_type_id`` is gpu node + or not. worker_node_type_flexibility: type: object properties: @@ -1950,10 +2073,10 @@ paths: Terminates the Spark cluster with the specified ID. The cluster is removed asynchronously. Once the - termination has completed, the cluster will be in a `TERMINATED` state. - If the cluster is already in a + termination has completed, the cluster will be in a ``TERMINATED`` + state. If the cluster is already in - `TERMINATING` or `TERMINATED` state, nothing will happen. + a ``TERMINATING`` or ``TERMINATED`` state, nothing will happen. :param cluster_id: str @@ -2005,23 +2128,23 @@ paths: Updates the configuration of a cluster to match the provided attributes and size. A cluster can be - updated if it is in a `RUNNING` or `TERMINATED` state. + updated if it is in a ``RUNNING`` or ``TERMINATED`` state. - If a cluster is updated while in a `RUNNING` state, it will be restarted - so that the new attributes + If a cluster is updated while in a ``RUNNING`` state, it will be + restarted so that the new attributes can take effect. - If a cluster is updated while in a `TERMINATED` state, it will remain - `TERMINATED`. The next time it + If a cluster is updated while in a ``TERMINATED`` state, it will remain + ``TERMINATED``. The next time - is started using the `clusters/start` API, the new attributes will take - effect. Any attempt to update + it is started using the ``clusters/start`` API, the new attributes will + take effect. Any attempt to - a cluster in any other state will be rejected with an `INVALID_STATE` - error code. + update a cluster in any other state will be rejected with an + ``INVALID_STATE`` error code. Clusters created by the Databricks Jobs service cannot be edited. @@ -2030,8 +2153,9 @@ paths: :param cluster_id: str ID of the cluster :param spark_version: str - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of available Spark versions can be - retrieved by using the :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list of available Spark versions can + be retrieved by using the `clusters/sparkVersions + `__ API call. :param apply_policy_default_values: bool (optional) When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied. @@ -2052,21 +2176,22 @@ paths: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every - `5 mins`. The destination of driver logs is `$destination/$clusterId/driver`, while the destination - of executor logs is `$destination/$clusterId/executor`. + ``5 mins``. The destination of driver logs is ``$destination/$clusterId/driver``, while the + destination of executor logs is ``$destination/$clusterId/executor``. :param cluster_name: str (optional) Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs. :param custom_tags: Dict[str,str] (optional) Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS - instances and EBS volumes) with these tags in addition to `default_tags`. Notes: + instances and EBS volumes) with these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags :param data_security_mode: :class:`DataSecurityMode` (optional) + :param dependency_mode: :class:`DependencyMode` (optional) + Controls dependency configuration for the cluster. :param docker_image: :class:`DockerImage` (optional) Custom docker image BYOC :param driver_instance_pool_id: str (optional) @@ -2077,7 +2202,7 @@ paths: Flexible node type configuration for the driver node. :param driver_node_type_id: str (optional) The node type of the Spark driver. Note that this field is optional; if unset, the driver node type - will be set as the same value as `node_type_id` defined above. + will be set as the same value as ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and @@ -2092,30 +2217,31 @@ paths: creation, a set of default values will be used. :param init_scripts: List[:class:`InitScriptInfo`] (optional) The configuration for storing init scripts. Any number of destinations can be specified. The scripts - are executed sequentially in the order provided. If `cluster_log_conf` is specified, init script - logs are sent to `//init_scripts`. + are executed sequentially in the order provided. If ``cluster_log_conf`` is specified, init script + logs are sent to ``//init_scripts``. :param instance_pool_id: str (optional) The optional ID of the instance pool to which the cluster belongs. :param is_single_node: bool (optional) - This field can only be used when `kind = CLASSIC_PREVIEW`. + This field can only be used when ``kind = CLASSIC_PREVIEW``. - When set to true, Databricks will automatically set single node related `custom_tags`, `spark_conf`, - and `num_workers` + When set to true, Databricks will automatically set single node related ``custom_tags``, + ``spark_conf``, and ``num_workers`` :param kind: :class:`Kind` (optional) :param node_type_id: str (optional) This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes `__ API + call. :param num_workers: int (optional) Number of worker nodes that this cluster should have. A cluster has one Spark Driver and - `num_workers` Executors for a total of `num_workers` + 1 Spark nodes. + ``num_workers`` Executors for a total of ``num_workers`` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas - the workers listed in `spark_info` will gradually increase from 5 to 10 as the new nodes are + the workers listed in ``spark_info`` will gradually increase from 5 to 10 as the new nodes are provisioned. :param policy_id: str (optional) The ID of the cluster policy used to create the cluster if applicable. @@ -2125,41 +2251,41 @@ paths: :param runtime_engine: :class:`RuntimeEngine` (optional) Determines the cluster's runtime engine, either standard or Photon. - This field is not compatible with legacy `spark_version` values that contain `-photon-`. Remove - `-photon-` from the `spark_version` and set `runtime_engine` to `PHOTON`. + This field is not compatible with legacy ``spark_version`` values that contain ``-photon-``. Remove + ``-photon-`` from the ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used. :param single_user_name: str (optional) - Single user name if data_security_mode is `SINGLE_USER` + Single user name if data_security_mode is ``SINGLE_USER`` :param spark_conf: Dict[str,str] (optional) An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via - `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` respectively. + ``spark.driver.extraJavaOptions`` and ``spark.executor.extraJavaOptions`` respectively. :param spark_env_vars: Dict[str,str] (optional) An object containing a set of optional, user-specified environment variable key-value pairs. Please - note that key-value pair of the form (X,Y) will be exported as is (i.e., `export X='Y'`) while + note that key-value pair of the form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. - In order to specify an additional set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This ensures that all default databricks + In order to specify an additional set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. - Example Spark environment variables: `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + Example Spark environment variables: ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` :param ssh_public_keys: List[str] (optional) SSH public key contents that will be added to each Spark node in this cluster. The corresponding - private keys can be used to login with the user name `ubuntu` on port `2200`. Up to 10 keys can be - specified. + private keys can be used to login with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can + be specified. :param total_initial_remote_disk_size: int (optional) If set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED disks. :param use_ml_runtime: bool (optional) - This field can only be used when `kind = CLASSIC_PREVIEW`. + This field can only be used when ``kind = CLASSIC_PREVIEW``. - `effective_spark_version` is determined by `spark_version` (DBR release), this field - `use_ml_runtime`, and whether `node_type_id` is gpu node or not. + ``effective_spark_version`` is determined by ``spark_version`` (DBR release), this field + ``use_ml_runtime``, and whether ``node_type_id`` is gpu node or not. :param worker_node_type_flexibility: :class:`NodeTypeFlexibility` (optional) Flexible node type configuration for worker nodes. @@ -2181,9 +2307,11 @@ paths: spark_version: type: string description: >- - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A - list of available Spark versions can be retrieved by using - the :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. + A list of available Spark versions can be retrieved by using + the `clusters/sparkVersions + `__ + API call. apply_policy_default_values: type: boolean description: >- @@ -2224,11 +2352,13 @@ paths: - SPOT_WITH_FALLBACK description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. + ``first_on_demand`` ones. + + Note: If ``first_on_demand`` is zero, this availability + type will be used for the entire - Note: If `first_on_demand` is zero, this availability - type will be used for the entire cluster. + cluster. ebs_volume_count: type: integer description: >- @@ -2238,17 +2368,17 @@ paths: cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail. These - EBS volumes will be mounted at `/ebs0`, `/ebs1`, and + EBS volumes will be mounted at ``/ebs0``, ``/ebs1``, and etc. Instance store volumes will be mounted at - `/local_disk0`, `/local_disk1`, and etc. If EBS volumes - are attached, Databricks will configure Spark to use - only the EBS volumes for scratch storage because + ``/local_disk0``, ``/local_disk1``, and etc. If EBS + volumes are attached, Databricks will configure Spark to + use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, Databricks will configure Spark to use instance store volumes. Please note that if EBS volumes are specified, then the Spark configuration - `spark.local.dir` will be overridden. + ``spark.local.dir`` will be overridden. ebs_volume_iops: type: integer description: >- @@ -2279,18 +2409,18 @@ paths: first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be - placed on on-demand instances. If this value is greater - than 0, the cluster driver node in particular will be - placed on an on-demand instance. If this value is - greater than or equal to the current cluster size, all - nodes will be placed on on-demand instances. If this + The first ``first_on_demand`` nodes of the cluster will + be placed on on-demand instances. If this value is + greater than 0, the cluster driver node in particular + will be placed on an on-demand instance. If this value + is greater than or equal to the current cluster size, + all nodes will be placed on on-demand instances. If this value is less than the current cluster size, - `first_on_demand` nodes will be placed on on-demand + ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not - affect cluster size and cannot currently be mutated over - the lifetime of a cluster. + ``availability`` instances. Note that this value does + not affect cluster size and cannot currently be mutated + over the lifetime of a cluster. instance_profile_arn: type: string description: >- @@ -2307,10 +2437,10 @@ paths: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster - needs a new `r3.xlarge` spot instance, then the bid - price is half of the price of on-demand `r3.xlarge` + needs a new ``r3.xlarge`` spot instance, then the bid + price is half of the price of on-demand ``r3.xlarge`` instances. Similarly, if this field is set to 200, the - bid price is twice the price of on-demand `r3.xlarge` + bid price is twice the price of on-demand ``r3.xlarge`` instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this @@ -2331,7 +2461,7 @@ paths: with high availability, and will retry placement in a different AZ if there is not enough capacity. The list of available zones as well as the default value can be - found by using the `List Zones` method. + found by using the ``List Zones`` method. description: >- Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of @@ -2347,24 +2477,43 @@ paths: - SPOT_WITH_FALLBACK_AZURE description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. Note: If `first_on_demand` is - zero, this availability type will be used for the entire - cluster. + ``first_on_demand`` ones. Note: If ``first_on_demand`` + is zero, this availability type will be used for the + entire cluster. + capacity_reservation_group: + type: string + description: >- + The Azure capacity reservation group resource ID to use + for launching VMs. When specified, VMs will be launched + using the provided capacity reservation. Capacity + reservations can only be specified when the workspace + uses injected vnet (i.e. customer defined vnet not + managed by databricks). Ensure the databricks-login-prod + Enterprise Application is granted the following four + permissions: 1. + Microsoft.Compute/capacityReservationGroups/read 2. + Microsoft.Compute/capacityReservationGroups/deploy/action + 3. + Microsoft.Compute/capacityReservationGroups/capacityReservations/read + 4. + Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action + Format: + ``/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}`` first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be - placed on on-demand instances. This value should be + The first ``first_on_demand`` nodes of the cluster will + be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, - `first_on_demand` nodes will be placed on on-demand + ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not - affect cluster size and cannot currently be mutated over - the lifetime of a cluster. + ``availability`` instances. Note that this value does + not affect cluster size and cannot currently be mutated + over the lifetime of a cluster. log_analytics_info: type: object properties: @@ -2397,19 +2546,19 @@ paths: properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: >- - destination needs to be provided. e.g. `{ "dbfs" : { - "destination" : "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs" : { + "destination" : "dbfs:/home/cluster_log" } }`` s3: type: object properties: destination: type: string description: >- - S3 destination, e.g. `s3://my-bucket/some-prefix` + S3 destination, e.g. ``s3://my-bucket/some-prefix`` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please @@ -2419,55 +2568,57 @@ paths: type: string description: >- (Optional) Set canned access control list for the - logs, e.g. `bucket-owner-full-control`. If - `canned_cal` is set, please make sure the cluster - iam role has `s3:PutObjectAcl` permission on the + logs, e.g. ``bucket-owner-full-control``. If + ``canned_cal`` is set, please make sure the cluster + iam role has ``s3:PutObjectAcl`` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set - `bucket-owner-full-control` to make bucket owner + ``bucket-owner-full-control`` to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- (Optional) Flag to enable server side encryption, - `false` by default. + ``false`` by default. encryption_type: type: string description: >- - (Optional) The encryption type, it could be `sse-s3` - or `sse-kms`. It will be used only when encryption - is enabled and the default type is `sse-s3`. + (Optional) The encryption type, it could be + ``sse-s3`` or ``sse-kms``. It will be used only when + encryption is enabled and the default type is + ``sse-s3``. endpoint: type: string description: >- S3 endpoint, e.g. - `https://s3-us-west-2.amazonaws.com`. Either region - or endpoint needs to be set. If both are set, + ``https://s3-us-west-2.amazonaws.com``. Either + region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: type: string description: >- (Optional) Kms key which will be used if encryption - is enabled and encryption type is set to `sse-kms`. + is enabled and encryption type is set to + ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or + S3 region, e.g. ``us-west-2``. Either region or endpoint needs to be set. If both are set, endpoint will be used. required: - destination description: >- destination and either the region or endpoint need to be - provided. e.g. `{ "s3": { "destination" : + provided. e.g. ``{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" - } }` Cluster iam role is used to access s3, please make - sure the cluster iam role in `instance_profile_arn` has - permission to write data to the s3 destination. + } }`` Cluster iam role is used to access s3, please make + sure the cluster iam role in ``instance_profile_arn`` + has permission to write data to the s3 destination. volumes: type: object properties: @@ -2475,24 +2626,24 @@ paths: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided, e.g. `{ "volumes": { + destination needs to be provided, e.g. ``{ "volumes": { "destination": - "/Volumes/catalog/schema/volume/cluster_log" } }` + "/Volumes/catalog/schema/volume/cluster_log" } }`` description: >- The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every - `5 mins`. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination of - executor logs is `$destination/$clusterId/executor`. + ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. cluster_name: type: string description: >- @@ -2505,7 +2656,7 @@ paths: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) - with these tags in addition to `default_tags`. Notes: - + with these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags @@ -2527,40 +2678,27 @@ paths: when accessing data from a cluster. - The following modes can only be used when `kind = - CLASSIC_PREVIEW`. * `DATA_SECURITY_MODE_AUTO`: - - Databricks will choose the most appropriate access mode - depending on your compute configuration. - - * `DATA_SECURITY_MODE_STANDARD`: Alias for `USER_ISOLATION`. - * `DATA_SECURITY_MODE_DEDICATED`: - - Alias for `SINGLE_USER`. - + - ``DATA_SECURITY_MODE_AUTO``: Databricks will choose the + most appropriate access mode depending + on your compute configuration. + - ``DATA_SECURITY_MODE_STANDARD``: A secure cluster that can + be shared by multiple users. + Cluster users are fully isolated so that they cannot see each other’s data and credentials. + Most data governance features are supported in this mode. But programming languages and + cluster features might be limited. + - ``DATA_SECURITY_MODE_DEDICATED``: A secure cluster that + can only be exclusively used by a + single user specified in ``single_user_name``. Most programming languages, cluster features + and data governance features are available in this mode. - The following modes can be used regardless of `kind`. * - `NONE`: No security isolation for + The following modes are legacy aliases for the above modes: - multiple users sharing the cluster. Data governance features - are not available in this mode. * - `SINGLE_USER`: A secure cluster that can only be exclusively - used by a single user specified in + - ``USER_ISOLATION``: Legacy alias for + ``DATA_SECURITY_MODE_STANDARD``. - `single_user_name`. Most programming languages, cluster - features and data governance features - - are available in this mode. * `USER_ISOLATION`: A secure - cluster that can be shared by multiple - - users. Cluster users are fully isolated so that they cannot - see each other's data and - - credentials. Most data governance features are supported in - this mode. But programming languages - - and cluster features might be limited. + - ``SINGLE_USER``: Legacy alias for + ``DATA_SECURITY_MODE_DEDICATED``. The following modes are deprecated starting with Databricks @@ -2569,19 +2707,25 @@ paths: future Databricks Runtime versions: - * `LEGACY_TABLE_ACL`: This mode is for users migrating from - legacy Table ACL clusters. * + - ``LEGACY_TABLE_ACL``: This mode is for users migrating + from legacy Table ACL clusters. - `LEGACY_PASSTHROUGH`: This mode is for users migrating from - legacy Passthrough on high - - concurrency clusters. * `LEGACY_SINGLE_USER`: This mode is - for users migrating from legacy - - Passthrough on standard clusters. * - `LEGACY_SINGLE_USER_STANDARD`: This mode provides a way that - - doesn’t have UC nor passthrough enabled. + - ``LEGACY_PASSTHROUGH``: This mode is for users migrating + from legacy Passthrough on high + concurrency clusters. + - ``LEGACY_SINGLE_USER``: This mode is for users migrating + from legacy Passthrough on standard + clusters. + - ``LEGACY_SINGLE_USER_STANDARD``: This mode provides a way + that doesn’t have UC nor + passthrough enabled. + dependency_mode: + type: string + x-enum: + - DEPENDENCY_MODE_AUTO + - DEPENDENCY_MODE_CLUSTER_LIBRARIES + - DEPENDENCY_MODE_ENVIRONMENTS + description: Controls dependency configuration for the cluster. docker_image: type: object properties: @@ -2620,7 +2764,7 @@ paths: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the - same value as `node_type_id` defined above. This field, + same value as ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, @@ -2651,21 +2795,31 @@ paths: boot_disk_size: type: integer description: Boot disk size in GB + confidential_compute_type: + type: string + x-enum: + - CONFIDENTIAL_COMPUTE_TYPE_NONE + - SEV_SNP + description: >- + The confidential computing technology for this cluster's + instances. Currently only SEV_SNP is supported, and only + on N2D instance types. When not set, no confidential + computing is applied. first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be - placed on on-demand instances. This value should be + The first ``first_on_demand`` nodes of the cluster will + be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, - `first_on_demand` nodes will be placed on on-demand + ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not - affect cluster size and cannot currently be mutated over - the lifetime of a cluster. + ``availability`` instances. Note that this value does + not affect cluster size and cannot currently be mutated + over the lifetime of a cluster. google_service_account: type: string description: >- @@ -2679,10 +2833,11 @@ paths: description: >- If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. - Each local SSD is 375GB in size. Refer to [GCP - documentation] for the supported number of local SSDs - for each instance type. [GCP documentation]: - https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds + Each local SSD is 375GB in size. Refer to `GCP + documentation + `__ + for the supported number of local SSDs for each instance + type. use_preemptible_executors: type: boolean description: >- @@ -2719,23 +2874,23 @@ paths: type: string description: >- abfss destination, e.g. - `abfss://@.dfs.core.windows.net/`. + ``abfss://@.dfs.core.windows.net/``. required: - destination description: >- destination needs to be provided, e.g. - `abfss://@.dfs.core.windows.net/` + ``abfss://@.dfs.core.windows.net/`` dbfs: type: object properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: >- - destination needs to be provided. e.g. `{ "dbfs": { - "destination" : "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs": { + "destination" : "dbfs:/home/cluster_log" } }`` file: type: object properties: @@ -2744,8 +2899,8 @@ paths: required: - destination description: >- - destination needs to be provided, e.g. `{ "file": { - "destination": "file:/my/local/file.sh" } }` + destination needs to be provided, e.g. ``{ "file": { + "destination": "file:/my/local/file.sh" } }`` gcs: type: object properties: @@ -2753,56 +2908,57 @@ paths: type: string description: >- GCS destination/URI, e.g. - `gs://my-bucket/some-prefix` + ``gs://my-bucket/some-prefix`` required: - destination description: >- - destination needs to be provided, e.g. `{ "gcs": { - "destination": "gs://my-bucket/file.sh" } }` + destination needs to be provided, e.g. ``{ "gcs": { + "destination": "gs://my-bucket/file.sh" } }`` s3: type: object properties: destination: type: string description: >- - S3 destination, e.g. `s3://my-bucket/some-prefix` - Note that logs will be delivered using cluster iam - role, please make sure you set cluster iam role - and the role has write access to the destination. - Please also note that you cannot use AWS keys to - deliver logs. + S3 destination, e.g. + ``s3://my-bucket/some-prefix`` Note that logs will + be delivered using cluster iam role, please make + sure you set cluster iam role and the role has + write access to the destination. Please also note + that you cannot use AWS keys to deliver logs. canned_acl: type: string description: >- (Optional) Set canned access control list for the - logs, e.g. `bucket-owner-full-control`. If - `canned_cal` is set, please make sure the cluster - iam role has `s3:PutObjectAcl` permission on the - destination bucket and prefix. The full list of - possible canned acl can be found at + logs, e.g. ``bucket-owner-full-control``. If + ``canned_cal`` is set, please make sure the + cluster iam role has ``s3:PutObjectAcl`` + permission on the destination bucket and prefix. + The full list of possible canned acl can be found + at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set - `bucket-owner-full-control` to make bucket owner + ``bucket-owner-full-control`` to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- (Optional) Flag to enable server side encryption, - `false` by default. + ``false`` by default. encryption_type: type: string description: >- (Optional) The encryption type, it could be - `sse-s3` or `sse-kms`. It will be used only when - encryption is enabled and the default type is - `sse-s3`. + ``sse-s3`` or ``sse-kms``. It will be used only + when encryption is enabled and the default type is + ``sse-s3``. endpoint: type: string description: >- S3 endpoint, e.g. - `https://s3-us-west-2.amazonaws.com`. Either + ``https://s3-us-west-2.amazonaws.com``. Either region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: @@ -2810,23 +2966,23 @@ paths: description: >- (Optional) Kms key which will be used if encryption is enabled and encryption type is set - to `sse-kms`. + to ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or + S3 region, e.g. ``us-west-2``. Either region or endpoint needs to be set. If both are set, endpoint will be used. required: - destination description: >- destination and either the region or endpoint need to - be provided. e.g. `{ \"s3\": { \"destination\": + be provided. e.g. ``{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": - \"us-west-2\" } }` Cluster iam role is used to access + \"us-west-2\" } }`` Cluster iam role is used to access s3, please make sure the cluster iam role in - `instance_profile_arn` has permission to write data to - the s3 destination. + ``instance_profile_arn`` has permission to write data + to the s3 destination. volumes: type: object properties: @@ -2834,14 +2990,14 @@ paths: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided. e.g. `{ \"volumes\" - : { \"destination\" : \"/Volumes/my-init.sh\" } }` + destination needs to be provided. e.g. ``{ \"volumes\" + : { \"destination\" : \"/Volumes/my-init.sh\" } }`` workspace: type: object properties: @@ -2849,20 +3005,20 @@ paths: type: string description: >- wsfs destination, e.g. - `workspace:/cluster-init-scripts/setup-datadog.sh` + ``workspace:/cluster-init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided, e.g. `{ "workspace": - { "destination": - "/cluster-init-scripts/setup-datadog.sh" } }` - description: 'Config for an individual init script Next ID: 11' + destination needs to be provided, e.g. ``{ + "workspace": { "destination": + "/cluster-init-scripts/setup-datadog.sh" } }`` + description: Config for an individual init script description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed - sequentially in the order provided. If `cluster_log_conf` is - specified, init script logs are sent to - `//init_scripts`. + sequentially in the order provided. If ``cluster_log_conf`` + is specified, init script logs are sent to + ``//init_scripts``. instance_pool_id: type: string description: >- @@ -2871,9 +3027,10 @@ paths: is_single_node: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. + This field can only be used when ``kind = CLASSIC_PREVIEW``. When set to true, Databricks will automatically set single - node related `custom_tags`, `spark_conf`, and `num_workers` + node related ``custom_tags``, ``spark_conf``, and + ``num_workers`` kind: type: string x-enum: @@ -2882,33 +3039,28 @@ paths: The kind of compute described by this compute specification. - Depending on `kind`, different validations and default + Depending on ``kind``, different validations and default values will be applied. - Clusters with `kind = CLASSIC_PREVIEW` support the following - fields, whereas clusters with no + Clusters with ``kind = CLASSIC_PREVIEW`` support the + following fields, whereas clusters with no - specified `kind` do not. * - [is_single_node](/api/workspace/clusters/create#is_single_node) - * - - [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - * + specified ``kind`` do not. - [data_security_mode](/api/workspace/clusters/create#data_security_mode) - set to - `DATA_SECURITY_MODE_AUTO`, `DATA_SECURITY_MODE_DEDICATED`, - or `DATA_SECURITY_MODE_STANDARD` + - + [is_single_node](/api/workspace/clusters/create#is_single_node) + - + [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - By using the [simple form], your clusters are automatically - using `kind = CLASSIC_PREVIEW`. + By using the `simple form + `__, + your - [simple form]: - https://docs.databricks.com/compute/simple-form.html + clusters are automatically using ``kind = CLASSIC_PREVIEW``. node_type_id: type: string description: >- @@ -2917,20 +3069,22 @@ paths: example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes + `__ + API call. num_workers: type: integer description: >- Number of worker nodes that this cluster should have. A - cluster has one Spark Driver and `num_workers` Executors for - a total of `num_workers` + 1 Spark nodes. Note: When reading - the properties of a cluster, this field reflects the desired - number of workers rather than the actual current number of - workers. For instance, if a cluster is resized from 5 to 10 - workers, this field will immediately be updated to reflect - the target size of 10 workers, whereas the workers listed in - `spark_info` will gradually increase from 5 to 10 as the new - nodes are provisioned. + cluster has one Spark Driver and ``num_workers`` Executors + for a total of ``num_workers`` + 1 Spark nodes. Note: When + reading the properties of a cluster, this field reflects the + desired number of workers rather than the actual current + number of workers. For instance, if a cluster is resized + from 5 to 10 workers, this field will immediately be updated + to reflect the target size of 10 workers, whereas the + workers listed in ``spark_info`` will gradually increase + from 5 to 10 as the new nodes are provisioned. policy_id: type: string description: >- @@ -2951,38 +3105,38 @@ paths: description: >- Determines the cluster's runtime engine, either standard or Photon. This field is not compatible with legacy - `spark_version` values that contain `-photon-`. Remove - `-photon-` from the `spark_version` and set `runtime_engine` - to `PHOTON`. If left unspecified, the runtime engine - defaults to standard unless the spark_version contains - -photon-, in which case Photon will be used. + ``spark_version`` values that contain ``-photon-``. Remove + ``-photon-`` from the ``spark_version`` and set + ``runtime_engine`` to ``PHOTON``. If left unspecified, the + runtime engine defaults to standard unless the spark_version + contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: Single user name if data_security_mode is ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors - via `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + via ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is - (i.e., `export X='Y'`) while launching the driver and + (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional set of - `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. + ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark - environment variables: `{"SPARK_WORKER_MEMORY": "28000m", - "SPARK_LOCAL_DIRS": "/local_disk0"}` or - `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + environment variables: ``{"SPARK_WORKER_MEMORY": "28000m", + "SPARK_LOCAL_DIRS": "/local_disk0"}`` or + ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` ssh_public_keys: type: array items: @@ -2990,8 +3144,8 @@ paths: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be - used to login with the user name `ubuntu` on port `2200`. Up - to 10 keys can be specified. + used to login with the user name ``ubuntu`` on port + ``2200``. Up to 10 keys can be specified. total_initial_remote_disk_size: type: integer description: >- @@ -3001,10 +3155,11 @@ paths: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. - `effective_spark_version` is determined by `spark_version` - (DBR release), this field `use_ml_runtime`, and whether - `node_type_id` is gpu node or not. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by + ``spark_version`` (DBR release), this field + ``use_ml_runtime``, and whether ``node_type_id`` is gpu node + or not. worker_node_type_flexibility: type: object properties: @@ -3133,6 +3288,8 @@ paths: - DBFS_DOWN - DECOMMISSION_ENDED - DECOMMISSION_STARTED + - DEFERRED_POLICY_ENFORCEMENT_FAILED + - DEFERRED_POLICY_ENFORCEMENT_SCHEDULED - DID_NOT_EXPAND_DISK - DRIVER_HEALTHY - DRIVER_NOT_RESPONDING @@ -3321,6 +3478,71 @@ paths: message: type: string x-stackql-sdk-source: ClustersAPI + /api/2.1/{name}/diagnostics: + get: + operationId: clusters_get_diagnostic + summary: Returns the most recent cluster diagnostics result for a cluster. + tags: + - compute + - clusters + description: >- + Returns the most recent cluster diagnostics result for a cluster. + + + Cluster diagnostics are a set of environment checks -- for example, + network reachability to the + + control plane, storage, DNS, and the SCC tunnel -- that verify the + cluster's environment is set up + + correctly for the cluster to provision. The checks run automatically + while the cluster is starting, + + once per cluster start, and their results are stored. + + + This method only reads the latest stored result; it does NOT run the + checks. The response reports an + + overall status plus a per-check breakdown (check status, failure reason, + remediation, and + + description). If diagnostics have not run for the cluster, the overall + status is NOT_RUN. + + + :param name: str + The resource name of the cluster whose diagnostics to retrieve. Format: clusters/{cluster_id} + + :returns: :class:`Diagnostic` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The resource name of the cluster whose diagnostics to retrieve. + Format: clusters/{cluster_id} + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Diagnostic' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ClustersAPI /api/2.0/permissions/clusters/{cluster_id}/permissionLevels: get: operationId: clusters_get_permission_levels @@ -3807,7 +4029,7 @@ paths: Resizes a cluster to have a desired number of workers. This will fail unless the cluster is in a - `RUNNING` state. + ``RUNNING`` state. :param cluster_id: str @@ -3817,12 +4039,12 @@ paths: autoscaling works best with DB runtime versions 3.0 or later. :param num_workers: int (optional) Number of worker nodes that this cluster should have. A cluster has one Spark Driver and - `num_workers` Executors for a total of `num_workers` + 1 Spark nodes. + ``num_workers`` Executors for a total of ``num_workers`` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas - the workers listed in `spark_info` will gradually increase from 5 to 10 as the new nodes are + the workers listed in ``spark_info`` will gradually increase from 5 to 10 as the new nodes are provisioned. :returns: @@ -3856,15 +4078,15 @@ paths: type: integer description: >- Number of worker nodes that this cluster should have. A - cluster has one Spark Driver and `num_workers` Executors for - a total of `num_workers` + 1 Spark nodes. Note: When reading - the properties of a cluster, this field reflects the desired - number of workers rather than the actual current number of - workers. For instance, if a cluster is resized from 5 to 10 - workers, this field will immediately be updated to reflect - the target size of 10 workers, whereas the workers listed in - `spark_info` will gradually increase from 5 to 10 as the new - nodes are provisioned. + cluster has one Spark Driver and ``num_workers`` Executors + for a total of ``num_workers`` + 1 Spark nodes. Note: When + reading the properties of a cluster, this field reflects the + desired number of workers rather than the actual current + number of workers. For instance, if a cluster is resized + from 5 to 10 workers, this field will immediately be updated + to reflect the target size of 10 workers, whereas the + workers listed in ``spark_info`` will gradually increase + from 5 to 10 as the new nodes are provisioned. required: - cluster_id responses: @@ -3891,13 +4113,13 @@ paths: operationId: clusters_restart summary: >- Restarts a Spark cluster with the supplied ID. If the cluster is not - currently in a `RUNNING` state, + currently in a ``RUNNING`` state, tags: - compute - clusters description: >- Restarts a Spark cluster with the supplied ID. If the cluster is not - currently in a `RUNNING` state, + currently in a ``RUNNING`` state, nothing will happen. @@ -3982,24 +4204,28 @@ paths: operationId: clusters_start summary: >- Starts a terminated Spark cluster with the supplied ID. This works - similar to `createCluster` except: + similar to ``createCluster`` tags: - compute - clusters description: >- Starts a terminated Spark cluster with the supplied ID. This works - similar to `createCluster` except: + similar to ``createCluster`` + + except: - - The previous cluster id and attributes are preserved. - The cluster - starts with the last specified - cluster size. - If the previous cluster was an autoscaling cluster, the - current cluster starts with + - The previous cluster id and attributes are preserved. - the minimum number of nodes. - If the cluster is not currently in a - ``TERMINATED`` state, nothing will + - The cluster starts with the last specified cluster size. - happen. - Clusters launched to run a job cannot be started. + - If the previous cluster was an autoscaling cluster, the current + cluster starts with the minimum + number of nodes. + - If the cluster is not currently in a ``TERMINATED`` state, nothing + will happen. + + - Clusters launched to run a job cannot be started. :param cluster_id: str @@ -4096,25 +4322,25 @@ paths: Updates the configuration of a cluster to match the partial set of attributes and size. Denote which - fields to update using the `update_mask` field in the request body. A - cluster can be updated if it is + fields to update using the ``update_mask`` field in the request body. A + cluster can be updated if it - in a `RUNNING` or `TERMINATED` state. If a cluster is updated while in a - `RUNNING` state, it will be + is in a ``RUNNING`` or ``TERMINATED`` state. If a cluster is updated + while in a ``RUNNING`` state, it - restarted so that the new attributes can take effect. If a cluster is - updated while in a `TERMINATED` + will be restarted so that the new attributes can take effect. If a + cluster is updated while in a - state, it will remain `TERMINATED`. The updated attributes will take - effect the next time the cluster + ``TERMINATED`` state, it will remain ``TERMINATED``. The updated + attributes will take effect the next - is started using the `clusters/start` API. Attempts to update a cluster - in any other state will be + time the cluster is started using the ``clusters/start`` API. Attempts + to update a cluster in any - rejected with an `INVALID_STATE` error code. Clusters created by the - Databricks Jobs service cannot be + other state will be rejected with an ``INVALID_STATE`` error code. + Clusters created by the Databricks - updated. + Jobs service cannot be updated. :param cluster_id: str @@ -4124,14 +4350,14 @@ paths: for more details. The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :param cluster: :class:`UpdateClusterResource` (optional) The cluster to be updated. @@ -4154,15 +4380,15 @@ paths: update. See https://google.aip.dev/161 for more details. The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative - to the resource object, using a dot (`.`) to navigate - sub-fields (e.g., `author.given_name`). Specification of + to the resource object, using a dot (``.``) to navigate + sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. A field mask of - `*` indicates full replacement. It’s recommended to always - explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API - changes in the future. + ``*`` indicates full replacement. It’s recommended to always + explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. cluster: type: object properties: @@ -4198,11 +4424,12 @@ paths: - SPOT_WITH_FALLBACK description: >- Availability type used for all subsequent nodes past - the `first_on_demand` ones. + the ``first_on_demand`` ones. - Note: If `first_on_demand` is zero, this + Note: If ``first_on_demand`` is zero, this availability type will be used for the entire + cluster. ebs_volume_count: type: integer @@ -4214,9 +4441,9 @@ paths: types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail. These EBS volumes will be - mounted at `/ebs0`, `/ebs1`, and etc. Instance store - volumes will be mounted at `/local_disk0`, - `/local_disk1`, and etc. If EBS volumes are + mounted at ``/ebs0``, ``/ebs1``, and etc. Instance + store volumes will be mounted at ``/local_disk0``, + ``/local_disk1``, and etc. If EBS volumes are attached, Databricks will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to @@ -4224,7 +4451,7 @@ paths: attached, Databricks will configure Spark to use instance store volumes. Please note that if EBS volumes are specified, then the Spark configuration - `spark.local.dir` will be overridden. + ``spark.local.dir`` will be overridden. ebs_volume_iops: type: integer description: >- @@ -4256,16 +4483,16 @@ paths: first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster + The first ``first_on_demand`` nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the - current cluster size, `first_on_demand` nodes will + current cluster size, ``first_on_demand`` nodes will be placed on on-demand instances and the remainder - will be placed on `availability` instances. Note + will be placed on ``availability`` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. @@ -4285,16 +4512,17 @@ paths: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set - to 50, and the cluster needs a new `r3.xlarge` spot - instance, then the bid price is half of the price of - on-demand `r3.xlarge` instances. Similarly, if this - field is set to 200, the bid price is twice the - price of on-demand `r3.xlarge` instances. If not - specified, the default value is 100. When spot - instances are requested for this cluster, only spot - instances whose bid price percentage matches this - field will be considered. Note that, for safety, we - enforce this field to be no more than 10000. + to 50, and the cluster needs a new ``r3.xlarge`` + spot instance, then the bid price is half of the + price of on-demand ``r3.xlarge`` instances. + Similarly, if this field is set to 200, the bid + price is twice the price of on-demand ``r3.xlarge`` + instances. If not specified, the default value is + 100. When spot instances are requested for this + cluster, only spot instances whose bid price + percentage matches this field will be considered. + Note that, for safety, we enforce this field to be + no more than 10000. zone_id: type: string description: >- @@ -4311,7 +4539,7 @@ paths: high availability, and will retry placement in a different AZ if there is not enough capacity. The list of available zones as well as the default value - can be found by using the `List Zones` method. + can be found by using the ``List Zones`` method. description: >- Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of @@ -4327,23 +4555,42 @@ paths: - SPOT_WITH_FALLBACK_AZURE description: >- Availability type used for all subsequent nodes past - the `first_on_demand` ones. Note: If - `first_on_demand` is zero, this availability type + the ``first_on_demand`` ones. Note: If + ``first_on_demand`` is zero, this availability type will be used for the entire cluster. + capacity_reservation_group: + type: string + description: >- + The Azure capacity reservation group resource ID to + use for launching VMs. When specified, VMs will be + launched using the provided capacity reservation. + Capacity reservations can only be specified when the + workspace uses injected vnet (i.e. customer defined + vnet not managed by databricks). Ensure the + databricks-login-prod Enterprise Application is + granted the following four permissions: 1. + Microsoft.Compute/capacityReservationGroups/read 2. + Microsoft.Compute/capacityReservationGroups/deploy/action + 3. + Microsoft.Compute/capacityReservationGroups/capacityReservations/read + 4. + Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action + Format: + ``/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}`` first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster + The first ``first_on_demand`` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current - cluster size, `first_on_demand` nodes will be placed - on on-demand instances and the remainder will be - placed on `availability` instances. Note that this - value does not affect cluster size and cannot + cluster size, ``first_on_demand`` nodes will be + placed on on-demand instances and the remainder will + be placed on ``availability`` instances. Note that + this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. log_analytics_info: type: object @@ -4377,12 +4624,12 @@ paths: properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: >- - destination needs to be provided. e.g. `{ "dbfs" : { - "destination" : "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs" : + { "destination" : "dbfs:/home/cluster_log" } }`` s3: type: object properties: @@ -4390,19 +4637,19 @@ paths: type: string description: >- S3 destination, e.g. - `s3://my-bucket/some-prefix` Note that logs will - be delivered using cluster iam role, please make - sure you set cluster iam role and the role has - write access to the destination. Please also + ``s3://my-bucket/some-prefix`` Note that logs + will be delivered using cluster iam role, please + make sure you set cluster iam role and the role + has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. canned_acl: type: string description: >- (Optional) Set canned access control list for - the logs, e.g. `bucket-owner-full-control`. If - `canned_cal` is set, please make sure the - cluster iam role has `s3:PutObjectAcl` + the logs, e.g. ``bucket-owner-full-control``. If + ``canned_cal`` is set, please make sure the + cluster iam role has ``s3:PutObjectAcl`` permission on the destination bucket and prefix. The full list of possible canned acl can be found at @@ -4410,25 +4657,25 @@ paths: Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to - set `bucket-owner-full-control` to make bucket + set ``bucket-owner-full-control`` to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- (Optional) Flag to enable server side - encryption, `false` by default. + encryption, ``false`` by default. encryption_type: type: string description: >- (Optional) The encryption type, it could be - `sse-s3` or `sse-kms`. It will be used only when - encryption is enabled and the default type is - `sse-s3`. + ``sse-s3`` or ``sse-kms``. It will be used only + when encryption is enabled and the default type + is ``sse-s3``. endpoint: type: string description: >- S3 endpoint, e.g. - `https://s3-us-west-2.amazonaws.com`. Either + ``https://s3-us-west-2.amazonaws.com``. Either region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: @@ -4436,23 +4683,23 @@ paths: description: >- (Optional) Kms key which will be used if encryption is enabled and encryption type is set - to `sse-kms`. + to ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or + S3 region, e.g. ``us-west-2``. Either region or endpoint needs to be set. If both are set, endpoint will be used. required: - destination description: >- destination and either the region or endpoint need - to be provided. e.g. `{ "s3": { "destination" : + to be provided. e.g. ``{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : - "us-west-2" } }` Cluster iam role is used to access + "us-west-2" } }`` Cluster iam role is used to access s3, please make sure the cluster iam role in - `instance_profile_arn` has permission to write data - to the s3 destination. + ``instance_profile_arn`` has permission to write + data to the s3 destination. volumes: type: object properties: @@ -4460,25 +4707,26 @@ paths: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided, e.g. `{ "volumes": - { "destination": - "/Volumes/catalog/schema/volume/cluster_log" } }` + destination needs to be provided, e.g. ``{ + "volumes": { "destination": + "/Volumes/catalog/schema/volume/cluster_log" } }`` description: >- The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be - delivered to the destination every `5 mins`. The + delivered to the destination every ``5 mins``. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination - of executor logs is `$destination/$clusterId/executor`. + ``$destination/$clusterId/driver``, while the + destination of executor logs is + ``$destination/$clusterId/executor``. cluster_name: type: string description: >- @@ -4492,10 +4740,11 @@ paths: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS - volumes) with these tags in addition to `default_tags`. - Notes: - Currently, Databricks allows at most 45 custom - tags - Clusters can only reuse cloud resources if the - resources' tags are a subset of the cluster tags + volumes) with these tags in addition to + ``default_tags``. Notes: - Currently, Databricks allows + at most 45 custom tags - Clusters can only reuse cloud + resources if the resources' tags are a subset of the + cluster tags data_security_mode: type: string x-enum: @@ -4514,40 +4763,28 @@ paths: use when accessing data from a cluster. - The following modes can only be used when `kind = - CLASSIC_PREVIEW`. * `DATA_SECURITY_MODE_AUTO`: + - ``DATA_SECURITY_MODE_AUTO``: Databricks will choose + the most appropriate access mode depending + on your compute configuration. + - ``DATA_SECURITY_MODE_STANDARD``: A secure cluster that + can be shared by multiple users. + Cluster users are fully isolated so that they cannot see each other’s data and credentials. + Most data governance features are supported in this mode. But programming languages and + cluster features might be limited. + - ``DATA_SECURITY_MODE_DEDICATED``: A secure cluster + that can only be exclusively used by a + single user specified in ``single_user_name``. Most programming languages, cluster features + and data governance features are available in this mode. - Databricks will choose the most appropriate access mode - depending on your compute configuration. + The following modes are legacy aliases for the above + modes: - * `DATA_SECURITY_MODE_STANDARD`: Alias for - `USER_ISOLATION`. * `DATA_SECURITY_MODE_DEDICATED`: - Alias for `SINGLE_USER`. + - ``USER_ISOLATION``: Legacy alias for + ``DATA_SECURITY_MODE_STANDARD``. - - The following modes can be used regardless of `kind`. * - `NONE`: No security isolation for - - multiple users sharing the cluster. Data governance - features are not available in this mode. * - - `SINGLE_USER`: A secure cluster that can only be - exclusively used by a single user specified in - - `single_user_name`. Most programming languages, cluster - features and data governance features - - are available in this mode. * `USER_ISOLATION`: A secure - cluster that can be shared by multiple - - users. Cluster users are fully isolated so that they - cannot see each other's data and - - credentials. Most data governance features are supported - in this mode. But programming languages - - and cluster features might be limited. + - ``SINGLE_USER``: Legacy alias for + ``DATA_SECURITY_MODE_DEDICATED``. The following modes are deprecated starting with @@ -4556,20 +4793,25 @@ paths: future Databricks Runtime versions: - * `LEGACY_TABLE_ACL`: This mode is for users migrating - from legacy Table ACL clusters. * + - ``LEGACY_TABLE_ACL``: This mode is for users migrating + from legacy Table ACL clusters. - `LEGACY_PASSTHROUGH`: This mode is for users migrating - from legacy Passthrough on high - - concurrency clusters. * `LEGACY_SINGLE_USER`: This mode - is for users migrating from legacy - - Passthrough on standard clusters. * - `LEGACY_SINGLE_USER_STANDARD`: This mode provides a way - that - - doesn’t have UC nor passthrough enabled. + - ``LEGACY_PASSTHROUGH``: This mode is for users + migrating from legacy Passthrough on high + concurrency clusters. + - ``LEGACY_SINGLE_USER``: This mode is for users + migrating from legacy Passthrough on standard + clusters. + - ``LEGACY_SINGLE_USER_STANDARD``: This mode provides a + way that doesn’t have UC nor + passthrough enabled. + dependency_mode: + type: string + x-enum: + - DEPENDENCY_MODE_AUTO + - DEPENDENCY_MODE_CLUSTER_LIBRARIES + - DEPENDENCY_MODE_ENVIRONMENTS + description: Controls dependency configuration for the cluster. docker_image: type: object properties: @@ -4608,9 +4850,9 @@ paths: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set - as the same value as `node_type_id` defined above. This - field, along with node_type_id, should not be set if - virtual_cluster_size is set. If both + as the same value as ``node_type_id`` defined above. + This field, along with node_type_id, should not be set + if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. @@ -4640,20 +4882,30 @@ paths: boot_disk_size: type: integer description: Boot disk size in GB + confidential_compute_type: + type: string + x-enum: + - CONFIDENTIAL_COMPUTE_TYPE_NONE + - SEV_SNP + description: >- + The confidential computing technology for this + cluster's instances. Currently only SEV_SNP is + supported, and only on N2D instance types. When not + set, no confidential computing is applied. first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster + The first ``first_on_demand`` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current - cluster size, `first_on_demand` nodes will be placed - on on-demand instances and the remainder will be - placed on `availability` instances. Note that this - value does not affect cluster size and cannot + cluster size, ``first_on_demand`` nodes will be + placed on on-demand instances and the remainder will + be placed on ``availability`` instances. Note that + this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. google_service_account: type: string @@ -4669,10 +4921,10 @@ paths: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to - [GCP documentation] for the supported number of - local SSDs for each instance type. [GCP - documentation]: - https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds + `GCP documentation + `__ + for the supported number of local SSDs for each + instance type. use_preemptible_executors: type: boolean description: >- @@ -4709,23 +4961,23 @@ paths: type: string description: >- abfss destination, e.g. - `abfss://@.dfs.core.windows.net/`. + ``abfss://@.dfs.core.windows.net/``. required: - destination description: >- destination needs to be provided, e.g. - `abfss://@.dfs.core.windows.net/` + ``abfss://@.dfs.core.windows.net/`` dbfs: type: object properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: >- - destination needs to be provided. e.g. `{ "dbfs": - { "destination" : "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs": + { "destination" : "dbfs:/home/cluster_log" } }`` file: type: object properties: @@ -4734,8 +4986,8 @@ paths: required: - destination description: >- - destination needs to be provided, e.g. `{ "file": - { "destination": "file:/my/local/file.sh" } }` + destination needs to be provided, e.g. ``{ "file": + { "destination": "file:/my/local/file.sh" } }`` gcs: type: object properties: @@ -4743,12 +4995,12 @@ paths: type: string description: >- GCS destination/URI, e.g. - `gs://my-bucket/some-prefix` + ``gs://my-bucket/some-prefix`` required: - destination description: >- - destination needs to be provided, e.g. `{ "gcs": { - "destination": "gs://my-bucket/file.sh" } }` + destination needs to be provided, e.g. ``{ "gcs": + { "destination": "gs://my-bucket/file.sh" } }`` s3: type: object properties: @@ -4756,7 +5008,7 @@ paths: type: string description: >- S3 destination, e.g. - `s3://my-bucket/some-prefix` Note that logs + ``s3://my-bucket/some-prefix`` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. @@ -4766,9 +5018,9 @@ paths: type: string description: >- (Optional) Set canned access control list for - the logs, e.g. `bucket-owner-full-control`. If - `canned_cal` is set, please make sure the - cluster iam role has `s3:PutObjectAcl` + the logs, e.g. ``bucket-owner-full-control``. + If ``canned_cal`` is set, please make sure the + cluster iam role has ``s3:PutObjectAcl`` permission on the destination bucket and prefix. The full list of possible canned acl can be found at @@ -4776,25 +5028,25 @@ paths: Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you - may want to set `bucket-owner-full-control` to - make bucket owner able to read the logs. + may want to set ``bucket-owner-full-control`` + to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- (Optional) Flag to enable server side - encryption, `false` by default. + encryption, ``false`` by default. encryption_type: type: string description: >- (Optional) The encryption type, it could be - `sse-s3` or `sse-kms`. It will be used only - when encryption is enabled and the default - type is `sse-s3`. + ``sse-s3`` or ``sse-kms``. It will be used + only when encryption is enabled and the + default type is ``sse-s3``. endpoint: type: string description: >- S3 endpoint, e.g. - `https://s3-us-west-2.amazonaws.com`. Either + ``https://s3-us-west-2.amazonaws.com``. Either region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: @@ -4802,23 +5054,24 @@ paths: description: >- (Optional) Kms key which will be used if encryption is enabled and encryption type is - set to `sse-kms`. + set to ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or - endpoint needs to be set. If both are set, + S3 region, e.g. ``us-west-2``. Either region + or endpoint needs to be set. If both are set, endpoint will be used. required: - destination description: >- destination and either the region or endpoint need - to be provided. e.g. `{ \"s3\": { \"destination\": + to be provided. e.g. ``{ \"s3\": { + \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": - \"us-west-2\" } }` Cluster iam role is used to + \"us-west-2\" } }`` Cluster iam role is used to access s3, please make sure the cluster iam role - in `instance_profile_arn` has permission to write - data to the s3 destination. + in ``instance_profile_arn`` has permission to + write data to the s3 destination. volumes: type: object properties: @@ -4826,15 +5079,15 @@ paths: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided. e.g. `{ + destination needs to be provided. e.g. ``{ \"volumes\" : { \"destination\" : - \"/Volumes/my-init.sh\" } }` + \"/Volumes/my-init.sh\" } }`` workspace: type: object properties: @@ -4842,20 +5095,20 @@ paths: type: string description: >- wsfs destination, e.g. - `workspace:/cluster-init-scripts/setup-datadog.sh` + ``workspace:/cluster-init-scripts/setup-datadog.sh`` required: - destination description: >- - destination needs to be provided, e.g. `{ + destination needs to be provided, e.g. ``{ "workspace": { "destination": - "/cluster-init-scripts/setup-datadog.sh" } }` - description: 'Config for an individual init script Next ID: 11' + "/cluster-init-scripts/setup-datadog.sh" } }`` + description: Config for an individual init script description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If - `cluster_log_conf` is specified, init script logs are - sent to `//init_scripts`. + ``cluster_log_conf`` is specified, init script logs are + sent to ``//init_scripts``. instance_pool_id: type: string description: >- @@ -4864,10 +5117,10 @@ paths: is_single_node: type: boolean description: >- - This field can only be used when `kind = - CLASSIC_PREVIEW`. When set to true, Databricks will - automatically set single node related `custom_tags`, - `spark_conf`, and `num_workers` + This field can only be used when ``kind = + CLASSIC_PREVIEW``. When set to true, Databricks will + automatically set single node related ``custom_tags``, + ``spark_conf``, and ``num_workers`` kind: type: string x-enum: @@ -4877,34 +5130,29 @@ paths: specification. - Depending on `kind`, different validations and default + Depending on ``kind``, different validations and default values will be applied. - Clusters with `kind = CLASSIC_PREVIEW` support the + Clusters with ``kind = CLASSIC_PREVIEW`` support the following fields, whereas clusters with no - specified `kind` do not. * - [is_single_node](/api/workspace/clusters/create#is_single_node) - * - - [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - * + specified ``kind`` do not. - [data_security_mode](/api/workspace/clusters/create#data_security_mode) - set to - `DATA_SECURITY_MODE_AUTO`, - `DATA_SECURITY_MODE_DEDICATED`, or - `DATA_SECURITY_MODE_STANDARD` + - + [is_single_node](/api/workspace/clusters/create#is_single_node) + - + [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - By using the [simple form], your clusters are - automatically using `kind = CLASSIC_PREVIEW`. + By using the `simple form + `__, + your - [simple form]: - https://docs.databricks.com/compute/simple-form.html + clusters are automatically using ``kind = + CLASSIC_PREVIEW``. node_type_id: type: string description: >- @@ -4913,21 +5161,23 @@ paths: cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using - the :method:clusters/listNodeTypes API call. + the `clusters/listNodeTypes + `__ + API call. num_workers: type: integer description: >- Number of worker nodes that this cluster should have. A - cluster has one Spark Driver and `num_workers` Executors - for a total of `num_workers` + 1 Spark nodes. Note: When - reading the properties of a cluster, this field reflects - the desired number of workers rather than the actual - current number of workers. For instance, if a cluster is - resized from 5 to 10 workers, this field will - immediately be updated to reflect the target size of 10 - workers, whereas the workers listed in `spark_info` will - gradually increase from 5 to 10 as the new nodes are - provisioned. + cluster has one Spark Driver and ``num_workers`` + Executors for a total of ``num_workers`` + 1 Spark + nodes. Note: When reading the properties of a cluster, + this field reflects the desired number of workers rather + than the actual current number of workers. For instance, + if a cluster is resized from 5 to 10 workers, this field + will immediately be updated to reflect the target size + of 10 workers, whereas the workers listed in + ``spark_info`` will gradually increase from 5 to 10 as + the new nodes are provisioned. policy_id: type: string description: >- @@ -4948,46 +5198,49 @@ paths: description: >- Determines the cluster's runtime engine, either standard or Photon. This field is not compatible with legacy - `spark_version` values that contain `-photon-`. Remove - `-photon-` from the `spark_version` and set - `runtime_engine` to `PHOTON`. If left unspecified, the - runtime engine defaults to standard unless the + ``spark_version`` values that contain ``-photon-``. + Remove ``-photon-`` from the ``spark_version`` and set + ``runtime_engine`` to ``PHOTON``. If left unspecified, + the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: >- + Single user name if data_security_mode is + ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the - executors via `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + executors via ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is - (i.e., `export X='Y'`) while launching the driver and + (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional set of - `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. - This ensures that all default databricks managed + ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them + to ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example + below. This ensures that all default databricks managed environmental variables are included as well. Example - Spark environment variables: `{"SPARK_WORKER_MEMORY": - "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}` or - `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + Spark environment variables: ``{"SPARK_WORKER_MEMORY": + "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}`` or + ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` spark_version: type: string description: >- The Spark version of the cluster, e.g. - `3.3.x-scala2.11`. A list of available Spark versions - can be retrieved by using the - :method:clusters/sparkVersions API call. + ``3.3.x-scala2.11``. A list of available Spark versions + can be retrieved by using the `clusters/sparkVersions + `__ + API call. ssh_public_keys: type: array items: @@ -4995,8 +5248,8 @@ paths: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can - be used to login with the user name `ubuntu` on port - `2200`. Up to 10 keys can be specified. + be used to login with the user name ``ubuntu`` on port + ``2200``. Up to 10 keys can be specified. total_initial_remote_disk_size: type: integer description: >- @@ -5006,11 +5259,11 @@ paths: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = - CLASSIC_PREVIEW`. `effective_spark_version` is - determined by `spark_version` (DBR release), this field - `use_ml_runtime`, and whether `node_type_id` is gpu node - or not. + This field can only be used when ``kind = + CLASSIC_PREVIEW``. ``effective_spark_version`` is + determined by ``spark_version`` (DBR release), this + field ``use_ml_runtime``, and whether ``node_type_id`` + is gpu node or not. worker_node_type_flexibility: type: object properties: @@ -5074,7 +5327,7 @@ paths: description: |- Cancels a currently running command within an execution context. - The command ID is obtained from a prior successful call to __execute__. + The command ID is obtained from a prior successful call to **execute**. :param cluster_id: str (optional) :param command_id: str (optional) @@ -5128,7 +5381,7 @@ paths: executing command. - The command ID is obtained from a prior successful call to __execute__. + The command ID is obtained from a prior successful call to **execute**. :param cluster_id: str @@ -5787,7 +6040,8 @@ paths: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes `__ API + call. :param aws_attributes: :class:`InstancePoolAwsAttributes` (optional) Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of default values will be used. @@ -5796,11 +6050,16 @@ paths: default values will be used. :param custom_tags: Dict[str,str] (optional) Additional tags for pool resources. Databricks will tag all pool resources (e.g., AWS instances and - EBS volumes) with these tags in addition to `default_tags`. Notes: + EBS volumes) with these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags :param disk_spec: :class:`DiskSpec` (optional) Defines the specification of the disks that will be attached to all spark containers. + :param enable_auto_alternate_node_types: bool (optional) + For pools with node type flexibility (Fleet-V2), whether auto generated alternate node type ids are + enabled. This field should not be true if node_type_flexibility is set. DEPRECATED: This field was + deprecated before entering PuPr and should no longer be used. TODO(CJ-71514): Remove this field + after sufficient time has passed for all clients to migrate. :param enable_elastic_disk: bool (optional) Autoscaling Local Storage: when enabled, this instances in this pool will dynamically acquire additional disk space when its Spark workers are running low on disk space. In AWS, this feature @@ -5827,7 +6086,8 @@ paths: :param preloaded_spark_versions: List[str] (optional) A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can - be retrieved by using the :method:clusters/sparkVersions API call. + be retrieved by using the `clusters/sparkVersions + `__ API call. :param remote_disk_throughput: int (optional) If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED types. @@ -5855,7 +6115,9 @@ paths: example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes + `__ + API call. aws_attributes: type: object properties: @@ -5886,11 +6148,11 @@ paths: Calculates the bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to - 50, and the cluster needs a new `r3.xlarge` spot + 50, and the cluster needs a new ``r3.xlarge`` spot instance, then the bid price is half of the price of - on-demand `r3.xlarge` instances. Similarly, if this + on-demand ``r3.xlarge`` instances. Similarly, if this field is set to 200, the bid price is twice the price of - on-demand `r3.xlarge` instances. If not specified, the + on-demand ``r3.xlarge`` instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note @@ -5908,7 +6170,7 @@ paths: This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as the default value can be - found by using the `List Zones` method. + found by using the ``List Zones`` method. description: >- Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of @@ -5922,6 +6184,27 @@ paths: - ON_DEMAND_AZURE - SPOT_AZURE description: Availability type used for the spot nodes. + capacity_reservation_group: + type: string + description: >- + The Azure capacity reservation group resource ID to use + for launching VMs in this pool. When specified, VMs will + be launched using the provided capacity reservation. + NOTE: Omitting this field will clear any existing + configured capacity reservation group on the pool. + Capacity reservations can only be specified when the + workspace uses injected vnet (i.e. customer defined vnet + not managed by databricks). Ensure the + databricks-login-prod Enterprise Application is granted + the following four permissions: 1. + Microsoft.Compute/capacityReservationGroups/read 2. + Microsoft.Compute/capacityReservationGroups/deploy/action + 3. + Microsoft.Compute/capacityReservationGroups/capacityReservations/read + 4. + Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action + Format: + ``/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}`` spot_bid_max_price: type: number description: >- @@ -5941,7 +6224,7 @@ paths: description: >- Additional tags for pool resources. Databricks will tag all pool resources (e.g., AWS instances and EBS volumes) with - these tags in addition to `default_tags`. Notes: - + these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags disk_spec: type: object @@ -5960,10 +6243,10 @@ paths: can lead to inefficient disk utilization. If no disks are attached, Databricks will configure Spark to use instance store disks. Note: If disks are specified, then - the Spark configuration `spark.local.dir` will be + the Spark configuration ``spark.local.dir`` will be overridden. Disks will be mounted at: - For AWS: - `/ebs0`, `/ebs1`, and etc. - For Azure: - `/remote_volume0`, `/remote_volume1`, and etc. + ``/ebs0``, ``/ebs1``, and etc. - For Azure: + ``/remote_volume0``, ``/remote_volume1``, and etc. disk_iops: type: integer disk_size: @@ -6005,6 +6288,16 @@ paths: description: >- Defines the specification of the disks that will be attached to all spark containers. + enable_auto_alternate_node_types: + type: boolean + description: >- + For pools with node type flexibility (Fleet-V2), whether + auto generated alternate node type ids are enabled. This + field should not be true if node_type_flexibility is set. + DEPRECATED: This field was deprecated before entering PuPr + and should no longer be used. TODO(CJ-71514): Remove this + field after sufficient time has passed for all clients to + migrate. enable_elastic_disk: type: boolean description: >- @@ -6034,10 +6327,10 @@ paths: description: >- If provided, each node in the instance pool will have this number of local SSDs attached. Each local SSD is - 375GB in size. Refer to [GCP documentation] for the - supported number of local SSDs for each instance type. - [GCP documentation]: - https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds + 375GB in size. Refer to `GCP documentation + `__ + for the supported number of local SSDs for each instance + type. zone_id: type: string description: >- @@ -6121,7 +6414,9 @@ paths: for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be retrieved by using the - :method:clusters/sparkVersions API call. + `clusters/sparkVersions + `__ + API call. remote_disk_throughput: type: integer description: >- @@ -6217,12 +6512,18 @@ paths: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes `__ API + call. :param custom_tags: Dict[str,str] (optional) Additional tags for pool resources. Databricks will tag all pool resources (e.g., AWS instances and - EBS volumes) with these tags in addition to `default_tags`. Notes: + EBS volumes) with these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags + :param enable_auto_alternate_node_types: bool (optional) + For pools with node type flexibility (Fleet-V2), whether auto generated alternate node type ids are + enabled. This field should not be true if node_type_flexibility is set. DEPRECATED: This field was + deprecated before entering PuPr and should no longer be used. TODO(CJ-71514): Remove this field + after sufficient time has passed for all clients to migrate. :param idle_instance_autotermination_minutes: int (optional) Automatically terminates the extra instances in the pool cache after they are inactive for this time in minutes if min_idle_instances requirement is already met. If not set, the extra pool instances @@ -6263,14 +6564,26 @@ paths: example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the - :method:clusters/listNodeTypes API call. + `clusters/listNodeTypes + `__ + API call. custom_tags: type: object description: >- Additional tags for pool resources. Databricks will tag all pool resources (e.g., AWS instances and EBS volumes) with - these tags in addition to `default_tags`. Notes: - + these tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags + enable_auto_alternate_node_types: + type: boolean + description: >- + For pools with node type flexibility (Fleet-V2), whether + auto generated alternate node type ids are enabled. This + field should not be true if node_type_flexibility is set. + DEPRECATED: This field was deprecated before entering PuPr + and should no longer be used. TODO(CJ-71514): Remove this + field after sufficient time has passed for all clients to + migrate. idle_instance_autotermination_minutes: type: integer description: >- @@ -6653,16 +6966,14 @@ paths: :param iam_role_arn: str (optional) The AWS IAM role ARN of the role associated with the instance profile. This field is required if your role name and instance profile name do not match and you want to use the instance profile with - [Databricks SQL Serverless]. + `Databricks SQL Serverless `__. Otherwise, this field is optional. - - [Databricks SQL Serverless]: https://docs.databricks.com/sql/admin/serverless.html :param is_meta_instance_profile: bool (optional) Boolean flag indicating whether the instance profile should only be used in credential passthrough scenarios. If true, it means the instance profile contains an meta IAM role which could assume a wide range of roles. Therefore it should always be used with authorization. This field is optional, - the default value is `false`. + the default value is ``false``. :param skip_validation: bool (optional) By default, Databricks validates that it has sufficient permissions to launch instances with the instance profile. This validation uses AWS dry-run mode for the RunInstances API. If validation @@ -6686,10 +6997,9 @@ paths: The AWS IAM role ARN of the role associated with the instance profile. This field is required if your role name and instance profile name do not match and you want to use - the instance profile with [Databricks SQL Serverless]. - Otherwise, this field is optional. [Databricks SQL - Serverless]: - https://docs.databricks.com/sql/admin/serverless.html + the instance profile with `Databricks SQL Serverless + `__. + Otherwise, this field is optional. is_meta_instance_profile: type: boolean description: >- @@ -6698,7 +7008,7 @@ paths: it means the instance profile contains an meta IAM role which could assume a wide range of roles. Therefore it should always be used with authorization. This field is - optional, the default value is `false`. + optional, the default value is ``false``. skip_validation: type: boolean description: >- @@ -6744,42 +7054,34 @@ paths: true: - * Your role name and instance profile name do not match. The name is the + - Your role name and instance profile name do not match. The name is the part after the last slash in + each ARN. + - You want to use the instance profile with `Databricks SQL Serverless + `__. - each ARN. * You want to use the instance profile with [Databricks SQL - Serverless]. - + To understand where these fields are in the AWS console, see `Enable + serverless SQL warehouses - To understand where these fields are in the AWS console, see [Enable - serverless SQL warehouses]. + `__. This API is only available to admin users. - [Databricks SQL Serverless]: - https://docs.databricks.com/sql/admin/serverless.html - - [Enable serverless SQL warehouses]: - https://docs.databricks.com/sql/admin/serverless.html - - :param instance_profile_arn: str The AWS ARN of the instance profile to register with Databricks. This field is required. :param iam_role_arn: str (optional) The AWS IAM role ARN of the role associated with the instance profile. This field is required if your role name and instance profile name do not match and you want to use the instance profile with - [Databricks SQL Serverless]. + `Databricks SQL Serverless `__. Otherwise, this field is optional. - - [Databricks SQL Serverless]: https://docs.databricks.com/sql/admin/serverless.html :param is_meta_instance_profile: bool (optional) Boolean flag indicating whether the instance profile should only be used in credential passthrough scenarios. If true, it means the instance profile contains an meta IAM role which could assume a wide range of roles. Therefore it should always be used with authorization. This field is optional, - the default value is `false`. + the default value is ``false``. requestBody: content: application/json: @@ -6797,10 +7099,9 @@ paths: The AWS IAM role ARN of the role associated with the instance profile. This field is required if your role name and instance profile name do not match and you want to use - the instance profile with [Databricks SQL Serverless]. - Otherwise, this field is optional. [Databricks SQL - Serverless]: - https://docs.databricks.com/sql/admin/serverless.html + the instance profile with `Databricks SQL Serverless + `__. + Otherwise, this field is optional. is_meta_instance_profile: type: boolean description: >- @@ -6809,7 +7110,7 @@ paths: it means the instance profile contains an meta IAM role which could assume a wide range of roles. Therefore it should always be used with authorization. This field is - optional, the default value is `false`. + optional, the default value is ``false``. required: - instance_profile_arn responses: @@ -6965,16 +7266,15 @@ paths: libraries installed on this cluster via the API or the libraries UI. The order of returned libraries - is as follows: 1. Libraries - - set to be installed on this cluster, in the order that the libraries - were added to the cluster, are - - returned first. 2. Libraries that were previously requested to be - installed on this cluster or, but + is as follows: - are now marked for removal, in no particular order, are returned last. + 1. Libraries set to be installed on this cluster, in the order that the + libraries were added to the + cluster, are returned first. + 2. Libraries that were previously requested to be installed on this + cluster or, but are now marked for + removal, in no particular order, are returned last. :param cluster_id: str Unique identifier of the cluster whose status should be retrieved. @@ -7006,97 +7306,888 @@ paths: message: type: string x-stackql-sdk-source: LibrariesAPI - /api/2.0/libraries/install: + /api/2.0/default-base-environments: post: - operationId: libraries_install + operationId: libraries_create_default_base_environment summary: >- - Add libraries to install on a cluster. The installation is asynchronous; - it happens in the background + Create a default base environment within workspaces to define the + environment version and a list of tags: - compute - libraries description: >- - Add libraries to install on a cluster. The installation is asynchronous; - it happens in the background + Create a default base environment within workspaces to define the + environment version and a list of - after the completion of this request. + dependencies to be used in serverless notebooks and jobs. This process + will asynchronously generate a + cache to optimize dependency resolution. - :param cluster_id: str - Unique identifier for the cluster on which to install these libraries. - :param libraries: List[:class:`Library`] - The libraries to install. + + :param default_base_environment: :class:`DefaultBaseEnvironment` + + :param request_id: str (optional) + A unique identifier for this request. A random UUID is recommended. This request is only idempotent + if a ``request_id`` is provided. + :param workspace_base_environment_id: str (optional) + + + :returns: :class:`DefaultBaseEnvironment` requestBody: content: application/json: schema: type: object properties: - cluster_id: - type: string - description: >- - Unique identifier for the cluster on which to install these - libraries. - libraries: - type: array - items: - type: object - properties: - cran: + default_base_environment: + type: object + properties: + base_environment_cache: + type: array + items: type: object properties: - package: - type: string - repo: - type: string + indefinite_materialized_environment: + type: object + properties: + last_updated_timestamp: + type: integer + description: >- + The timestamp (in epoch milliseconds) when the + materialized env is updated. description: >- - The repository where the package can be found. If - not specified, the default CRAN repo is used. - required: - - package - egg: - type: string - description: >- - Deprecated. URI of the egg library to install. - Installing Python egg files is deprecated and is not - supported in Databricks Runtime 14.0 and above. - jar: - type: string - description: >- - URI of the JAR library to install. Supported URIs - include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "jar": - "/Workspace/path/to/library.jar" }`, `{ "jar" : - "/Volumes/path/to/library.jar" }` or `{ "jar": - "s3://my-bucket/library.jar" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. - maven: - type: object - properties: - coordinates: - type: string - exclusions: - type: array - items: - type: string + Materialized Environment information enables + environment sharing and reuse via Environment + Caching during library installations. Currently this feature is only supported for Python + libraries. + + - If the env cache entry in LMv2 DB doesn't exist or invalid, library installations and + environment materialization will occur. A new Materialized Environment metadata will be sent + from DP upon successful library installations and env materialization, and is persisted into + database by LMv2. + - If the env cache entry in LMv2 DB is valid, the Materialized Environment will be sent to DP by + LMv2, and DP will restore the cached environment from a store instead of reinstalling + libraries from scratch. + + If changed, also update estore/namespaces/defaultbaseenvironments/latest.proto with new version + If changed, also update estore/namespaces/envspecenvironments/latest.proto with new version + materialized_environment: + type: object + properties: + last_updated_timestamp: + type: integer + description: >- + The timestamp (in epoch milliseconds) when the + materialized env is updated. description: >- - List of dependences to exclude. For example: - `["slf4j:slf4j", "*:hadoop-client"]`. Maven - dependency exclusions: - https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html. - repo: + Materialized Environment information enables + environment sharing and reuse via Environment + Caching during library installations. Currently this feature is only supported for Python + libraries. + + - If the env cache entry in LMv2 DB doesn't exist or invalid, library installations and + environment materialization will occur. A new Materialized Environment metadata will be sent + from DP upon successful library installations and env materialization, and is persisted into + database by LMv2. + - If the env cache entry in LMv2 DB is valid, the Materialized Environment will be sent to DP by + LMv2, and DP will restore the cached environment from a store instead of reinstalling + libraries from scratch. + + If changed, also update estore/namespaces/defaultbaseenvironments/latest.proto with new version + If changed, also update estore/namespaces/envspecenvironments/latest.proto with new version + message: type: string + status: + type: string + x-enum: + - CREATED + - EXPIRED + - FAILED + - INVALID + - PENDING + - REFRESHING description: >- - Maven repo to install the Maven package from. If - omitted, both Maven Central Repository and Spark - Packages are searched. - required: - - coordinates - description: >- - Specification of a maven library to be installed. For - example: `{ "coordinates": "org.jsoup:jsoup:1.7.2" }` + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how + many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + base_environment_type: + type: string + x-enum: + - CPU + - GPU + description: >- + If changed, also update + estore/namespaces/defaultbaseenvironments/latest.proto + created_timestamp: + type: integer + creator_user_id: + type: integer + environment: + type: object + properties: + base_environment: + type: string + description: >- + The base environment this environment is built on + top of. A base environment defines the environment + version and a list of dependencies for serverless + compute. The value can be a file path to a custom + ``env.yaml`` file (e.g., + ``/Workspace/path/to/env.yaml``). Support for a + Databricks-provided base environment ID (e.g., + ``workspace-base-environments/databricks_ai_v4``) + and workspace base environment ID (e.g., + ``workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7``) + is in Beta. Either ``environment_version`` or + ``base_environment`` can be provided. For more + information about Databricks-provided base + environments, see the [list workspace base + environments](:method:Environments/ListWorkspaceBaseEnvironments) + API. For more information, see + client: + type: string + description: Use ``environment_version`` instead. + dependencies: + type: array + items: + type: string + description: >- + List of pip dependencies, as supported by the + version of pip in this environment. Each dependency + is a valid pip requirements file line per + https://pip.pypa.io/en/stable/reference/requirements-file-format/. + Allowed dependencies include a requirement + specifier, an archive URL, a local project path + (such as WSFS or UC Volumes in Databricks), or a VCS + project URL. + environment_version: + type: string + description: >- + Either ``environment_version`` or + ``base_environment`` needs to be provided. + Environment version used by the environment. Each + version comes with a specific Python version and a + set of Python packages. The version is a string, + consisting of an integer. + java_dependencies: + type: array + items: + type: string + description: >- + List of java dependencies. Each dependency is a + string representing a java library path. For + example: ``/Volumes/path/to/test.jar``. + description: >- + Note: we made ``environment`` non-internal because we + need to expose its ``client`` field. All other fields + should be treated as internal. + filepath: + type: string + id: + type: string + is_default: + type: boolean + last_updated_timestamp: + type: integer + last_updated_user_id: + type: integer + message: + type: string + name: + type: string + principal_ids: + type: array + items: + type: integer + status: + type: string + x-enum: + - CREATED + - EXPIRED + - FAILED + - INVALID + - PENDING + - REFRESHING + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + request_id: + type: string + description: >- + A unique identifier for this request. A random UUID is + recommended. This request is only idempotent if a + ``request_id`` is provided. + workspace_base_environment_id: + type: string + required: + - default_base_environment + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DefaultBaseEnvironment' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + get: + operationId: libraries_list_default_base_environments + summary: >- + List default base environments defined in the workspaces for the + requested user. + tags: + - compute + - libraries + description: >- + List default base environments defined in the workspaces for the + requested user. + + + :param page_size: int (optional) + + :param page_token: str (optional) + + + :returns: Iterator over :class:`DefaultBaseEnvironment` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + - name: page_token + in: query + required: false + schema: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListDefaultBaseEnvironmentsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + /api/2.0/default-base-environments/{id}: + delete: + operationId: libraries_delete_default_base_environment + summary: >- + Delete the default base environment given an ID. The default base + environment may be used by + tags: + - compute + - libraries + description: >- + Delete the default base environment given an ID. The default base + environment may be used by + + downstream workloads. Please ensure that the deletion is intentional. + + + :param id: str + parameters: + - name: id + in: path + required: true + schema: + type: string + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + patch: + operationId: libraries_update_default_base_environment + summary: >- + Update the default base environment for the given ID. This process will + asynchronously regenerate the + tags: + - compute + - libraries + description: >- + Update the default base environment for the given ID. This process will + asynchronously regenerate the + + cache. The existing cache remains available until it expires. + + + :param id: str + + :param default_base_environment: :class:`DefaultBaseEnvironment` + + + :returns: :class:`DefaultBaseEnvironment` + parameters: + - name: id + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + default_base_environment: + type: object + properties: + base_environment_cache: + type: array + items: + type: object + properties: + indefinite_materialized_environment: + type: object + properties: + last_updated_timestamp: + type: integer + description: >- + The timestamp (in epoch milliseconds) when the + materialized env is updated. + description: >- + Materialized Environment information enables + environment sharing and reuse via Environment + Caching during library installations. Currently this feature is only supported for Python + libraries. + + - If the env cache entry in LMv2 DB doesn't exist or invalid, library installations and + environment materialization will occur. A new Materialized Environment metadata will be sent + from DP upon successful library installations and env materialization, and is persisted into + database by LMv2. + - If the env cache entry in LMv2 DB is valid, the Materialized Environment will be sent to DP by + LMv2, and DP will restore the cached environment from a store instead of reinstalling + libraries from scratch. + + If changed, also update estore/namespaces/defaultbaseenvironments/latest.proto with new version + If changed, also update estore/namespaces/envspecenvironments/latest.proto with new version + materialized_environment: + type: object + properties: + last_updated_timestamp: + type: integer + description: >- + The timestamp (in epoch milliseconds) when the + materialized env is updated. + description: >- + Materialized Environment information enables + environment sharing and reuse via Environment + Caching during library installations. Currently this feature is only supported for Python + libraries. + + - If the env cache entry in LMv2 DB doesn't exist or invalid, library installations and + environment materialization will occur. A new Materialized Environment metadata will be sent + from DP upon successful library installations and env materialization, and is persisted into + database by LMv2. + - If the env cache entry in LMv2 DB is valid, the Materialized Environment will be sent to DP by + LMv2, and DP will restore the cached environment from a store instead of reinstalling + libraries from scratch. + + If changed, also update estore/namespaces/defaultbaseenvironments/latest.proto with new version + If changed, also update estore/namespaces/envspecenvironments/latest.proto with new version + message: + type: string + status: + type: string + x-enum: + - CREATED + - EXPIRED + - FAILED + - INVALID + - PENDING + - REFRESHING + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how + many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + base_environment_type: + type: string + x-enum: + - CPU + - GPU + description: >- + If changed, also update + estore/namespaces/defaultbaseenvironments/latest.proto + created_timestamp: + type: integer + creator_user_id: + type: integer + environment: + type: object + properties: + base_environment: + type: string + description: >- + The base environment this environment is built on + top of. A base environment defines the environment + version and a list of dependencies for serverless + compute. The value can be a file path to a custom + ``env.yaml`` file (e.g., + ``/Workspace/path/to/env.yaml``). Support for a + Databricks-provided base environment ID (e.g., + ``workspace-base-environments/databricks_ai_v4``) + and workspace base environment ID (e.g., + ``workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7``) + is in Beta. Either ``environment_version`` or + ``base_environment`` can be provided. For more + information about Databricks-provided base + environments, see the [list workspace base + environments](:method:Environments/ListWorkspaceBaseEnvironments) + API. For more information, see + client: + type: string + description: Use ``environment_version`` instead. + dependencies: + type: array + items: + type: string + description: >- + List of pip dependencies, as supported by the + version of pip in this environment. Each dependency + is a valid pip requirements file line per + https://pip.pypa.io/en/stable/reference/requirements-file-format/. + Allowed dependencies include a requirement + specifier, an archive URL, a local project path + (such as WSFS or UC Volumes in Databricks), or a VCS + project URL. + environment_version: + type: string + description: >- + Either ``environment_version`` or + ``base_environment`` needs to be provided. + Environment version used by the environment. Each + version comes with a specific Python version and a + set of Python packages. The version is a string, + consisting of an integer. + java_dependencies: + type: array + items: + type: string + description: >- + List of java dependencies. Each dependency is a + string representing a java library path. For + example: ``/Volumes/path/to/test.jar``. + description: >- + Note: we made ``environment`` non-internal because we + need to expose its ``client`` field. All other fields + should be treated as internal. + filepath: + type: string + id: + type: string + is_default: + type: boolean + last_updated_timestamp: + type: integer + last_updated_user_id: + type: integer + message: + type: string + name: + type: string + principal_ids: + type: array + items: + type: integer + status: + type: string + x-enum: + - CREATED + - EXPIRED + - FAILED + - INVALID + - PENDING + - REFRESHING + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members can + have their own + + attributes -- see the documentation for details. + required: + - default_base_environment + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DefaultBaseEnvironment' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + /api/2.0/default-base-environments:getDefaultBaseEnvironment: + get: + operationId: libraries_get_default_base_environment + summary: Return the default base environment details for a given ID. + tags: + - compute + - libraries + description: |- + Return the default base environment details for a given ID. + + :param id: str + :param trace_id: str (optional) + Deprecated: use ctx.requestId instead + + :returns: :class:`DefaultBaseEnvironment` + parameters: + - name: id + in: query + required: true + schema: + type: string + - name: trace_id + in: query + required: false + schema: + type: string + description: 'Deprecated: use ctx.requestId instead' + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DefaultBaseEnvironment' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + /api/2.0/libraries/install: + post: + operationId: libraries_install + summary: >- + Add libraries to install on a cluster. The installation is asynchronous; + it happens in the background + tags: + - compute + - libraries + description: >- + Add libraries to install on a cluster. The installation is asynchronous; + it happens in the background + + after the completion of this request. + + + :param cluster_id: str + Unique identifier for the cluster on which to install these libraries. + :param libraries: List[:class:`Library`] + The libraries to install. + requestBody: + content: + application/json: + schema: + type: object + properties: + cluster_id: + type: string + description: >- + Unique identifier for the cluster on which to install these + libraries. + libraries: + type: array + items: + type: object + properties: + cran: + type: object + properties: + package: + type: string + repo: + type: string + description: >- + The repository where the package can be found. If + not specified, the default CRAN repo is used. + required: + - package + egg: + type: string + description: >- + Deprecated. URI of the egg library to install. + Installing Python egg files is deprecated and is not + supported in Databricks Runtime 14.0 and above. + jar: + type: string + description: >- + URI of the JAR library to install. Supported URIs + include Workspace paths, Unity Catalog Volumes paths, + and S3 URIs. For example: ``{ "jar": + "/Workspace/path/to/library.jar" }``, ``{ "jar" : + "/Volumes/path/to/library.jar" }`` or ``{ "jar": + "s3://my-bucket/library.jar" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. + maven: + type: object + properties: + coordinates: + type: string + exclusions: + type: array + items: + type: string + description: >- + List of dependences to exclude. For example: + ``["slf4j:slf4j", "*:hadoop-client"]``. Maven + dependency exclusions: + https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html. + repo: + type: string + description: >- + Maven repo to install the Maven package from. If + omitted, both Maven Central Repository and Spark + Packages are searched. + required: + - coordinates + description: >- + Specification of a maven library to be installed. For + example: ``{ "coordinates": "org.jsoup:jsoup:1.7.2" + }`` pypi: type: object properties: @@ -7111,28 +8202,28 @@ paths: - package description: >- Specification of a PyPi library to be installed. For - example: `{ "package": "simplejson" }` + example: ``{ "package": "simplejson" }`` requirements: type: string description: >- URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are - supported. For example: `{ "requirements": - "/Workspace/path/to/requirements.txt" }` or `{ + supported. For example: ``{ "requirements": + "/Workspace/path/to/requirements.txt" }`` or ``{ "requirements" : "/Volumes/path/to/requirements.txt" - }` + }`` whl: type: string description: >- URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "whl": - "/Workspace/path/to/library.whl" }`, `{ "whl" : - "/Volumes/path/to/library.whl" }` or `{ "whl": - "s3://my-bucket/library.whl" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "whl": + "/Workspace/path/to/library.whl" }``, ``{ "whl" : + "/Volumes/path/to/library.whl" }`` or ``{ "whl": + "s3://my-bucket/library.whl" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. description: The libraries to install. required: - cluster_id @@ -7152,6 +8243,51 @@ paths: message: type: string x-stackql-sdk-source: LibrariesAPI + /api/2.0/default-base-environments/refresh: + post: + operationId: libraries_refresh_default_base_environments + summary: >- + Refresh the cached default base environments for the given IDs. This + process will asynchronously + tags: + - compute + - libraries + description: >- + Refresh the cached default base environments for the given IDs. This + process will asynchronously + + regenerate the caches. The existing caches remains available until it + expires. + + + :param ids: List[str] + requestBody: + content: + application/json: + schema: + type: object + properties: + ids: + type: array + items: + type: string + required: + - ids + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI /api/2.0/libraries/uninstall: post: operationId: libraries_uninstall @@ -7212,13 +8348,13 @@ paths: description: >- URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "jar": - "/Workspace/path/to/library.jar" }`, `{ "jar" : - "/Volumes/path/to/library.jar" }` or `{ "jar": - "s3://my-bucket/library.jar" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "jar": + "/Workspace/path/to/library.jar" }``, ``{ "jar" : + "/Volumes/path/to/library.jar" }`` or ``{ "jar": + "s3://my-bucket/library.jar" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. maven: type: object properties: @@ -7230,7 +8366,7 @@ paths: type: string description: >- List of dependences to exclude. For example: - `["slf4j:slf4j", "*:hadoop-client"]`. Maven + ``["slf4j:slf4j", "*:hadoop-client"]``. Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html. repo: @@ -7243,7 +8379,8 @@ paths: - coordinates description: >- Specification of a maven library to be installed. For - example: `{ "coordinates": "org.jsoup:jsoup:1.7.2" }` + example: ``{ "coordinates": "org.jsoup:jsoup:1.7.2" + }`` pypi: type: object properties: @@ -7258,35 +8395,153 @@ paths: - package description: >- Specification of a PyPi library to be installed. For - example: `{ "package": "simplejson" }` + example: ``{ "package": "simplejson" }`` requirements: type: string description: >- URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are - supported. For example: `{ "requirements": - "/Workspace/path/to/requirements.txt" }` or `{ + supported. For example: ``{ "requirements": + "/Workspace/path/to/requirements.txt" }`` or ``{ "requirements" : "/Volumes/path/to/requirements.txt" - }` + }`` whl: type: string description: >- URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, - and S3 URIs. For example: `{ "whl": - "/Workspace/path/to/library.whl" }`, `{ "whl" : - "/Volumes/path/to/library.whl" }` or `{ "whl": - "s3://my-bucket/library.whl" }`. If S3 is used, please - make sure the cluster has read access on the library. - You may need to launch the cluster with an IAM role to - access the S3 URI. + and S3 URIs. For example: ``{ "whl": + "/Workspace/path/to/library.whl" }``, ``{ "whl" : + "/Volumes/path/to/library.whl" }`` or ``{ "whl": + "s3://my-bucket/library.whl" }``. If S3 is used, + please make sure the cluster has read access on the + library. You may need to launch the cluster with an + IAM role to access the S3 URI. description: The libraries to uninstall. required: - cluster_id - - libraries + - libraries + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + /api/2.0/default-base-environments:setDefault: + post: + operationId: libraries_update_default_default_base_environment + summary: >- + Set the default base environment for the workspace. This marks the + specified DBE as the workspace + tags: + - compute + - libraries + description: >- + Set the default base environment for the workspace. This marks the + specified DBE as the workspace + + default. + + + :param base_environment_type: :class:`BaseEnvironmentType` (optional) + + :param id: str (optional) + + + :returns: :class:`DefaultBaseEnvironment` + requestBody: + content: + application/json: + schema: + type: object + properties: + base_environment_type: + type: string + x-enum: + - CPU + - GPU + description: >- + If changed, also update + estore/namespaces/defaultbaseenvironments/latest.proto + id: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DefaultBaseEnvironment' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LibrariesAPI + /api/2.0/policies/clusters:cancelPendingClusterEnforcement: + post: + operationId: policy_compliance_for_clusters_cancel_pending_cluster_enforcement + summary: >- + Cancels a pending enforcement on a cluster. After canceling the pending + enforcement, the cluster will + tags: + - compute + - policy_compliance_for_clusters + description: >- + Cancels a pending enforcement on a cluster. After canceling the pending + enforcement, the cluster will + + no longer update on the next termination or restart. Pending + enforcements cannot be canceled when a + + cluster is in ``TERMINATING`` state. Only workspace admins can cancel + pending enforcements. + + + :param cluster_id: str + The ID of the cluster to cancel the pending enforcement for. + :param allow_missing: bool (optional) + If true and no pending enforcement exists, the request will succeed but no action will be taken. + + :returns: :class:`CancelPendingClusterEnforcementResponse` + requestBody: + content: + application/json: + schema: + type: object + properties: + cluster_id: + type: string + description: The ID of the cluster to cancel the pending enforcement for. + allow_missing: + type: boolean + description: >- + If true and no pending enforcement exists, the request will + succeed but no action will be taken. + required: + - cluster_id responses: '200': description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/CancelPendingClusterEnforcementResponse' default: description: Error response content: @@ -7298,36 +8553,32 @@ paths: type: string message: type: string - x-stackql-sdk-source: LibrariesAPI + x-stackql-sdk-source: PolicyComplianceForClustersAPI /api/2.0/policies/clusters/enforce-compliance: post: operationId: policy_compliance_for_clusters_enforce_compliance summary: >- Updates a cluster to be compliant with the current version of its - policy. A cluster can be updated if + policy. tags: - compute - policy_compliance_for_clusters description: >- Updates a cluster to be compliant with the current version of its - policy. A cluster can be updated if - - it is in a `RUNNING` or `TERMINATED` state. - + policy. - If a cluster is updated while in a `RUNNING` state, it will be restarted - so that the new attributes - can take effect. + If a cluster is updated while in a ``TERMINATED`` state, it will remain + ``TERMINATED``. The next time + the cluster is started, the new attributes will take effect. - If a cluster is updated while in a `TERMINATED` state, it will remain - `TERMINATED`. The next time the - cluster is started, the new attributes will take effect. + For clusters in other states, the behavior depends on the + ``enforce_mode`` used. - Clusters created by the Databricks Jobs, DLT, or Models services cannot + Clusters created by the Databricks Jobs, SDP, or Models services cannot be enforced by this API. Instead, use the "Enforce job policy compliance" API to enforce policy @@ -7336,6 +8587,18 @@ paths: :param cluster_id: str The ID of the cluster you want to enforce policy compliance on. + :param enforce_mode: + :class:`EnforcePolicyComplianceForClusterEnforceMode` (optional) + Determines how changes should be made to clusters that are not in ``TERMINATED`` state. + + - ``ENFORCE_IMMEDIATELY``: If the cluster is in a ``RUNNING`` state, it will be restarted so that + the new attributes can take effect. For other states aside from ``TERMINATED`` state, the request + will be rejected. + - ``WAIT_FOR_TERMINATION``: The cluster is not immediately edited. Instead, a pending enforcement is + scheduled to update the cluster when it terminates or restarts. When this occurs, + ``enforce_result`` will contain ``DEFERRED``. Only workspace admins can use this mode. + + Regardless of the enforce mode, clusters in ``TERMINATED`` state are immediately edited. :param validate_only: bool (optional) If set, previews the changes that would be made to a cluster to enforce compliance but does not update the cluster. @@ -7352,6 +8615,24 @@ paths: description: >- The ID of the cluster you want to enforce policy compliance on. + enforce_mode: + type: string + x-enum: + - ENFORCE_IMMEDIATELY + - WAIT_FOR_TERMINATION + description: >- + Determines how changes should be made to clusters that are + not in ``TERMINATED`` state. - ``ENFORCE_IMMEDIATELY``: If + the cluster is in a ``RUNNING`` state, it will be restarted + so that the new attributes can take effect. For other states + aside from ``TERMINATED`` state, the request will be + rejected. - ``WAIT_FOR_TERMINATION``: The cluster is not + immediately edited. Instead, a pending enforcement is + scheduled to update the cluster when it terminates or + restarts. When this occurs, ``enforce_result`` will contain + ``DEFERRED``. Only workspace admins can use this mode. + Regardless of the enforce mode, clusters in ``TERMINATED`` + state are immediately edited. validate_only: type: boolean description: >- @@ -7448,7 +8729,7 @@ paths: further constrain the maximum number of results returned in a single page. :param page_token: str (optional) A page token that can be used to navigate to the next page or previous page as returned by - `next_page_token` or `prev_page_token`. + ``next_page_token`` or ``prev_page_token``. :returns: Iterator over :class:`ClusterCompliance` parameters: @@ -7474,7 +8755,8 @@ paths: type: string description: >- A page token that can be used to navigate to the next page or - previous page as returned by `next_page_token` or `prev_page_token`. + previous page as returned by ``next_page_token`` or + ``prev_page_token``. responses: '200': description: Success @@ -7614,7 +8896,7 @@ components: type: string description: >- abfss destination, e.g. - `abfss://@.dfs.core.windows.net/`. + ``abfss://@.dfs.core.windows.net/``. required: - destination description: A storage location in Adls Gen2 @@ -7642,14 +8924,14 @@ components: types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail. These EBS volumes - will be mounted at `/ebs0`, `/ebs1`, and etc. Instance store volumes - will be mounted at `/local_disk0`, `/local_disk1`, and etc. If EBS - volumes are attached, Databricks will configure Spark to use only - the EBS volumes for scratch storage because heterogenously sized - scratch devices can lead to inefficient disk utilization. If no EBS - volumes are attached, Databricks will configure Spark to use + will be mounted at ``/ebs0``, ``/ebs1``, and etc. Instance store + volumes will be mounted at ``/local_disk0``, ``/local_disk1``, and + etc. If EBS volumes are attached, Databricks will configure Spark to + use only the EBS volumes for scratch storage because heterogenously + sized scratch devices can lead to inefficient disk utilization. If + no EBS volumes are attached, Databricks will configure Spark to use instance store volumes. Please note that if EBS volumes are - specified, then the Spark configuration `spark.local.dir` will be + specified, then the Spark configuration ``spark.local.dir`` will be overridden. ebs_volume_iops: type: integer @@ -7676,14 +8958,14 @@ components: first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be placed on + The first ``first_on_demand`` nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is - less than the current cluster size, `first_on_demand` nodes will be - placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not affect + less than the current cluster size, ``first_on_demand`` nodes will + be placed on on-demand instances and the remainder will be placed on + ``availability`` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. instance_profile_arn: @@ -7700,11 +8982,11 @@ components: description: >- The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this - field is set to 50, and the cluster needs a new `r3.xlarge` spot + field is set to 50, and the cluster needs a new ``r3.xlarge`` spot instance, then the bid price is half of the price of on-demand - `r3.xlarge` instances. Similarly, if this field is set to 200, the - bid price is twice the price of on-demand `r3.xlarge` instances. If - not specified, the default value is 100. When spot instances are + ``r3.xlarge`` instances. Similarly, if this field is set to 200, the + bid price is twice the price of on-demand ``r3.xlarge`` instances. + If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000. @@ -7721,7 +9003,7 @@ components: try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. The list of available zones as well as the default value can be - found by using the `List Zones` method. + found by using the ``List Zones`` method. description: >- Attributes set during cluster creation which are related to Amazon Web Services. @@ -7732,19 +9014,36 @@ components: $ref: '#/components/schemas/AzureAvailability' description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. Note: If `first_on_demand` is zero, this + ``first_on_demand`` ones. Note: If ``first_on_demand`` is zero, this availability type will be used for the entire cluster. + capacity_reservation_group: + type: string + description: >- + The Azure capacity reservation group resource ID to use for + launching VMs. When specified, VMs will be launched using the + provided capacity reservation. Capacity reservations can only be + specified when the workspace uses injected vnet (i.e. customer + defined vnet not managed by databricks). Ensure the + databricks-login-prod Enterprise Application is granted the + following four permissions: 1. + Microsoft.Compute/capacityReservationGroups/read 2. + Microsoft.Compute/capacityReservationGroups/deploy/action 3. + Microsoft.Compute/capacityReservationGroups/capacityReservations/read + 4. + Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action + Format: + ``/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}`` first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be placed on + The first ``first_on_demand`` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less - than the current cluster size, `first_on_demand` nodes will be + than the current cluster size, ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not affect + ``availability`` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. log_analytics_info: @@ -7763,6 +9062,13 @@ components: description: >- Attributes set during cluster creation which are related to Microsoft Azure. + CancelPendingClusterEnforcementResponse: + type: object + properties: {} + description: >- + Response for canceling the pending enforcement for a cluster. If the + cancel request succeeds, an + empty response object is returned. Otherwise, an error response is returned. CancelResponse: type: object properties: {} @@ -7829,9 +9135,11 @@ components: spark_version: type: string description: >- - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of - available Spark versions can be retrieved by using the - :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list + of available Spark versions can be retrieved by using the + `clusters/sparkVersions + `__ + API call. autotermination_minutes: type: integer description: >- @@ -7858,9 +9166,9 @@ components: destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the - destination every `5 mins`. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination of executor - logs is `$destination/$clusterId/executor`. + destination every ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. cluster_name: type: string description: >- @@ -7873,11 +9181,14 @@ components: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these - tags in addition to `default_tags`. Notes: - Currently, Databricks + tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags data_security_mode: $ref: '#/components/schemas/DataSecurityMode' + dependency_mode: + $ref: '#/components/schemas/DependencyMode' + description: Controls dependency configuration for the cluster. docker_image: $ref: '#/components/schemas/DockerImage' description: Custom docker image BYOC @@ -7895,7 +9206,7 @@ components: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as - `node_type_id` defined above. This field, along with node_type_id, + ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. @@ -7921,17 +9232,17 @@ components: description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially - in the order provided. If `cluster_log_conf` is specified, init - script logs are sent to `//init_scripts`. + in the order provided. If ``cluster_log_conf`` is specified, init + script logs are sent to ``//init_scripts``. instance_pool_id: type: string description: The optional ID of the instance pool to which the cluster belongs. is_single_node: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. When set - to true, Databricks will automatically set single node related - `custom_tags`, `spark_conf`, and `num_workers` + This field can only be used when ``kind = CLASSIC_PREVIEW``. When + set to true, Databricks will automatically set single node related + ``custom_tags``, ``spark_conf``, and ``num_workers`` kind: $ref: '#/components/schemas/Kind' node_type_id: @@ -7941,7 +9252,9 @@ components: to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved - by using the :method:clusters/listNodeTypes API call. + by using the `clusters/listNodeTypes + `__ + API call. policy_id: type: string description: >- @@ -7956,36 +9269,36 @@ components: $ref: '#/components/schemas/RuntimeEngine' description: >- Determines the cluster's runtime engine, either standard or Photon. - This field is not compatible with legacy `spark_version` values that - contain `-photon-`. Remove `-photon-` from the `spark_version` and - set `runtime_engine` to `PHOTON`. If left unspecified, the runtime - engine defaults to standard unless the spark_version contains - -photon-, in which case Photon will be used. + This field is not compatible with legacy ``spark_version`` values + that contain ``-photon-``. Remove ``-photon-`` from the + ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left + unspecified, the runtime engine defaults to standard unless the + spark_version contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: Single user name if data_security_mode is ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via - `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the - form (X,Y) will be exported as is (i.e., `export X='Y'`) while + form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional - set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This + set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` + ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": + "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}`` ssh_public_keys: type: array items: @@ -7993,7 +9306,7 @@ components: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login - with the user name `ubuntu` on port `2200`. Up to 10 keys can be + with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can be specified. total_initial_remote_disk_size: type: integer @@ -8004,10 +9317,10 @@ components: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. - `effective_spark_version` is determined by `spark_version` (DBR - release), this field `use_ml_runtime`, and whether `node_type_id` is - gpu node or not. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by ``spark_version`` (DBR + release), this field ``use_ml_runtime``, and whether + ``node_type_id`` is gpu node or not. worker_node_type_flexibility: $ref: '#/components/schemas/NodeTypeFlexibility' description: Flexible node type configuration for worker nodes. @@ -8029,6 +9342,11 @@ components: description: >- Whether this cluster is in compliance with the latest version of its policy. + pending_enforcement: + $ref: '#/components/schemas/PendingEnforcement' + description: >- + Information about the pending enforcement for the cluster. Only + present if a pending enforcement is scheduled for the cluster. violations: type: object description: >- @@ -8085,9 +9403,9 @@ components: destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the - destination every `5 mins`. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination of executor - logs is `$destination/$clusterId/executor`. + destination every ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. cluster_log_status: $ref: '#/components/schemas/LogSyncStatus' description: Cluster log delivery status. @@ -8116,7 +9434,7 @@ components: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these - tags in addition to `default_tags`. Notes: - Currently, Databricks + tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags data_security_mode: @@ -8124,10 +9442,13 @@ components: default_tags: type: object description: >- - Tags that are added by Databricks regardless of any `custom_tags`, + Tags that are added by Databricks regardless of any ``custom_tags``, including: - Vendor: Databricks - Creator: - ClusterName: - ClusterId: - Name: + dependency_mode: + $ref: '#/components/schemas/DependencyMode' + description: Controls dependency configuration for the cluster. docker_image: $ref: '#/components/schemas/DockerImage' description: Custom docker image BYOC @@ -8151,7 +9472,7 @@ components: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as - `node_type_id` defined above. This field, along with node_type_id, + ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. @@ -8182,17 +9503,17 @@ components: description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially - in the order provided. If `cluster_log_conf` is specified, init - script logs are sent to `//init_scripts`. + in the order provided. If ``cluster_log_conf`` is specified, init + script logs are sent to ``//init_scripts``. instance_pool_id: type: string description: The optional ID of the instance pool to which the cluster belongs. is_single_node: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. When set - to true, Databricks will automatically set single node related - `custom_tags`, `spark_conf`, and `num_workers` + This field can only be used when ``kind = CLASSIC_PREVIEW``. When + set to true, Databricks will automatically set single node related + ``custom_tags``, ``spark_conf``, and ``num_workers`` jdbc_port: type: integer description: >- @@ -8215,19 +9536,21 @@ components: to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved - by using the :method:clusters/listNodeTypes API call. + by using the `clusters/listNodeTypes + `__ + API call. num_workers: type: integer description: >- Number of worker nodes that this cluster should have. A cluster has - one Spark Driver and `num_workers` Executors for a total of - `num_workers` + 1 Spark nodes. Note: When reading the properties of - a cluster, this field reflects the desired number of workers rather - than the actual current number of workers. For instance, if a + one Spark Driver and ``num_workers`` Executors for a total of + ``num_workers`` + 1 Spark nodes. Note: When reading the properties + of a cluster, this field reflects the desired number of workers + rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase from 5 to 10 - as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually increase from 5 to + 10 as the new nodes are provisioned. policy_id: type: string description: >- @@ -8242,48 +9565,51 @@ components: $ref: '#/components/schemas/RuntimeEngine' description: >- Determines the cluster's runtime engine, either standard or Photon. - This field is not compatible with legacy `spark_version` values that - contain `-photon-`. Remove `-photon-` from the `spark_version` and - set `runtime_engine` to `PHOTON`. If left unspecified, the runtime - engine defaults to standard unless the spark_version contains - -photon-, in which case Photon will be used. + This field is not compatible with legacy ``spark_version`` values + that contain ``-photon-``. Remove ``-photon-`` from the + ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left + unspecified, the runtime engine defaults to standard unless the + spark_version contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: Single user name if data_security_mode is ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via - `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_context_id: type: integer description: >- A canonical SparkContext identifier. This value *does* change when - the Spark driver restarts. The pair `(cluster_id, spark_context_id)` - is a globally unique identifier over all Spark contexts. + the Spark driver restarts. The pair ``(cluster_id, + spark_context_id)`` is a globally unique identifier over all Spark + contexts. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the - form (X,Y) will be exported as is (i.e., `export X='Y'`) while + form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional - set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This + set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` + ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": + "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}`` spark_version: type: string description: >- - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of - available Spark versions can be retrieved by using the - :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list + of available Spark versions can be retrieved by using the + `clusters/sparkVersions + `__ + API call. spec: $ref: '#/components/schemas/ClusterSpec' description: >- @@ -8297,13 +9623,13 @@ components: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login - with the user name `ubuntu` on port `2200`. Up to 10 keys can be + with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can be specified. start_time: type: integer description: >- Time (in epoch milliseconds) when the cluster creation request was - received (when the cluster entered a `PENDING` state). + received (when the cluster entered a ``PENDING`` state). state: $ref: '#/components/schemas/State' description: Current state of the cluster. @@ -8311,7 +9637,7 @@ components: type: string description: >- A message associated with the most recent state transition (e.g., - the reason why the cluster entered a `TERMINATED` state). + the reason why the cluster entered a ``TERMINATED`` state). terminated_time: type: integer description: >- @@ -8321,7 +9647,7 @@ components: $ref: '#/components/schemas/TerminationReason' description: >- Information about why the cluster was terminated. This field only - appears when the cluster is in a `TERMINATING` or `TERMINATED` + appears when the cluster is in a ``TERMINATING`` or ``TERMINATED`` state. total_initial_remote_disk_size: type: integer @@ -8332,10 +9658,10 @@ components: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. - `effective_spark_version` is determined by `spark_version` (DBR - release), this field `use_ml_runtime`, and whether `node_type_id` is - gpu node or not. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by ``spark_version`` (DBR + release), this field ``use_ml_runtime``, and whether + ``node_type_id`` is gpu node or not. worker_node_type_flexibility: $ref: '#/components/schemas/NodeTypeFlexibility' description: Flexible node type configuration for worker nodes. @@ -8379,21 +9705,21 @@ components: dbfs: $ref: '#/components/schemas/DbfsStorageInfo' description: >- - destination needs to be provided. e.g. `{ "dbfs" : { "destination" : - "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs" : { "destination" + : "dbfs:/home/cluster_log" } }`` s3: $ref: '#/components/schemas/S3StorageInfo' description: >- destination and either the region or endpoint need to be provided. - e.g. `{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", - "region" : "us-west-2" } }` Cluster iam role is used to access s3, - please make sure the cluster iam role in `instance_profile_arn` has - permission to write data to the s3 destination. + e.g. ``{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", + "region" : "us-west-2" } }`` Cluster iam role is used to access s3, + please make sure the cluster iam role in ``instance_profile_arn`` + has permission to write data to the s3 destination. volumes: $ref: '#/components/schemas/VolumesStorageInfo' description: >- - destination needs to be provided, e.g. `{ "volumes": { - "destination": "/Volumes/catalog/schema/volume/cluster_log" } }` + destination needs to be provided, e.g. ``{ "volumes": { + "destination": "/Volumes/catalog/schema/volume/cluster_log" } }`` description: Cluster log delivery config ClusterPermission: type: object @@ -8519,14 +9845,14 @@ components: type: integer description: >- Number of worker nodes that this cluster should have. A cluster has - one Spark Driver and `num_workers` Executors for a total of - `num_workers` + 1 Spark nodes. Note: When reading the properties of - a cluster, this field reflects the desired number of workers rather - than the actual current number of workers. For instance, if a + one Spark Driver and ``num_workers`` Executors for a total of + ``num_workers`` + 1 Spark nodes. Note: When reading the properties + of a cluster, this field reflects the desired number of workers + rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase from 5 to 10 - as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually increase from 5 to + 10 as the new nodes are provisioned. ClusterSpec: type: object properties: @@ -8568,9 +9894,9 @@ components: destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the - destination every `5 mins`. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination of executor - logs is `$destination/$clusterId/executor`. + destination every ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. cluster_name: type: string description: >- @@ -8583,11 +9909,14 @@ components: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these - tags in addition to `default_tags`. Notes: - Currently, Databricks + tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags data_security_mode: $ref: '#/components/schemas/DataSecurityMode' + dependency_mode: + $ref: '#/components/schemas/DependencyMode' + description: Controls dependency configuration for the cluster. docker_image: $ref: '#/components/schemas/DockerImage' description: Custom docker image BYOC @@ -8605,7 +9934,7 @@ components: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as - `node_type_id` defined above. This field, along with node_type_id, + ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. @@ -8631,17 +9960,17 @@ components: description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially - in the order provided. If `cluster_log_conf` is specified, init - script logs are sent to `//init_scripts`. + in the order provided. If ``cluster_log_conf`` is specified, init + script logs are sent to ``//init_scripts``. instance_pool_id: type: string description: The optional ID of the instance pool to which the cluster belongs. is_single_node: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. When set - to true, Databricks will automatically set single node related - `custom_tags`, `spark_conf`, and `num_workers` + This field can only be used when ``kind = CLASSIC_PREVIEW``. When + set to true, Databricks will automatically set single node related + ``custom_tags``, ``spark_conf``, and ``num_workers`` kind: $ref: '#/components/schemas/Kind' node_type_id: @@ -8651,19 +9980,21 @@ components: to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved - by using the :method:clusters/listNodeTypes API call. + by using the `clusters/listNodeTypes + `__ + API call. num_workers: type: integer description: >- Number of worker nodes that this cluster should have. A cluster has - one Spark Driver and `num_workers` Executors for a total of - `num_workers` + 1 Spark nodes. Note: When reading the properties of - a cluster, this field reflects the desired number of workers rather - than the actual current number of workers. For instance, if a + one Spark Driver and ``num_workers`` Executors for a total of + ``num_workers`` + 1 Spark nodes. Note: When reading the properties + of a cluster, this field reflects the desired number of workers + rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase from 5 to 10 - as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually increase from 5 to + 10 as the new nodes are provisioned. policy_id: type: string description: >- @@ -8678,42 +10009,44 @@ components: $ref: '#/components/schemas/RuntimeEngine' description: >- Determines the cluster's runtime engine, either standard or Photon. - This field is not compatible with legacy `spark_version` values that - contain `-photon-`. Remove `-photon-` from the `spark_version` and - set `runtime_engine` to `PHOTON`. If left unspecified, the runtime - engine defaults to standard unless the spark_version contains - -photon-, in which case Photon will be used. + This field is not compatible with legacy ``spark_version`` values + that contain ``-photon-``. Remove ``-photon-`` from the + ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left + unspecified, the runtime engine defaults to standard unless the + spark_version contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: Single user name if data_security_mode is ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via - `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the - form (X,Y) will be exported as is (i.e., `export X='Y'`) while + form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional - set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This + set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` + ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": + "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}`` spark_version: type: string description: >- - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of - available Spark versions can be retrieved by using the - :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list + of available Spark versions can be retrieved by using the + `clusters/sparkVersions + `__ + API call. ssh_public_keys: type: array items: @@ -8721,7 +10054,7 @@ components: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login - with the user name `ubuntu` on port `2200`. Up to 10 keys can be + with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can be specified. total_initial_remote_disk_size: type: integer @@ -8732,10 +10065,10 @@ components: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. - `effective_spark_version` is determined by `spark_version` (DBR - release), this field `use_ml_runtime`, and whether `node_type_id` is - gpu node or not. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by ``spark_version`` (DBR + release), this field ``use_ml_runtime``, and whether + ``node_type_id`` is gpu node or not. worker_node_type_flexibility: $ref: '#/components/schemas/NodeTypeFlexibility' description: Flexible node type configuration for worker nodes. @@ -8812,10 +10145,60 @@ components: properties: destination: type: string - description: dbfs destination, e.g. `dbfs:/my/path` + description: dbfs destination, e.g. ``dbfs:/my/path`` required: - destination description: A storage location in DBFS + DefaultBaseEnvironment: + type: object + properties: + base_environment_cache: + type: array + items: + $ref: '#/components/schemas/DefaultBaseEnvironmentCache' + base_environment_type: + $ref: '#/components/schemas/BaseEnvironmentType' + created_timestamp: + type: integer + creator_user_id: + type: integer + environment: + $ref: '#/components/schemas/Environment' + description: >- + Note: we made ``environment`` non-internal because we need to expose + its ``client`` field. All other fields should be treated as + internal. + filepath: + type: string + id: + type: string + is_default: + type: boolean + last_updated_timestamp: + type: integer + last_updated_user_id: + type: integer + message: + type: string + name: + type: string + principal_ids: + type: array + items: + type: integer + status: + $ref: '#/components/schemas/DefaultBaseEnvironmentCacheStatus' + DefaultBaseEnvironmentCache: + type: object + properties: + indefinite_materialized_environment: + $ref: '#/components/schemas/MaterializedEnvironment' + materialized_environment: + $ref: '#/components/schemas/MaterializedEnvironment' + message: + type: string + status: + $ref: '#/components/schemas/DefaultBaseEnvironmentCacheStatus' DeleteClusterResponse: type: object properties: {} @@ -8831,6 +10214,48 @@ components: DestroyResponse: type: object properties: {} + Diagnostic: + type: object + properties: + checks: + type: array + items: + $ref: '#/components/schemas/DiagnosticCheck' + description: List of individual checks (maps to UI rows). + diagnostics_status: + $ref: '#/components/schemas/DiagnosticsStatus' + description: >- + Overall run status (PASSED / FAILED / NOT_RUN). FAILED if any + individual check fails. + description: >- + The cluster diagnostics singleton resource: the latest diagnostics run + for a cluster. + DiagnosticCheck: + type: object + properties: + check_id: + $ref: '#/components/schemas/CheckId' + description: Identifies the probed target (from the CheckId enum). + check_status: + $ref: '#/components/schemas/DiagnosticsStatus' + description: Outcome of this specific check (PASSED / FAILED / NOT_RUN). + description: + type: string + description: >- + Static summary of what this check tests (e.g., "Control-plane REST + reachability"). + reason: + $ref: '#/components/schemas/DiagnosticsErrorReason' + description: >- + The error reason that caused the check failure, mapping to the first + failing layer (DNS -> TCP -> TLS -> HTTP). Set ONLY when + check_status = FAILED. + remediation: + type: string + description: >- + Static, human-readable instructions to resolve the failure. Set ONLY + when check_status = FAILED. + description: A single check (one row of the cluster's Diagnostics tab). DiskSpec: type: object properties: @@ -8846,9 +10271,9 @@ components: heterogenously sized scratch devices can lead to inefficient disk utilization. If no disks are attached, Databricks will configure Spark to use instance store disks. Note: If disks are specified, - then the Spark configuration `spark.local.dir` will be overridden. - Disks will be mounted at: - For AWS: `/ebs0`, `/ebs1`, and etc. - - For Azure: `/remote_volume0`, `/remote_volume1`, and etc. + then the Spark configuration ``spark.local.dir`` will be overridden. + Disks will be mounted at: - For AWS: ``/ebs0``, ``/ebs1``, and etc. + - For Azure: ``/remote_volume0``, ``/remote_volume1``, and etc. disk_iops: type: integer disk_size: @@ -8911,29 +10336,249 @@ components: changes: type: array items: - $ref: '#/components/schemas/ClusterSettingsChange' - has_changes: + $ref: '#/components/schemas/ClusterSettingsChange' + enforce_result: + $ref: >- + #/components/schemas/EnforcePolicyComplianceForClusterResponseEnforceResult + description: Describes whether changes have been applied to the cluster. + has_changes: + type: boolean + description: >- + Whether any changes have been made to the cluster settings for the + cluster to become compliant with its policy. + EnforcePolicyComplianceForClusterResponseClusterSettings: + type: object + properties: + autoscale: + $ref: '#/components/schemas/AutoScale' + autotermination_minutes: + type: integer + description: >- + Automatically terminates the cluster after it is inactive for this + time in minutes. If not set, this cluster will not be automatically + terminated. If specified, the threshold must be between 10 and 10000 + minutes. Users can also set this value to 0 to explicitly disable + automatic termination. + aws_attributes: + $ref: '#/components/schemas/AwsAttributes' + description: >- + Attributes related to clusters running on Amazon Web Services. If + not specified at cluster creation, a set of default values will be + used. + azure_attributes: + $ref: '#/components/schemas/AzureAttributes' + description: >- + Attributes related to clusters running on Microsoft Azure. If not + specified at cluster creation, a set of default values will be used. + cluster_log_conf: + $ref: '#/components/schemas/ClusterLogConf' + description: >- + The configuration for delivering spark logs to a long-term storage + destination. Three kinds of destinations (DBFS, S3 and Unity Catalog + volumes) are supported. Only one destination can be specified for + one cluster. If the conf is given, the logs will be delivered to the + destination every ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. + cluster_name: + type: string + description: >- + Cluster name requested by the user. This doesn't have to be unique. + If not specified at creation, the cluster name will be an empty + string. For job clusters, the cluster name is automatically set + based on the job and job run IDs. + custom_tags: + type: object + description: >- + Additional tags for cluster resources. Databricks will tag all + cluster resources (e.g., AWS instances and EBS volumes) with these + tags in addition to ``default_tags``. Notes: - Currently, Databricks + allows at most 45 custom tags - Clusters can only reuse cloud + resources if the resources' tags are a subset of the cluster tags + data_security_mode: + $ref: '#/components/schemas/DataSecurityMode' + dependency_mode: + $ref: '#/components/schemas/DependencyMode' + description: Controls dependency configuration for the cluster. + docker_image: + $ref: '#/components/schemas/DockerImage' + description: Custom docker image BYOC + driver_instance_pool_id: + type: string + description: >- + The optional ID of the instance pool for the driver of the cluster + belongs. The pool cluster uses the instance pool with id + (instance_pool_id) if the driver pool is not assigned. + driver_node_type_flexibility: + $ref: '#/components/schemas/NodeTypeFlexibility' + description: Flexible node type configuration for the driver node. + driver_node_type_id: + type: string + description: >- + The node type of the Spark driver. Note that this field is optional; + if unset, the driver node type will be set as the same value as + ``node_type_id`` defined above. This field, along with node_type_id, + should not be set if virtual_cluster_size is set. If both + driver_node_type_id, node_type_id, and virtual_cluster_size are + specified, driver_node_type_id and node_type_id take precedence. + enable_elastic_disk: + type: boolean + description: >- + Autoscaling Local Storage: when enabled, this cluster will + dynamically acquire additional disk space when its Spark workers are + running low on disk space. + enable_local_disk_encryption: + type: boolean + description: Whether to enable LUKS on cluster VMs' local disks + gcp_attributes: + $ref: '#/components/schemas/GcpAttributes' + description: >- + Attributes related to clusters running on Google Cloud Platform. If + not specified at cluster creation, a set of default values will be + used. + init_scripts: + type: array + items: + $ref: '#/components/schemas/InitScriptInfo' + description: >- + The configuration for storing init scripts. Any number of + destinations can be specified. The scripts are executed sequentially + in the order provided. If ``cluster_log_conf`` is specified, init + script logs are sent to ``//init_scripts``. + instance_pool_id: + type: string + description: The optional ID of the instance pool to which the cluster belongs. + is_single_node: + type: boolean + description: >- + This field can only be used when ``kind = CLASSIC_PREVIEW``. When + set to true, Databricks will automatically set single node related + ``custom_tags``, ``spark_conf``, and ``num_workers`` + kind: + $ref: '#/components/schemas/Kind' + node_type_id: + type: string + description: >- + This field encodes, through a single value, the resources available + to each of the Spark nodes in this cluster. For example, the Spark + nodes can be provisioned and optimized for memory or compute + intensive workloads. A list of available node types can be retrieved + by using the `clusters/listNodeTypes + `__ + API call. + num_workers: + type: integer + description: >- + Number of worker nodes that this cluster should have. A cluster has + one Spark Driver and ``num_workers`` Executors for a total of + ``num_workers`` + 1 Spark nodes. Note: When reading the properties + of a cluster, this field reflects the desired number of workers + rather than the actual current number of workers. For instance, if a + cluster is resized from 5 to 10 workers, this field will immediately + be updated to reflect the target size of 10 workers, whereas the + workers listed in ``spark_info`` will gradually increase from 5 to + 10 as the new nodes are provisioned. + policy_id: + type: string + description: >- + The ID of the cluster policy used to create the cluster if + applicable. + remote_disk_throughput: + type: integer + description: >- + If set, what the configurable throughput (in Mb/s) for the remote + disk is. Currently only supported for GCP HYPERDISK_BALANCED disks. + runtime_engine: + $ref: '#/components/schemas/RuntimeEngine' + description: >- + Determines the cluster's runtime engine, either standard or Photon. + This field is not compatible with legacy ``spark_version`` values + that contain ``-photon-``. Remove ``-photon-`` from the + ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left + unspecified, the runtime engine defaults to standard unless the + spark_version contains -photon-, in which case Photon will be used. + single_user_name: + type: string + description: Single user name if data_security_mode is ``SINGLE_USER`` + spark_conf: + type: object + description: >- + An object containing a set of optional, user-specified Spark + configuration key-value pairs. Users can also pass in a string of + extra JVM options to the driver and the executors via + ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. + spark_env_vars: + type: object + description: >- + An object containing a set of optional, user-specified environment + variable key-value pairs. Please note that key-value pair of the + form (X,Y) will be exported as is (i.e., ``export X='Y'``) while + launching the driver and workers. In order to specify an additional + set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This + ensures that all default databricks managed environmental variables + are included as well. Example Spark environment variables: + ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": + "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}`` + spark_version: + type: string + description: >- + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list + of available Spark versions can be retrieved by using the + `clusters/sparkVersions + `__ + API call. + ssh_public_keys: + type: array + items: + type: string + description: >- + SSH public key contents that will be added to each Spark node in + this cluster. The corresponding private keys can be used to login + with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can be + specified. + total_initial_remote_disk_size: + type: integer + description: >- + If set, what the total initial volume size (in GB) of the remote + disks should be. Currently only supported for GCP HYPERDISK_BALANCED + disks. + use_ml_runtime: type: boolean description: >- - Whether any changes have been made to the cluster settings for the - cluster to become compliant with its policy. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by ``spark_version`` (DBR + release), this field ``use_ml_runtime``, and whether + ``node_type_id`` is gpu node or not. + worker_node_type_flexibility: + $ref: '#/components/schemas/NodeTypeFlexibility' + description: Flexible node type configuration for worker nodes. + workload_type: + $ref: '#/components/schemas/WorkloadType' Environment: type: object properties: base_environment: type: string description: >- - The `base_environment` key refers to an `env.yaml` file that - specifies an environment version and a collection of dependencies - required for the environment setup. This `env.yaml` file may itself - include a `base_environment` reference pointing to another - `env_1.yaml` file. However, when used as a base environment, - `env_1.yaml` (or further nested references) will not be processed or - included in the final environment, meaning that the resolution of - `base_environment` references is not recursive. + The base environment this environment is built on top of. A base + environment defines the environment version and a list of + dependencies for serverless compute. The value can be a file path to + a custom ``env.yaml`` file (e.g., ``/Workspace/path/to/env.yaml``). + Support for a Databricks-provided base environment ID (e.g., + ``workspace-base-environments/databricks_ai_v4``) and workspace base + environment ID (e.g., + ``workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7``) + is in Beta. Either ``environment_version`` or ``base_environment`` + can be provided. For more information about Databricks-provided base + environments, see the [list workspace base + environments](:method:Environments/ListWorkspaceBaseEnvironments) + API. For more information, see client: type: string - description: Use `environment_version` instead. + description: Use ``environment_version`` instead. dependencies: type: array items: @@ -8949,7 +10594,7 @@ components: environment_version: type: string description: >- - Either `environment_version` or `base_environment` needs to be + Either ``environment_version`` or ``base_environment`` needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer. @@ -8959,11 +10604,11 @@ components: type: string description: >- List of java dependencies. Each dependency is a string representing - a java library path. For example: `/Volumes/path/to/test.jar`. + a java library path. For example: ``/Volumes/path/to/test.jar``. description: >- The environment entity used to preserve serverless environment side panel, jobs' environment for - non-notebook task, and DLT's environment for classic and serverless pipelines. In this minimal + non-notebook task, and SDP's environment for classic and serverless pipelines. In this minimal environment spec, only pip and java dependencies are supported. EventDetails: type: object @@ -9011,7 +10656,7 @@ components: type: string description: >- Unique identifier of the specific job run associated with this - cluster event * For clusters created for jobs, this will be the same + cluster event - For clusters created for jobs, this will be the same as the cluster name previous_attributes: $ref: '#/components/schemas/ClusterAttributes' @@ -9025,8 +10670,8 @@ components: reason: $ref: '#/components/schemas/TerminationReason' description: >- - A termination reason: * On a TERMINATED event, this is the reason of - the termination. * On a RESIZE_COMPLETE event, this indicates the + A termination reason: - On a TERMINATED event, this is the reason of + the termination. - On a RESIZE_COMPLETE event, this indicates the reason that we failed to acquire some nodes. target_num_vcpus: type: integer @@ -9051,17 +10696,23 @@ components: boot_disk_size: type: integer description: Boot disk size in GB + confidential_compute_type: + $ref: '#/components/schemas/ConfidentialComputeType' + description: >- + The confidential computing technology for this cluster's instances. + Currently only SEV_SNP is supported, and only on N2D instance types. + When not set, no confidential computing is applied. first_on_demand: type: integer description: >- - The first `first_on_demand` nodes of the cluster will be placed on + The first ``first_on_demand`` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less - than the current cluster size, `first_on_demand` nodes will be + than the current cluster size, ``first_on_demand`` nodes will be placed on on-demand instances and the remainder will be placed on - `availability` instances. Note that this value does not affect + ``availability`` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. google_service_account: @@ -9076,9 +10727,9 @@ components: description: >- If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. - Refer to [GCP documentation] for the supported number of local SSDs - for each instance type. [GCP documentation]: - https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds + Refer to `GCP documentation + `__ + for the supported number of local SSDs for each instance type. use_preemptible_executors: type: boolean description: >- @@ -9102,7 +10753,7 @@ components: properties: destination: type: string - description: GCS destination/URI, e.g. `gs://my-bucket/some-prefix` + description: GCS destination/URI, e.g. ``gs://my-bucket/some-prefix`` required: - destination description: A storage location in Google Cloud Platform's GCS @@ -9111,6 +10762,11 @@ components: properties: is_compliant: type: boolean + pending_enforcement: + $ref: '#/components/schemas/PendingEnforcement' + description: >- + Information about the pending enforcement for the cluster. Only + present if a pending enforcement is scheduled for the cluster. violations: type: object description: >- @@ -9241,8 +10897,8 @@ components: description: >- Additional tags for pool resources. Databricks will tag all pool resources (e.g., AWS instances and EBS volumes) with these tags in - addition to `default_tags`. Notes: - Currently, Databricks allows at - most 45 custom tags + addition to ``default_tags``. Notes: - Currently, Databricks allows + at most 45 custom tags default_tags: type: object description: >- @@ -9255,6 +10911,15 @@ components: description: >- Defines the specification of the disks that will be attached to all spark containers. + enable_auto_alternate_node_types: + type: boolean + description: >- + For pools with node type flexibility (Fleet-V2), whether auto + generated alternate node type ids are enabled. This field should not + be true if node_type_flexibility is set. DEPRECATED: This field was + deprecated before entering PuPr and should no longer be used. + TODO(CJ-71514): Remove this field after sufficient time has passed + for all clients to migrate. enable_elastic_disk: type: boolean description: >- @@ -9304,7 +10969,9 @@ components: to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved - by using the :method:clusters/listNodeTypes API call. + by using the `clusters/listNodeTypes + `__ + API call. preloaded_docker_images: type: array items: @@ -9318,7 +10985,9 @@ components: A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be - retrieved by using the :method:clusters/sparkVersions API call. + retrieved by using the `clusters/sparkVersions + `__ + API call. remote_disk_throughput: type: integer description: >- @@ -9449,42 +11118,42 @@ components: $ref: '#/components/schemas/Adlsgen2Info' description: >- destination needs to be provided, e.g. - `abfss://@.dfs.core.windows.net/` + ``abfss://@.dfs.core.windows.net/`` dbfs: $ref: '#/components/schemas/DbfsStorageInfo' description: >- - destination needs to be provided. e.g. `{ "dbfs": { "destination" : - "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs": { "destination" : + "dbfs:/home/cluster_log" } }`` file: $ref: '#/components/schemas/LocalFileInfo' description: >- - destination needs to be provided, e.g. `{ "file": { "destination": - "file:/my/local/file.sh" } }` + destination needs to be provided, e.g. ``{ "file": { "destination": + "file:/my/local/file.sh" } }`` gcs: $ref: '#/components/schemas/GcsStorageInfo' description: >- - destination needs to be provided, e.g. `{ "gcs": { "destination": - "gs://my-bucket/file.sh" } }` + destination needs to be provided, e.g. ``{ "gcs": { "destination": + "gs://my-bucket/file.sh" } }`` s3: $ref: '#/components/schemas/S3StorageInfo' description: >- destination and either the region or endpoint need to be provided. - e.g. `{ \"s3\": { \"destination\": - \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }` + e.g. ``{ \"s3\": { \"destination\": + \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }`` Cluster iam role is used to access s3, please make sure the cluster - iam role in `instance_profile_arn` has permission to write data to + iam role in ``instance_profile_arn`` has permission to write data to the s3 destination. volumes: $ref: '#/components/schemas/VolumesStorageInfo' description: >- - destination needs to be provided. e.g. `{ \"volumes\" : { - \"destination\" : \"/Volumes/my-init.sh\" } }` + destination needs to be provided. e.g. ``{ \"volumes\" : { + \"destination\" : \"/Volumes/my-init.sh\" } }`` workspace: $ref: '#/components/schemas/WorkspaceStorageInfo' description: >- - destination needs to be provided, e.g. `{ "workspace": { - "destination": "/cluster-init-scripts/setup-datadog.sh" } }` - description: 'Config for an individual init script Next ID: 11' + destination needs to be provided, e.g. ``{ "workspace": { + "destination": "/cluster-init-scripts/setup-datadog.sh" } }`` + description: Config for an individual init script InitScriptInfoAndExecutionDetails: type: object properties: @@ -9493,8 +11162,8 @@ components: dbfs: $ref: '#/components/schemas/DbfsStorageInfo' description: >- - destination needs to be provided. e.g. `{ "dbfs": { "destination" : - "dbfs:/home/cluster_log" } }` + destination needs to be provided. e.g. ``{ "dbfs": { "destination" : + "dbfs:/home/cluster_log" } }`` error_message: type: string description: >- @@ -9508,21 +11177,21 @@ components: file: $ref: '#/components/schemas/LocalFileInfo' description: >- - destination needs to be provided, e.g. `{ "file": { "destination": - "file:/my/local/file.sh" } }` + destination needs to be provided, e.g. ``{ "file": { "destination": + "file:/my/local/file.sh" } }`` gcs: $ref: '#/components/schemas/GcsStorageInfo' description: >- - destination needs to be provided, e.g. `{ "gcs": { "destination": - "gs://my-bucket/file.sh" } }` + destination needs to be provided, e.g. ``{ "gcs": { "destination": + "gs://my-bucket/file.sh" } }`` s3: $ref: '#/components/schemas/S3StorageInfo' description: >- destination and either the region or endpoint need to be provided. - e.g. `{ \"s3\": { \"destination\": - \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }` + e.g. ``{ \"s3\": { \"destination\": + \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }`` Cluster iam role is used to access s3, please make sure the cluster - iam role in `instance_profile_arn` has permission to write data to + iam role in ``instance_profile_arn`` has permission to write data to the s3 destination. status: $ref: >- @@ -9536,13 +11205,13 @@ components: volumes: $ref: '#/components/schemas/VolumesStorageInfo' description: >- - destination needs to be provided. e.g. `{ \"volumes\" : { - \"destination\" : \"/Volumes/my-init.sh\" } }` + destination needs to be provided. e.g. ``{ \"volumes\" : { + \"destination\" : \"/Volumes/my-init.sh\" } }`` workspace: $ref: '#/components/schemas/WorkspaceStorageInfo' description: >- - destination needs to be provided, e.g. `{ "workspace": { - "destination": "/cluster-init-scripts/setup-datadog.sh" } }` + destination needs to be provided, e.g. ``{ "workspace": { + "destination": "/cluster-init-scripts/setup-datadog.sh" } }`` InstallLibrariesResponse: type: object properties: {} @@ -9593,8 +11262,8 @@ components: description: >- Additional tags for pool resources. Databricks will tag all pool resources (e.g., AWS instances and EBS volumes) with these tags in - addition to `default_tags`. Notes: - Currently, Databricks allows at - most 45 custom tags + addition to ``default_tags``. Notes: - Currently, Databricks allows + at most 45 custom tags default_tags: type: object description: >- @@ -9607,6 +11276,15 @@ components: description: >- Defines the specification of the disks that will be attached to all spark containers. + enable_auto_alternate_node_types: + type: boolean + description: >- + For pools with node type flexibility (Fleet-V2), whether auto + generated alternate node type ids are enabled. This field should not + be true if node_type_flexibility is set. DEPRECATED: This field was + deprecated before entering PuPr and should no longer be used. + TODO(CJ-71514): Remove this field after sufficient time has passed + for all clients to migrate. enable_elastic_disk: type: boolean description: >- @@ -9659,7 +11337,9 @@ components: to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved - by using the :method:clusters/listNodeTypes API call. + by using the `clusters/listNodeTypes + `__ + API call. preloaded_docker_images: type: array items: @@ -9673,7 +11353,9 @@ components: A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be - retrieved by using the :method:clusters/sparkVersions API call. + retrieved by using the `clusters/sparkVersions + `__ + API call. remote_disk_throughput: type: integer description: >- @@ -9718,11 +11400,11 @@ components: description: >- Calculates the bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if - this field is set to 50, and the cluster needs a new `r3.xlarge` + this field is set to 50, and the cluster needs a new ``r3.xlarge`` spot instance, then the bid price is half of the price of on-demand - `r3.xlarge` instances. Similarly, if this field is set to 200, the - bid price is twice the price of on-demand `r3.xlarge` instances. If - not specified, the default value is 100. When spot instances are + ``r3.xlarge`` instances. Similarly, if this field is set to 200, the + bid price is twice the price of on-demand ``r3.xlarge`` instances. + If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000. @@ -9736,7 +11418,7 @@ components: id if the Databricks deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as - the default value can be found by using the `List Zones` method. + the default value can be found by using the ``List Zones`` method. description: >- Attributes set during instance pool creation which are related to Amazon Web Services. @@ -9746,6 +11428,24 @@ components: availability: $ref: '#/components/schemas/InstancePoolAzureAttributesAvailability' description: Availability type used for the spot nodes. + capacity_reservation_group: + type: string + description: >- + The Azure capacity reservation group resource ID to use for + launching VMs in this pool. When specified, VMs will be launched + using the provided capacity reservation. NOTE: Omitting this field + will clear any existing configured capacity reservation group on the + pool. Capacity reservations can only be specified when the workspace + uses injected vnet (i.e. customer defined vnet not managed by + databricks). Ensure the databricks-login-prod Enterprise Application + is granted the following four permissions: 1. + Microsoft.Compute/capacityReservationGroups/read 2. + Microsoft.Compute/capacityReservationGroups/deploy/action 3. + Microsoft.Compute/capacityReservationGroups/capacityReservations/read + 4. + Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action + Format: + ``/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}`` spot_bid_max_price: type: number description: >- @@ -9765,10 +11465,10 @@ components: type: integer description: >- If provided, each node in the instance pool will have this number of - local SSDs attached. Each local SSD is 375GB in size. Refer to [GCP - documentation] for the supported number of local SSDs for each - instance type. [GCP documentation]: - https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds + local SSDs attached. Each local SSD is 375GB in size. Refer to `GCP + documentation + `__ + for the supported number of local SSDs for each instance type. zone_id: type: string description: >- @@ -9850,9 +11550,9 @@ components: The AWS IAM role ARN of the role associated with the instance profile. This field is required if your role name and instance profile name do not match and you want to use the instance profile - with [Databricks SQL Serverless]. Otherwise, this field is optional. - [Databricks SQL Serverless]: - https://docs.databricks.com/sql/admin/serverless.html + with `Databricks SQL Serverless + `__. + Otherwise, this field is optional. is_meta_instance_profile: type: boolean description: >- @@ -9860,7 +11560,8 @@ components: used in credential passthrough scenarios. If true, it means the instance profile contains an meta IAM role which could assume a wide range of roles. Therefore it should always be used with - authorization. This field is optional, the default value is `false`. + authorization. This field is optional, the default value is + ``false``. required: - instance_profile_arn Library: @@ -9878,39 +11579,39 @@ components: type: string description: >- URI of the JAR library to install. Supported URIs include Workspace - paths, Unity Catalog Volumes paths, and S3 URIs. For example: `{ - "jar": "/Workspace/path/to/library.jar" }`, `{ "jar" : - "/Volumes/path/to/library.jar" }` or `{ "jar": - "s3://my-bucket/library.jar" }`. If S3 is used, please make sure the - cluster has read access on the library. You may need to launch the - cluster with an IAM role to access the S3 URI. + paths, Unity Catalog Volumes paths, and S3 URIs. For example: ``{ + "jar": "/Workspace/path/to/library.jar" }``, ``{ "jar" : + "/Volumes/path/to/library.jar" }`` or ``{ "jar": + "s3://my-bucket/library.jar" }``. If S3 is used, please make sure + the cluster has read access on the library. You may need to launch + the cluster with an IAM role to access the S3 URI. maven: $ref: '#/components/schemas/MavenLibrary' description: >- - Specification of a maven library to be installed. For example: `{ - "coordinates": "org.jsoup:jsoup:1.7.2" }` + Specification of a maven library to be installed. For example: ``{ + "coordinates": "org.jsoup:jsoup:1.7.2" }`` pypi: $ref: '#/components/schemas/PythonPyPiLibrary' description: >- - Specification of a PyPi library to be installed. For example: `{ - "package": "simplejson" }` + Specification of a PyPi library to be installed. For example: ``{ + "package": "simplejson" }`` requirements: type: string description: >- URI of the requirements.txt file to install. Only Workspace paths - and Unity Catalog Volumes paths are supported. For example: `{ - "requirements": "/Workspace/path/to/requirements.txt" }` or `{ - "requirements" : "/Volumes/path/to/requirements.txt" }` + and Unity Catalog Volumes paths are supported. For example: ``{ + "requirements": "/Workspace/path/to/requirements.txt" }`` or ``{ + "requirements" : "/Volumes/path/to/requirements.txt" }`` whl: type: string description: >- URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For - example: `{ "whl": "/Workspace/path/to/library.whl" }`, `{ "whl" : - "/Volumes/path/to/library.whl" }` or `{ "whl": - "s3://my-bucket/library.whl" }`. If S3 is used, please make sure the - cluster has read access on the library. You may need to launch the - cluster with an IAM role to access the S3 URI. + example: ``{ "whl": "/Workspace/path/to/library.whl" }``, ``{ "whl" + : "/Volumes/path/to/library.whl" }`` or ``{ "whl": + "s3://my-bucket/library.whl" }``. If S3 is used, please make sure + the cluster has read access on the library. You may need to launch + the cluster with an IAM role to access the S3 URI. LibraryFullStatus: type: object properties: @@ -10019,6 +11720,15 @@ components: The sorting criteria. By default, clusters are sorted by 3 columns from highest to lowest precedence: cluster state, pinned or unpinned, then cluster name. + ListDefaultBaseEnvironmentsResponse: + type: object + properties: + default_base_environments: + type: array + items: + $ref: '#/components/schemas/DefaultBaseEnvironment' + next_page_token: + type: string ListGlobalInitScriptsResponse: type: object properties: @@ -10085,13 +11795,37 @@ components: type: integer description: >- The timestamp of last attempt. If the last attempt fails, - `last_exception` will contain the exception in the last attempt. + ``last_exception`` will contain the exception in the last attempt. last_exception: type: string description: >- The exception thrown in the last attempt, it would be null (omitted in the response) if there is no exception in last attempted. description: The log delivery status + MaterializedEnvironment: + type: object + properties: + last_updated_timestamp: + type: integer + description: >- + The timestamp (in epoch milliseconds) when the materialized env is + updated. + description: >- + Materialized Environment information enables environment sharing and + reuse via Environment + Caching during library installations. Currently this feature is only supported for Python + libraries. + + - If the env cache entry in LMv2 DB doesn't exist or invalid, library installations and + environment materialization will occur. A new Materialized Environment metadata will be sent + from DP upon successful library installations and env materialization, and is persisted into + database by LMv2. + - If the env cache entry in LMv2 DB is valid, the Materialized Environment will be sent to DP by + LMv2, and DP will restore the cached environment from a store instead of reinstalling + libraries from scratch. + + If changed, also update estore/namespaces/defaultbaseenvironments/latest.proto with new version + If changed, also update estore/namespaces/envspecenvironments/latest.proto with new version MavenLibrary: type: object properties: @@ -10102,8 +11836,8 @@ components: items: type: string description: >- - List of dependences to exclude. For example: `["slf4j:slf4j", - "*:hadoop-client"]`. Maven dependency exclusions: + List of dependences to exclude. For example: ``["slf4j:slf4j", + "*:hadoop-client"]``. Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html. repo: type: string @@ -10154,9 +11888,9 @@ components: type: number description: >- Number of CPU cores available for this node type. Note that this can - be fractional, e.g., 2.5 cores, if the the number of cores on a - machine instance is not divisible by the number of Spark nodes on - that machine. + be fractional, e.g., 2.5 cores, if the number of cores on a machine + instance is not divisible by the number of Spark nodes on that + machine. description: type: string description: >- @@ -10246,6 +11980,39 @@ components: Configuration for flexible node types, allowing fallback to alternate node types during cluster launch and upscale. + PendingEnforcement: + type: object + properties: + enforcement_status: + $ref: '#/components/schemas/PendingEnforcementEnforcementStatus' + description: >- + Whether the pending enforcement will be applied. A pending + enforcement begins in ``ACTIVE`` state. If the enforcement fails to + apply too many times, the state transitions to ``INACTIVE``. + Afterwards, the enforcement must be re-scheduled to become + ``ACTIVE`` again. + initiate_time: + type: string + format: date-time + description: The time the pending enforcement was initiated. + initiator_user: + type: string + description: The user who initiated the pending enforcement. + target_changes: + type: array + items: + $ref: '#/components/schemas/ClusterSettingsChange' + description: >- + A list of changes that will be made to the cluster configuration + when the pending enforcement is applied. + target_spec: + $ref: >- + #/components/schemas/EnforcePolicyComplianceForClusterResponseClusterSettings + description: The new configuration to apply upon cluster termination or restart. + description: >- + Represents a pending enforcement on a cluster, which contains the + changes to make to the cluster + configuration when the cluster is next terminated or restarted. PendingInstanceError: type: object properties: @@ -10263,6 +12030,28 @@ components: Policy: type: object properties: + auto_enforcement_config: + $ref: '#/components/schemas/PolicyAutoEnforcementConfig' + description: >- + If present, auto enforcement is enabled for the policy. After the + policy is edited, a background operation may be scheduled to scan + and edit all clusters and jobs using this policy to be in compliance + with the policy. The background operation is created when a policy + edit does one of the following: - Changes the policy definition, or + - Changes auto enforcement from disabled to enabled Additionally, + changes to the policy definition or auto enforcement configuration + may cause an in-progress operation to be restarted. To cancel an + in-progress operation, edit the policy to delete this auto + enforcement configuration. The background operation status is + reported via the ``background_enforcement`` field when reading the + policy. + background_enforcement: + $ref: '#/components/schemas/PolicyBackgroundEnforcement' + description: >- + Status and results of the background auto-enforcement operation for + this policy. Only returned if calling "Get a cluster policy" with + ``POLICY_VIEW_FULL``. Must be a workspace admin for this field to be + populated. created_at_timestamp: type: integer description: >- @@ -10276,10 +12065,9 @@ components: definition: type: string description: >- - Policy definition document expressed in [Databricks Cluster Policy - Definition Language]. [Databricks Cluster Policy Definition - Language]: - https://docs.databricks.com/administration-guide/clusters/policy-definition.html + Policy definition document expressed in `Databricks Cluster Policy + Definition Language + `__. description: type: string description: Additional human-readable description of the cluster policy. @@ -10309,24 +12097,74 @@ components: policy_family_definition_overrides: type: string description: >- - Policy definition JSON document expressed in [Databricks Policy - Definition Language]. The JSON document must be passed as a string - and cannot be embedded in the requests. You can use this to - customize the policy definition inherited from the policy family. - Policy rules specified here are merged into the inherited policy - definition. [Databricks Policy Definition Language]: - https://docs.databricks.com/administration-guide/clusters/policy-definition.html + Policy definition JSON document expressed in `Databricks Policy + Definition Language + `__. + The JSON document must be passed as a string and cannot be embedded + in the requests. You can use this to customize the policy definition + inherited from the policy family. Policy rules specified here are + merged into the inherited policy definition. policy_family_id: type: string description: >- ID of the policy family. The cluster policy's policy definition inherits the policy family's policy definition. Cannot be used with - `definition`. Use `policy_family_definition_overrides` instead to - customize the policy definition. + ``definition``. Use ``policy_family_definition_overrides`` instead + to customize the policy definition. policy_id: type: string description: Canonical unique identifier for the Cluster Policy. description: Describes a Cluster Policy entity. + PolicyAutoEnforcementConfig: + type: object + properties: + enforce_mode: + $ref: >- + #/components/schemas/PolicyAutoEnforcementConfigPolicyAutoEnforcementEnforceMode + description: >- + For running clusters, whether to defer enforcement until the cluster + terminates or to restart it immediately. + description: Configures how auto enforcement should be executed for a cluster policy. + PolicyBackgroundEnforcement: + type: object + properties: + deferred_cluster_enforce_count: + type: integer + description: >- + Number of clusters that are scheduled to enforce on next termination + or restart. + end_time: + type: string + format: date-time + description: >- + The time the enforcement operation finished. Only set once the + operation reaches a COMPLETED or ABORTED state. + failed_cluster_enforce_count: + type: integer + description: Number of clusters that could not be enforced due to an error. + failed_job_enforce_count: + type: integer + description: Number of jobs that could not be enforced due to an error. + initiate_time: + type: string + format: date-time + description: The time the enforcement operation was initiated. + initiator_user: + type: string + description: >- + The user who edited the policy to initiate the enforcement + operation. + status: + $ref: >- + #/components/schemas/PolicyBackgroundEnforcementPolicyBackgroundEnforcementStatus + description: Whether the enforcement operation is still in-progress or completed. + success_cluster_enforce_count: + type: integer + description: Number of clusters that were successfully enforced. + success_job_enforce_count: + type: integer + description: Number of jobs that were successfully enforced. + description: Status and results of the background enforcement operation for a policy. PolicyFamily: type: object properties: @@ -10365,6 +12203,9 @@ components: default CRAN repo is used. required: - package + RefreshDefaultBaseEnvironmentsResponse: + type: object + properties: {} RemoveResponse: type: object properties: {} @@ -10385,10 +12226,11 @@ components: type: string description: >- The image data in one of the following formats: 1. A Data URL with - base64-encoded image data: `data:image/{type};base64,{base64-data}`. - Example: `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA...` 2. - A FileStore file path for large images: `/plots/{filename}.png`. - Example: `/plots/b6a7ad70-fb2c-4353-8aed-3f1e015174a4.png` + base64-encoded image data: + ``data:image/{type};base64,{base64-data}``. Example: + ``data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA...`` 2. A + FileStore file path for large images: ``/plots/{filename}.png``. + Example: ``/plots/b6a7ad70-fb2c-4353-8aed-3f1e015174a4.png`` fileNames: type: array items: @@ -10423,7 +12265,7 @@ components: destination: type: string description: >- - S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs + S3 destination, e.g. ``s3://my-bucket/some-prefix`` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. @@ -10431,41 +12273,41 @@ components: type: string description: >- (Optional) Set canned access control list for the logs, e.g. - `bucket-owner-full-control`. If `canned_cal` is set, please make - sure the cluster iam role has `s3:PutObjectAcl` permission on the + ``bucket-owner-full-control``. If ``canned_cal`` is set, please make + sure the cluster iam role has ``s3:PutObjectAcl`` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you - may want to set `bucket-owner-full-control` to make bucket owner + may want to set ``bucket-owner-full-control`` to make bucket owner able to read the logs. enable_encryption: type: boolean description: >- - (Optional) Flag to enable server side encryption, `false` by + (Optional) Flag to enable server side encryption, ``false`` by default. encryption_type: type: string description: >- - (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. - It will be used only when encryption is enabled and the default type - is `sse-s3`. + (Optional) The encryption type, it could be ``sse-s3`` or + ``sse-kms``. It will be used only when encryption is enabled and the + default type is ``sse-s3``. endpoint: type: string description: >- - S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either + S3 endpoint, e.g. ``https://s3-us-west-2.amazonaws.com``. Either region or endpoint needs to be set. If both are set, endpoint will be used. kms_key: type: string description: >- (Optional) Kms key which will be used if encryption is enabled and - encryption type is set to `sse-kms`. + encryption type is set to ``sse-kms``. region: type: string description: >- - S3 region, e.g. `us-west-2`. Either region or endpoint needs to be + S3 region, e.g. ``us-west-2``. Either region or endpoint needs to be set. If both are set, endpoint will be used. required: - destination @@ -10573,9 +12415,9 @@ components: destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the - destination every `5 mins`. The destination of driver logs is - `$destination/$clusterId/driver`, while the destination of executor - logs is `$destination/$clusterId/executor`. + destination every ``5 mins``. The destination of driver logs is + ``$destination/$clusterId/driver``, while the destination of + executor logs is ``$destination/$clusterId/executor``. cluster_name: type: string description: >- @@ -10588,11 +12430,14 @@ components: description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these - tags in addition to `default_tags`. Notes: - Currently, Databricks + tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags data_security_mode: $ref: '#/components/schemas/DataSecurityMode' + dependency_mode: + $ref: '#/components/schemas/DependencyMode' + description: Controls dependency configuration for the cluster. docker_image: $ref: '#/components/schemas/DockerImage' description: Custom docker image BYOC @@ -10610,7 +12455,7 @@ components: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as - `node_type_id` defined above. This field, along with node_type_id, + ``node_type_id`` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. @@ -10636,17 +12481,17 @@ components: description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially - in the order provided. If `cluster_log_conf` is specified, init - script logs are sent to `//init_scripts`. + in the order provided. If ``cluster_log_conf`` is specified, init + script logs are sent to ``//init_scripts``. instance_pool_id: type: string description: The optional ID of the instance pool to which the cluster belongs. is_single_node: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. When set - to true, Databricks will automatically set single node related - `custom_tags`, `spark_conf`, and `num_workers` + This field can only be used when ``kind = CLASSIC_PREVIEW``. When + set to true, Databricks will automatically set single node related + ``custom_tags``, ``spark_conf``, and ``num_workers`` kind: $ref: '#/components/schemas/Kind' node_type_id: @@ -10656,19 +12501,21 @@ components: to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved - by using the :method:clusters/listNodeTypes API call. + by using the `clusters/listNodeTypes + `__ + API call. num_workers: type: integer description: >- Number of worker nodes that this cluster should have. A cluster has - one Spark Driver and `num_workers` Executors for a total of - `num_workers` + 1 Spark nodes. Note: When reading the properties of - a cluster, this field reflects the desired number of workers rather - than the actual current number of workers. For instance, if a + one Spark Driver and ``num_workers`` Executors for a total of + ``num_workers`` + 1 Spark nodes. Note: When reading the properties + of a cluster, this field reflects the desired number of workers + rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase from 5 to 10 - as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually increase from 5 to + 10 as the new nodes are provisioned. policy_id: type: string description: >- @@ -10683,42 +12530,44 @@ components: $ref: '#/components/schemas/RuntimeEngine' description: >- Determines the cluster's runtime engine, either standard or Photon. - This field is not compatible with legacy `spark_version` values that - contain `-photon-`. Remove `-photon-` from the `spark_version` and - set `runtime_engine` to `PHOTON`. If left unspecified, the runtime - engine defaults to standard unless the spark_version contains - -photon-, in which case Photon will be used. + This field is not compatible with legacy ``spark_version`` values + that contain ``-photon-``. Remove ``-photon-`` from the + ``spark_version`` and set ``runtime_engine`` to ``PHOTON``. If left + unspecified, the runtime engine defaults to standard unless the + spark_version contains -photon-, in which case Photon will be used. single_user_name: type: string - description: Single user name if data_security_mode is `SINGLE_USER` + description: Single user name if data_security_mode is ``SINGLE_USER`` spark_conf: type: object description: >- An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via - `spark.driver.extraJavaOptions` and - `spark.executor.extraJavaOptions` respectively. + ``spark.driver.extraJavaOptions`` and + ``spark.executor.extraJavaOptions`` respectively. spark_env_vars: type: object description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the - form (X,Y) will be exported as is (i.e., `export X='Y'`) while + form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional - set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This + set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` + ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": + "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}`` spark_version: type: string description: >- - The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of - available Spark versions can be retrieved by using the - :method:clusters/sparkVersions API call. + The Spark version of the cluster, e.g. ``3.3.x-scala2.11``. A list + of available Spark versions can be retrieved by using the + `clusters/sparkVersions + `__ + API call. ssh_public_keys: type: array items: @@ -10726,7 +12575,7 @@ components: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login - with the user name `ubuntu` on port `2200`. Up to 10 keys can be + with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can be specified. total_initial_remote_disk_size: type: integer @@ -10737,10 +12586,10 @@ components: use_ml_runtime: type: boolean description: >- - This field can only be used when `kind = CLASSIC_PREVIEW`. - `effective_spark_version` is determined by `spark_version` (DBR - release), this field `use_ml_runtime`, and whether `node_type_id` is - gpu node or not. + This field can only be used when ``kind = CLASSIC_PREVIEW``. + ``effective_spark_version`` is determined by ``spark_version`` (DBR + release), this field ``use_ml_runtime``, and whether + ``node_type_id`` is gpu node or not. worker_node_type_flexibility: $ref: '#/components/schemas/NodeTypeFlexibility' description: Flexible node type configuration for worker nodes. @@ -10759,8 +12608,8 @@ components: type: string description: >- UC Volumes destination, e.g. - `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or - `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` + ``/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` or + ``dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh`` required: - destination description: A storage location back by UC Volumes. @@ -10782,7 +12631,7 @@ components: type: string description: >- wsfs destination, e.g. - `workspace:/cluster-init-scripts/setup-datadog.sh` + ``workspace:/cluster-init-scripts/setup-datadog.sh`` required: - destination description: A storage location in Workspace Filesystem (WSFS) @@ -10794,11 +12643,13 @@ components: - SPOT_WITH_FALLBACK description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. + ``first_on_demand`` ones. - Note: If `first_on_demand` is zero, this availability type will be used - for the entire cluster. + Note: If ``first_on_demand`` is zero, this availability type will be + used for the entire + + cluster. AzureAvailability: type: string x-enum: @@ -10807,10 +12658,36 @@ components: - SPOT_WITH_FALLBACK_AZURE description: >- Availability type used for all subsequent nodes past the - `first_on_demand` ones. Note: If + ``first_on_demand`` ones. Note: If - `first_on_demand` is zero, this availability type will be used for the + ``first_on_demand`` is zero, this availability type will be used for the entire cluster. + BaseEnvironmentType: + type: string + x-enum: + - CPU + - GPU + description: >- + If changed, also update + estore/namespaces/defaultbaseenvironments/latest.proto + CheckId: + type: string + x-enum: + - CHECK_ID_NETWORK_BIFROST + - CHECK_ID_NETWORK_CONTROL_PLANE + - CHECK_ID_NETWORK_CP_MTLS + - CHECK_ID_NETWORK_DNS_SERVER + - CHECK_ID_NETWORK_INTERNET + - CHECK_ID_NETWORK_LOG_ARTIFACT_BUCKET + - CHECK_ID_NETWORK_METADATA_ENDPOINT + - CHECK_ID_NETWORK_NIC + - CHECK_ID_NETWORK_SCC_TUNNEL + - CHECK_ID_NETWORK_STORAGE_BUCKET + description: >- + Identifies the specific diagnostic check the user will see. Each value + maps to one row in the + + cluster's Diagnostics tab. CloudProviderNodeStatus: type: string x-enum: @@ -10927,12 +12804,290 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. - ContextStatus: + ConfidentialComputeType: + type: string + x-enum: + - CONFIDENTIAL_COMPUTE_TYPE_NONE + - SEV_SNP + description: >- + Confidential computing technology for GCP instances. Aligns with + gcloud's + + --confidential-compute-type flag and the REST API's + + confidentialInstanceConfig.confidentialInstanceType field. See: + + https://cloud.google.com/confidential-computing/confidential-vm/docs/create-a-confidential-vm-instance + ContextStatus: + type: string + x-enum: + - Error + - Pending + - Running + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + DataPlaneEventDetailsEventType: + type: string + x-enum: + - NODE_BLACKLISTED + - NODE_EXCLUDED_DECOMMISSIONED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + DataSecurityMode: + type: string + x-enum: + - DATA_SECURITY_MODE_AUTO + - DATA_SECURITY_MODE_DEDICATED + - DATA_SECURITY_MODE_STANDARD + - LEGACY_PASSTHROUGH + - LEGACY_SINGLE_USER + - LEGACY_SINGLE_USER_STANDARD + - LEGACY_TABLE_ACL + - NONE + - SINGLE_USER + - USER_ISOLATION + description: >- + Data security mode decides what data governance model to use when + accessing data from a cluster. + + + - ``DATA_SECURITY_MODE_AUTO``: Databricks will choose the most + appropriate access mode depending + on your compute configuration. + - ``DATA_SECURITY_MODE_STANDARD``: A secure cluster that can be shared + by multiple users. + Cluster users are fully isolated so that they cannot see each other’s data and credentials. + Most data governance features are supported in this mode. But programming languages and + cluster features might be limited. + - ``DATA_SECURITY_MODE_DEDICATED``: A secure cluster that can only be + exclusively used by a + single user specified in ``single_user_name``. Most programming languages, cluster features + and data governance features are available in this mode. + + The following modes are legacy aliases for the above modes: + + + - ``USER_ISOLATION``: Legacy alias for ``DATA_SECURITY_MODE_STANDARD``. + + - ``SINGLE_USER``: Legacy alias for ``DATA_SECURITY_MODE_DEDICATED``. + + + The following modes are deprecated starting with Databricks Runtime 15.0 + and will be removed for + + future Databricks Runtime versions: + + + - ``LEGACY_TABLE_ACL``: This mode is for users migrating from legacy + Table ACL clusters. + + - ``LEGACY_PASSTHROUGH``: This mode is for users migrating from legacy + Passthrough on high + concurrency clusters. + - ``LEGACY_SINGLE_USER``: This mode is for users migrating from legacy + Passthrough on standard + clusters. + - ``LEGACY_SINGLE_USER_STANDARD``: This mode provides a way that doesn’t + have UC nor + passthrough enabled. + DefaultBaseEnvironmentCacheStatus: + type: string + x-enum: + - CREATED + - EXPIRED + - FAILED + - INVALID + - PENDING + - REFRESHING + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + DependencyMode: type: string x-enum: - - Error - - Pending - - Running + - DEPENDENCY_MODE_AUTO + - DEPENDENCY_MODE_CLUSTER_LIBRARIES + - DEPENDENCY_MODE_ENVIRONMENTS + description: >- + Controls dependency configuration for the cluster. + + + - ``DEPENDENCY_MODE_AUTO``: Databricks will choose the most appropriate + dependency mode based on + your compute configuration. + - ``DEPENDENCY_MODE_ENVIRONMENTS``: Enables a unified dependency + management experience across + classic and serverless, resulting in increased stability and performance. Supported only on + DBR 19+ in Standard access mode. + - ``DEPENDENCY_MODE_CLUSTER_LIBRARIES``: Legacy mode: dependencies come + from cluster libraries + and init scripts. + DiagnosticsErrorReason: + type: string + x-enum: + - ERROR_REASON_CERT_SAN_MISMATCH + - ERROR_REASON_DNS_RESOLVE_FAIL + - ERROR_REASON_HTTP_3XX + - ERROR_REASON_HTTP_4XX + - ERROR_REASON_HTTP_5XX + - ERROR_REASON_HTTP_TIMEOUT + - ERROR_REASON_NOT_RUN + - ERROR_REASON_TCP_REFUSED + - ERROR_REASON_TCP_TIMEOUT + - ERROR_REASON_TLS_HANDSHAKE_FAIL + description: >- + Diagnostic error reason for a failed check, identifying the first + failing network layer (DNS -> + + TCP -> TLS -> HTTP). Set on Check.reason only when check_status = + FAILED. + DiagnosticsStatus: + type: string + x-enum: + - DIAGNOSTICS_STATUS_FAILED + - DIAGNOSTICS_STATUS_NOT_RUN + - DIAGNOSTICS_STATUS_PASSED + description: >- + Overall status of a cluster diagnostics run, and of each individual + check within it. + DiskTypeAzureDiskVolumeType: + type: string + x-enum: + - PREMIUM_LRS + - STANDARD_LRS + description: >- + All Azure Disk types that Databricks supports. See + + https://docs.microsoft.com/en-us/azure/storage/storage-about-disks-and-vhds-linux#types-of-disks + DiskTypeEbsVolumeType: + type: string + x-enum: + - GENERAL_PURPOSE_SSD + - THROUGHPUT_OPTIMIZED_HDD + description: >- + All EBS volume types that Databricks supports. See + https://aws.amazon.com/ebs/details/ for + + details. + EbsVolumeType: + type: string + x-enum: + - GENERAL_PURPOSE_SSD + - THROUGHPUT_OPTIMIZED_HDD + description: >- + All EBS volume types that Databricks supports. See + https://aws.amazon.com/ebs/details/ for + + details. + EnforcePolicyComplianceForClusterEnforceMode: + type: string + x-enum: + - ENFORCE_IMMEDIATELY + - WAIT_FOR_TERMINATION description: |- Create a collection of name/value pairs. @@ -10970,11 +13125,12 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. - DataPlaneEventDetailsEventType: + EnforcePolicyComplianceForClusterResponseEnforceResult: type: string x-enum: - - NODE_BLACKLISTED - - NODE_EXCLUDED_DECOMMISSIONED + - APPLIED + - DEFERRED + - NO_CHANGES description: |- Create a collection of name/value pairs. @@ -11012,108 +13168,6 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. - DataSecurityMode: - type: string - x-enum: - - DATA_SECURITY_MODE_AUTO - - DATA_SECURITY_MODE_DEDICATED - - DATA_SECURITY_MODE_STANDARD - - LEGACY_PASSTHROUGH - - LEGACY_SINGLE_USER - - LEGACY_SINGLE_USER_STANDARD - - LEGACY_TABLE_ACL - - NONE - - SINGLE_USER - - USER_ISOLATION - description: >- - Data security mode decides what data governance model to use when - accessing data from a cluster. - - - The following modes can only be used when `kind = CLASSIC_PREVIEW`. * - `DATA_SECURITY_MODE_AUTO`: - - Databricks will choose the most appropriate access mode depending on - your compute configuration. - - * `DATA_SECURITY_MODE_STANDARD`: Alias for `USER_ISOLATION`. * - `DATA_SECURITY_MODE_DEDICATED`: - - Alias for `SINGLE_USER`. - - - The following modes can be used regardless of `kind`. * `NONE`: No - security isolation for - - multiple users sharing the cluster. Data governance features are not - available in this mode. * - - `SINGLE_USER`: A secure cluster that can only be exclusively used by a - single user specified in - - `single_user_name`. Most programming languages, cluster features and - data governance features - - are available in this mode. * `USER_ISOLATION`: A secure cluster that - can be shared by multiple - - users. Cluster users are fully isolated so that they cannot see each - other's data and - - credentials. Most data governance features are supported in this mode. - But programming languages - - and cluster features might be limited. - - - The following modes are deprecated starting with Databricks Runtime 15.0 - and will be removed for - - future Databricks Runtime versions: - - - * `LEGACY_TABLE_ACL`: This mode is for users migrating from legacy Table - ACL clusters. * - - `LEGACY_PASSTHROUGH`: This mode is for users migrating from legacy - Passthrough on high - - concurrency clusters. * `LEGACY_SINGLE_USER`: This mode is for users - migrating from legacy - - Passthrough on standard clusters. * `LEGACY_SINGLE_USER_STANDARD`: This - mode provides a way that - - doesn’t have UC nor passthrough enabled. - DiskTypeAzureDiskVolumeType: - type: string - x-enum: - - PREMIUM_LRS - - STANDARD_LRS - description: >- - All Azure Disk types that Databricks supports. See - - https://docs.microsoft.com/en-us/azure/storage/storage-about-disks-and-vhds-linux#types-of-disks - DiskTypeEbsVolumeType: - type: string - x-enum: - - GENERAL_PURPOSE_SSD - - THROUGHPUT_OPTIMIZED_HDD - description: >- - All EBS volume types that Databricks supports. See - https://aws.amazon.com/ebs/details/ for - - details. - EbsVolumeType: - type: string - x-enum: - - GENERAL_PURPOSE_SSD - - THROUGHPUT_OPTIMIZED_HDD - description: >- - All EBS volume types that Databricks supports. See - https://aws.amazon.com/ebs/details/ for - - details. EventDetailsCause: type: string x-enum: @@ -11137,6 +13191,8 @@ components: - DBFS_DOWN - DECOMMISSION_ENDED - DECOMMISSION_STARTED + - DEFERRED_POLICY_ENFORCEMENT_FAILED + - DEFERRED_POLICY_ENFORCEMENT_SCHEDULED - DID_NOT_EXPAND_DISK - DRIVER_HEALTHY - DRIVER_NOT_RESPONDING @@ -11255,6 +13311,8 @@ components: type: string x-enum: - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 - GPU_8xH100 description: >- HardwareAcceleratorType: The type of hardware accelerator to use for @@ -11308,10 +13366,13 @@ components: follows: - - ``ACTIVE`` -> ``STOPPED`` - ``ACTIVE`` -> ``DELETED`` - ``STOPPED`` -> - ``ACTIVE`` - + - ``ACTIVE`` -> ``STOPPED`` + + - ``ACTIVE`` -> ``DELETED`` - ``STOPPED`` -> ``DELETED`` + - ``STOPPED`` -> ``ACTIVE`` + + - ``STOPPED`` -> ``DELETED`` Kind: type: string x-enum: @@ -11320,30 +13381,25 @@ components: The kind of compute described by this compute specification. - Depending on `kind`, different validations and default values will be + Depending on ``kind``, different validations and default values will be applied. - Clusters with `kind = CLASSIC_PREVIEW` support the following fields, + Clusters with ``kind = CLASSIC_PREVIEW`` support the following fields, whereas clusters with no - specified `kind` do not. * - [is_single_node](/api/workspace/clusters/create#is_single_node) * - - [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) * + specified ``kind`` do not. - [data_security_mode](/api/workspace/clusters/create#data_security_mode) - set to - `DATA_SECURITY_MODE_AUTO`, `DATA_SECURITY_MODE_DEDICATED`, or - `DATA_SECURITY_MODE_STANDARD` + - [is_single_node](/api/workspace/clusters/create#is_single_node) + - [use_ml_runtime](/api/workspace/clusters/create#use_ml_runtime) - By using the [simple form], your clusters are automatically using `kind - = CLASSIC_PREVIEW`. + By using the `simple form + `__, your - [simple form]: https://docs.databricks.com/compute/simple-form.html + clusters are automatically using ``kind = CLASSIC_PREVIEW``. Language: type: string x-enum: @@ -11568,6 +13624,71 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + PendingEnforcementEnforcementStatus: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + PolicyAutoEnforcementConfigPolicyAutoEnforcementEnforceMode: + type: string + x-enum: + - ENFORCE_IMMEDIATELY + - WAIT_FOR_TERMINATION + description: >- + Behavior for running clusters when auto-enforcement is applied. Mirrors + ``enforce_mode`` on the + + enforce compliance API. + PolicyBackgroundEnforcementPolicyBackgroundEnforcementStatus: + type: string + x-enum: + - ABORTED + - COMPLETED + - IN_PROGRESS + description: The status of a background policy enforcement operation. + PolicyView: + type: string + x-enum: + - POLICY_VIEW_BASIC + - POLICY_VIEW_FULL + description: Controls which fields are returned when reading a policy. ResultType: type: string x-enum: @@ -11672,16 +13793,25 @@ components: follows: - - `PENDING` -> `RUNNING` - `PENDING` -> `TERMINATING` - `RUNNING` -> - `RESIZING` - `RUNNING` -> + - ``PENDING`` -> ``RUNNING`` + + - ``PENDING`` -> ``TERMINATING`` + + - ``RUNNING`` -> ``RESIZING`` + + - ``RUNNING`` -> ``RESTARTING`` + + - ``RUNNING`` -> ``TERMINATING`` - `RESTARTING` - `RUNNING` -> `TERMINATING` - `RESTARTING` -> `RUNNING` - - `RESTARTING` -> + - ``RESTARTING`` -> ``RUNNING`` - `TERMINATING` - `RESIZING` -> `RUNNING` - `RESIZING` -> `TERMINATING` - - `TERMINATING` -> + - ``RESTARTING`` -> ``TERMINATING`` - `TERMINATED` + - ``RESIZING`` -> ``RUNNING`` + + - ``RESIZING`` -> ``TERMINATING`` + + - ``TERMINATING`` -> ``TERMINATED`` TerminationReasonCode: type: string x-enum: @@ -11723,6 +13853,7 @@ components: - BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG - BUDGET_POLICY_LIMIT_ENFORCEMENT_ACTIVATED - BUDGET_POLICY_RESOLUTION_FAILURE + - CERT_ROTATION - CLOUD_ACCOUNT_POD_QUOTA_EXCEEDED - CLOUD_ACCOUNT_SETUP_FAILURE - CLOUD_OPERATION_CANCELLED @@ -11741,6 +13872,7 @@ components: - CONTROL_PLANE_CONNECTION_FAILURE_DUE_TO_MISCONFIG - CONTROL_PLANE_REQUEST_FAILURE - CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG + - COST_CONTROL_ENTITLEMENT_DENIED - DATABASE_CONNECTION_FAILURE - DATA_ACCESS_CONFIG_CHANGED - DBFS_COMPONENT_UNHEALTHY @@ -11751,6 +13883,7 @@ components: - DOCKER_IMAGE_PULL_FAILURE - DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION - DOCKER_INVALID_OS_EXCEPTION + - DRIVER_DNS_RESOLUTION_FAILURE - DRIVER_EVICTION - DRIVER_LAUNCH_TIMEOUT - DRIVER_NODE_UNREACHABLE @@ -11827,6 +13960,8 @@ components: - NETWORK_CHECK_STORAGE_FAILURE_DUE_TO_MISCONFIG - NETWORK_CONFIGURATION_FAILURE - NFS_MOUNT_FAILURE + - NO_ACTIVATED_K8S + - NO_ACTIVATED_K8S_TESTING_TAG - NO_MATCHED_K8S - NO_MATCHED_K8S_TESTING_TAG - NPIP_TUNNEL_SETUP_FAILURE @@ -11837,9 +13972,11 @@ components: - REQUEST_REJECTED - REQUEST_THROTTLED - RESOURCE_USAGE_BLOCKED + - SECRET_CREATION_ACCESS_DENIED - SECRET_CREATION_FAILURE - SECRET_PERMISSION_DENIED - SECRET_RESOLUTION_ERROR + - SECURITY_AGENTS_FAILED_INITIAL_VERIFICATION - SECURITY_DAEMON_REGISTRATION_EXCEPTION - SELF_BOOTSTRAP_FAILURE - SERVERLESS_LONG_RUNNING_TERMINATED @@ -11871,6 +14008,7 @@ components: - WORKER_SETUP_FAILURE - WORKSPACE_CANCELLED_ERROR - WORKSPACE_CONFIGURATION_ERROR + - WORKSPACE_DELEGATION_KEY_MISCONFIGURED - WORKSPACE_UPDATE description: The status code indicating why the cluster was terminated TerminationReasonType: @@ -12058,6 +14196,12 @@ components: response: mediaType: application/json openAPIDocKey: '200' + clusters_get_diagnostic: + operation: + $ref: '#/paths/~1api~12.1~1{name}~1diagnostics/get' + response: + mediaType: application/json + openAPIDocKey: '200' list: operation: $ref: '#/paths/~1api~12.1~1clusters~1list/get' @@ -12130,6 +14274,8 @@ components: openAPIDocKey: '200' sqlVerbs: select: + - $ref: >- + #/components/x-stackQL-resources/clusters/methods/clusters_get_diagnostic - $ref: '#/components/x-stackQL-resources/clusters/methods/get' - $ref: '#/components/x-stackQL-resources/clusters/methods/list' insert: @@ -12607,11 +14753,109 @@ components: update: [] delete: [] replace: [] + default_base_environments: + id: databricks_workspace.compute.default_base_environments + name: default_base_environments + title: Default Base Environments + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1default-base-environments/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1default-base-environments/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.default_base_environments + delete: + operation: + $ref: '#/paths/~1api~12.0~1default-base-environments~1{id}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1default-base-environments~1{id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + refresh: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1default-base-environments~1refresh/post' + response: + mediaType: application/json + openAPIDocKey: '200' + set_default: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1default-base-environments:setDefault/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/default_base_environments/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/default_base_environments/methods/create + update: + - $ref: >- + #/components/x-stackQL-resources/default_base_environments/methods/update + delete: + - $ref: >- + #/components/x-stackQL-resources/default_base_environments/methods/delete + replace: [] + default_base_environment: + id: databricks_workspace.compute.default_base_environment + name: default_base_environment + title: Default Base Environment + methods: + get: + operation: + $ref: >- + #/paths/~1api~12.0~1default-base-environments:getDefaultBaseEnvironment/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/default_base_environment/methods/get + insert: [] + update: [] + delete: [] + replace: [] policy_compliance_for_clusters: id: databricks_workspace.compute.policy_compliance_for_clusters name: policy_compliance_for_clusters title: Policy Compliance For Clusters methods: + cancel_pending_enforcement: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1policies~1clusters:cancelPendingClusterEnforcement/post + response: + mediaType: application/json + openAPIDocKey: '200' enforce: config: requestBodyTranslate: diff --git a/providers/src/databricks_workspace/v00.00.00000/services/dashboards.yaml b/providers/src/databricks_workspace/v00.00.00000/services/dashboards.yaml index d0562138..7593e887 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/dashboards.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/dashboards.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks dashboards service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.dashboards servers: - url: https://{deployment_name}.cloud.databricks.com @@ -37,6 +37,8 @@ paths: The ID associated with the conversation. :param content: str User message content. + :param enable_visualization: bool (optional) + Enable visualization generation. :returns: Long-running operation waiter for :class:`GenieMessage`. @@ -65,6 +67,9 @@ paths: content: type: string description: User message content. + enable_visualization: + type: boolean + description: Enable visualization generation. required: - content responses: @@ -151,6 +156,146 @@ paths: message: type: string x-stackql-sdk-source: GenieAPI + /api/2.0/genie/spaces/{space_id}/conversations/{conversation_id}/messages/{message_id}/comments: + post: + operationId: genie_create_message_comment + summary: Create a comment on a conversation message. + tags: + - dashboards + - genie + description: |- + Create a comment on a conversation message. + + :param space_id: str + The ID associated with the Genie space. + :param conversation_id: str + The ID associated with the conversation. + :param message_id: str + The ID associated with the message. + :param content: str + Comment text content. + + :returns: :class:`GenieMessageComment` + parameters: + - name: space_id + in: path + required: true + schema: + type: string + description: The ID associated with the Genie space. + - name: conversation_id + in: path + required: true + schema: + type: string + description: The ID associated with the conversation. + - name: message_id + in: path + required: true + schema: + type: string + description: The ID associated with the message. + requestBody: + content: + application/json: + schema: + type: object + properties: + content: + type: string + description: Comment text content. + required: + - content + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GenieMessageComment' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GenieAPI + get: + operationId: genie_list_message_comments + summary: List comments on a specific conversation message. + tags: + - dashboards + - genie + description: |- + List comments on a specific conversation message. + + :param space_id: str + The ID associated with the Genie space. + :param conversation_id: str + The ID associated with the conversation. + :param message_id: str + The ID associated with the message. + :param page_size: int (optional) + Maximum number of comments to return per page. + :param page_token: str (optional) + Pagination token for getting the next page of results. + + :returns: :class:`GenieListMessageCommentsResponse` + parameters: + - name: space_id + in: path + required: true + schema: + type: string + description: The ID associated with the Genie space. + - name: conversation_id + in: path + required: true + schema: + type: string + description: The ID associated with the conversation. + - name: message_id + in: path + required: true + schema: + type: string + description: The ID associated with the message. + - name: page_size + in: query + required: false + schema: + type: integer + description: Maximum number of comments to return per page. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token for getting the next page of results. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GenieListMessageCommentsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GenieAPI /api/2.0/genie/spaces: post: operationId: genie_create_space @@ -166,7 +311,7 @@ paths: :param serialized_space: str The contents of the Genie Space in serialized string form. Use the [Get Genie Space](:method:genie/getspace) API to retrieve an example response, which includes the - `serialized_space` field. This field provides the structure of the JSON string that represents the + ``serialized_space`` field. This field provides the structure of the JSON string that represents the space's layout and components. :param description: str (optional) Optional description @@ -191,9 +336,9 @@ paths: The contents of the Genie Space in serialized string form. Use the [Get Genie Space](:method:genie/getspace) API to retrieve an example response, which includes the - `serialized_space` field. This field provides the structure - of the JSON string that represents the space's layout and - components. + ``serialized_space`` field. This field provides the + structure of the JSON string that represents the space's + layout and components. description: type: string description: Optional description @@ -425,6 +570,60 @@ paths: message: type: string x-stackql-sdk-source: GenieAPI + /api/2.0/genie/{name}/download-visualization: + get: + operationId: genie_download_message_attachment_visualization + summary: >- + Download a rendered image of a message visualization attachment. The + response body is the raw PNG + tags: + - dashboards + - genie + description: >- + Download a rendered image of a message visualization attachment. The + response body is the raw PNG + + image, not a JSON payload. This is only available if the attachment is a + visualization and the message + + status is ``COMPLETED``. This endpoint is not supported for Private Link + workspaces. + + + :param name: str + The resource name of the attachment to render, in the format + ``spaces/{space_id}/conversations/{conversation_id}/messages/{message_id}/attachments/{attachment_id}``. + + :returns: :class:`DownloadMessageAttachmentVisualizationResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The resource name of the attachment to render, in the format + ``spaces/{space_id}/conversations/{conversation_id}/messages/{message_id}/attachments/{attachment_id}``. + responses: + '200': + description: Success + content: + application/octet-stream: + schema: + type: string + format: binary + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GenieAPI /api/2.0/genie/spaces/{space_id}/conversations/{conversation_id}/messages/{message_id}/attachments/{attachment_id}/execute-query: post: operationId: genie_execute_message_attachment_query @@ -561,35 +760,32 @@ paths: post: operationId: genie_generate_download_full_query_result summary: >- - Initiates a new SQL execution and returns a `download_id` and - `download_id_signature` that you can use + Initiates a new SQL execution and returns a ``download_id`` and + ``download_id_signature`` that you can tags: - dashboards - genie description: >- - Initiates a new SQL execution and returns a `download_id` and - `download_id_signature` that you can use + Initiates a new SQL execution and returns a ``download_id`` and + ``download_id_signature`` that you can - to track the progress of the download. The query result is stored in an - external link and can be + use to track the progress of the download. The query result is stored in + an external link and can be retrieved using the [Get Download Full Query Result](:method:genie/getdownloadfullqueryresult) API. - Both `download_id` and `download_id_signature` must be provided when + Both ``download_id`` and ``download_id_signature`` must be provided when calling the Get endpoint. - ---- - - - ### **Warning: Databricks strongly recommends that you protect the URLs - that are returned by the + **Warning: Databricks strongly recommends that you protect the URLs that + are returned by the - `EXTERNAL_LINKS` disposition.** + ``EXTERNAL_LINKS`` disposition.** - When you use the `EXTERNAL_LINKS` disposition, a short-lived, URL is + When you use the ``EXTERNAL_LINKS`` disposition, a short-lived, URL is generated, which can be used to download the results directly from . As a short-lived is embedded in @@ -599,7 +795,7 @@ paths: Because URLs are already generated with embedded temporary s, you must - not set an `Authorization` + not set an ``Authorization`` header in the download requests. @@ -608,9 +804,6 @@ paths: more details. - ---- - - :param space_id: str Genie space ID :param conversation_id: str @@ -971,28 +1164,26 @@ paths: After [Generating a Full Query Result Download](:method:genie/generatedownloadfullqueryresult) and - successfully receiving a `download_id` and `download_id_signature`, use - this API to poll the download - - progress. Both `download_id` and `download_id_signature` are required to - call this endpoint. When the - - download is complete, the API returns the result in the `EXTERNAL_LINKS` - disposition, containing one + successfully receiving a ``download_id`` and ``download_id_signature``, + use this API to poll the - or more external links to the query result files. + download progress. Both ``download_id`` and ``download_id_signature`` + are required to call this + endpoint. When the download is complete, the API returns the result in + the ``EXTERNAL_LINKS`` - ---- + disposition, containing one or more external links to the query result + files. - ### **Warning: Databricks strongly recommends that you protect the URLs - that are returned by the + **Warning: Databricks strongly recommends that you protect the URLs that + are returned by the - `EXTERNAL_LINKS` disposition.** + ``EXTERNAL_LINKS`` disposition.** - When you use the `EXTERNAL_LINKS` disposition, a short-lived, URL is + When you use the ``EXTERNAL_LINKS`` disposition, a short-lived, URL is generated, which can be used to download the results directly from . As a short-lived is embedded in @@ -1002,7 +1193,7 @@ paths: Because URLs are already generated with embedded temporary s, you must - not set an `Authorization` + not set an ``Authorization`` header in the download requests. @@ -1011,9 +1202,6 @@ paths: more details. - ---- - - :param space_id: str Genie space ID :param conversation_id: str @@ -1102,8 +1290,8 @@ paths: Get the result of SQL query if the message has a query attachment. This is only available if a message - has a query attachment and the message status is `EXECUTING_QUERY` OR - `COMPLETED`. + has a query attachment and the message status is ``EXECUTING_QUERY`` OR + ``COMPLETED``. :param space_id: str @@ -1389,10 +1577,15 @@ paths: Genie space ID :param description: str (optional) Optional description + :param etag: str (optional) + ETag returned by a previous GET or UPDATE. When set, the update will fail if the space has been + modified since. Omit to apply the update unconditionally. + :param parent_path: str (optional) + Parent workspace folder path to move this Genie space under. :param serialized_space: str (optional) The contents of the Genie Space in serialized string form (full replacement). Use the [Get Genie Space](:method:genie/getspace) API to retrieve an example response, which includes the - `serialized_space` field. This field provides the structure of the JSON string that represents the + ``serialized_space`` field. This field provides the structure of the JSON string that represents the space's layout and components. :param title: str (optional) Optional title override @@ -1416,14 +1609,23 @@ paths: description: type: string description: Optional description + etag: + type: string + description: >- + ETag returned by a previous GET or UPDATE. When set, the + update will fail if the space has been modified since. Omit + to apply the update unconditionally. + parent_path: + type: string + description: Parent workspace folder path to move this Genie space under. serialized_space: type: string description: >- The contents of the Genie Space in serialized string form (full replacement). Use the [Get Genie Space](:method:genie/getspace) API to retrieve an example - response, which includes the `serialized_space` field. This - field provides the structure of the JSON string that + response, which includes the ``serialized_space`` field. + This field provides the structure of the JSON string that represents the space's layout and components. title: type: string @@ -1450,6 +1652,70 @@ paths: message: type: string x-stackql-sdk-source: GenieAPI + /api/2.0/genie/spaces/{space_id}/conversations/{conversation_id}/list-comments: + get: + operationId: genie_list_conversation_comments + summary: List all comments across all messages in a conversation. + tags: + - dashboards + - genie + description: |- + List all comments across all messages in a conversation. + + :param space_id: str + The ID associated with the Genie space. + :param conversation_id: str + The ID associated with the conversation. + :param page_size: int (optional) + Maximum number of comments to return per page. + :param page_token: str (optional) + Pagination token for getting the next page of results. + + :returns: :class:`GenieListConversationCommentsResponse` + parameters: + - name: space_id + in: path + required: true + schema: + type: string + description: The ID associated with the Genie space. + - name: conversation_id + in: path + required: true + schema: + type: string + description: The ID associated with the conversation. + - name: page_size + in: query + required: false + schema: + type: integer + description: Maximum number of comments to return per page. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token for getting the next page of results. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GenieListConversationCommentsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: GenieAPI /api/2.0/genie/spaces/{space_id}/conversations: get: operationId: genie_list_conversations @@ -1535,6 +1801,8 @@ paths: The ID associated with the message to provide feedback for. :param rating: :class:`GenieFeedbackRating` The rating (POSITIVE, NEGATIVE, or NONE). + :param comment: str (optional) + Optional text feedback that will be stored as a comment. parameters: - name: space_id in: path @@ -1567,6 +1835,9 @@ paths: - NONE - POSITIVE description: The rating (POSITIVE, NEGATIVE, or NONE). + comment: + type: string + description: Optional text feedback that will be stored as a comment. required: - rating responses: @@ -1598,6 +1869,8 @@ paths: The ID associated with the Genie space where you want to start a conversation. :param content: str The text of the message that starts the conversation. + :param enable_visualization: bool (optional) + Enable visualization generation. :returns: Long-running operation waiter for :class:`GenieMessage`. @@ -1620,6 +1893,9 @@ paths: content: type: string description: The text of the message that starts the conversation. + enable_visualization: + type: boolean + description: Enable visualization generation. required: - content responses: @@ -1726,19 +2002,19 @@ paths: description: >- The workspace path of the dashboard asset, including the file name. Exported dashboards always have the file - extension `.lvdash.json`. This field is excluded in List - Dashboards responses. + extension ``.lvdash.json``. This field is excluded in + List Dashboards responses. serialized_dashboard: type: string description: >- The contents of the dashboard in serialized string form. This field is excluded in List Dashboards responses. Use - the [get dashboard API] to retrieve an example response, - which includes the `serialized_dashboard` field. This - field provides the structure of the JSON string that - represents the dashboard's layout and components. [get - dashboard API]: - https://docs.databricks.com/api/workspace/lakeview/get + the `get dashboard API + `__ + to retrieve an example response, which includes the + ``serialized_dashboard`` field. This field provides the + structure of the JSON string that represents the + dashboard's layout and components. update_time: type: string description: >- @@ -1781,13 +2057,13 @@ paths: :param page_size: int (optional) The number of dashboards to return per page. :param page_token: str (optional) - A page token, received from a previous `ListDashboards` call. This token can be used to retrieve the - subsequent page. + A page token, received from a previous ``ListDashboards`` call. This token can be used to retrieve + the subsequent page. :param show_trashed: bool (optional) The flag to include dashboards located in the trash. If unspecified, only active dashboards will be returned. :param view: :class:`DashboardView` (optional) - `DASHBOARD_VIEW_BASIC`only includes summary metadata from the dashboard. + ``DASHBOARD_VIEW_BASIC`` only includes summary metadata from the dashboard. :returns: Iterator over :class:`Dashboard` parameters: @@ -1803,7 +2079,7 @@ paths: schema: type: string description: >- - A page token, received from a previous `ListDashboards` call. This + A page token, received from a previous ``ListDashboards`` call. This token can be used to retrieve the subsequent page. - name: show_trashed in: query @@ -1819,7 +2095,7 @@ paths: schema: $ref: '#/components/schemas/DashboardView' description: >- - `DASHBOARD_VIEW_BASIC`only includes summary metadata from the + ``DASHBOARD_VIEW_BASIC`` only includes summary metadata from the dashboard. responses: '200': @@ -1881,9 +2157,9 @@ paths: type: string description: >- A Java timezone id. The schedule will be resolved - with respect to this timezone. See [Java TimeZone] - for details. [Java TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + with respect to this timezone. See `Java TimeZone + `__ + for details. required: - quartz_cron_expression - timezone_id @@ -1966,7 +2242,7 @@ paths: :param page_size: int (optional) The number of schedules to return per page. :param page_token: str (optional) - A page token, received from a previous `ListSchedules` call. Use this to retrieve the subsequent + A page token, received from a previous ``ListSchedules`` call. Use this to retrieve the subsequent page. :returns: Iterator over :class:`Schedule` @@ -1989,7 +2265,7 @@ paths: schema: type: string description: >- - A page token, received from a previous `ListSchedules` call. Use + A page token, received from a previous ``ListSchedules`` call. Use this to retrieve the subsequent page. responses: '200': @@ -2070,7 +2346,7 @@ paths: description: >- The user to receive the subscription email. This parameter is mutually exclusive with - `destination_subscriber`. + ``destination_subscriber``. create_time: type: string description: >- @@ -2155,8 +2431,8 @@ paths: :param page_size: int (optional) The number of subscriptions to return per page. :param page_token: str (optional) - A page token, received from a previous `ListSubscriptions` call. Use this to retrieve the subsequent - page. + A page token, received from a previous ``ListSubscriptions`` call. Use this to retrieve the + subsequent page. :returns: Iterator over :class:`Subscription` parameters: @@ -2184,8 +2460,8 @@ paths: schema: type: string description: >- - A page token, received from a previous `ListSubscriptions` call. Use - this to retrieve the subsequent page. + A page token, received from a previous ``ListSubscriptions`` call. + Use this to retrieve the subsequent page. responses: '200': description: Success @@ -2353,9 +2629,9 @@ paths: type: string description: >- A Java timezone id. The schedule will be resolved - with respect to this timezone. See [Java TimeZone] - for details. [Java TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + with respect to this timezone. See `Java TimeZone + `__ + for details. required: - quartz_cron_expression - timezone_id @@ -2703,19 +2979,19 @@ paths: description: >- The workspace path of the dashboard asset, including the file name. Exported dashboards always have the file - extension `.lvdash.json`. This field is excluded in List - Dashboards responses. + extension ``.lvdash.json``. This field is excluded in + List Dashboards responses. serialized_dashboard: type: string description: >- The contents of the dashboard in serialized string form. This field is excluded in List Dashboards responses. Use - the [get dashboard API] to retrieve an example response, - which includes the `serialized_dashboard` field. This - field provides the structure of the JSON string that - represents the dashboard's layout and components. [get - dashboard API]: - https://docs.databricks.com/api/workspace/lakeview/get + the `get dashboard API + `__ + to retrieve an example response, which includes the + ``serialized_dashboard`` field. This field provides the + structure of the JSON string that represents the + dashboard's layout and components. update_time: type: string description: >- @@ -2946,47 +3222,140 @@ paths: message: type: string x-stackql-sdk-source: LakeviewAPI - /api/2.0/lakeview/dashboards/{dashboard_id}/published/tokeninfo: - get: - operationId: lakeview_embedded_get_published_dashboard_token_info + /api/2.0/lakeview/dashboards/{dashboard_id}/revert: + post: + operationId: lakeview_revert summary: >- - Get a required authorization details and scopes of a published dashboard - to mint an OAuth token. + Revert a dashboard's definition in draft mode to the last published + version. tags: - dashboards - - lakeview_embedded + - lakeview description: >- - Get a required authorization details and scopes of a published dashboard - to mint an OAuth token. + Revert a dashboard's definition in draft mode to the last published + version. :param dashboard_id: str - UUID identifying the published dashboard. - :param external_value: str (optional) - Provided external value to be included in the custom claim. - :param external_viewer_id: str (optional) - Provided external viewer id to be included in the custom claim. + UUID identifying the dashboard. + :param etag: str (optional) + The etag for the dashboard. Optionally, it can be provided to verify that the dashboard has not been + modified from its last retrieval. - :returns: :class:`GetPublishedDashboardTokenInfoResponse` + :returns: :class:`RevertDashboardResponse` parameters: - name: dashboard_id in: path required: true schema: type: string - description: UUID identifying the published dashboard. - - name: external_value - in: query - required: false - schema: - type: string - description: Provided external value to be included in the custom claim. - - name: external_viewer_id - in: query - required: false - schema: - type: string - description: Provided external viewer id to be included in the custom claim. + description: UUID identifying the dashboard. + requestBody: + content: + application/json: + schema: + type: object + properties: + etag: + type: string + description: >- + The etag for the dashboard. Optionally, it can be provided + to verify that the dashboard has not been modified from its + last retrieval. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/RevertDashboardResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LakeviewAPI + /api/2.0/lakeview/dashboards/{dashboard_id}/published/embedded: + get: + operationId: lakeview_embedded_get_published_dashboard_embedded + summary: Get the current published dashboard within an embedded context. + tags: + - dashboards + - lakeview_embedded + description: |- + Get the current published dashboard within an embedded context. + + :param dashboard_id: str + UUID identifying the published dashboard. + parameters: + - name: dashboard_id + in: path + required: true + schema: + type: string + description: UUID identifying the published dashboard. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: LakeviewEmbeddedAPI + /api/2.0/lakeview/dashboards/{dashboard_id}/published/tokeninfo: + get: + operationId: lakeview_embedded_get_published_dashboard_token_info + summary: >- + Get a required authorization details and scopes of a published dashboard + to mint an OAuth token. + tags: + - dashboards + - lakeview_embedded + description: >- + Get a required authorization details and scopes of a published dashboard + to mint an OAuth token. + + + :param dashboard_id: str + UUID identifying the published dashboard. + :param external_value: str (optional) + Provided external value to be included in the custom claim. + :param external_viewer_id: str (optional) + Provided external viewer id to be included in the custom claim. + + :returns: :class:`GetPublishedDashboardTokenInfoResponse` + parameters: + - name: dashboard_id + in: path + required: true + schema: + type: string + description: UUID identifying the published dashboard. + - name: external_value + in: query + required: false + schema: + type: string + description: Provided external value to be included in the custom claim. + - name: external_viewer_id + in: query + required: false + schema: + type: string + description: Provided external viewer id to be included in the custom claim. responses: '200': description: Success @@ -3022,13 +3391,13 @@ components: type: string description: >- The resource name to which the authorization rule applies. This - field is specific to `workspace_rule_set` constraint. Format: - `workspaces/{workspace_id}/dashboards/{dashboard_id}` + field is specific to ``workspace_rule_set`` constraint. Format: + ``workspaces/{workspace_id}/dashboards/{dashboard_id}`` type: type: string description: >- The type of authorization downscoping policy. Ex: - `workspace_rule_set` defines access rules for a specific workspace + ``workspace_rule_set`` defines access rules for a specific workspace resource AuthorizationDetailsGrantRule: type: object @@ -3044,8 +3413,9 @@ components: type: string description: >- A Java timezone id. The schedule will be resolved with respect to - this timezone. See [Java TimeZone] for details. [Java TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + this timezone. See `Java TimeZone + `__ + for details. required: - quartz_cron_expression - timezone_id @@ -3081,18 +3451,18 @@ components: type: string description: >- The workspace path of the dashboard asset, including the file name. - Exported dashboards always have the file extension `.lvdash.json`. + Exported dashboards always have the file extension ``.lvdash.json``. This field is excluded in List Dashboards responses. serialized_dashboard: type: string description: >- The contents of the dashboard in serialized string form. This field - is excluded in List Dashboards responses. Use the [get dashboard - API] to retrieve an example response, which includes the - `serialized_dashboard` field. This field provides the structure of + is excluded in List Dashboards responses. Use the `get dashboard API + `__ to + retrieve an example response, which includes the + ``serialized_dashboard`` field. This field provides the structure of the JSON string that represents the dashboard's layout and - components. [get dashboard API]: - https://docs.databricks.com/api/workspace/lakeview/get + components. update_time: type: string description: >- @@ -3101,6 +3471,12 @@ components: warehouse_id: type: string description: The warehouse ID used to run the dashboard. + DownloadMessageAttachmentVisualizationResponse: + type: object + properties: + contents: + type: string + format: binary GenieAttachment: type: object properties: @@ -3118,6 +3494,11 @@ components: description: >- Text Attachment if Genie responds with text This also contains the final summary when available. + viz: + $ref: '#/components/schemas/GenieVizAttachment' + description: >- + Visualization generated by Genie, if requested via + ``enable_visualization`` description: Genie AI Response GenieConversation: type: object @@ -3225,7 +3606,57 @@ components: type: array items: $ref: '#/components/schemas/ScoreReason' - description: Reasons for the assessment score. + description: >- + Reasons for the assessment score. Assessment reasons describe why a + Genie response was scored as BAD. Deterministic values (compared + against the ground truth result): - EMPTY_RESULT: Genie's generated + SQL results were empty for this benchmark question. - + RESULT_MISSING_ROWS: Genie's generated SQL response is missing rows + from the provided ground truth SQL. - RESULT_EXTRA_ROWS: Genie's + generated SQL response has more rows than the provided ground truth + SQL. - RESULT_MISSING_COLUMNS: Genie's generated SQL response is + missing columns from the provided ground truth SQL. - + RESULT_EXTRA_COLUMNS: Genie's generated SQL response has more + columns than the provided ground truth SQL. - + SINGLE_CELL_DIFFERENCE: Single value result was produced but differs + from ground truth result. - EMPTY_GOOD_SQL: The benchmark SQL + returned an empty result. - COLUMN_TYPE_DIFFERENCE: The values + between the results match but the column type is different. LLM + judge ratings explain the factors driving BAD results: - + LLM_JUDGE_MISSING_OR_INCORRECT_FILTER: Genie's generated SQL is + missing a WHERE clause condition or has incorrect filter logic that + excludes/includes wrong data. - + LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT: Genie's generated SQL + returns only some of the requested data or columns, missing parts of + what the ground truth SQL returns. - + LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST: Genie's generated SQL + fundamentally misunderstands what the user is asking for, addressing + the wrong question or goal. - + LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC: Genie's + generated SQL fails to apply specified instructions or business + logic that should be followed. - + LLM_JUDGE_INCORRECT_METRIC_CALCULATION: Genie's generated SQL uses + incorrect logic or makes wrong assumptions when calculating metrics. + - LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE: Genie's generated SQL + references wrong tables, columns, or uses fields that don't match + the ground truth SQL's intent. - LLM_JUDGE_INCORRECT_FUNCTION_USAGE: + Genie's generated SQL uses SQL functions incorrectly or + inappropriately (wrong parameters, wrong function for the task, + etc.). - LLM_JUDGE_MISSING_OR_INCORRECT_JOIN: Genie's generated SQL + is missing necessary joins between tables or has incorrect join + conditions/types that produce wrong results. - + LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION: Genie's generated SQL is + missing GROUP BY clauses or has incorrect grouping that doesn't + match the requested aggregation level. - LLM_JUDGE_FORMATTING_ERROR: + Genie's generated SQL output has incorrect formatting, ordering + (ORDER BY), or presentation issues that don't match expectations. - + LLM_JUDGE_OTHER: LLM judge identified an error that doesn't fall + into other categories. Deprecated LLM judge values (kept for + backward compatibility, do not use): - LLM_JUDGE_MISSING_JOIN + (deprecated) - LLM_JUDGE_WRONG_FILTER (deprecated) - + LLM_JUDGE_WRONG_AGGREGATION (deprecated) - LLM_JUDGE_WRONG_COLUMNS + (deprecated) - LLM_JUDGE_SYNTAX_ERROR (deprecated) - + LLM_JUDGE_SEMANTIC_ERROR (deprecated) eval_run_status: $ref: '#/components/schemas/EvaluationStatusType' description: Current status of the evaluation run. @@ -3280,6 +3711,9 @@ components: GenieFeedback: type: object properties: + comment: + type: string + description: Optional feedback comment text rating: $ref: '#/components/schemas/GenieFeedbackRating' description: The feedback rating @@ -3304,6 +3738,16 @@ components: properties: statement_response: type: string + GenieListConversationCommentsResponse: + type: object + properties: + comments: + type: array + items: + $ref: '#/components/schemas/GenieMessageComment' + next_page_token: + type: string + description: Token to get the next page of results. GenieListConversationMessagesResponse: type: object properties: @@ -3344,6 +3788,16 @@ components: next_page_token: type: string description: The token to use for retrieving the next page of results. + GenieListMessageCommentsResponse: + type: object + properties: + comments: + type: array + items: + $ref: '#/components/schemas/GenieMessageComment' + next_page_token: + type: string + description: Token to get the next page of results. GenieListSpacesResponse: type: object properties: @@ -3392,8 +3846,8 @@ components: $ref: '#/components/schemas/Result' description: >- The result of SQL query if the message includes a query attachment. - Deprecated. Use `query_result_metadata` in `GenieQueryAttachment` - instead. + Deprecated. Use ``query_result_metadata`` in + ``GenieQueryAttachment`` instead. status: $ref: '#/components/schemas/MessageStatus' user_id: @@ -3404,6 +3858,37 @@ components: - conversation_id - content - message_id + GenieMessageComment: + type: object + properties: + space_id: + type: string + description: Genie space ID + conversation_id: + type: string + description: Conversation ID + message_id: + type: string + description: Message ID + message_comment_id: + type: string + description: Comment ID + content: + type: string + description: Comment text content + created_timestamp: + type: integer + description: Timestamp when the comment was created + user_id: + type: integer + description: ID of the user who created the comment + required: + - space_id + - conversation_id + - message_id + - message_comment_id + - content + description: A comment on a Genie conversation message. GenieQueryAttachment: type: object properties: @@ -3430,6 +3915,11 @@ components: Statement Execution API statement id. Use [Get status, manifest, and result first chunk](:method:statementexecution/getstatement) to get the full result data. + thoughts: + type: array + items: + $ref: '#/components/schemas/Thought' + description: Insights into how Genie came to generate the SQL. title: type: string description: Name of the query @@ -3452,6 +3942,11 @@ components: description: type: string description: Description of the Genie Space + etag: + type: string + description: >- + ETag for this space. Pass this value back in the update request to + prevent overwriting concurrent changes. parent_path: type: string description: Parent folder path of the Genie Space @@ -3461,9 +3956,9 @@ components: The contents of the Genie Space in serialized string form. This field is excluded in List Genie spaces responses. Use the [Get Genie Space](:method:genie/getspace) API to retrieve an example response, - which includes the `serialized_space` field. This field provides the - structure of the JSON string that represents the space's layout and - components. + which includes the ``serialized_space`` field. This field provides + the structure of the JSON string that represents the space's layout + and components. warehouse_id: type: string description: Warehouse associated with the Genie Space @@ -3494,6 +3989,22 @@ components: type: string description: The suggested follow-up questions description: Follow-up questions suggested by Genie + GenieVizAttachment: + type: object + properties: + query_attachment_id: + type: string + description: The ID of the query attachment the visualization was generated from + title: + type: string + description: Name of the visualization + description: >- + Visualization generated by Genie for a query result. Use the attachment + ID with the download + visualization API to retrieve the rendered image. + GetPublishedDashboardEmbeddedResponse: + type: object + properties: {} GetPublishedDashboardTokenInfoResponse: type: object properties: @@ -3506,7 +4017,7 @@ components: description: >- Custom claim generated from external_value and external_viewer_id. Format: - `urn:aibi:external_data:::` + ``urn:aibi:external_data:::`` scope: type: string description: Scope defining access permissions. @@ -3520,7 +4031,7 @@ components: next_page_token: type: string description: >- - A token, which can be sent as `page_token` to retrieve the next + A token, which can be sent as ``page_token`` to retrieve the next page. If this field is omitted, there are no subsequent dashboards. ListSchedulesResponse: type: object @@ -3589,6 +4100,13 @@ components: statement_id_signature: type: string description: JWT corresponding to the statement contained in this result + RevertDashboardResponse: + type: object + properties: + dashboard: + $ref: '#/components/schemas/Dashboard' + description: The reverted dashboard. + description: Response to revert a dashboard draft to its last published state. Schedule: type: object properties: @@ -3633,7 +4151,7 @@ components: $ref: '#/components/schemas/SubscriptionSubscriberUser' description: >- The user to receive the subscription email. This parameter is - mutually exclusive with `destination_subscriber`. + mutually exclusive with ``destination_subscriber``. Subscription: type: object properties: @@ -3694,15 +4212,50 @@ components: type: string id: type: string + phase: + $ref: '#/components/schemas/ResponsePhase' purpose: $ref: '#/components/schemas/TextAttachmentPurpose' description: Purpose/intent of this text attachment + verification_metadata: + $ref: '#/components/schemas/VerificationMetadata' + description: >- + Metadata for verification phase attachments. Only set when phase = + RESPONSE_PHASE_VERIFYING. + Thought: + type: object + properties: + content: + type: string + description: The md formatted content for this thought. + thought_type: + $ref: '#/components/schemas/ThoughtType' + description: >- + The category of this thought. The possible values are: - + ``THOUGHT_TYPE_DESCRIPTION``: A high-level description of how the + question was interpreted. - ``THOUGHT_TYPE_UNDERSTANDING``: How + ambiguous parts of the question were resolved. - + ``THOUGHT_TYPE_DATA_SOURCING``: Which tables or datasets were + identified as relevant. - ``THOUGHT_TYPE_INSTRUCTIONS``: Which + author-defined instructions were referenced. - + ``THOUGHT_TYPE_STEPS``: The logical steps taken to compute the + answer. + description: A single thought in the AI's reasoning process for a query. TrashDashboardResponse: type: object properties: {} UnpublishDashboardResponse: type: object properties: {} + VerificationMetadata: + type: object + properties: + index: + type: integer + description: Optional index to help order attachments within the same section + section: + $ref: '#/components/schemas/VerificationSection' + description: Metadata for verification phase attachments DashboardView: type: string x-enum: @@ -3928,6 +4481,8 @@ components: type: string x-enum: - BLOCK_MULTIPLE_EXECUTIONS_EXCEPTION + - BUDGET_EXCEEDED_EXCEPTION + - CERTIFIED_ANSWERS_MISSING_EXCEPTION - CHAT_COMPLETION_CLIENT_EXCEPTION - CHAT_COMPLETION_CLIENT_TIMEOUT_EXCEPTION - CHAT_COMPLETION_NETWORK_EXCEPTION @@ -3936,6 +4491,8 @@ components: - COULD_NOT_GET_DASHBOARD_SCHEMA_EXCEPTION - COULD_NOT_GET_MODEL_DEPLOYMENTS_EXCEPTION - COULD_NOT_GET_UC_SCHEMA_EXCEPTION + - DASHBOARD_PERMISSION_DENIED_EXCEPTION + - DELEGATION_NOT_FOUND_EXCEPTION - DEPLOYMENT_NOT_FOUND_EXCEPTION - DESCRIBE_QUERY_INVALID_SQL_ERROR - DESCRIBE_QUERY_TIMEOUT @@ -3958,6 +4515,7 @@ components: - INTERNAL_CATALOG_PATH_OVERLAP_EXCEPTION - INVALID_CERTIFIED_ANSWER_FUNCTION_EXCEPTION - INVALID_CERTIFIED_ANSWER_IDENTIFIER_EXCEPTION + - INVALID_CHAT_COMPLETION_ARGUMENTS_JSON_EXCEPTION - INVALID_CHAT_COMPLETION_JSON_EXCEPTION - INVALID_COMPLETION_REQUEST_EXCEPTION - INVALID_FUNCTION_CALL_EXCEPTION @@ -3974,6 +4532,7 @@ components: - NO_DEPLOYMENTS_AVAILABLE_TO_WORKSPACE - NO_QUERY_TO_VISUALIZE_EXCEPTION - NO_TABLES_TO_QUERY_EXCEPTION + - PAY_PER_TOKEN_DISABLED_EXCEPTION - RATE_LIMIT_EXCEEDED_GENERIC_EXCEPTION - RATE_LIMIT_EXCEEDED_SPECIFIED_WAIT_EXCEPTION - REPLY_PROCESS_TIMEOUT_EXCEPTION @@ -4039,44 +4598,78 @@ components: - QUERY_RESULT_EXPIRED - SUBMITTED description: >- - MessageStatus. The possible values are: * `FETCHING_METADATA`: Fetching - metadata from the data + MessageStatus. The possible values are: + + + - ``FETCHING_METADATA``: Fetching metadata from the data sources. + + - ``FILTERING_CONTEXT``: Running smart context step to determine + relevant context. - sources. * `FILTERING_CONTEXT`: Running smart context step to determine - relevant context. * + - ``ASKING_AI``: Waiting for the LLM to respond to the user's question. + + - ``PENDING_WAREHOUSE``: Waiting for warehouse before the SQL query can + start executing. + + - ``EXECUTING_QUERY``: Executing a generated SQL query. Get the SQL + query result by calling + [getMessageAttachmentQueryResult](:method:genie/getMessageAttachmentQueryResult) API. + - ``FAILED``: The response generation or query execution failed. See + ``error`` field. + + - ``COMPLETED``: Message processing is completed. Results are in the + ``attachments`` field. Get + the SQL query result by calling + [getMessageAttachmentQueryResult](:method:genie/getMessageAttachmentQueryResult) API. + - ``SUBMITTED``: Message has been submitted. + + - ``QUERY_RESULT_EXPIRED``: SQL result is not available anymore. The + user needs to rerun the + query. Rerun the SQL query result by calling + [executeMessageAttachmentQuery](:method:genie/executeMessageAttachmentQuery) API. + - ``CANCELLED``: Message has been cancelled. + ResponsePhase: + type: string + x-enum: + - RESPONSE_PHASE_THINKING + - RESPONSE_PHASE_VERIFYING + description: |- + Create a collection of name/value pairs. + + Example enumeration: - `ASKING_AI`: Waiting for the LLM to respond to the user's question. * - `PENDING_WAREHOUSE`: + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 - Waiting for warehouse before the SQL query can start executing. * - `EXECUTING_QUERY`: Executing a + Access them by: - generated SQL query. Get the SQL query result by calling + - attribute access: - [getMessageAttachmentQueryResult](:method:genie/getMessageAttachmentQueryResult) - API. * + >>> Color.RED + - `FAILED`: The response generation or query execution failed. See `error` - field. * `COMPLETED`: + - value lookup: - Message processing is completed. Results are in the `attachments` field. - Get the SQL query + >>> Color(1) + - result by calling + - name lookup: - [getMessageAttachmentQueryResult](:method:genie/getMessageAttachmentQueryResult) - API. * + >>> Color['RED'] + - `SUBMITTED`: Message has been submitted. * `QUERY_RESULT_EXPIRED`: SQL - result is not available + Enumerations can be iterated over, and know how many members they have: - anymore. The user needs to rerun the query. Rerun the SQL query result - by calling + >>> len(Color) + 3 - [executeMessageAttachmentQuery](:method:genie/executeMessageAttachmentQuery) - API. * `CANCELLED`: + >>> list(Color) + [, , ] - Message has been cancelled. + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. SchedulePauseStatus: type: string x-enum: @@ -4147,6 +4740,7 @@ components: - RESULT_MISSING_COLUMNS - RESULT_MISSING_ROWS - SINGLE_CELL_DIFFERENCE + - TRANSIENT_ERROR description: |- Create a collection of name/value pairs. @@ -4189,6 +4783,48 @@ components: x-enum: - FOLLOW_UP_QUESTION description: Purpose/intent of a text attachment + ThoughtType: + type: string + x-enum: + - THOUGHT_TYPE_DATA_SOURCING + - THOUGHT_TYPE_DESCRIPTION + - THOUGHT_TYPE_INSTRUCTIONS + - THOUGHT_TYPE_STEPS + - THOUGHT_TYPE_UNDERSTANDING + description: >- + ThoughtType. The possible values are: + + + - ``THOUGHT_TYPE_UNSPECIFIED``: Default value that should not be used. + + - ``THOUGHT_TYPE_DESCRIPTION``: A high-level description of how the + question was interpreted. + + - ``THOUGHT_TYPE_UNDERSTANDING``: How ambiguous parts of the question + were resolved. + + - ``THOUGHT_TYPE_DATA_SOURCING``: Which tables or datasets were + identified as relevant. + + - ``THOUGHT_TYPE_INSTRUCTIONS``: Which author-defined instructions were + referenced. + + - ``THOUGHT_TYPE_STEPS``: The logical steps taken to compute the answer. + The category of a + Thought. Additional values may be added in the future. + VerificationSection: + type: string + x-enum: + - VERIFICATION_SECTION_FINAL_DECISION + - VERIFICATION_SECTION_PROPOSED_IMPROVEMENT + - VERIFICATION_SECTION_SQL_EXAMPLES_VALIDATION + - VERIFICATION_SECTION_VERIFICATION_QUERIES + description: >- + Verification workflow section - indicates which stage of verification + this attachment belongs to + + These sections are used for grouping and ordering attachments in the + frontend UI x-stackQL-resources: genie_messages: id: databricks_workspace.dashboards.genie_messages @@ -4274,56 +4910,43 @@ components: name: genie title: Genie methods: - create: + genie_create_message_comment: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1genie~1spaces/post' + $ref: >- + #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1conversations~1{conversation_id}~1messages~1{message_id}~1comments/post response: mediaType: application/json openAPIDocKey: '200' - list: + genie_list_message_comments: operation: - $ref: '#/paths/~1api~12.0~1genie~1spaces/get' + $ref: >- + #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1conversations~1{conversation_id}~1messages~1{message_id}~1comments/get response: mediaType: application/json openAPIDocKey: '200' - genie_genie_create_eval_run: + create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs/post' - response: - mediaType: application/json - openAPIDocKey: '200' - genie_genie_list_eval_runs: - operation: - $ref: '#/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs/get' + $ref: '#/paths/~1api~12.0~1genie~1spaces/post' response: mediaType: application/json openAPIDocKey: '200' - genie_genie_get_eval_result_details: + list: operation: - $ref: >- - #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs~1{eval_run_id}~1results~1{result_id}/get + $ref: '#/paths/~1api~12.0~1genie~1spaces/get' response: mediaType: application/json openAPIDocKey: '200' - genie_genie_get_eval_run: + genie_download_message_attachment_visualization: operation: - $ref: >- - #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs~1{eval_run_id}/get + $ref: '#/paths/~1api~12.0~1genie~1{name}~1download-visualization/get' response: - mediaType: application/json - openAPIDocKey: '200' - genie_genie_list_eval_results: - operation: - $ref: >- - #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs~1{eval_run_id}~1results/get - response: - mediaType: application/json + mediaType: application/octet-stream openAPIDocKey: '200' get: operation: @@ -4346,21 +4969,26 @@ components: response: mediaType: application/json openAPIDocKey: '200' + genie_list_conversation_comments: + operation: + $ref: >- + #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1conversations~1{conversation_id}~1list-comments/get + response: + mediaType: application/json + openAPIDocKey: '200' sqlVerbs: select: - $ref: >- - #/components/x-stackQL-resources/genie/methods/genie_genie_get_eval_result_details + #/components/x-stackQL-resources/genie/methods/genie_list_message_comments - $ref: >- - #/components/x-stackQL-resources/genie/methods/genie_genie_get_eval_run + #/components/x-stackQL-resources/genie/methods/genie_list_conversation_comments - $ref: >- - #/components/x-stackQL-resources/genie/methods/genie_genie_list_eval_results - - $ref: >- - #/components/x-stackQL-resources/genie/methods/genie_genie_list_eval_runs + #/components/x-stackQL-resources/genie/methods/genie_download_message_attachment_visualization - $ref: '#/components/x-stackQL-resources/genie/methods/get' - $ref: '#/components/x-stackQL-resources/genie/methods/list' insert: - $ref: >- - #/components/x-stackQL-resources/genie/methods/genie_genie_create_eval_run + #/components/x-stackQL-resources/genie/methods/genie_create_message_comment - $ref: '#/components/x-stackQL-resources/genie/methods/create' update: - $ref: '#/components/x-stackQL-resources/genie/methods/update' @@ -4458,6 +5086,72 @@ components: update: [] delete: [] replace: [] + genie_eval_runs: + id: databricks_workspace.dashboards.genie_eval_runs + name: genie_eval_runs + title: Genie Eval Runs + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.eval_runs + get: + operation: + $ref: >- + #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs~1{eval_run_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/genie_eval_runs/methods/get' + - $ref: '#/components/x-stackQL-resources/genie_eval_runs/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/genie_eval_runs/methods/create' + update: [] + delete: [] + replace: [] + genie_eval_results: + id: databricks_workspace.dashboards.genie_eval_results + name: genie_eval_results + title: Genie Eval Results + methods: + get_details: + operation: + $ref: >- + #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs~1{eval_run_id}~1results~1{result_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: >- + #/paths/~1api~12.0~1genie~1spaces~1{space_id}~1eval-runs~1{eval_run_id}~1results/get + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.eval_results + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/genie_eval_results/methods/get_details + - $ref: '#/components/x-stackQL-resources/genie_eval_results/methods/list' + insert: [] + update: [] + delete: [] + replace: [] lakeview: id: databricks_workspace.dashboards.lakeview name: lakeview @@ -4509,11 +5203,22 @@ components: response: mediaType: application/json openAPIDocKey: '200' + lakeview_revert: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1lakeview~1dashboards~1{dashboard_id}~1revert/post + response: + mediaType: application/json + openAPIDocKey: '200' sqlVerbs: select: - $ref: '#/components/x-stackQL-resources/lakeview/methods/get' - $ref: '#/components/x-stackQL-resources/lakeview/methods/list' insert: + - $ref: '#/components/x-stackQL-resources/lakeview/methods/lakeview_revert' - $ref: '#/components/x-stackQL-resources/lakeview/methods/create' update: - $ref: '#/components/x-stackQL-resources/lakeview/methods/update' @@ -4655,6 +5360,24 @@ components: update: [] delete: [] replace: [] + published_dashboard_embedded: + id: databricks_workspace.dashboards.published_dashboard_embedded + name: published_dashboard_embedded + title: Published Dashboard Embedded + methods: + embed: + operation: + $ref: >- + #/paths/~1api~12.0~1lakeview~1dashboards~1{dashboard_id}~1published~1embedded/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: [] + update: [] + delete: [] + replace: [] lakeview_embedded: id: databricks_workspace.dashboards.lakeview_embedded name: lakeview_embedded diff --git a/providers/src/databricks_workspace/v00.00.00000/services/database.yaml b/providers/src/databricks_workspace/v00.00.00000/services/database.yaml index 6b80a50a..00f0f7f9 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/database.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/database.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks database service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.database servers: - url: https://{deployment_name}.cloud.databricks.com @@ -44,10 +44,20 @@ paths: database_name: type: string description: >- - The name of the database (in a instance) associated with - the catalog. + The name of the database (in an instance) associated + with the catalog. create_database_if_not_exists: type: boolean + database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the + catalog. + database_project_id: + type: string + description: >- + The project_id of the database project associated with + the catalog. uid: type: string required: @@ -346,6 +356,7 @@ paths: - STARTING - STOPPED - UPDATING + - UPGRADING description: The current state of the instance. stopped: type: boolean @@ -655,6 +666,9 @@ paths: postgres database. Registration of database tables via /database/tables is currently only supported in standard catalogs. + table_serving_url: + type: string + description: Data serving REST API URL for this table required: - name required: @@ -967,6 +981,11 @@ paths: synced table is in the SYNCED_TRIGGERED_UPDATE or the SYNCED_NO_PENDING_UPDATE state. description: Synced Table data synchronization status + database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the + table. database_instance_name: type: string description: >- @@ -978,6 +997,19 @@ paths: registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected). + database_project_id: + type: string + description: >- + The project_id of the database project associated with + the table. + effective_database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the + table. This is an output only field that contains the + value computed from the input field combined with server + side defaults. Use the field without the effective_ + prefix to set the value. effective_database_instance_name: type: string description: >- @@ -988,6 +1020,14 @@ paths: only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value. + effective_database_project_id: + type: string + description: >- + The project_id of the database project associated with + the table. This is an output only field that contains + the value computed from the input field combined with + server side defaults. Use the field without the + effective_ prefix to set the value. effective_logical_database_name: type: string description: >- @@ -1010,10 +1050,18 @@ paths: field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. Note that this has implications for the - `create_database_objects_is_missing` field in `spec`. + ``create_database_objects_is_missing`` field in + ``spec``. spec: type: object properties: + accelerated_sync: + type: boolean + description: >- + When true, enables accelerated sync mode for the + initial data load. This significantly improves + performance for large tables. Requires + workspace-level enablement. create_database_objects_if_missing: type: boolean description: >- @@ -1031,6 +1079,74 @@ paths: allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline. + extra_columns: + type: array + items: + type: object + properties: + column_name: + type: string + description: Name of the column. + column_type: + type: string + description: >- + PostgreSQL type of the column, for example + "tsvector" or "vector(1024)". + compute: + type: string + description: >- + SQL expression used to compute the column's + value, for example "to_tsvector('english', + content)". + maintenance: + type: string + x-enum: + - DEFAULT_VALUE + - STORED_GENERATED + description: >- + How the column's value is populated and kept + up to date. + required: + - column_name + - column_type + description: >- + An extra PostgreSQL column to add to the synced + table. + description: >- + Extra PostgreSQL-only columns to add to the synced + table. + extra_index_definitions: + type: array + items: + type: object + properties: + name: + type: string + description: >- + Name of the index as it will appear in + PostgreSQL. + definition: + type: string + description: >- + The definition portion of a CREATE INDEX + statement, placed after ON table_name. For + example: USING hnsw (embedding + vector_cosine_ops) WITH (m = 16, + ef_construction = 64). + creation_point: + type: string + x-enum: + - CREATION_POINT_AFTER_DATA_LOAD + description: >- + When the index should be created relative to + the initial data load. + required: + - name + - definition + description: >- + Definition of a secondary index to create on the + synced table. + description: Secondary indexes to create on the synced table. new_pipeline_spec: type: object properties: @@ -1039,6 +1155,16 @@ paths: description: >- Budget policy to set on the newly created pipeline. + pipeline_channel: + type: string + x-enum: + - CURRENT + - PREVIEW + description: >- + Release channel of the underlying pipeline's + runtime. Some source table configurations (e.g., + read-time CDF) require PREVIEW. Defaults to + CURRENT if not specified. storage_catalog: type: string description: >- @@ -1093,7 +1219,47 @@ paths: description: >- Time series key to deduplicate (tie-break) rows with the same primary key. + type_overrides: + type: array + items: + type: object + properties: + column_name: + type: string + description: >- + Name of the source column whose target + PostgreSQL type should be overridden. + pg_type: + type: string + x-enum: + - PG_SPECIFIC_TYPE_HALFVEC + - PG_SPECIFIC_TYPE_VARCHAR + - PG_SPECIFIC_TYPE_VECTOR + description: >- + PostgreSQL-specific target type to use for the + column. + size: + type: integer + description: >- + Size parameter for the target type, for types + that take one (e.g. vector dimension, varchar + length). Required when the chosen pg_type + needs a size. + required: + - column_name + - pg_type + description: >- + Overrides the default Delta-to-PostgreSQL type + mapping for a single column. + description: >- + Override the default Delta->PG type mapping for + specific columns. A TypeOverride with + PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid + pg_type must be set. description: Specification of a synced database table. + table_serving_url: + type: string + description: Data serving REST API URL for this table unity_catalog_provisioning_state: type: string x-enum: @@ -1247,10 +1413,20 @@ paths: database_name: type: string description: >- - The name of the database (in a instance) associated with - the catalog. + The name of the database (in an instance) associated + with the catalog. create_database_if_not_exists: type: boolean + database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the + catalog. + database_project_id: + type: string + description: >- + The project_id of the database project associated with + the catalog. uid: type: string required: @@ -1291,7 +1467,7 @@ paths: :param name: str Name of the instance to delete. :param force: bool (optional) - By default, a instance cannot be deleted if it has descendant instances created via PITR. If this + By default, an instance cannot be deleted if it has descendant instances created via PITR. If this flag is specified as true, all descendent instances will be deleted as well. :param purge: bool (optional) Deprecated. Omitting the field or setting it to true will result in the field being hard deleted. @@ -1309,7 +1485,7 @@ paths: schema: type: boolean description: >- - By default, a instance cannot be deleted if it has descendant + By default, an instance cannot be deleted if it has descendant instances created via PITR. If this flag is specified as true, all descendent instances will be deleted as well. - name: purge @@ -1664,6 +1840,7 @@ paths: - STARTING - STOPPED - UPDATING + - UPGRADING description: The current state of the instance. stopped: type: boolean @@ -1715,7 +1892,7 @@ paths: :param instance_name: str :param name: str :param allow_missing: bool (optional) - This is the AIP standard name for the equivalent of Postgres' `IF EXISTS` option + This is the AIP standard name for the equivalent of Postgres' ``IF EXISTS`` option :param reassign_owned_to: str (optional) parameters: - name: instance_name @@ -1734,8 +1911,8 @@ paths: schema: type: boolean description: >- - This is the AIP standard name for the equivalent of Postgres' `IF - EXISTS` option + This is the AIP standard name for the equivalent of Postgres' ``IF + EXISTS`` option - name: reassign_owned_to in: query required: false @@ -1798,6 +1975,124 @@ paths: message: type: string x-stackql-sdk-source: DatabaseAPI + patch: + operationId: database_update_database_instance_role + summary: Update a role for a Database Instance. + tags: + - database + description: |- + Update a role for a Database Instance. + + :param instance_name: str + :param name: str + The name of the role. This is the unique identifier for the role in an instance. + :param database_instance_role: :class:`DatabaseInstanceRole` + :param database_instance_name: str (optional) + + :returns: :class:`DatabaseInstanceRole` + parameters: + - name: instance_name + in: path + required: true + schema: + type: string + - name: name + in: path + required: true + schema: + type: string + description: >- + The name of the role. This is the unique identifier for the role in + an instance. + - name: database_instance_name + in: query + required: false + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + database_instance_role: + type: object + properties: + name: + type: string + description: >- + The name of the role. This is the unique identifier for + the role in an instance. + attributes: + type: object + properties: + bypassrls: + type: boolean + createdb: + type: boolean + createrole: + type: boolean + description: >- + The desired API-exposed Postgres role attribute to + associate with the role. Optional. + effective_attributes: + type: object + properties: + bypassrls: + type: boolean + createdb: + type: boolean + createrole: + type: boolean + description: >- + The attributes that are applied to the role. This is an + output only field that contains the value computed from + the input field combined with server side defaults. Use + the field without the effective_ prefix to set the + value. + identity_type: + type: string + x-enum: + - GROUP + - PG_ONLY + - SERVICE_PRINCIPAL + - USER + description: The type of the role. + instance_name: + type: string + membership_role: + type: string + x-enum: + - DATABRICKS_SUPERUSER + description: >- + An enum value for a standard role that this role is a + member of. + required: + - name + description: >- + A DatabaseInstanceRole represents a Postgres role in a + database instance. + required: + - database_instance_role + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DatabaseInstanceRole' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DatabaseAPI /api/2.0/database/tables/{name}: delete: operationId: database_delete_database_table @@ -2250,6 +2545,11 @@ paths: synced table is in the SYNCED_TRIGGERED_UPDATE or the SYNCED_NO_PENDING_UPDATE state. description: Synced Table data synchronization status + database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the + table. database_instance_name: type: string description: >- @@ -2261,6 +2561,19 @@ paths: registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected). + database_project_id: + type: string + description: >- + The project_id of the database project associated with + the table. + effective_database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the + table. This is an output only field that contains the + value computed from the input field combined with server + side defaults. Use the field without the effective_ + prefix to set the value. effective_database_instance_name: type: string description: >- @@ -2271,6 +2584,14 @@ paths: only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value. + effective_database_project_id: + type: string + description: >- + The project_id of the database project associated with + the table. This is an output only field that contains + the value computed from the input field combined with + server side defaults. Use the field without the + effective_ prefix to set the value. effective_logical_database_name: type: string description: >- @@ -2293,10 +2614,18 @@ paths: field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. Note that this has implications for the - `create_database_objects_is_missing` field in `spec`. + ``create_database_objects_is_missing`` field in + ``spec``. spec: type: object properties: + accelerated_sync: + type: boolean + description: >- + When true, enables accelerated sync mode for the + initial data load. This significantly improves + performance for large tables. Requires + workspace-level enablement. create_database_objects_if_missing: type: boolean description: >- @@ -2314,6 +2643,74 @@ paths: allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline. + extra_columns: + type: array + items: + type: object + properties: + column_name: + type: string + description: Name of the column. + column_type: + type: string + description: >- + PostgreSQL type of the column, for example + "tsvector" or "vector(1024)". + compute: + type: string + description: >- + SQL expression used to compute the column's + value, for example "to_tsvector('english', + content)". + maintenance: + type: string + x-enum: + - DEFAULT_VALUE + - STORED_GENERATED + description: >- + How the column's value is populated and kept + up to date. + required: + - column_name + - column_type + description: >- + An extra PostgreSQL column to add to the synced + table. + description: >- + Extra PostgreSQL-only columns to add to the synced + table. + extra_index_definitions: + type: array + items: + type: object + properties: + name: + type: string + description: >- + Name of the index as it will appear in + PostgreSQL. + definition: + type: string + description: >- + The definition portion of a CREATE INDEX + statement, placed after ON table_name. For + example: USING hnsw (embedding + vector_cosine_ops) WITH (m = 16, + ef_construction = 64). + creation_point: + type: string + x-enum: + - CREATION_POINT_AFTER_DATA_LOAD + description: >- + When the index should be created relative to + the initial data load. + required: + - name + - definition + description: >- + Definition of a secondary index to create on the + synced table. + description: Secondary indexes to create on the synced table. new_pipeline_spec: type: object properties: @@ -2322,6 +2719,16 @@ paths: description: >- Budget policy to set on the newly created pipeline. + pipeline_channel: + type: string + x-enum: + - CURRENT + - PREVIEW + description: >- + Release channel of the underlying pipeline's + runtime. Some source table configurations (e.g., + read-time CDF) require PREVIEW. Defaults to + CURRENT if not specified. storage_catalog: type: string description: >- @@ -2376,7 +2783,47 @@ paths: description: >- Time series key to deduplicate (tie-break) rows with the same primary key. + type_overrides: + type: array + items: + type: object + properties: + column_name: + type: string + description: >- + Name of the source column whose target + PostgreSQL type should be overridden. + pg_type: + type: string + x-enum: + - PG_SPECIFIC_TYPE_HALFVEC + - PG_SPECIFIC_TYPE_VARCHAR + - PG_SPECIFIC_TYPE_VECTOR + description: >- + PostgreSQL-specific target type to use for the + column. + size: + type: integer + description: >- + Size parameter for the target type, for types + that take one (e.g. vector dimension, varchar + length). Required when the chosen pg_type + needs a size. + required: + - column_name + - pg_type + description: >- + Overrides the default Delta-to-PostgreSQL type + mapping for a single column. + description: >- + Override the default Delta->PG type mapping for + specific columns. A TypeOverride with + PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid + pg_type must be set. description: Specification of a synced database table. + table_serving_url: + type: string + description: Data serving REST API URL for this table unity_catalog_provisioning_state: type: string x-enum: @@ -2418,6 +2865,54 @@ paths: message: type: string x-stackql-sdk-source: DatabaseAPI + /api/2.0/database/instances/{name}/failover: + post: + operationId: database_failover_database_instance + summary: Failover the primary node of a Database Instance to a secondary. + tags: + - database + description: |- + Failover the primary node of a Database Instance to a secondary. + + :param name: str + Name of the instance to failover. + :param failover_target_database_instance_name: str (optional) + + :returns: :class:`DatabaseInstance` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the instance to failover. + requestBody: + content: + application/json: + schema: + type: object + properties: + failover_target_database_instance_name: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DatabaseInstance' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DatabaseAPI /api/2.0/database/instances:findByUid: get: operationId: database_find_database_instance_by_uid @@ -2653,6 +3148,39 @@ paths: message: type: string x-stackql-sdk-source: DatabaseAPI + /api/2.0/database/instances/{name}/upgrade: + patch: + operationId: database_upgrade_instance_to_autoscaling + summary: Upgrade a Database Instance to Autoscaling. + tags: + - database + description: |- + Upgrade a Database Instance to Autoscaling. + + :param name: str + Name of the instance to upgrade. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the instance to upgrade. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: DatabaseAPI components: schemas: CustomTag: @@ -2674,10 +3202,16 @@ components: database_name: type: string description: >- - The name of the database (in a instance) associated with the + The name of the database (in an instance) associated with the catalog. create_database_if_not_exists: type: boolean + database_branch_id: + type: string + description: The branch_id of the database branch associated with the catalog. + database_project_id: + type: string + description: The project_id of the database project associated with the catalog. uid: type: string required: @@ -2959,6 +3493,9 @@ components: targeting an arbitrary postgres database. Registration of database tables via /database/tables is currently only supported in standard catalogs. + table_serving_url: + type: string + description: Data serving REST API URL for this table required: - name DeltaTableSyncInfo: @@ -3014,6 +3551,12 @@ components: budget_policy_id: type: string description: Budget policy to set on the newly created pipeline. + pipeline_channel: + $ref: '#/components/schemas/PipelineChannel' + description: >- + Release channel of the underlying pipeline's runtime. Some source + table configurations (e.g., read-time CDF) require PREVIEW. Defaults + to CURRENT if not specified. storage_catalog: type: string description: >- @@ -3058,6 +3601,9 @@ components: data_synchronization_status: $ref: '#/components/schemas/SyncedTableStatus' description: Synced Table data synchronization status + database_branch_id: + type: string + description: The branch_id of the database branch associated with the table. database_instance_name: type: string description: >- @@ -3067,6 +3613,16 @@ components: field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected). + database_project_id: + type: string + description: The project_id of the database project associated with the table. + effective_database_branch_id: + type: string + description: >- + The branch_id of the database branch associated with the table. This + is an output only field that contains the value computed from the + input field combined with server side defaults. Use the field + without the effective_ prefix to set the value. effective_database_instance_name: type: string description: >- @@ -3076,6 +3632,13 @@ components: This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value. + effective_database_project_id: + type: string + description: >- + The project_id of the database project associated with the table. + This is an output only field that contains the value computed from + the input field combined with server side defaults. Use the field + without the effective_ prefix to set the value. effective_logical_database_name: type: string description: >- @@ -3095,9 +3658,12 @@ components: standard catalog, this field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. Note that this has implications for the - `create_database_objects_is_missing` field in `spec`. + ``create_database_objects_is_missing`` field in ``spec``. spec: $ref: '#/components/schemas/SyncedTableSpec' + table_serving_url: + type: string + description: Data serving REST API URL for this table unity_catalog_provisioning_state: $ref: '#/components/schemas/ProvisioningInfoState' description: >- @@ -3208,6 +3774,12 @@ components: SyncedTableSpec: type: object properties: + accelerated_sync: + type: boolean + description: >- + When true, enables accelerated sync mode for the initial data load. + This significantly improves performance for large tables. Requires + workspace-level enablement. create_database_objects_if_missing: type: boolean description: >- @@ -3222,6 +3794,16 @@ components: creating a new pipeline and allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline. + extra_columns: + type: array + items: + $ref: '#/components/schemas/SyncedTableSpecExtraColumn' + description: Extra PostgreSQL-only columns to add to the synced table. + extra_index_definitions: + type: array + items: + $ref: '#/components/schemas/SyncedTableSpecSecondaryIndex' + description: Secondary indexes to create on the synced table. new_pipeline_spec: $ref: '#/components/schemas/NewPipelineSpec' description: >- @@ -3250,7 +3832,79 @@ components: description: >- Time series key to deduplicate (tie-break) rows with the same primary key. + type_overrides: + type: array + items: + $ref: '#/components/schemas/SyncedTableSpecTypeOverride' + description: >- + Override the default Delta->PG type mapping for specific columns. A + TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid + pg_type must be set. description: Specification of a synced database table. + SyncedTableSpecExtraColumn: + type: object + properties: + column_name: + type: string + description: Name of the column. + column_type: + type: string + description: >- + PostgreSQL type of the column, for example "tsvector" or + "vector(1024)". + compute: + type: string + description: >- + SQL expression used to compute the column's value, for example + "to_tsvector('english', content)". + maintenance: + $ref: '#/components/schemas/SyncedTableSpecExtraColumnMaintenance' + required: + - column_name + - column_type + description: An extra PostgreSQL column to add to the synced table. + SyncedTableSpecSecondaryIndex: + type: object + properties: + name: + type: string + description: Name of the index as it will appear in PostgreSQL. + definition: + type: string + description: >- + The definition portion of a CREATE INDEX statement, placed after ON + table_name. For example: USING hnsw (embedding vector_cosine_ops) + WITH (m = 16, ef_construction = 64). + creation_point: + $ref: '#/components/schemas/SyncedTableSpecSecondaryIndexCreationPoint' + description: When the index should be created relative to the initial data load. + required: + - name + - definition + description: Definition of a secondary index to create on the synced table. + SyncedTableSpecTypeOverride: + type: object + properties: + column_name: + type: string + description: >- + Name of the source column whose target PostgreSQL type should be + overridden. + pg_type: + $ref: '#/components/schemas/SyncedTableSpecPgSpecificType' + description: PostgreSQL-specific target type to use for the column. + size: + type: integer + description: >- + Size parameter for the target type, for types that take one (e.g. + vector dimension, varchar length). Required when the chosen pg_type + needs a size. + required: + - column_name + - pg_type + description: >- + Overrides the default Delta-to-PostgreSQL type mapping for a single + column. SyncedTableStatus: type: object properties: @@ -3366,6 +4020,7 @@ components: - STARTING - STOPPED - UPDATING + - UPGRADING description: |- Create a collection of name/value pairs. @@ -3403,6 +4058,19 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + PipelineChannel: + type: string + x-enum: + - CURRENT + - PREVIEW + description: >- + Release channel of the underlying pipeline's runtime. PREVIEW provides + early access to the + + latest features but may be less stable. Some source table configurations + (e.g., read-time CDF) + + require PREVIEW. Defaults to CURRENT if not specified. ProvisioningInfoState: type: string x-enum: @@ -3540,6 +4208,26 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + SyncedTableSpecExtraColumnMaintenance: + type: string + x-enum: + - DEFAULT_VALUE + - STORED_GENERATED + description: How the column's value is populated and kept up to date. + SyncedTableSpecPgSpecificType: + type: string + x-enum: + - PG_SPECIFIC_TYPE_HALFVEC + - PG_SPECIFIC_TYPE_VARCHAR + - PG_SPECIFIC_TYPE_VECTOR + description: >- + PostgreSQL-specific target types that can override the default + Delta-to-PG mapping. + SyncedTableSpecSecondaryIndexCreationPoint: + type: string + x-enum: + - CREATION_POINT_AFTER_DATA_LOAD + description: Controls when the index is created relative to the initial data load. SyncedTableState: type: string x-enum: @@ -3816,6 +4504,48 @@ components: - $ref: >- #/components/x-stackQL-resources/synced_database_tables/methods/delete replace: [] + database: + id: databricks_workspace.database.database + name: database + title: Database + methods: + database_update_database_instance_role: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1database~1instances~1{instance_name}~1roles~1{name}/patch + response: + mediaType: application/json + openAPIDocKey: '200' + database_failover_database_instance: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1database~1instances~1{name}~1failover/post' + response: + mediaType: application/json + openAPIDocKey: '200' + database_upgrade_instance_to_autoscaling: + operation: + $ref: '#/paths/~1api~12.0~1database~1instances~1{name}~1upgrade/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: + - $ref: >- + #/components/x-stackQL-resources/database/methods/database_failover_database_instance + update: + - $ref: >- + #/components/x-stackQL-resources/database/methods/database_update_database_instance_role + - $ref: >- + #/components/x-stackQL-resources/database/methods/database_upgrade_instance_to_autoscaling + delete: [] + replace: [] database_credentials: id: databricks_workspace.database.database_credentials name: database_credentials diff --git a/providers/src/databricks_workspace/v00.00.00000/services/dataquality.yaml b/providers/src/databricks_workspace/v00.00.00000/services/dataquality.yaml index 3e1221ec..1dbc81b0 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/dataquality.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/dataquality.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks dataquality service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.dataquality servers: - url: https://{deployment_name}.cloud.databricks.com @@ -20,45 +20,48 @@ paths: operationId: data_quality_cancel_refresh summary: >- Cancels a data quality monitor refresh. Currently only supported for the - `table` `object_type`. The + ``table`` ``object_type``. tags: - dataquality - data_quality description: >- Cancels a data quality monitor refresh. Currently only supported for the - `table` `object_type`. The + ``table`` ``object_type``. - call must be made in the same workspace as where the monitor was + The call must be made in the same workspace as where the monitor was created. - The caller must have either of the following sets of permissions: 1. - **MANAGE** and **USE_CATALOG** on + The caller must have either of the following sets of permissions: - the table's parent catalog. 2. **USE_CATALOG** on the table's parent - catalog, and **MANAGE** and - **USE_SCHEMA** on the table's parent schema. 3. **USE_CATALOG** on the - table's parent catalog, - - **USE_SCHEMA** on the table's parent schema, and **MANAGE** on the - table. + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog. + 2. **USE_CATALOG** on the table's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the table's + parent schema. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **MANAGE** on the table. :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + Find the ``schema_id`` from either: - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + Find the ``table_id`` from either: + + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param refresh_id: int Unique id of the refresh operation. @@ -71,24 +74,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. - name: refresh_id in: path required: true @@ -127,39 +131,36 @@ paths: Create a data quality monitor on a Unity Catalog object. The caller must provide either - `anomaly_detection_config` for a schema monitor or - `data_profiling_config` for a table monitor. - - - For the `table` `object_type`, the caller must have either of the - following sets of permissions: 1. - - **MANAGE** and **USE_CATALOG** on the table's parent catalog, - **USE_SCHEMA** on the table's parent + ``anomaly_detection_config`` for a schema monitor or + ``data_profiling_config`` for a table monitor. - schema, and **SELECT** on the table 2. **USE_CATALOG** on the table's - parent catalog, **MANAGE** and - **USE_SCHEMA** on the table's parent schema, and **SELECT** on the - table. 3. **USE_CATALOG** on the - - table's parent catalog, **USE_SCHEMA** on the table's parent schema, and - **MANAGE** and **SELECT** on + For the ``table`` ``object_type``, the caller must have either of the + following sets of permissions: - the table. + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog, + **USE_SCHEMA** on the table's parent + schema, and **SELECT** on the table + 2. **USE_CATALOG** on the table's parent catalog, **MANAGE** and + **USE_SCHEMA** on the table's parent + schema, and **SELECT** on the table. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **MANAGE** and **SELECT** on the table. Workspace assets, such as the dashboard, will be created in the workspace where this call was made. - For the `schema` `object_type`, the caller must have either of the - following sets of permissions: 1. + For the ``schema`` ``object_type``, the caller must have either of the + following sets of permissions: + - **MANAGE** and **USE_CATALOG** on the schema's parent catalog. 2. - **USE_CATALOG** on the schema's + 1. **MANAGE** and **USE_CATALOG** on the schema's parent catalog. - parent catalog, and **MANAGE** and **USE_SCHEMA** on the schema. + 2. **USE_CATALOG** on the schema's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the schema. :param monitor: :class:`Monitor` @@ -179,27 +180,34 @@ paths: type: string description: >- The type of the monitored object. Can be one of the - following: `schema` or `table`. + following: ``schema`` or ``table``. object_id: type: string description: >- - The UUID of the request object. It is `schema_id` for - `schema`, and `table_id` for `table`. Find the - `schema_id` from either: 1. The [schema_id] of the - `Schemas` resource. 2. In [Catalog Explorer] > select - the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The - [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab - > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for + ``schema``, and ``table_id`` for ``table``. Find the + ``schema_id`` from either: 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ + > select the ``schema`` > go to the ``Details`` tab > + the ``Schema ID`` field. Find the ``table_id`` from + either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ + > select the ``table`` > go to the ``Details`` tab > the + ``Table ID`` field. anomaly_detection_config: type: object properties: + anomaly_detection_workflow_id: + type: integer + description: >- + The id of the workflow that detects the anomaly. + This field will only be returned in the Get/Update + response, if the request comes from the workspace + where this anomaly detection job is created. excluded_table_full_names: type: array items: @@ -207,9 +215,58 @@ paths: description: >- List of fully qualified table names to exclude from anomaly detection. + job_type: + type: string + x-enum: + - ANOMALY_DETECTION_JOB_TYPE_INTERNAL_HIDDEN + - ANOMALY_DETECTION_JOB_TYPE_NORMAL + description: The type of the last run of the workflow. + publish_health_indicator: + type: boolean + description: If the health indicator should be shown. + validity_check_configurations: + type: array + items: + type: object + properties: + name: + type: string + percent_null_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + upper_bound: + type: number + description: >- + Optional upper bound; we should use auto + determined bounds for now + range_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + lower_bound: + type: number + description: Lower bound for the range + upper_bound: + type: number + description: Upper bound for the range + uniqueness_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + description: Validity check configurations for anomaly detection. description: >- - Anomaly Detection Configuration, applicable to `schema` - object types. + Anomaly Detection Configuration, applicable to + ``schema`` object types. data_profiling_config: type: object properties: @@ -228,7 +285,7 @@ paths: description: >- Baseline table name. Baseline data is used to compute drift from the data in the monitored - `table_name`. The baseline table and the monitored + ``table_name``. The baseline table and the monitored table shall have the same schema. custom_metrics: type: array @@ -243,9 +300,8 @@ paths: description: >- Jinja template for a SQL expression that specifies how to compute the metric. See - [create metric definition]. [create metric - definition]: - https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition + `create metric definition + `__. input_columns: type: array items: @@ -283,7 +339,7 @@ paths: type: string description: >- Table that stores drift metrics data. Format: - `catalog.schema.table_name`. + ``catalog.schema.table_name``. effective_warehouse_id: type: string description: The warehouse for dashboard creation @@ -329,6 +385,9 @@ paths: label_column: type: string description: Column for the label. + prediction_probability_column: + type: string + description: Column for prediction probabilities required: - problem_type - timestamp_column @@ -336,7 +395,7 @@ paths: - prediction_column - model_id_column description: >- - `Analysis Configuration` for monitoring inference + ``Analysis Configuration`` for monitoring inference log tables. latest_monitor_failure_message: type: string @@ -353,7 +412,7 @@ paths: type: string description: >- Unity Catalog table to monitor. Format: - `catalog.schema.table_name` + ``catalog.schema.table_name`` notification_settings: type: object properties: @@ -376,7 +435,7 @@ paths: type: string description: >- Table that stores profile metrics data. Format: - `catalog.schema.table_name`. + ``catalog.schema.table_name``. schedule: type: object properties: @@ -384,8 +443,8 @@ paths: type: string description: >- The expression that determines when to run the - monitor. See [examples]. [examples]: - https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html + monitor. See `examples + `__. timezone_id: type: string description: >- @@ -422,17 +481,17 @@ paths: targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For - example `slicing_exprs=[“col_1”, “col_2 > 10”]` will - generate the following slices: two slices for `col_2 - > 10` (True and False), and one slice per unique - value in `col1`. For high-cardinality columns, only - the top 100 unique values by frequency will generate - slices. + example ``slicing_exprs=[“col_1”, “col_2 > 10”]`` + will generate the following slices: two slices for + ``col_2 > 10`` (True and False), and one slice per + unique value in ``col1``. For high-cardinality + columns, only the top 100 unique values by frequency + will generate slices. snapshot: type: object properties: {} description: >- - `Analysis Configuration` for monitoring snapshot + ``Analysis Configuration`` for monitoring snapshot tables. status: type: string @@ -474,8 +533,8 @@ paths: - timestamp_column - granularities description: >- - `Analysis Configuration` for monitoring time series - tables. + ``Analysis Configuration`` for monitoring time + series tables. warehouse_id: type: string description: >- @@ -485,9 +544,9 @@ paths: required: - output_schema_id description: >- - Data Profiling Configuration, applicable to `table` - object types. Exactly one `Analysis Configuration` must - be present. + Data Profiling Configuration, applicable to ``table`` + object types. Exactly one ``Analysis Configuration`` + must be present. required: - object_type - object_id @@ -560,45 +619,48 @@ paths: post: operationId: data_quality_create_refresh summary: >- - Creates a refresh. Currently only supported for the `table` - `object_type`. The call must be made in + Creates a refresh. Currently only supported for the ``table`` + ``object_type``. The call must be made tags: - dataquality - data_quality description: >- - Creates a refresh. Currently only supported for the `table` - `object_type`. The call must be made in - - the same workspace as where the monitor was created. + Creates a refresh. Currently only supported for the ``table`` + ``object_type``. The call must be made + in the same workspace as where the monitor was created. - The caller must have either of the following sets of permissions: 1. - **MANAGE** and **USE_CATALOG** on - the table's parent catalog. 2. **USE_CATALOG** on the table's parent - catalog, and **MANAGE** and + The caller must have either of the following sets of permissions: - **USE_SCHEMA** on the table's parent schema. 3. **USE_CATALOG** on the - table's parent catalog, - **USE_SCHEMA** on the table's parent schema, and **MANAGE** on the - table. + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog. + 2. **USE_CATALOG** on the table's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the table's + parent schema. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **MANAGE** on the table. :param object_type: str - The type of the monitored object. Can be one of the following: `schema`or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param refresh: :class:`Refresh` The refresh to create @@ -611,24 +673,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema`or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. requestBody: content: application/json: @@ -642,24 +705,24 @@ paths: type: string description: >- The type of the monitored object. Can be one of the - following: `schema`or `table`. + following: ``schema`` or ``table``. object_id: type: string description: >- - The UUID of the request object. It is `schema_id` for - `schema`, and `table_id` for `table`. Find the - `schema_id` from either: 1. The [schema_id] of the - `Schemas` resource. 2. In [Catalog Explorer] > select - the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The - [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab - > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for + ``schema``, and ``table_id`` for ``table``. Find the + ``schema_id`` from either: 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ + > select the ``schema`` > go to the ``Details`` tab > + the ``Schema ID`` field. Find the ``table_id`` from + either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ + > select the ``table`` > go to the ``Details`` tab > the + ``Table ID`` field. end_time_ms: type: integer description: >- @@ -736,42 +799,47 @@ paths: was created. - For the `table` `object_type`, the caller must have either of the - following sets of permissions: 1. + For the ``table`` ``object_type``, the caller must have either of the + following sets of permissions: - **MANAGE** and **USE_CATALOG** on the table's parent catalog. 2. - **USE_CATALOG** on the table's parent - catalog, and **MANAGE** and **USE_SCHEMA** on the table's parent schema. - 3. **USE_CATALOG** on the + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog. - table's parent catalog, **USE_SCHEMA** on the table's parent schema, and - **SELECT** on the table. + 2. **USE_CATALOG** on the table's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the table's + parent schema. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **SELECT** on the table. + For the ``schema`` ``object_type``, the caller must have either of the + following sets of permissions: - For the `schema` `object_type`, the caller must have either of the - following sets of permissions: 1. - **MANAGE** and **USE_CATALOG** on the schema's parent catalog. 2. - **USE_CATALOG** on the schema's + 1. **MANAGE** and **USE_CATALOG** on the schema's parent catalog. - parent catalog, and **USE_SCHEMA** on the schema. + 2. **USE_CATALOG** on the schema's parent catalog, and **USE_SCHEMA** on + the schema. :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param page_size: int (optional) :param page_token: str (optional) @@ -786,24 +854,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. - name: page_size in: query required: false @@ -844,7 +913,7 @@ paths: Delete a data quality monitor on Unity Catalog object. - For the `table` `object_type`, the caller must have either of the + For the ``table`` ``object_type``, the caller must have either of the following sets of permissions: **MANAGE** and **USE_CATALOG** on the table's parent catalog. @@ -863,29 +932,34 @@ paths: be manually cleaned up (if desired). - For the `schema` `object_type`, the caller must have either of the - following sets of permissions: 1. + For the ``schema`` ``object_type``, the caller must have either of the + following sets of permissions: + - **MANAGE** and **USE_CATALOG** on the schema's parent catalog. 2. - **USE_CATALOG** on the schema's + 1. **MANAGE** and **USE_CATALOG** on the schema's parent catalog. - parent catalog, and **MANAGE** and **USE_SCHEMA** on the schema. + 2. **USE_CATALOG** on the schema's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the schema. :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. parameters: - name: object_type in: path @@ -894,24 +968,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. responses: '200': description: Success @@ -937,26 +1012,27 @@ paths: Read a data quality monitor on a Unity Catalog object. - For the `table` `object_type`, the caller must have either of the - following sets of permissions: 1. + For the ``table`` ``object_type``, the caller must have either of the + following sets of permissions: - **MANAGE** and **USE_CATALOG** on the table's parent catalog. 2. - **USE_CATALOG** on the table's parent - catalog, and **MANAGE** and **USE_SCHEMA** on the table's parent schema. - 3. **USE_CATALOG** on the + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog. - table's parent catalog, **USE_SCHEMA** on the table's parent schema, and - **SELECT** on the table. + 2. **USE_CATALOG** on the table's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the table's + parent schema. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **SELECT** on the table. + For the ``schema`` ``object_type``, the caller must have either of the + following sets of permissions: - For the `schema` `object_type`, the caller must have either of the - following sets of permissions: 1. - **MANAGE** and **USE_CATALOG** on the schema's parent catalog. 2. - **USE_CATALOG** on the schema's + 1. **MANAGE** and **USE_CATALOG** on the schema's parent catalog. - parent catalog, and **USE_SCHEMA** on the schema. + 2. **USE_CATALOG** on the schema's parent catalog, and **USE_SCHEMA** on + the schema. The returned information includes configuration values on the entity and @@ -970,19 +1046,23 @@ paths: :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :returns: :class:`Monitor` parameters: @@ -993,24 +1073,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. responses: '200': description: Success @@ -1040,47 +1121,52 @@ paths: Update a data quality monitor on Unity Catalog object. - For the `table` `object_type`, the caller must have either of the - following sets of permissions: 1. + For the ``table`` ``object_type``, the caller must have either of the + following sets of permissions: - **MANAGE** and **USE_CATALOG** on the table's parent catalog. 2. - **USE_CATALOG** on the table's parent - catalog, and **MANAGE** and **USE_SCHEMA** on the table's parent schema. - 3. **USE_CATALOG** on the + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog. - table's parent catalog, **USE_SCHEMA** on the table's parent schema, and - **MANAGE** on the table. + 2. **USE_CATALOG** on the table's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the table's + parent schema. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **MANAGE** on the table. + For the ``schema`` ``object_type``, the caller must have either of the + following sets of permissions: - For the `schema` `object_type`, the caller must have either of the - following sets of permissions: 1. - **MANAGE** and **USE_CATALOG** on the schema's parent catalog. 2. - **USE_CATALOG** on the schema's + 1. **MANAGE** and **USE_CATALOG** on the schema's parent catalog. - parent catalog, and **MANAGE** and **USE_SCHEMA** on the schema. + 2. **USE_CATALOG** on the schema's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the schema. :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + Find the ``schema_id`` from either: - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + Find the ``table_id`` from either: + + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param monitor: :class:`Monitor` The monitor to update. :param update_mask: str The field mask to specify which fields to update as a comma-separated list. Example value: - `data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression` + ``data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression`` :returns: :class:`Monitor` parameters: @@ -1091,24 +1177,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. - name: update_mask in: query required: true @@ -1117,7 +1204,7 @@ paths: description: >- The field mask to specify which fields to update as a comma-separated list. Example value: - `data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression` + ``data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression`` requestBody: content: application/json: @@ -1131,27 +1218,34 @@ paths: type: string description: >- The type of the monitored object. Can be one of the - following: `schema` or `table`. + following: ``schema`` or ``table``. object_id: type: string description: >- - The UUID of the request object. It is `schema_id` for - `schema`, and `table_id` for `table`. Find the - `schema_id` from either: 1. The [schema_id] of the - `Schemas` resource. 2. In [Catalog Explorer] > select - the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The - [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab - > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for + ``schema``, and ``table_id`` for ``table``. Find the + ``schema_id`` from either: 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ + > select the ``schema`` > go to the ``Details`` tab > + the ``Schema ID`` field. Find the ``table_id`` from + either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ + > select the ``table`` > go to the ``Details`` tab > the + ``Table ID`` field. anomaly_detection_config: type: object properties: + anomaly_detection_workflow_id: + type: integer + description: >- + The id of the workflow that detects the anomaly. + This field will only be returned in the Get/Update + response, if the request comes from the workspace + where this anomaly detection job is created. excluded_table_full_names: type: array items: @@ -1159,9 +1253,58 @@ paths: description: >- List of fully qualified table names to exclude from anomaly detection. + job_type: + type: string + x-enum: + - ANOMALY_DETECTION_JOB_TYPE_INTERNAL_HIDDEN + - ANOMALY_DETECTION_JOB_TYPE_NORMAL + description: The type of the last run of the workflow. + publish_health_indicator: + type: boolean + description: If the health indicator should be shown. + validity_check_configurations: + type: array + items: + type: object + properties: + name: + type: string + percent_null_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + upper_bound: + type: number + description: >- + Optional upper bound; we should use auto + determined bounds for now + range_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + lower_bound: + type: number + description: Lower bound for the range + upper_bound: + type: number + description: Upper bound for the range + uniqueness_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + description: Validity check configurations for anomaly detection. description: >- - Anomaly Detection Configuration, applicable to `schema` - object types. + Anomaly Detection Configuration, applicable to + ``schema`` object types. data_profiling_config: type: object properties: @@ -1180,7 +1323,7 @@ paths: description: >- Baseline table name. Baseline data is used to compute drift from the data in the monitored - `table_name`. The baseline table and the monitored + ``table_name``. The baseline table and the monitored table shall have the same schema. custom_metrics: type: array @@ -1195,9 +1338,8 @@ paths: description: >- Jinja template for a SQL expression that specifies how to compute the metric. See - [create metric definition]. [create metric - definition]: - https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition + `create metric definition + `__. input_columns: type: array items: @@ -1235,7 +1377,7 @@ paths: type: string description: >- Table that stores drift metrics data. Format: - `catalog.schema.table_name`. + ``catalog.schema.table_name``. effective_warehouse_id: type: string description: The warehouse for dashboard creation @@ -1281,6 +1423,9 @@ paths: label_column: type: string description: Column for the label. + prediction_probability_column: + type: string + description: Column for prediction probabilities required: - problem_type - timestamp_column @@ -1288,7 +1433,7 @@ paths: - prediction_column - model_id_column description: >- - `Analysis Configuration` for monitoring inference + ``Analysis Configuration`` for monitoring inference log tables. latest_monitor_failure_message: type: string @@ -1305,7 +1450,7 @@ paths: type: string description: >- Unity Catalog table to monitor. Format: - `catalog.schema.table_name` + ``catalog.schema.table_name`` notification_settings: type: object properties: @@ -1328,7 +1473,7 @@ paths: type: string description: >- Table that stores profile metrics data. Format: - `catalog.schema.table_name`. + ``catalog.schema.table_name``. schedule: type: object properties: @@ -1336,8 +1481,8 @@ paths: type: string description: >- The expression that determines when to run the - monitor. See [examples]. [examples]: - https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html + monitor. See `examples + `__. timezone_id: type: string description: >- @@ -1374,17 +1519,17 @@ paths: targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For - example `slicing_exprs=[“col_1”, “col_2 > 10”]` will - generate the following slices: two slices for `col_2 - > 10` (True and False), and one slice per unique - value in `col1`. For high-cardinality columns, only - the top 100 unique values by frequency will generate - slices. + example ``slicing_exprs=[“col_1”, “col_2 > 10”]`` + will generate the following slices: two slices for + ``col_2 > 10`` (True and False), and one slice per + unique value in ``col1``. For high-cardinality + columns, only the top 100 unique values by frequency + will generate slices. snapshot: type: object properties: {} description: >- - `Analysis Configuration` for monitoring snapshot + ``Analysis Configuration`` for monitoring snapshot tables. status: type: string @@ -1426,8 +1571,8 @@ paths: - timestamp_column - granularities description: >- - `Analysis Configuration` for monitoring time series - tables. + ``Analysis Configuration`` for monitoring time + series tables. warehouse_id: type: string description: >- @@ -1437,9 +1582,9 @@ paths: required: - output_schema_id description: >- - Data Profiling Configuration, applicable to `table` - object types. Exactly one `Analysis Configuration` must - be present. + Data Profiling Configuration, applicable to ``table`` + object types. Exactly one ``Analysis Configuration`` + must be present. required: - object_type - object_id @@ -1476,19 +1621,23 @@ paths: (Unimplemented) Delete a refresh :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param refresh_id: int Unique id of the refresh operation. parameters: @@ -1499,24 +1648,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. - name: refresh_id in: path required: true @@ -1553,42 +1703,47 @@ paths: created. - For the `table` `object_type`, the caller must have either of the - following sets of permissions: 1. + For the ``table`` ``object_type``, the caller must have either of the + following sets of permissions: - **MANAGE** and **USE_CATALOG** on the table's parent catalog. 2. - **USE_CATALOG** on the table's parent - catalog, and **MANAGE** and **USE_SCHEMA** on the table's parent schema. - 3. **USE_CATALOG** on the + 1. **MANAGE** and **USE_CATALOG** on the table's parent catalog. - table's parent catalog, **USE_SCHEMA** on the table's parent schema, and - **SELECT** on the table. + 2. **USE_CATALOG** on the table's parent catalog, and **MANAGE** and + **USE_SCHEMA** on the table's + parent schema. + 3. **USE_CATALOG** on the table's parent catalog, **USE_SCHEMA** on the + table's parent schema, and + **SELECT** on the table. + For the ``schema`` ``object_type``, the caller must have either of the + following sets of permissions: - For the `schema` `object_type`, the caller must have either of the - following sets of permissions: 1. - **MANAGE** and **USE_CATALOG** on the schema's parent catalog. 2. - **USE_CATALOG** on the schema's + 1. **MANAGE** and **USE_CATALOG** on the schema's parent catalog. - parent catalog, and **USE_SCHEMA** on the schema. + 2. **USE_CATALOG** on the schema's parent catalog, and **USE_SCHEMA** on + the schema. :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param refresh_id: int Unique id of the refresh operation. @@ -1601,24 +1756,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. - name: refresh_id in: path required: true @@ -1654,19 +1810,23 @@ paths: (Unimplemented) Update a refresh :param object_type: str - The type of the monitored object. Can be one of the following: `schema` or `table`. + The type of the monitored object. Can be one of the following: ``schema`` or ``table``. :param object_id: str - The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. + The UUID of the request object. It is ``schema_id`` for ``schema``, and ``table_id`` for ``table``. + + Find the ``schema_id`` from either: - Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog - Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. + 1. The `schema_id `__ of the + ``Schemas`` resource. + 2. In `Catalog Explorer `__ > select the + ``schema`` > go to the ``Details`` tab > the ``Schema ID`` field. - Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. + Find the ``table_id`` from either: - [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id + 1. The `table_id `__ of the + ``Tables`` resource. + 2. In `Catalog Explorer `__ > select the + ``table`` > go to the ``Details`` tab > the ``Table ID`` field. :param refresh_id: int Unique id of the refresh operation. :param refresh: :class:`Refresh` @@ -1683,24 +1843,25 @@ paths: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. - name: object_id in: path required: true schema: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. - name: refresh_id in: path required: true @@ -1726,24 +1887,24 @@ paths: type: string description: >- The type of the monitored object. Can be one of the - following: `schema`or `table`. + following: ``schema`` or ``table``. object_id: type: string description: >- - The UUID of the request object. It is `schema_id` for - `schema`, and `table_id` for `table`. Find the - `schema_id` from either: 1. The [schema_id] of the - `Schemas` resource. 2. In [Catalog Explorer] > select - the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The - [table_id] of the `Tables` resource. 2. In [Catalog - Explorer] > select the `table` > go to the `Details` tab - > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ - [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for + ``schema``, and ``table_id`` for ``table``. Find the + ``schema_id`` from either: 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ + > select the ``schema`` > go to the ``Details`` tab > + the ``Schema ID`` field. Find the ``table_id`` from + either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ + > select the ``table`` > go to the ``Details`` tab > the + ``Table ID`` field. end_time_ms: type: integer description: >- @@ -1810,6 +1971,12 @@ components: AnomalyDetectionConfig: type: object properties: + anomaly_detection_workflow_id: + type: integer + description: >- + The id of the workflow that detects the anomaly. This field will + only be returned in the Get/Update response, if the request comes + from the workspace where this anomaly detection job is created. excluded_table_full_names: type: array items: @@ -1817,6 +1984,17 @@ components: description: >- List of fully qualified table names to exclude from anomaly detection. + job_type: + $ref: '#/components/schemas/AnomalyDetectionJobType' + description: The type of the last run of the workflow. + publish_health_indicator: + type: boolean + description: If the health indicator should be shown. + validity_check_configurations: + type: array + items: + $ref: '#/components/schemas/ValidityCheckConfiguration' + description: Validity check configurations for anomaly detection. description: Anomaly Detection Configurations. CancelRefreshResponse: type: object @@ -1832,8 +2010,8 @@ components: type: string description: >- The expression that determines when to run the monitor. See - [examples]. [examples]: - https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html + `examples + `__. timezone_id: type: string description: >- @@ -1867,7 +2045,7 @@ components: type: string description: >- Baseline table name. Baseline data is used to compute drift from the - data in the monitored `table_name`. The baseline table and the + data in the monitored ``table_name``. The baseline table and the monitored table shall have the same schema. custom_metrics: type: array @@ -1883,13 +2061,13 @@ components: type: string description: >- Table that stores drift metrics data. Format: - `catalog.schema.table_name`. + ``catalog.schema.table_name``. effective_warehouse_id: type: string description: The warehouse for dashboard creation inference_log: $ref: '#/components/schemas/InferenceLogConfig' - description: '`Analysis Configuration` for monitoring inference log tables.' + description: '``Analysis Configuration`` for monitoring inference log tables.' latest_monitor_failure_message: type: string description: The latest error message for a monitor failure. @@ -1902,7 +2080,9 @@ components: corrupted monitor_version numbers. monitored_table_name: type: string - description: 'Unity Catalog table to monitor. Format: `catalog.schema.table_name`' + description: >- + Unity Catalog table to monitor. Format: + ``catalog.schema.table_name`` notification_settings: $ref: '#/components/schemas/NotificationSettings' description: Field for specifying notification settings. @@ -1910,7 +2090,7 @@ components: type: string description: >- Table that stores profile metrics data. Format: - `catalog.schema.table_name`. + ``catalog.schema.table_name``. schedule: $ref: '#/components/schemas/CronSchedule' description: The cron schedule. @@ -1927,19 +2107,20 @@ components: List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example - `slicing_exprs=[“col_1”, “col_2 > 10”]` will generate the following - slices: two slices for `col_2 > 10` (True and False), and one slice - per unique value in `col1`. For high-cardinality columns, only the - top 100 unique values by frequency will generate slices. + ``slicing_exprs=[“col_1”, “col_2 > 10”]`` will generate the + following slices: two slices for ``col_2 > 10`` (True and False), + and one slice per unique value in ``col1``. For high-cardinality + columns, only the top 100 unique values by frequency will generate + slices. snapshot: $ref: '#/components/schemas/SnapshotConfig' - description: '`Analysis Configuration` for monitoring snapshot tables.' + description: '``Analysis Configuration`` for monitoring snapshot tables.' status: $ref: '#/components/schemas/DataProfilingStatus' description: The data profiling monitor status. time_series: $ref: '#/components/schemas/TimeSeriesConfig' - description: '`Analysis Configuration` for monitoring time series tables.' + description: '``Analysis Configuration`` for monitoring time series tables.' warehouse_id: type: string description: >- @@ -1958,9 +2139,8 @@ components: type: string description: >- Jinja template for a SQL expression that specifies how to compute - the metric. See [create metric definition]. [create metric - definition]: - https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html#create-definition + the metric. See `create metric definition + `__. input_columns: type: array items: @@ -2007,6 +2187,9 @@ components: label_column: type: string description: Column for the label. + prediction_probability_column: + type: string + description: Column for prediction probabilities required: - problem_type - timestamp_column @@ -2041,31 +2224,32 @@ components: type: string description: >- The type of the monitored object. Can be one of the following: - `schema` or `table`. + ``schema`` or ``table``. object_id: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. anomaly_detection_config: $ref: '#/components/schemas/AnomalyDetectionConfig' description: >- - Anomaly Detection Configuration, applicable to `schema` object + Anomaly Detection Configuration, applicable to ``schema`` object types. data_profiling_config: $ref: '#/components/schemas/DataProfilingConfig' description: >- - Data Profiling Configuration, applicable to `table` object types. - Exactly one `Analysis Configuration` must be present. + Data Profiling Configuration, applicable to ``table`` object types. + Exactly one ``Analysis Configuration`` must be present. required: - object_type - object_id @@ -2090,6 +2274,29 @@ components: $ref: '#/components/schemas/NotificationDestination' description: Destinations to send notifications on failure/timeout. description: Settings for sending notifications on the data quality monitoring. + PercentNullValidityCheck: + type: object + properties: + column_names: + type: array + items: + type: string + upper_bound: + type: number + description: Optional upper bound; we should use auto determined bounds for now + RangeValidityCheck: + type: object + properties: + column_names: + type: array + items: + type: string + lower_bound: + type: number + description: Lower bound for the range + upper_bound: + type: number + description: Upper bound for the range Refresh: type: object properties: @@ -2097,21 +2304,22 @@ components: type: string description: >- The type of the monitored object. Can be one of the following: - `schema`or `table`. + ``schema`` or ``table``. object_id: type: string description: >- - The UUID of the request object. It is `schema_id` for `schema`, and - `table_id` for `table`. Find the `schema_id` from either: 1. The - [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > - select the `schema` > go to the `Details` tab > the `Schema ID` - field. Find the `table_id` from either: 1. The [table_id] of the - `Tables` resource. 2. In [Catalog Explorer] > select the `table` > - go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: - https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: - https://docs.databricks.com/api/workspace/schemas/get#schema_id - [table_id]: - https://docs.databricks.com/api/workspace/tables/get#table_id + The UUID of the request object. It is ``schema_id`` for ``schema``, + and ``table_id`` for ``table``. Find the ``schema_id`` from either: + 1. The `schema_id + `__ + of the ``Schemas`` resource. 2. In `Catalog Explorer + `__ > select + the ``schema`` > go to the ``Details`` tab > the ``Schema ID`` + field. Find the ``table_id`` from either: 1. The `table_id + `__ + of the ``Tables`` resource. 2. In `Catalog Explorer + `__ > select + the ``table`` > go to the ``Details`` tab > the ``Table ID`` field. end_time_ms: type: integer description: Time when the refresh ended (milliseconds since 1/1/1970 UTC). @@ -2159,6 +2367,24 @@ components: - timestamp_column - granularities description: Time series analysis configuration. + UniquenessValidityCheck: + type: object + properties: + column_names: + type: array + items: + type: string + ValidityCheckConfiguration: + type: object + properties: + name: + type: string + percent_null_validity_check: + $ref: '#/components/schemas/PercentNullValidityCheck' + range_validity_check: + $ref: '#/components/schemas/RangeValidityCheck' + uniqueness_validity_check: + $ref: '#/components/schemas/UniquenessValidityCheck' AggregationGranularity: type: string x-enum: @@ -2175,6 +2401,12 @@ components: description: >- The granularity for aggregating data into time windows based on their timestamp. + AnomalyDetectionJobType: + type: string + x-enum: + - ANOMALY_DETECTION_JOB_TYPE_INTERNAL_HIDDEN + - ANOMALY_DETECTION_JOB_TYPE_NORMAL + description: Anomaly Detection job type. CronSchedulePauseStatus: type: string x-enum: diff --git a/providers/src/databricks_workspace/v00.00.00000/services/files.yaml b/providers/src/databricks_workspace/v00.00.00000/services/files.yaml index b2111dea..e0e7feeb 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/files.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/files.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks files service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.files servers: - url: https://{deployment_name}.cloud.databricks.com @@ -129,7 +129,7 @@ paths: There is a 10 minute idle timeout on this handle. If a file or directory already exists on the - given path and __overwrite__ is + given path and **overwrite** is set to false, this call will throw an exception with ``RESOURCE_ALREADY_EXISTS``. @@ -138,10 +138,11 @@ paths: A typical workflow for file upload would be: - 1. Issue a ``create`` call and get a handle. 2. Issue one or more - ``add-block`` calls with the handle + 1. Issue a ``create`` call and get a handle. - you have. 3. Issue a ``close`` call with the handle you have. + 2. Issue one or more ``add-block`` calls with the handle you have. + + 3. Issue a ``close`` call with the handle you have. :param path: str @@ -200,10 +201,10 @@ paths: Delete the file or directory (optionally recursively delete all files in the directory). This call - throws an exception with `IO_ERROR` if the path is a non-empty directory - and `recursive` is set to + throws an exception with ``IO_ERROR`` if the path is a non-empty + directory and ``recursive`` is set to - `false` or on other similar errors. + ``false`` or on other similar errors. When you delete a large number of files, the delete operation is done in @@ -222,8 +223,8 @@ paths: to perform such operations in the context of a cluster, using the [File system utility - (dbutils.fs)](/dev-tools/databricks-utils.html#dbutils-fs). `dbutils.fs` - covers the functional scope + (dbutils.fs)](/dev-tools/databricks-utils.html#dbutils-fs). + ``dbutils.fs`` covers the functional scope of the DBFS REST API, but from notebooks. Running such operations using notebooks provides better @@ -286,7 +287,7 @@ paths: Gets the file information for a file or directory. If the file or directory does not exist, this call - throws an exception with `RESOURCE_DOES_NOT_EXIST`. + throws an exception with ``RESOURCE_DOES_NOT_EXIST``. :param path: str @@ -334,7 +335,7 @@ paths: List the contents of a directory, or details of the file. If the file or directory does not exist, - this call throws an exception with `RESOURCE_DOES_NOT_EXIST`. + this call throws an exception with ``RESOURCE_DOES_NOT_EXIST``. When calling list on a large directory, the list operation will time out @@ -403,10 +404,10 @@ paths: directory) exists at any prefix of the input path, this call throws an exception with - `RESOURCE_ALREADY_EXISTS`. **Note**: If this operation fails, it might - have succeeded in creating some + ``RESOURCE_ALREADY_EXISTS``. **Note**: If this operation fails, it might + have succeeded in creating - of the necessary parent directories. + some of the necessary parent directories. :param path: str @@ -452,11 +453,11 @@ paths: Moves a file from one location to another location within DBFS. If the source file does not exist, - this call throws an exception with `RESOURCE_DOES_NOT_EXIST`. If a file - already exists in the + this call throws an exception with ``RESOURCE_DOES_NOT_EXIST``. If a + file already exists in the destination path, this call throws an exception with - `RESOURCE_ALREADY_EXISTS`. If the given source + ``RESOURCE_ALREADY_EXISTS``. If the given source path is a directory, this call always recursively moves all files. @@ -519,9 +520,9 @@ paths: The amount of data that can be passed (when not streaming) using the - __contents__ parameter is limited + **contents** parameter is limited - to 1 MB. `MAX_BLOCK_SIZE_EXCEEDED` will be thrown if this limit is + to 1 MB. ``MAX_BLOCK_SIZE_EXCEEDED`` will be thrown if this limit is exceeded. @@ -588,17 +589,18 @@ paths: Returns the contents of a file. If the file does not exist, this call throws an exception with - `RESOURCE_DOES_NOT_EXIST`. If the path is a directory, the read length + ``RESOURCE_DOES_NOT_EXIST``. If the path is a directory, the read length is negative, or if the offset is negative, this call throws an exception with - `INVALID_PARAMETER_VALUE`. If the read length exceeds + ``INVALID_PARAMETER_VALUE``. If the read length - 1 MB, this call throws an exception with `MAX_READ_SIZE_EXCEEDED`. + exceeds 1 MB, this call throws an exception with + ``MAX_READ_SIZE_EXCEEDED``. - If `offset + length` exceeds the number of bytes in a file, it reads the - contents until the end of + If ``offset + length`` exceeds the number of bytes in a file, it reads + the contents until the end of file. @@ -666,8 +668,8 @@ paths: Creates an empty directory. If necessary, also creates any parent directories of the new, empty - directory (like the shell command `mkdir -p`). If called on an existing - directory, returns a success + directory (like the shell command ``mkdir -p``). If called on an + existing directory, returns a success response; this method is idempotent (it will succeed if the directory already exists). @@ -754,11 +756,11 @@ paths: access to it. - If you wish to ensure the directory exists, you can instead use `PUT`, - which will create the directory + If you wish to ensure the directory exists, you can instead use ``PUT``, + which will create the - if it does not exist, and is idempotent (it will succeed if the - directory already exists). + directory if it does not exist, and is idempotent (it will succeed if + the directory already exists). :param directory_path: str @@ -789,21 +791,21 @@ paths: operationId: files_list_directory_contents summary: >- Returns the contents of a directory. If there is no directory at the - specified path, the API returns a + specified path, the API returns tags: - files description: >- Returns the contents of a directory. If there is no directory at the - specified path, the API returns a + specified path, the API returns - HTTP 404 error. + an HTTP 404 error. :param directory_path: str The absolute path of a directory. :param page_size: int (optional) The maximum number of directory entries to return. The response may contain fewer entries. If the - response contains a `next_page_token`, there may be more entries, even if fewer than `page_size` + response contains a ``next_page_token``, there may be more entries, even if fewer than ``page_size`` entries are in the response. We recommend not to set this value unless you are intentionally listing less than the complete @@ -812,12 +814,12 @@ paths: If unspecified, at most 1000 directory entries will be returned. The maximum value is 1000. Values above 1000 will be coerced to 1000. :param page_token: str (optional) - An opaque page token which was the `next_page_token` in the response of the previous request to list - the contents of this directory. Provide this token to retrieve the next page of directory entries. - When providing a `page_token`, all other parameters provided to the request must match the previous - request. To list all of the entries in a directory, it is necessary to continue requesting pages of - entries until the response contains no `next_page_token`. Note that the number of entries returned - must not be used to determine when the listing is complete. + An opaque page token which was the ``next_page_token`` in the response of the previous request to + list the contents of this directory. Provide this token to retrieve the next page of directory + entries. When providing a ``page_token``, all other parameters provided to the request must match + the previous request. To list all of the entries in a directory, it is necessary to continue + requesting pages of entries until the response contains no ``next_page_token``. Note that the number + of entries returned must not be used to determine when the listing is complete. :returns: Iterator over :class:`DirectoryEntry` parameters: @@ -834,27 +836,28 @@ paths: type: integer description: >- The maximum number of directory entries to return. The response may - contain fewer entries. If the response contains a `next_page_token`, - there may be more entries, even if fewer than `page_size` entries - are in the response. We recommend not to set this value unless you - are intentionally listing less than the complete directory contents. - If unspecified, at most 1000 directory entries will be returned. The - maximum value is 1000. Values above 1000 will be coerced to 1000. + contain fewer entries. If the response contains a + ``next_page_token``, there may be more entries, even if fewer than + ``page_size`` entries are in the response. We recommend not to set + this value unless you are intentionally listing less than the + complete directory contents. If unspecified, at most 1000 directory + entries will be returned. The maximum value is 1000. Values above + 1000 will be coerced to 1000. - name: page_token in: query required: false schema: type: string description: >- - An opaque page token which was the `next_page_token` in the response - of the previous request to list the contents of this directory. - Provide this token to retrieve the next page of directory entries. - When providing a `page_token`, all other parameters provided to the - request must match the previous request. To list all of the entries - in a directory, it is necessary to continue requesting pages of - entries until the response contains no `next_page_token`. Note that - the number of entries returned must not be used to determine when - the listing is complete. + An opaque page token which was the ``next_page_token`` in the + response of the previous request to list the contents of this + directory. Provide this token to retrieve the next page of directory + entries. When providing a ``page_token``, all other parameters + provided to the request must match the previous request. To list all + of the entries in a directory, it is necessary to continue + requesting pages of entries until the response contains no + ``next_page_token``. Note that the number of entries returned must + not be used to determine when the listing is complete. responses: '200': description: Success @@ -1147,7 +1150,7 @@ components: next_page_token: type: string description: >- - A token, which can be sent as `page_token` to retrieve the next + A token, which can be sent as ``page_token`` to retrieve the next page. ListStatusResponse: type: object diff --git a/providers/src/databricks_workspace/v00.00.00000/services/iam.yaml b/providers/src/databricks_workspace/v00.00.00000/services/iam.yaml index 7a32e3bf..6ae202c9 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/iam.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/iam.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks iam service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.iam servers: - url: https://{deployment_name}.cloud.databricks.com @@ -100,10 +100,10 @@ paths: :param resource: str The resource name for which assignable roles will be listed. - Examples | Summary :--- | :--- `resource=accounts/` | A resource name for the account. - `resource=accounts//groups/` | A resource name for the group. - `resource=accounts//servicePrincipals/` | A resource name for the service - principal. `resource=accounts//tagPolicies/` | A resource name for the + Examples | Summary :--- | :--- ``resource=accounts/`` | A resource name for the account. + ``resource=accounts//groups/`` | A resource name for the group. + ``resource=accounts//servicePrincipals/`` | A resource name for the service + principal. ``resource=accounts//tagPolicies/`` | A resource name for the tag policy. :returns: :class:`GetAssignableRolesForResourceResponse` @@ -113,10 +113,10 @@ paths: required: true schema: type: string - description: The resource name for which assignable roles will be listed. Examples | Summary :--- | :--- `resource=accounts/` - | A resource name for the account. `resource=accounts//groups/` | A resource name for the - group. `resource=accounts//servicePrincipals/` | A resource name for the service principal. `resource=accounts//tagPolicies/` - | A resource name for the tag policy. + description: The resource name for which assignable roles will be listed. Examples | Summary :--- | :--- ``resource=accounts/`` + | A resource name for the account. ``resource=accounts//groups/`` | A resource name for the + group. ``resource=accounts//servicePrincipals/`` | A resource name for the service principal. + ``resource=accounts//tagPolicies/`` | A resource name for the tag policy. responses: '200': description: Success @@ -150,13 +150,13 @@ paths: :param name: str The ruleset name associated with the request. - Examples | Summary :--- | :--- `name=accounts//ruleSets/default` | A name for a rule set - on the account. `name=accounts//groups//ruleSets/default` | A name for a rule - set on the group. - `name=accounts//servicePrincipals//ruleSets/default` | - A name for a rule set on the service principal. - `name=accounts//tagPolicies//ruleSets/default` | A name for a rule set on - the tag policy. + Examples | Summary :--- | :--- ``name=accounts//ruleSets/default`` | A name for a rule + set on the account. ``name=accounts//groups//ruleSets/default`` | A name for a + rule set on the group. + ``name=accounts//servicePrincipals//ruleSets/default`` + | A name for a rule set on the service principal. + ``name=accounts//tagPolicies//ruleSets/default`` | A name for a rule set + on the tag policy. :param etag: str Etag used for versioning. The response is at least as fresh as the eTag provided. Etag is used for optimistic concurrency control as a way to help prevent simultaneous updates of a rule set from @@ -165,8 +165,8 @@ paths: etag from a GET rule set request, and pass it with the PUT update request to identify the rule set version you are updating. - Examples | Summary :--- | :--- `etag=` | An empty etag can only be used in GET to indicate no - freshness requirements. `etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==` | An + Examples | Summary :--- | :--- ``etag=`` | An empty etag can only be used in GET to indicate no + freshness requirements. ``etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==`` | An etag encoded a specific version of the rule set to get or to be updated. :returns: :class:`RuleSetResponse` @@ -176,10 +176,10 @@ paths: required: true schema: type: string - description: The ruleset name associated with the request. Examples | Summary :--- | :--- `name=accounts//ruleSets/default` - | A name for a rule set on the account. `name=accounts//groups//ruleSets/default` | A name - for a rule set on the group. `name=accounts//servicePrincipals//ruleSets/default` - | A name for a rule set on the service principal. `name=accounts//tagPolicies//ruleSets/default` + description: The ruleset name associated with the request. Examples | Summary :--- | :--- ``name=accounts//ruleSets/default`` + | A name for a rule set on the account. ``name=accounts//groups//ruleSets/default`` | A name + for a rule set on the group. ``name=accounts//servicePrincipals//ruleSets/default`` + | A name for a rule set on the service principal. ``name=accounts//tagPolicies//ruleSets/default`` | A name for a rule set on the tag policy. - name: etag in: query @@ -190,8 +190,8 @@ paths: concurrency control as a way to help prevent simultaneous updates of a rule set from overwriting each other. It is strongly suggested that systems make use of the etag in the read -> modify -> write pattern to perform rule set updates in order to avoid race conditions that is get an etag from a GET rule set request, and pass it with the - PUT update request to identify the rule set version you are updating. Examples | Summary :--- | :--- `etag=` | An - empty etag can only be used in GET to indicate no freshness requirements. `etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==` + PUT update request to identify the rule set version you are updating. Examples | Summary :--- | :--- ``etag=`` | + An empty etag can only be used in GET to indicate no freshness requirements. ``etag=RENUAAABhSweA4NvVmmUYdiU717H3Tgy0UJdor3gE4a+mq/oj9NjAf8ZsQ==`` | An etag encoded a specific version of the rule set to get or to be updated. responses: '200': @@ -262,7 +262,7 @@ paths: type: string description: 'Principals this grant rule applies to. A principal can be a user (for end users), a service principal (for applications and compute workloads), or an account group. Each principal - has its own identifier format: * users/ * groups/ * servicePrincipals/' + has its own identifier format: - users/ - groups/ - servicePrincipals/' required: - role required: @@ -300,8 +300,25 @@ paths: description: |- Get details about the current method caller's identity. + :param attributes: str (optional) + Comma-separated list of attributes to return in response. + :param excluded_attributes: str (optional) + Comma-separated list of attributes to exclude in response. :returns: :class:`User` + parameters: + - name: attributes + in: query + required: false + schema: + type: string + description: Comma-separated list of attributes to return in response. + - name: excluded_attributes + in: query + required: false + schema: + type: string + description: Comma-separated list of attributes to exclude in response. responses: '200': description: Success @@ -334,10 +351,9 @@ paths: :param display_name: str (optional) String that represents a human-readable group name :param entitlements: List[:class:`ComplexValue`] (optional) - Entitlements assigned to the group. See [assigning entitlements] for a full list of supported - values. - - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements + Entitlements assigned to the group. See `assigning entitlements + `__ + for a full list of supported values. :param external_id: str (optional) :param groups: List[:class:`ComplexValue`] (optional) :param id: str (optional) @@ -375,8 +391,8 @@ paths: type: string value: type: string - description: 'Entitlements assigned to the group. See [assigning entitlements] for a full list of supported - values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the group. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string groups: @@ -513,12 +529,11 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) Attribute to sort the results. :param sort_order: :class:`ListSortOrder` (optional) @@ -551,9 +566,10 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false @@ -782,10 +798,9 @@ paths: :param display_name: str (optional) String that represents a human-readable group name :param entitlements: List[:class:`ComplexValue`] (optional) - Entitlements assigned to the group. See [assigning entitlements] for a full list of supported - values. - - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements + Entitlements assigned to the group. See `assigning entitlements + `__ + for a full list of supported values. :param external_id: str (optional) :param groups: List[:class:`ComplexValue`] (optional) :param members: List[:class:`ComplexValue`] (optional) @@ -826,8 +841,8 @@ paths: type: string value: type: string - description: 'Entitlements assigned to the group. See [assigning entitlements] for a full list of supported - values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the group. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string groups: @@ -1016,8 +1031,9 @@ paths: :param request_object_type: str The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, - experiments, files, genie, instance-pools, jobs, notebooks, pipelines, queries, registered-models, - repos, serving-endpoints, or warehouses. + experiments, files, genie, instance-pools, jobs, knowledge-assistants, notebooks, pipelines, + queries, registered-models, repos, serving-endpoints, supervisor-agents, vector-search-endpoints, or + warehouses. :param request_object_id: str The id of the request object. @@ -1030,7 +1046,8 @@ paths: type: string description: 'The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, experiments, files, genie, instance-pools, - jobs, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, or warehouses.' + jobs, knowledge-assistants, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, supervisor-agents, + vector-search-endpoints, or warehouses.' - name: request_object_id in: path required: true @@ -1070,8 +1087,9 @@ paths: :param request_object_type: str The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, - experiments, files, genie, instance-pools, jobs, notebooks, pipelines, queries, registered-models, - repos, serving-endpoints, or warehouses. + experiments, files, genie, instance-pools, jobs, knowledge-assistants, notebooks, pipelines, + queries, registered-models, repos, serving-endpoints, supervisor-agents, vector-search-endpoints, or + warehouses. :param request_object_id: str The id of the request object. :param access_control_list: List[:class:`AccessControlRequest`] (optional) @@ -1085,7 +1103,8 @@ paths: type: string description: 'The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, experiments, files, genie, instance-pools, - jobs, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, or warehouses.' + jobs, knowledge-assistants, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, supervisor-agents, + vector-search-endpoints, or warehouses.' - name: request_object_id in: path required: true @@ -1128,6 +1147,7 @@ paths: - CAN_VIEW - CAN_VIEW_METADATA - IS_OWNER + - UNSPECIFIED description: Permission level service_principal_name: type: string @@ -1167,8 +1187,9 @@ paths: :param request_object_type: str The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, - experiments, files, genie, instance-pools, jobs, notebooks, pipelines, queries, registered-models, - repos, serving-endpoints, or warehouses. + experiments, files, genie, instance-pools, jobs, knowledge-assistants, notebooks, pipelines, + queries, registered-models, repos, serving-endpoints, supervisor-agents, vector-search-endpoints, or + warehouses. :param request_object_id: str The id of the request object. :param access_control_list: List[:class:`AccessControlRequest`] (optional) @@ -1182,7 +1203,8 @@ paths: type: string description: 'The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, experiments, files, genie, instance-pools, - jobs, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, or warehouses.' + jobs, knowledge-assistants, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, supervisor-agents, + vector-search-endpoints, or warehouses.' - name: request_object_id in: path required: true @@ -1225,6 +1247,7 @@ paths: - CAN_VIEW - CAN_VIEW_METADATA - IS_OWNER + - UNSPECIFIED description: Permission level service_principal_name: type: string @@ -1264,8 +1287,9 @@ paths: :param request_object_type: str The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, - experiments, files, genie, instance-pools, jobs, notebooks, pipelines, queries, registered-models, - repos, serving-endpoints, or warehouses. + experiments, files, genie, instance-pools, jobs, knowledge-assistants, notebooks, pipelines, + queries, registered-models, repos, serving-endpoints, supervisor-agents, vector-search-endpoints, or + warehouses. :param request_object_id: str :returns: :class:`GetPermissionLevelsResponse` @@ -1277,7 +1301,8 @@ paths: type: string description: 'The type of the request object. Can be one of the following: alerts, alertsv2, authorization, clusters, cluster-policies, dashboards, database-projects, dbsql-dashboards, directories, experiments, files, genie, instance-pools, - jobs, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, or warehouses.' + jobs, knowledge-assistants, notebooks, pipelines, queries, registered-models, repos, serving-endpoints, supervisor-agents, + vector-search-endpoints, or warehouses.' - name: request_object_id in: path required: true @@ -1319,10 +1344,9 @@ paths: :param display_name: str (optional) String that represents a concatenation of given and family names. :param entitlements: List[:class:`ComplexValue`] (optional) - Entitlements assigned to the service principal. See [assigning entitlements] for a full list of - supported values. - - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements + Entitlements assigned to the service principal. See `assigning entitlements + `__ + for a full list of supported values. :param external_id: str (optional) :param groups: List[:class:`ComplexValue`] (optional) :param id: str (optional) @@ -1363,8 +1387,8 @@ paths: type: string value: type: string - description: 'Entitlements assigned to the service principal. See [assigning entitlements] for a full list - of supported values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the service principal. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string groups: @@ -1480,12 +1504,11 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) Attribute to sort the results. :param sort_order: :class:`ListSortOrder` (optional) @@ -1518,9 +1541,10 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false @@ -1755,10 +1779,9 @@ paths: :param display_name: str (optional) String that represents a concatenation of given and family names. :param entitlements: List[:class:`ComplexValue`] (optional) - Entitlements assigned to the service principal. See [assigning entitlements] for a full list of - supported values. - - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements + Entitlements assigned to the service principal. See `assigning entitlements + `__ + for a full list of supported values. :param external_id: str (optional) :param groups: List[:class:`ComplexValue`] (optional) :param roles: List[:class:`ComplexValue`] (optional) @@ -1802,8 +1825,8 @@ paths: type: string value: type: string - description: 'Entitlements assigned to the service principal. See [assigning entitlements] for a full list - of supported values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the service principal. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string groups: @@ -1910,17 +1933,16 @@ paths: :param active: bool (optional) If this user is active :param display_name: str (optional) - String that represents a concatenation of given and family names. For example `John Smith`. This - field cannot be updated through the Workspace SCIM APIs when [identity federation is enabled]. Use - Account SCIM APIs to update `displayName`. - - [identity federation is enabled]: https://docs.databricks.com/administration-guide/users-groups/best-practices.html#enable-identity-federation + String that represents a concatenation of given and family names. For example ``John Smith``. This + field cannot be updated through the Workspace SCIM APIs when `identity federation is enabled + `__. + Use Account SCIM APIs to update ``displayName``. :param emails: List[:class:`ComplexValue`] (optional) All the emails associated with the Databricks user. :param entitlements: List[:class:`ComplexValue`] (optional) - Entitlements assigned to the user. See [assigning entitlements] for a full list of supported values. - - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements + Entitlements assigned to the user. See `assigning entitlements + `__ + for a full list of supported values. :param external_id: str (optional) External ID is not currently supported. It is reserved for future use. :param groups: List[:class:`ComplexValue`] (optional) @@ -1946,9 +1968,9 @@ paths: description: If this user is active displayName: type: string - description: 'String that represents a concatenation of given and family names. For example `John Smith`. - This field cannot be updated through the Workspace SCIM APIs when [identity federation is enabled]. Use - Account SCIM APIs to update `displayName`. [identity federation is enabled]: https://docs.databricks.com/administration-guide/users-groups/best-practices.html#enable-identity-federation' + description: String that represents a concatenation of given and family names. For example ``John Smith``. + This field cannot be updated through the Workspace SCIM APIs when `identity federation is enabled `__. + Use Account SCIM APIs to update ``displayName``. emails: type: array items: @@ -1980,8 +2002,8 @@ paths: type: string value: type: string - description: 'Entitlements assigned to the user. See [assigning entitlements] for a full list of supported - values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the user. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string description: External ID is not currently supported. It is reserved for future use. @@ -2110,15 +2132,14 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) - Attribute to sort the results. Multi-part paths are supported. For example, `userName`, - `name.givenName`, and `emails`. + Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, + ``name.givenName``, and ``emails``. :param sort_order: :class:`ListSortOrder` (optional) The order to sort the results. :param start_index: int (optional) @@ -2149,16 +2170,17 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false schema: type: string - description: Attribute to sort the results. Multi-part paths are supported. For example, `userName`, `name.givenName`, - and `emails`. + description: Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, ``name.givenName``, + and ``emails``. - name: sort_order in: query required: false @@ -2243,15 +2265,14 @@ paths: :param excluded_attributes: str (optional) Comma-separated list of attributes to exclude in response. :param filter: str (optional) - Query by which the results have to be filtered. Supported operators are equals(`eq`), - contains(`co`), starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be - formed using logical operators - `and` and `or`. The [SCIM RFC] has more details but we currently - only support simple expressions. - - [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2 + Query by which the results have to be filtered. Supported operators are equals(``eq``), + contains(``co``), starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can + be formed using logical operators - ``and`` and ``or``. The `SCIM RFC + `__ has more details but we currently only + support simple expressions. :param sort_by: str (optional) - Attribute to sort the results. Multi-part paths are supported. For example, `userName`, - `name.givenName`, and `emails`. + Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, + ``name.givenName``, and ``emails``. :param sort_order: :class:`GetSortOrder` (optional) The order to sort the results. :param start_index: int (optional) @@ -2288,16 +2309,17 @@ paths: required: false schema: type: string - description: 'Query by which the results have to be filtered. Supported operators are equals(`eq`), contains(`co`), - starts with(`sw`) and not equals(`ne`). Additionally, simple expressions can be formed using logical operators - - `and` and `or`. The [SCIM RFC] has more details but we currently only support simple expressions. [SCIM RFC]: https://tools.ietf.org/html/rfc7644#section-3.4.2.2' + description: Query by which the results have to be filtered. Supported operators are equals(``eq``), contains(``co``), + starts with(``sw``) and not equals(``ne``). Additionally, simple expressions can be formed using logical operators + - ``and`` and ``or``. The `SCIM RFC `__ has more details but + we currently only support simple expressions. - name: sort_by in: query required: false schema: type: string - description: Attribute to sort the results. Multi-part paths are supported. For example, `userName`, `name.givenName`, - and `emails`. + description: Attribute to sort the results. Multi-part paths are supported. For example, ``userName``, ``name.givenName``, + and ``emails``. - name: sort_order in: query required: false @@ -2331,12 +2353,14 @@ paths: x-stackql-sdk-source: UsersV2API patch: operationId: users_v2_patch - summary: Partially updates a user resource by applying the supplied operations on specific user attributes. + summary: Partially updates a user resource by applying the supplied operations on specific user attributes. The tags: - iam - users_v2 description: |- - Partially updates a user resource by applying the supplied operations on specific user attributes. + Partially updates a user resource by applying the supplied operations on specific user attributes. The + ``userName`` and ``emails`` attributes cannot be updated through this API; any supplied changes to + them are ignored (no-op). :param id: str Unique ID in the Databricks workspace. @@ -2447,17 +2471,16 @@ paths: :param active: bool (optional) If this user is active :param display_name: str (optional) - String that represents a concatenation of given and family names. For example `John Smith`. This - field cannot be updated through the Workspace SCIM APIs when [identity federation is enabled]. Use - Account SCIM APIs to update `displayName`. - - [identity federation is enabled]: https://docs.databricks.com/administration-guide/users-groups/best-practices.html#enable-identity-federation + String that represents a concatenation of given and family names. For example ``John Smith``. This + field cannot be updated through the Workspace SCIM APIs when `identity federation is enabled + `__. + Use Account SCIM APIs to update ``displayName``. :param emails: List[:class:`ComplexValue`] (optional) All the emails associated with the Databricks user. :param entitlements: List[:class:`ComplexValue`] (optional) - Entitlements assigned to the user. See [assigning entitlements] for a full list of supported values. - - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements + Entitlements assigned to the user. See `assigning entitlements + `__ + for a full list of supported values. :param external_id: str (optional) External ID is not currently supported. It is reserved for future use. :param groups: List[:class:`ComplexValue`] (optional) @@ -2486,9 +2509,9 @@ paths: description: If this user is active displayName: type: string - description: 'String that represents a concatenation of given and family names. For example `John Smith`. - This field cannot be updated through the Workspace SCIM APIs when [identity federation is enabled]. Use - Account SCIM APIs to update `displayName`. [identity federation is enabled]: https://docs.databricks.com/administration-guide/users-groups/best-practices.html#enable-identity-federation' + description: String that represents a concatenation of given and family names. For example ``John Smith``. + This field cannot be updated through the Workspace SCIM APIs when `identity federation is enabled `__. + Use Account SCIM APIs to update ``displayName``. emails: type: array items: @@ -2520,8 +2543,8 @@ paths: type: string value: type: string - description: 'Entitlements assigned to the user. See [assigning entitlements] for a full list of supported - values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the user. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string description: External ID is not currently supported. It is reserved for future use. @@ -2894,12 +2917,13 @@ components: description: If this user is active displayName: type: string - description: String that represents a concatenation of given and family names. For example `John Smith`. + description: String that represents a concatenation of given and family names. For example ``John Smith``. emails: type: array items: $ref: '#/components/schemas/ComplexValue' - description: All the emails associated with the Databricks user. + description: All the emails associated with the Databricks user. This attribute cannot be updated through the SCIM + PATCH or PUT APIs; any supplied change is ignored. externalId: type: string description: External ID is not currently supported. It is reserved for future use. @@ -2915,7 +2939,8 @@ components: description: Indicates if the group has the admin role. userName: type: string - description: Email address of the Databricks user. + description: Email address of the Databricks user. This attribute cannot be updated through the SCIM PATCH or PUT + APIs; any supplied change is ignored. Actor: type: object properties: @@ -2987,8 +3012,8 @@ components: items: type: string description: 'Principals this grant rule applies to. A principal can be a user (for end users), a service principal - (for applications and compute workloads), or an account group. Each principal has its own identifier format: * - users/ * groups/ * servicePrincipals/' + (for applications and compute workloads), or an account group. Each principal has its own identifier format: - + users/ - groups/ - servicePrincipals/' required: - role Group: @@ -3000,8 +3025,8 @@ components: type: array items: $ref: '#/components/schemas/ComplexValue' - description: 'Entitlements assigned to the group. See [assigning entitlements] for a full list of supported values. - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the group. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string description: external_id should be unique for identifying groups @@ -3270,10 +3295,16 @@ components: PermissionAssignments: type: object properties: + next_page_token: + type: string permission_assignments: type: array items: $ref: '#/components/schemas/PermissionAssignment' + description: Array of permissions assignments defined for a workspace. + prev_page_token: + type: string + description: Token to retrieve the previous page of results. PermissionOutput: type: object properties: @@ -3387,8 +3418,8 @@ components: type: array items: $ref: '#/components/schemas/ComplexValue' - description: 'Entitlements assigned to the service principal. See [assigning entitlements] for a full list of supported - values. [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the service principal. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string groups: @@ -3415,20 +3446,21 @@ components: type: boolean displayName: type: string - description: 'String that represents a concatenation of given and family names. For example `John Smith`. This field - cannot be updated through the Workspace SCIM APIs when [identity federation is enabled]. Use Account SCIM APIs - to update `displayName`. [identity federation is enabled]: https://docs.databricks.com/administration-guide/users-groups/best-practices.html#enable-identity-federation' + description: String that represents a concatenation of given and family names. For example ``John Smith``. This + field cannot be updated through the Workspace SCIM APIs when `identity federation is enabled `__. + Use Account SCIM APIs to update ``displayName``. emails: type: array items: $ref: '#/components/schemas/ComplexValue' - description: All the emails associated with the Databricks user. + description: All the emails associated with the Databricks user. This attribute cannot be updated through the SCIM + PATCH or PUT APIs; any supplied change is ignored. entitlements: type: array items: $ref: '#/components/schemas/ComplexValue' - description: 'Entitlements assigned to the user. See [assigning entitlements] for a full list of supported values. - [assigning entitlements]: https://docs.databricks.com/administration-guide/users-groups/index.html#assigning-entitlements' + description: Entitlements assigned to the user. See `assigning entitlements `__ + for a full list of supported values. externalId: type: string description: External ID is not currently supported. It is reserved for future use. @@ -3453,7 +3485,8 @@ components: description: The schema of the user. userName: type: string - description: Email address of the Databricks user. + description: Email address of the Databricks user. This attribute cannot be updated through the SCIM PATCH or PUT + APIs; any supplied change is ignored. WorkspacePermissions: type: object properties: @@ -3461,6 +3494,21 @@ components: type: array items: $ref: '#/components/schemas/PermissionOutput' + AutoscopeState: + type: string + x-enum: + - AUTOSCOPE_STATE_API_NOT_COVERED + - AUTOSCOPE_STATE_BACKFILLED + - AUTOSCOPE_STATE_COMPLETED + - AUTOSCOPE_STATE_DISABLED + - AUTOSCOPE_STATE_RUNNING + - AUTOSCOPE_STATE_USER_SELECTED + description: |- + State of inferred scope collection (autoscope) for an external PAT. Mirrored in + databricks.identity.AutoscopeState in common/principal-context/api/proto/tokendetails.proto. + Token store and token management proto can depend on this. Principal context proto should NOT + depend on this proto definitions because too many services depend on the principal context + proto. GetSortOrder: type: string x-enum: @@ -3703,6 +3751,7 @@ components: - CAN_VIEW - CAN_VIEW_METADATA - IS_OWNER + - UNSPECIFIED description: Permission level RequestAuthzIdentity: type: string diff --git a/providers/src/databricks_workspace/v00.00.00000/services/iamv2.yaml b/providers/src/databricks_workspace/v00.00.00000/services/iamv2.yaml index 61492ee8..b98f7aac 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/iamv2.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/iamv2.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks iamv2 service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.iamv2 servers: - url: https://{deployment_name}.cloud.databricks.com @@ -14,47 +14,1873 @@ servers: description: The Databricks Workspace Deployment Name default: dbc-abcd0123-a1bc paths: - /api/2.0/identity/workspaceAccessDetails/{principal_id}: + /api/2.0/identity/groups/{group_id}/direct-members: + post: + operationId: workspace_iam_v2_create_direct_group_member_proxy + summary: Creates a group membership (assigns a principal to a group). + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Creates a group membership (assigns a principal to a group). + + :param group_id: int + Required. Internal ID of the group in Databricks. + :param direct_group_member: :class:`DirectGroupMember` + Required. The group membership to create. + + :returns: :class:`DirectGroupMember` + parameters: + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks. + requestBody: + content: + application/json: + schema: + type: object + properties: + direct_group_member: + type: object + properties: + principal_id: + type: integer + description: Internal ID of the principal in Databricks. + display_name: + type: string + description: Display name of the principal. + external_id: + type: string + description: The external ID of the principal in Databricks. + membership_source: + type: string + x-enum: + - IDENTITY_PROVIDER + - INTERNAL + description: The source of group membership (internal or from identity provider). + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the principal (user/service principal/group). + required: + - principal_id + description: Required. The group membership to create. + required: + - direct_group_member + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DirectGroupMember' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_list_direct_group_members_proxy + summary: Lists provisioned direct members of a group with their membership source (internal or from identity + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists provisioned direct members of a group with their membership source (internal or from identity + provider). + + :param group_id: int + Required. Internal ID of the group in Databricks whose direct members are being listed. + :param page_size: int (optional) + The maximum number of members to return. The service may return fewer than this value. If not + provided, defaults to 1000 (also the maximum allowed). + :param page_token: str (optional) + A page token, received from a previous ListDirectGroupMembersProxy call. Provide this to retrieve + the subsequent page. + + :returns: :class:`ListDirectGroupMembersResponse` + parameters: + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks whose direct members are being listed. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of members to return. The service may return fewer than this value. If not provided, + defaults to 1000 (also the maximum allowed). + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListDirectGroupMembersProxy call. Provide this to retrieve the + subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListDirectGroupMembersResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/groups: + post: + operationId: workspace_iam_v2_create_group_proxy + summary: Creates a group in the Databricks account that parents the calling workspace and returns the resulting + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Creates a group in the Databricks account that parents the calling workspace and returns the resulting + Group resource. + + :param group: :class:`Group` + Required. Group to be created in + + :returns: :class:`Group` + requestBody: + content: + application/json: + schema: + type: object + properties: + group: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + group_name: + type: string + description: Display name of the group. + internal_id: + type: string + description: Internal group ID of the group in Databricks. + description: Required. Group to be created in + required: + - group + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Group' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_list_groups_proxy + summary: Lists the groups in the Databricks account that parents the calling workspace, returning one page per + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists the groups in the Databricks account that parents the calling workspace, returning one page per + call. Supports filtering by group name or external ID. + + :param filter: str (optional) + Optional. Allows filtering groups by group name or external id. + :param page_size: int (optional) + The maximum number of groups to return. The service may return fewer than this value. + :param page_token: str (optional) + A page token, received from a previous ListGroups call. Provide this to retrieve the subsequent + page. + + :returns: :class:`ListGroupsResponse` + parameters: + - name: filter + in: query + required: false + schema: + type: string + description: Optional. Allows filtering groups by group name or external id. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of groups to return. The service may return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListGroups call. Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListGroupsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/service-principals: + post: + operationId: workspace_iam_v2_create_service_principal_proxy + summary: Creates a service principal in the Databricks account that parents the calling workspace and returns + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Creates a service principal in the Databricks account that parents the calling workspace and returns + the resulting ServicePrincipal resource. + + :param service_principal: :class:`ServicePrincipal` + Required. Service principal to be created in + + :returns: :class:`ServicePrincipal` + requestBody: + content: + application/json: + schema: + type: object + properties: + service_principal: + type: object + properties: + account_id: + type: string + description: The parent account ID for the service principal in Databricks. + account_sp_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of a service principal in a Databricks account. + application_id: + type: string + description: Application ID of the service principal. + display_name: + type: string + description: Display name of the service principal. + external_id: + type: string + description: ExternalId of the service principal in the customer's IdP. + internal_id: + type: string + description: Internal service principal ID of the service principal in Databricks. + description: Required. Service principal to be created in + required: + - service_principal + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServicePrincipal' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_list_service_principals_proxy + summary: Lists the service principals in the Databricks account that parents the calling workspace, returning + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists the service principals in the Databricks account that parents the calling workspace, returning + one page per call. Supports filtering by application ID or external ID. + + :param filter: str (optional) + Optional. Allows filtering service principals by application id or external id. + :param page_size: int (optional) + The maximum number of SPs to return. The service may return fewer than this value. + :param page_token: str (optional) + A page token, received from a previous ListServicePrincipals call. Provide this to retrieve the + subsequent page. + + :returns: :class:`ListServicePrincipalsResponse` + parameters: + - name: filter + in: query + required: false + schema: + type: string + description: Optional. Allows filtering service principals by application id or external id. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of SPs to return. The service may return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListServicePrincipals call. Provide this to retrieve the subsequent + page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListServicePrincipalsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/users: + post: + operationId: workspace_iam_v2_create_user_proxy + summary: Creates a user in the Databricks account that parents the calling workspace and returns the resulting + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Creates a user in the Databricks account that parents the calling workspace and returns the resulting + User resource. The same AIM-dependent provisioning behavior described on CreateUser applies. + + :param user: :class:`User` + Required. User to be created in + + :returns: :class:`User` + requestBody: + content: + application/json: + schema: + type: object + properties: + user: + type: object + properties: + account_id: + type: string + description: The accountId parent of the user in Databricks. + account_user_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of a user in a Databricks account. + external_id: + type: string + description: ExternalId of the user in the customer's IdP. + full_name: + type: object + properties: + family_name: + type: string + given_name: + type: string + internal_id: + type: string + description: Internal userId of the user in Databricks. + username: + type: string + description: Username/email of the user. + description: Required. User to be created in + required: + - user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/User' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_list_users_proxy + summary: Lists the users in the Databricks account that parents the calling workspace, returning one page per + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists the users in the Databricks account that parents the calling workspace, returning one page per + call. Supports filtering by username or external ID. + + :param filter: str (optional) + Optional. Allows filtering users by username or external id. + :param page_size: int (optional) + The maximum number of users to return. The service may return fewer than this value. + :param page_token: str (optional) + A page token, received from a previous ListUsers call. Provide this to retrieve the subsequent page. + + :returns: :class:`ListUsersResponse` + parameters: + - name: filter + in: query + required: false + schema: + type: string + description: Optional. Allows filtering users by username or external id. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of users to return. The service may return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListUsers call. Provide this to retrieve the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListUsersResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/workspace-assignment-details: + post: + operationId: workspace_iam_v2_create_workspace_assignment_detail_proxy + summary: Creates a workspace assignment detail for a principal (workspace-level proxy). Entitlement grants are + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Creates a workspace assignment detail for a principal (workspace-level proxy). Entitlement grants are + applied individually and non-atomically — if a failure occurs partway through, the principal will be + assigned to the workspace but with only a subset of the requested entitlements. Use + GetWorkspaceAssignmentDetail to confirm which entitlements were successfully granted. + + :param workspace_assignment_detail: :class:`WorkspaceAssignmentDetail` + Required. Workspace assignment detail to be created in . + + :returns: :class:`WorkspaceAssignmentDetail` + requestBody: + content: + application/json: + schema: + type: object + properties: + workspace_assignment_detail: + type: object + properties: + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + account_id: + type: string + description: The account ID parent of the workspace where the principal is assigned + effective_entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: 'The principal''s full effective entitlements granted in this workspace: every entitlement + it holds whether granted directly or via group membership. Populated on Get; empty on List.' + entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: 'Entitlements granted directly to the principal on this workspace. The only client-settable + field: create and update manage exactly this set (including entitlements the principal also holds + via a group). Not populated by ListWorkspaceAssignmentDetails (omitted for scalability); call GetWorkspaceAssignmentDetail + to read the entitlements for a single principal.' + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the principal (user/sp/group). + workspace_id: + type: integer + description: The workspace ID where the principal is assigned + required: + - principal_id + description: Required. Workspace assignment detail to be created in . + required: + - workspace_assignment_detail + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_list_workspace_assignment_details_proxy + summary: Lists workspace assignment details for a workspace (workspace-level proxy). For scalability, the + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists workspace assignment details for a workspace (workspace-level proxy). For scalability, the + response omits the per-principal entitlement fields (``entitlements`` and ``effective_entitlements``); + call GetWorkspaceAssignmentDetailProxy to read entitlements for a single principal. + + :param page_size: int (optional) + The maximum number of workspace assignment details to return. The service may return fewer than this + value. + :param page_token: str (optional) + A page token, received from a previous ListWorkspaceAssignmentDetailsProxy call. Provide this to + retrieve the subsequent page. + + :returns: :class:`ListWorkspaceAssignmentDetailsResponse` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of workspace assignment details to return. The service may return fewer than this + value. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListWorkspaceAssignmentDetailsProxy call. Provide this to retrieve + the subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListWorkspaceAssignmentDetailsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/groups/{group_id}/direct-members/{principal_id}: + delete: + operationId: workspace_iam_v2_delete_direct_group_member_proxy + summary: Deletes a group membership (unassigns a principal from a group). + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Deletes a group membership (unassigns a principal from a group). + + :param group_id: int + Required. Internal ID of the group in Databricks. + :param principal_id: int + Required. Internal ID of the principal to be unassigned from the group. + parameters: + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks. + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the principal to be unassigned from the group. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_get_direct_group_member_proxy + summary: Gets a provisioned direct member of a group. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Gets a provisioned direct member of a group. + + :param group_id: int + Required. Internal ID of the group in Databricks. + :param principal_id: int + Required. Internal ID of the principal belonging to the group in Databricks. + + :returns: :class:`DirectGroupMember` + parameters: + - name: group_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the group in Databricks. + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the principal belonging to the group in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DirectGroupMember' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/groups/{internal_id}: + delete: + operationId: workspace_iam_v2_delete_group_proxy + summary: Deletes a group by its internal ID from the Databricks account that parents the calling workspace. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Deletes a group by its internal ID from the Databricks account that parents the calling workspace. + + :param internal_id: str + Required. Internal ID of the group in Databricks. + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the group in Databricks. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_get_group_proxy + summary: Fetches a group by its internal ID from the Databricks account that parents the calling workspace. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Fetches a group by its internal ID from the Databricks account that parents the calling workspace. + + :param internal_id: str + Required. Internal ID of the group in Databricks. + + :returns: :class:`Group` + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the group in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Group' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + patch: + operationId: workspace_iam_v2_update_group_proxy + summary: Updates an existing group in the Databricks account that parents the calling workspace. Only the + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Updates an existing group in the Databricks account that parents the calling workspace. Only the + fields named in the update mask are modified. Returns the updated Group resource. + + :param internal_id: str + Required. Internal ID of the group in Databricks. + :param group: :class:`Group` + Required. Group to be updated in + :param update_mask: str + Optional. The list of fields to update. + + :returns: :class:`Group` + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the group in Databricks. + - name: update_mask + in: query + required: true + schema: + type: string + description: Optional. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + group: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + group_name: + type: string + description: Display name of the group. + internal_id: + type: string + description: Internal group ID of the group in Databricks. + description: Required. Group to be updated in + required: + - group + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Group' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/service-principals/{internal_id}: + delete: + operationId: workspace_iam_v2_delete_service_principal_proxy + summary: Deletes a service principal by its internal ID from the Databricks account that parents the calling + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Deletes a service principal by its internal ID from the Databricks account that parents the calling + workspace. + + :param internal_id: str + Required. Internal ID of the service principal in Databricks. + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the service principal in Databricks. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_get_service_principal_proxy + summary: Fetches a service principal by its internal ID from the Databricks account that parents the calling + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Fetches a service principal by its internal ID from the Databricks account that parents the calling + workspace. + + :param internal_id: str + Required. Internal ID of the service principal in Databricks. + + :returns: :class:`ServicePrincipal` + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the service principal in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServicePrincipal' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + patch: + operationId: workspace_iam_v2_update_service_principal_proxy + summary: Updates an existing service principal in the Databricks account that parents the calling workspace. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Updates an existing service principal in the Databricks account that parents the calling workspace. + Only the fields named in the update mask are modified. Returns the updated ServicePrincipal resource. + + :param internal_id: str + Required. Internal ID of the service principal in Databricks. + :param service_principal: :class:`ServicePrincipal` + Required. Service principal to be updated in + :param update_mask: str + Optional. The list of fields to update. + + :returns: :class:`ServicePrincipal` + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the service principal in Databricks. + - name: update_mask + in: query + required: true + schema: + type: string + description: Optional. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + service_principal: + type: object + properties: + account_id: + type: string + description: The parent account ID for the service principal in Databricks. + account_sp_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of a service principal in a Databricks account. + application_id: + type: string + description: Application ID of the service principal. + display_name: + type: string + description: Display name of the service principal. + external_id: + type: string + description: ExternalId of the service principal in the customer's IdP. + internal_id: + type: string + description: Internal service principal ID of the service principal in Databricks. + description: Required. Service principal to be updated in + required: + - service_principal + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServicePrincipal' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/users/{internal_id}: + delete: + operationId: workspace_iam_v2_delete_user_proxy + summary: Deletes a user by its internal ID from the Databricks account that parents the calling workspace. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Deletes a user by its internal ID from the Databricks account that parents the calling workspace. + + :param internal_id: str + Required. Internal ID of the user in Databricks. + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the user in Databricks. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_get_user_proxy + summary: Fetches a user by its internal ID from the Databricks account that parents the calling workspace. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Fetches a user by its internal ID from the Databricks account that parents the calling workspace. + + :param internal_id: str + Required. Internal ID of the user in Databricks. + + :returns: :class:`User` + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the user in Databricks. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/User' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + patch: + operationId: workspace_iam_v2_update_user_proxy + summary: Updates an existing user in the Databricks account that parents the calling workspace. Only the fields + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Updates an existing user in the Databricks account that parents the calling workspace. Only the fields + named in the update mask are modified; the updatable fields are fullName.givenName, + fullName.familyName, status, and externalId. Returns the updated User resource. + + :param internal_id: str + Required. Internal ID of the user in Databricks. + :param user: :class:`User` + Required. User to be updated in + :param update_mask: str + Optional. The list of fields to update. + + :returns: :class:`User` + parameters: + - name: internal_id + in: path + required: true + schema: + type: string + description: Required. Internal ID of the user in Databricks. + - name: update_mask + in: query + required: true + schema: + type: string + description: Optional. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + user: + type: object + properties: + account_id: + type: string + description: The accountId parent of the user in Databricks. + account_user_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of a user in a Databricks account. + external_id: + type: string + description: ExternalId of the user in the customer's IdP. + full_name: + type: object + properties: + family_name: + type: string + given_name: + type: string + internal_id: + type: string + description: Internal userId of the user in Databricks. + username: + type: string + description: Username/email of the user. + description: Required. User to be updated in + required: + - user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/User' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/workspace-assignment-details/{principal_id}: + delete: + operationId: workspace_iam_v2_delete_workspace_assignment_detail_proxy + summary: Deletes a workspace assignment detail for a principal (workspace-level proxy), revoking all associated + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Deletes a workspace assignment detail for a principal (workspace-level proxy), revoking all associated + entitlements. Entitlement revocations are applied individually and non-atomically — if a failure + occurs partway through, the principal remains assigned with a subset of its original entitlements, and + the operation is safe to retry. + + :param principal_id: int + Required. ID of the principal in Databricks to delete workspace assignment for. + parameters: + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. ID of the principal in Databricks to delete workspace assignment for. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + get: + operationId: workspace_iam_v2_get_workspace_assignment_detail_proxy + summary: Returns the assignment details for a principal in a workspace (workspace-level proxy). + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Returns the assignment details for a principal in a workspace (workspace-level proxy). + + :param principal_id: int + Required. The internal ID of the principal (user/sp/group) for which the assignment details are + being requested. + + :returns: :class:`WorkspaceAssignmentDetail` + parameters: + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. The internal ID of the principal (user/sp/group) for which the assignment details are being + requested. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + patch: + operationId: workspace_iam_v2_update_workspace_assignment_detail_proxy + summary: Updates the entitlements of a directly assigned principal in a workspace (workspace-level proxy). + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Updates the entitlements of a directly assigned principal in a workspace (workspace-level proxy). + Entitlement changes are applied individually and non-atomically — if a failure occurs partway + through, only a subset of the requested changes may have been applied. Use + GetWorkspaceAssignmentDetail to confirm the final state. + + :param principal_id: int + Required. ID of the principal in Databricks. + :param workspace_assignment_detail: :class:`WorkspaceAssignmentDetail` + Required. Workspace assignment detail to be updated in . + :param update_mask: FieldMask + Required. The list of fields to update. + + :returns: :class:`WorkspaceAssignmentDetail` + parameters: + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. ID of the principal in Databricks. + - name: update_mask + in: query + required: true + schema: + type: object + description: Required. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + workspace_assignment_detail: + type: object + properties: + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + account_id: + type: string + description: The account ID parent of the workspace where the principal is assigned + effective_entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: 'The principal''s full effective entitlements granted in this workspace: every entitlement + it holds whether granted directly or via group membership. Populated on Get; empty on List.' + entitlements: + type: array + items: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + description: 'Entitlements granted directly to the principal on this workspace. The only client-settable + field: create and update manage exactly this set (including entitlements the principal also holds + via a group). Not populated by ListWorkspaceAssignmentDetails (omitted for scalability); call GetWorkspaceAssignmentDetail + to read the entitlements for a single principal.' + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the principal (user/sp/group). + workspace_id: + type: integer + description: The workspace ID where the principal is assigned + required: + - principal_id + description: Required. Workspace assignment detail to be updated in . + required: + - workspace_assignment_detail + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/{name}: + get: + operationId: workspace_iam_v2_get_external_user_proxy + summary: Retrieves an external user with the given external ID from the customer's IdP. If the user does not + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Retrieves an external user with the given external ID from the customer's IdP. If the user does not + exist, it will be created. If the customer is not onboarded onto Automatic Identity Management (AIM), + this will return an error. Workspace-scoped variant for workspace-authenticated callers. + + :param name: str + Required. The resource name of the external user. Format: external-users/{external_user_id} + + :returns: :class:`ExternalUser` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: 'Required. The resource name of the external user. Format: external-users/{external_user_id}' + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ExternalUser' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/workspace-access-details/{principal_id}: + get: + operationId: workspace_iam_v2_get_workspace_access_detail_local + summary: Returns the access details for a principal in the current workspace. Allows for checking access + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Returns the access details for a principal in the current workspace. Allows for checking access + details for any provisioned principal (user, service principal, or group) in the current workspace. + + - Provisioned principal here refers to one that has been synced into Databricks from the customer's + IdP or added explicitly to Databricks via SCIM/UI. Allows for passing in a "view" parameter to + control what fields are returned (BASIC by default or FULL). + + :param principal_id: int + Required. The internal ID of the principal (user/sp/group) for which the access details are being + requested. + :param view: :class:`WorkspaceAccessDetailView` (optional) + Controls what fields are returned. + + :returns: :class:`WorkspaceAccessDetail` + parameters: + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. The internal ID of the principal (user/sp/group) for which the access details are being requested. + - name: view + in: query + required: false + schema: + $ref: '#/components/schemas/WorkspaceAccessDetailView' + description: Controls what fields are returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceAccessDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/workspace-identity-details/{principal_id}: get: - operationId: workspace_iam_v2_get_workspace_access_detail_local - summary: Returns the access details for a principal in the current workspace. Allows for checking access + operationId: workspace_iam_v2_get_workspace_identity_detail + summary: Returns the identity details for a principal in a workspace. tags: - iamv2 - workspace_iam_v2 description: |- - Returns the access details for a principal in the current workspace. Allows for checking access - details for any provisioned principal (user, service principal, or group) in the current workspace. * - Provisioned principal here refers to one that has been synced into Databricks from the customer's IdP - or added explicitly to Databricks via SCIM/UI. Allows for passing in a "view" parameter to control - what fields are returned (BASIC by default or FULL). + Returns the identity details for a principal in a workspace. :param principal_id: int - Required. The internal ID of the principal (user/sp/group) for which the access details are being + Required. The internal ID of the principal (user/sp/group) for which the identity details are being requested. - :param view: :class:`WorkspaceAccessDetailView` (optional) - Controls what fields are returned. - :returns: :class:`WorkspaceAccessDetail` + :returns: :class:`WorkspaceIdentityDetail` parameters: - name: principal_id in: path required: true schema: type: integer - description: Required. The internal ID of the principal (user/sp/group) for which the access details are being requested. - - name: view + description: Required. The internal ID of the principal (user/sp/group) for which the identity details are being requested. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceIdentityDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + patch: + operationId: workspace_iam_v2_update_workspace_identity_detail + summary: Updates a workspace identity detail for a principal. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Updates a workspace identity detail for a principal. + + :param principal_id: int + Required. ID of the principal in Databricks. + :param workspace_identity_detail: :class:`WorkspaceIdentityDetail` + Required. Workspace identity detail to be updated in . + :param update_mask: FieldMask + Required. The list of fields to update. + + :returns: :class:`WorkspaceIdentityDetail` + parameters: + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. ID of the principal in Databricks. + - name: update_mask + in: query + required: true + schema: + type: object + description: Required. The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + workspace_identity_detail: + type: object + properties: + assignment_type: + type: string + x-enum: + - DIRECT + - INDIRECT + description: The type of assignment the principal has to the workspace (direct or indirect). + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + principal_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the principal (user/service principal/group). + workspace_identity_status: + type: string + x-enum: + - ACTIVE + - INACTIVE + description: The activity status of an identity in a Databricks workspace. + description: Required. Workspace identity detail to be updated in . + required: + - workspace_identity_detail + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/WorkspaceIdentityDetail' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/principals/{principal_id}/transitive-parent-groups: + get: + operationId: workspace_iam_v2_list_transitive_parent_groups_proxy + summary: Lists all transitive parent groups of a principal. + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists all transitive parent groups of a principal. + + :param principal_id: int + Required. Internal ID of the principal in Databricks whose transitive parent groups are being + listed. + :param page_size: int (optional) + The maximum number of parent groups to return. The service may return fewer than this value. If not + provided, defaults to 1000 (also the maximum allowed). + :param page_token: str (optional) + A page token, received from a previous ListTransitiveParentGroups call. Provide this to retrieve the + subsequent page. + + :returns: :class:`ListTransitiveParentGroupsResponse` + parameters: + - name: principal_id + in: path + required: true + schema: + type: integer + description: Required. Internal ID of the principal in Databricks whose transitive parent groups are being listed. + - name: page_size in: query required: false schema: - $ref: '#/components/schemas/WorkspaceAccessDetailView' - description: Controls what fields are returned. + type: integer + description: The maximum number of parent groups to return. The service may return fewer than this value. If not provided, + defaults to 1000 (also the maximum allowed). + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListTransitiveParentGroups call. Provide this to retrieve the + subsequent page. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/WorkspaceAccessDetail' + $ref: '#/components/schemas/ListTransitiveParentGroupsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: WorkspaceIamV2API + /api/2.0/identity/workspace-access-details: + get: + operationId: workspace_iam_v2_list_workspace_access_details_local + summary: Lists the access details of every provisioned principal (user, service principal, or group) with + tags: + - iamv2 + - workspace_iam_v2 + description: |- + Lists the access details of every provisioned principal (user, service principal, or group) with + access to the current workspace, returning one page per call. + + - Provisioned principal here refers to one that has been synced into Databricks from the customer's + IdP or added explicitly to Databricks via SCIM/UI. + + :param page_size: int (optional) + The maximum number of workspace access details to return. The service may return fewer than this + value. + :param page_token: str (optional) + A page token, received from a previous ListWorkspaceAccessDetails call. Provide this to retrieve the + subsequent page. + + :returns: :class:`ListWorkspaceAccessDetailsResponse` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of workspace access details to return. The service may return fewer than this value. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ListWorkspaceAccessDetails call. Provide this to retrieve the + subsequent page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListWorkspaceAccessDetailsResponse' default: description: Error response content: @@ -67,7 +1893,7 @@ paths: message: type: string x-stackql-sdk-source: WorkspaceIamV2API - /api/2.0/identity/groups/resolveByExternalId: + /api/2.0/identity/groups/resolve-by-external-id: post: operationId: workspace_iam_v2_resolve_group_proxy summary: Resolves a group with the given external ID from the customer's IdP. If the group does not exist, it @@ -113,7 +1939,7 @@ paths: message: type: string x-stackql-sdk-source: WorkspaceIamV2API - /api/2.0/identity/servicePrincipals/resolveByExternalId: + /api/2.0/identity/service-principals/resolve-by-external-id: post: operationId: workspace_iam_v2_resolve_service_principal_proxy summary: Resolves an SP with the given external ID from the customer's IdP. If the SP does not exist, it will @@ -159,7 +1985,7 @@ paths: message: type: string x-stackql-sdk-source: WorkspaceIamV2API - /api/2.0/identity/users/resolveByExternalId: + /api/2.0/identity/users/resolve-by-external-id: post: operationId: workspace_iam_v2_resolve_user_proxy summary: Resolves a user with the given external ID from the customer's IdP. If the user does not exist, it @@ -207,6 +2033,160 @@ paths: x-stackql-sdk-source: WorkspaceIamV2API components: schemas: + AccountAccessIdentityRule: + type: object + properties: + action: + $ref: '#/components/schemas/AccountAccessRuleAction' + description: Currently, only DENY action is supported. + display_name: + type: string + description: Display name of the principal. + external_principal_id: + type: string + description: External ID of the principal in the customer's IdP. + name: + type: string + description: 'Fully qualified name for the rule. Format: accounts/{account_id}/account-access-identity-rules/{external_principal_id}' + principal_type: + $ref: '#/components/schemas/PrincipalType' + description: The type of the principal (user/service principal/group). This field is populated by the server based + on the external_principal_id. + required: + - action + description: An identity rule that controls which principals can access an account. + AttributeControlEntry: + type: object + properties: + is_allowed: + type: boolean + description: Whether the attribute is permitted. Always true in the Identity Attributes PrPr. + name: + type: string + description: 'The resource name of the entry. Format: accounts/{account_id}/attribute-control-entries/{attribute_name} + where {attribute_name} is the IdP attribute being governed (e.g. "department").' + description: |- + A single entry in an account's identity attribute control list. The presence of an entry means + the governed attribute (the last segment of ``name``) is controlled across all write sources + (SCIM, Identity APIs, AIM); is_allowed records whether the attribute is permitted (always true + in the Identity Attributes PrPr). For the Identity Attributes PrPr the control list is an + explicit allowlist: an attribute that is NOT present in the control list is treated as not + allowed. This may change post-PrPr. + DirectGroupMember: + type: object + properties: + principal_id: + type: integer + description: Internal ID of the principal in Databricks. + display_name: + type: string + description: Display name of the principal. + external_id: + type: string + description: The external ID of the principal in Databricks. + membership_source: + $ref: '#/components/schemas/GroupMembershipSource' + description: The source of group membership (internal or from identity provider). + principal_type: + $ref: '#/components/schemas/PrincipalType' + description: The type of the principal (user/service principal/group). + required: + - principal_id + description: Represents a principal that is a direct member of a group, with its source of membership. + ExternalGroup: + type: object + properties: + account_id: + type: string + description: The parent account ID, from Databricks. + display_name: + type: string + description: Display name of the group from the customer's IdP. + external_group_id: + type: string + description: The external ID of the group in the customer's IdP. + internal_id: + type: string + description: Internal groupId of the group in Databricks. + name: + type: string + description: 'The resource name of the external group. Format: accounts/{account_id}/external-groups/{external_group_id}' + description: |- + An external group from the customer's Identity Provider, resolved into Databricks. This is a + read-only resource keyed by the IdP external ID. The Get method may trigger an idempotent sync + from the customer's IdP to provision or refresh the group's data in Databricks. + ExternalServicePrincipal: + type: object + properties: + account_id: + type: string + description: The parent account ID, from Databricks. + account_sp_status: + $ref: '#/components/schemas/State' + description: The activity status of the service principal in the Databricks account. + application_id: + type: string + description: Application ID of the service principal, from the customer's IdP. + display_name: + type: string + description: Display name of the service principal, from the customer's IdP. + external_service_principal_id: + type: string + description: The external ID of the service principal in the customer's IdP. + internal_id: + type: string + description: Internal servicePrincipalId of the service principal in Databricks. + name: + type: string + description: 'The resource name of the external service principal. Format: accounts/{account_id}/external-service-principals/{external_service_principal_id}' + description: |- + An external service principal from the customer's Identity Provider, resolved into Databricks. + This is a read-only resource keyed by the IdP external ID. The Get method may trigger an + idempotent sync from the customer's IdP to provision or refresh the service principal's data in + Databricks. + ExternalUser: + type: object + properties: + account_id: + type: string + description: The parent account ID, from Databricks. + account_user_status: + $ref: '#/components/schemas/State' + description: The activity status of the user in the Databricks account. + display_name: + type: string + description: Display name of the user from the customer's IdP. + external_user_id: + type: string + description: The external ID of the user in the customer's IdP. + full_name: + $ref: '#/components/schemas/FullName' + description: The full name of the user, from the customer's IdP. + internal_id: + type: string + description: Internal userId of the user in Databricks. + name: + type: string + description: 'The resource name of the external user. Format: accounts/{account_id}/external-users/{external_user_id}' + username: + type: string + description: Username/email of the user, from Databricks. + description: |- + An external user from the customer's Identity Provider, resolved into Databricks. This is a + read-only resource that allows customers to look up external user identities by their IdP + external ID and retrieve the corresponding Databricks internal ID and metadata. The Get method + may trigger an idempotent sync from the customer's IdP to provision or refresh the user's data + in Databricks. + FullName: + type: object + properties: + family_name: + type: string + description: The family (last) name of the user, from the customer's IdP. + given_name: + type: string + description: The given (first) name of the user, from the customer's IdP. + description: The full name of a user. Group: type: object properties: @@ -220,9 +2200,115 @@ components: type: string description: Display name of the group. internal_id: - type: integer + type: string description: Internal group ID of the group in Databricks. description: The details of a Group resource. + ListAccountAccessIdentityRulesResponse: + type: object + properties: + account_access_identity_rules: + type: array + items: + $ref: '#/components/schemas/AccountAccessIdentityRule' + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + description: Response message for listing account access identity rules. + ListAttributeControlEntriesResponse: + type: object + properties: + attribute_control_entries: + type: array + items: + $ref: '#/components/schemas/AttributeControlEntry' + next_page_token: + type: string + description: A token to retrieve the next page; empty when there are no more results. + description: Response message for listing identity attribute control-list entries. + ListDirectGroupMembersResponse: + type: object + properties: + direct_group_members: + type: array + items: + $ref: '#/components/schemas/DirectGroupMember' + description: The list of direct group members with their membership source type. + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + description: Response message for listing direct group members. + ListGroupsResponse: + type: object + properties: + groups: + type: array + items: + $ref: '#/components/schemas/Group' + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + description: Response message containing a page of groups in the account. + ListServicePrincipalsResponse: + type: object + properties: + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + service_principals: + type: array + items: + $ref: '#/components/schemas/ServicePrincipal' + description: Response message containing a page of service principals in the account. + ListTransitiveParentGroupsResponse: + type: object + properties: + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + transitive_parent_groups: + type: array + items: + $ref: '#/components/schemas/TransitiveParentGroup' + description: The list of transitive parent groups. + description: Response message for listing all transitive parent groups of a principal. + ListUsersResponse: + type: object + properties: + next_page_token: + type: string + users: + type: array + items: + $ref: '#/components/schemas/User' + ListWorkspaceAccessDetailsResponse: + type: object + properties: + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + workspace_access_details: + type: array + items: + $ref: '#/components/schemas/WorkspaceAccessDetail' + description: Response message containing a page of workspace access details. + ListWorkspaceAssignmentDetailsResponse: + type: object + properties: + next_page_token: + type: string + description: A token, which can be sent as page_token to retrieve the next page. If this field is omitted, there + are no subsequent pages. + workspace_assignment_details: + type: array + items: + $ref: '#/components/schemas/WorkspaceAssignmentDetail' + description: Response message for listing workspace assignment details. ResolveGroupResponse: type: object properties: @@ -257,9 +2343,22 @@ components: type: string description: ExternalId of the service principal in the customer's IdP. internal_id: - type: integer + type: string description: Internal service principal ID of the service principal in Databricks. description: The details of a ServicePrincipal resource. + TransitiveParentGroup: + type: object + properties: + account_id: + type: string + description: The parent account ID for group in Databricks. + external_id: + type: string + description: ExternalId of the group in the customer's IdP. + internal_id: + type: string + description: Internal group ID of the group in Databricks. + description: Represents a group that is a transitive parent of a principal. User: type: object properties: @@ -272,16 +2371,16 @@ components: external_id: type: string description: ExternalId of the user in the customer's IdP. + full_name: + $ref: '#/components/schemas/UserFullName' internal_id: - type: integer + type: string description: Internal userId of the user in Databricks. - name: - $ref: '#/components/schemas/UserName' username: type: string description: Username/email of the user. description: The details of a User resource. - UserName: + UserFullName: type: object properties: family_name: @@ -313,6 +2412,128 @@ components: type: integer description: The workspace ID where the principal has access. description: The details of a principal's access to a workspace. + WorkspaceAssignmentDetail: + type: object + properties: + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + account_id: + type: string + description: The account ID parent of the workspace where the principal is assigned + effective_entitlements: + type: array + items: + $ref: '#/components/schemas/Entitlement' + description: 'The principal''s full effective entitlements granted in this workspace: every entitlement it holds + whether granted directly or via group membership. Populated on Get; empty on List.' + entitlements: + type: array + items: + $ref: '#/components/schemas/Entitlement' + description: 'Entitlements granted directly to the principal on this workspace. The only client-settable field: + create and update manage exactly this set (including entitlements the principal also holds via a group). Not populated + by ListWorkspaceAssignmentDetails (omitted for scalability); call GetWorkspaceAssignmentDetail to read the entitlements + for a single principal.' + principal_type: + $ref: '#/components/schemas/PrincipalType' + workspace_id: + type: integer + description: The workspace ID where the principal is assigned + required: + - principal_id + description: |- + The direct assignment of a provisioned account-level principal (user, service principal, or + group) to a workspace, together with the entitlements that assignment grants in the workspace. + + A WorkspaceAssignmentDetail exists only for principals that are directly assigned to the + workspace; principals that merely inherit workspace access through a group are not represented + here (see WorkspaceAccessDetail / WorkspaceIdentityDetail for the effective, direct-or-indirect + view). Creating the resource assigns the principal to the workspace; deleting it removes the + assignment. The ``entitlements`` field is the only client-settable field and defines the + entitlements granted directly on this assignment; ``effective_entitlements`` is the read-only + union of those plus any granted via group membership. + + A direct assignment always carries at least one directly-assigned entitlement: the assignment is + what grants the entitlement, so a WorkspaceAssignmentDetail with an empty ``entitlements`` set + is not a valid state. Both create and update require a non-empty ``entitlements`` set (an empty + set is rejected); to remove a principal's assignment entirely, delete the resource. + + This resource replaces workspace assignment previously managed through the workspace SCIM and + permission-assignment APIs, and is intended for account and workspace admins. + WorkspaceIdentityDetail: + type: object + properties: + assignment_type: + $ref: '#/components/schemas/WorkspaceIdentityDetailAssignmentType' + description: The type of assignment the principal has to the workspace (direct or indirect). + principal_id: + type: integer + description: The internal ID of the principal (user/sp/group) in Databricks. + principal_type: + $ref: '#/components/schemas/PrincipalType' + description: The type of the principal (user/service principal/group). + workspace_identity_status: + $ref: '#/components/schemas/State' + description: The activity status of an identity in a Databricks workspace. + description: The details of a directly or indirectly assigned principal's details in a workspace. + AccountAccessRuleAction: + type: string + x-enum: + - DENY + description: The action type for an account access identity rule (currently DENY only). + Entitlement: + type: string + x-enum: + - ALLOW_CLUSTER_CREATE + - ALLOW_INSTANCE_POOL_CREATE + - DATABRICKS_SQL_ACCESS + - WORKSPACE_ACCESS + - WORKSPACE_ADMIN + - WORKSPACE_CONSUME + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + GroupMembershipSource: + type: string + x-enum: + - IDENTITY_PROVIDER + - INTERNAL + description: The source of the group membership (internal or from identity provider). PrincipalType: type: string x-enum: @@ -338,6 +2559,12 @@ components: - BASIC - FULL description: Controls what fields are returned in the GetWorkspaceAccessDetail response. + WorkspaceIdentityDetailAssignmentType: + type: string + x-enum: + - DIRECT + - INDIRECT + description: The type of assignment the principal has to the workspace. WorkspacePermission: type: string x-enum: @@ -345,47 +2572,375 @@ components: - USER_PERMISSION description: The type of permission a principal has to a workspace (admin/user). x-stackQL-resources: - workspace_iam_v2: - id: databricks_workspace.iamv2.workspace_iam_v2 - name: workspace_iam_v2 - title: Workspace Iam V2 + direct_group_members: + id: databricks_workspace.iamv2.direct_group_members + name: direct_group_members + title: Direct Group Members + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{group_id}~1direct-members/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{group_id}~1direct-members/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.direct_group_members + delete: + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{group_id}~1direct-members~1{principal_id}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{group_id}~1direct-members~1{principal_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/direct_group_members/methods/get' + - $ref: '#/components/x-stackQL-resources/direct_group_members/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/direct_group_members/methods/create' + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/direct_group_members/methods/delete' + replace: [] + groups: + id: databricks_workspace.iamv2.groups + name: groups + title: Groups + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.groups + delete: + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{internal_id}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{internal_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1{internal_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + resolve: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1groups~1resolve-by-external-id/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/groups/methods/get' + - $ref: '#/components/x-stackQL-resources/groups/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/groups/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/groups/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/groups/methods/delete' + replace: [] + service_principals: + id: databricks_workspace.iamv2.service_principals + name: service_principals + title: Service Principals + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1service-principals/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1service-principals/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.service_principals + delete: + operation: + $ref: '#/paths/~1api~12.0~1identity~1service-principals~1{internal_id}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1service-principals~1{internal_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1service-principals~1{internal_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + resolve: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1service-principals~1resolve-by-external-id/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/get' + - $ref: '#/components/x-stackQL-resources/service_principals/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/service_principals/methods/delete' + replace: [] + users: + id: databricks_workspace.iamv2.users + name: users + title: Users methods: - get_workspace_access_detail_local: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1users/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1users/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.users + delete: + operation: + $ref: '#/paths/~1api~12.0~1identity~1users~1{internal_id}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1users~1{internal_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1identity~1users~1{internal_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + resolve: + config: + requestBodyTranslate: + algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1identity~1workspaceAccessDetails~1{principal_id}/get' + $ref: '#/paths/~1api~12.0~1identity~1users~1resolve-by-external-id/post' response: mediaType: application/json openAPIDocKey: '200' - resolve_group_proxy: + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/users/methods/get' + - $ref: '#/components/x-stackQL-resources/users/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/users/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/users/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/users/methods/delete' + replace: [] + workspace_assignment_details: + id: databricks_workspace.iamv2.workspace_assignment_details + name: workspace_assignment_details + title: Workspace Assignment Details + methods: + create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1identity~1groups~1resolveByExternalId/post' + $ref: '#/paths/~1api~12.0~1identity~1workspace-assignment-details/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1workspace-assignment-details/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.workspace_assignment_details + delete: + operation: + $ref: '#/paths/~1api~12.0~1identity~1workspace-assignment-details~1{principal_id}/delete' response: mediaType: application/json openAPIDocKey: '200' - resolve_service_principal_proxy: + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1workspace-assignment-details~1{principal_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1identity~1servicePrincipals~1resolveByExternalId/post' + $ref: '#/paths/~1api~12.0~1identity~1workspace-assignment-details~1{principal_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/workspace_assignment_details/methods/get' + - $ref: '#/components/x-stackQL-resources/workspace_assignment_details/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/workspace_assignment_details/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/workspace_assignment_details/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/workspace_assignment_details/methods/delete' + replace: [] + external_users: + id: databricks_workspace.iamv2.external_users + name: external_users + title: External Users + methods: + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/external_users/methods/get' + insert: [] + update: [] + delete: [] + replace: [] + workspace_access_details: + id: databricks_workspace.iamv2.workspace_access_details + name: workspace_access_details + title: Workspace Access Details + methods: + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1workspace-access-details~1{principal_id}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1workspace-access-details/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.workspace_access_details + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/workspace_access_details/methods/get' + - $ref: '#/components/x-stackQL-resources/workspace_access_details/methods/list' + insert: [] + update: [] + delete: [] + replace: [] + workspace_identity_details: + id: databricks_workspace.iamv2.workspace_identity_details + name: workspace_identity_details + title: Workspace Identity Details + methods: + get: + operation: + $ref: '#/paths/~1api~12.0~1identity~1workspace-identity-details~1{principal_id}/get' response: mediaType: application/json openAPIDocKey: '200' - resolve_user_proxy: + update: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1identity~1users~1resolveByExternalId/post' + $ref: '#/paths/~1api~12.0~1identity~1workspace-identity-details~1{principal_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/workspace_identity_details/methods/get' + insert: [] + update: + - $ref: '#/components/x-stackQL-resources/workspace_identity_details/methods/update' + delete: [] + replace: [] + transitive_parent_groups: + id: databricks_workspace.iamv2.transitive_parent_groups + name: transitive_parent_groups + title: Transitive Parent Groups + methods: + list: + operation: + $ref: '#/paths/~1api~12.0~1identity~1principals~1{principal_id}~1transitive-parent-groups/get' response: mediaType: application/json openAPIDocKey: '200' + objectKey: $.transitive_parent_groups sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/workspace_iam_v2/methods/get_workspace_access_detail_local' + - $ref: '#/components/x-stackQL-resources/transitive_parent_groups/methods/list' insert: [] update: [] delete: [] @@ -440,7 +2995,7 @@ components: wa.account_id, wa.workspace_id, p.value AS permission - FROM databricks_workspace.iamv2.workspace_iam_v2 wa, + FROM databricks_workspace.iamv2.workspace_access_details wa, JSON_EACH(wa.permissions) p WHERE wa.deployment_name = '{{ deployment_name }}' AND wa.principal_id = '{{ principal_id }}' @@ -456,7 +3011,7 @@ components: wa.account_id, wa.workspace_id, p.value AS permission - FROM databricks_workspace.iamv2.workspace_iam_v2 wa, + FROM databricks_workspace.iamv2.workspace_access_details wa, jsonb_array_elements(wa.permissions::jsonb) AS p WHERE wa.deployment_name = '{{ deployment_name }}' AND wa.principal_id = '{{ principal_id }}' diff --git a/providers/src/databricks_workspace/v00.00.00000/services/jobs.yaml b/providers/src/databricks_workspace/v00.00.00000/services/jobs.yaml index bc9431c9..38ebc047 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/jobs.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/jobs.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks jobs service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.jobs servers: - url: https://{deployment_name}.cloud.databricks.com @@ -131,11 +131,11 @@ paths: List of permissions to set on the job. :param budget_policy_id: str (optional) The id of the user specified budget policy to use for this job. If not specified, a default budget - policy may be applied when creating or modifying the job. See `effective_budget_policy_id` for the + policy may be applied when creating or modifying the job. See ``effective_budget_policy_id`` for the budget policy used by this workload. :param continuous: :class:`Continuous` (optional) An optional continuous property for this job. The continuous property will ensure that there is - always one run executing. Only one of `schedule` and `continuous` can be used. + always one run executing. Only one of ``schedule`` and ``continuous`` can be used. :param deployment: :class:`JobDeployment` (optional) Deployment information for jobs managed by external sources. :param description: str (optional) @@ -143,11 +143,17 @@ paths: :param edit_mode: :class:`JobEditMode` (optional) Edit mode of the job. - * `UI_LOCKED`: The job is in a locked UI state and cannot be modified. * `EDITABLE`: The job is in - an editable state and can be modified. + - ``UI_LOCKED``: The job is in a locked UI state and cannot be modified. + - ``EDITABLE``: The job is in an editable state and can be modified. :param email_notifications: :class:`JobEmailNotifications` (optional) An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted. + :param environment_variables: List[:class:`JobEnvironmentVariables`] + (optional) + Named environment-variable entries that tasks can reference by key from + ``TaskSettings.environment_variables_key``. Each entry holds inline ``variables`` plus optional + ``.env`` ``files``. Maximum 10 entries per job. Entries are independent of one another — there is + no cross-entry merging. :param environments: List[:class:`JobEnvironment`] (optional) A list of task execution environment specifications that can be referenced by serverless tasks of this job. For serverless notebook tasks, if the environment_key is not specified, the notebook @@ -156,16 +162,16 @@ paths: using environment_key in the task settings. :param format: :class:`Format` (optional) Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When - using the Jobs API 2.1 this value is always set to `"MULTI_TASK"`. + using the Jobs API 2.1 this value is always set to ``"MULTI_TASK"``. :param git_source: :class:`GitSource` (optional) An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. - If `git_source` is set, these tasks retrieve the file from the remote repository by default. - However, this behavior can be overridden by setting `source` to `WORKSPACE` on the task. + If ``git_source`` is set, these tasks retrieve the file from the remote repository by default. + However, this behavior can be overridden by setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are - used, `git_source` must be defined on the job. + used, ``git_source`` must be defined on the job. :param health: :class:`JobsHealthRules` (optional) :param job_clusters: List[:class:`JobCluster`] (optional) @@ -179,32 +185,35 @@ paths: affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed - 1000. Setting this value to `0` causes all new runs to be skipped. + 1000. Setting this value to ``0`` causes all new runs to be skipped. :param name: str (optional) An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding. :param notification_settings: :class:`JobNotificationSettings` (optional) Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` for this job. + ``email_notifications`` and ``webhook_notifications`` for this job. :param parameters: List[:class:`JobParameterDefinition`] (optional) Job-level parameter definitions + :param parent_path: str (optional) + Path of the job parent folder in workspace file tree. If absent, the job doesn't have a workspace + object. :param performance_target: :class:`PerformanceTarget` (optional) The performance mode on a serverless job. This field determines the level of compute performance or cost-efficiency for the run. The performance target does not apply to tasks that run on Serverless GPU compute. - * `STANDARD`: Enables cost-efficient execution of serverless workloads. * `PERFORMANCE_OPTIMIZED`: - Prioritizes fast startup and execution times through rapid scaling and optimized cluster - performance. + - ``STANDARD``: Enables cost-efficient execution of serverless workloads. + - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and + optimized cluster performance. :param queue: :class:`QueueSettings` (optional) The queue settings of the job. :param run_as: :class:`JobRunAs` (optional) The user or service principal that the job runs as, if specified in the request. This field - indicates the explicit configuration of `run_as` for the job. To find the value in all cases, - explicit or implicit, use `run_as_user_name`. + indicates the explicit configuration of ``run_as`` for the job. To find the value in all cases, + explicit or implicit, use ``run_as_user_name``. :param schedule: :class:`CronSchedule` (optional) An optional periodic schedule for this job. The default behavior is that the job only runs when - triggered by clicking “Run Now” in the Jobs UI or sending an API request to `runNow`. + triggered by clicking “Run Now” in the Jobs UI or sending an API request to ``runNow``. :param tags: Dict[str,str] (optional) A map of tags associated with the job. These are forwarded to the cluster as cluster tags for jobs clusters, and are subject to the same limitations as cluster tags. A maximum of 25 tags can be added @@ -213,17 +222,17 @@ paths: A list of task specifications to be executed by this job. It supports up to 1000 elements in write endpoints (:method:jobs/create, :method:jobs/reset, :method:jobs/update, :method:jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them - using :method:jobs/get. Use the `next_page_token` field at the object root to determine if more + using :method:jobs/get. Use the ``next_page_token`` field at the object root to determine if more results are available. :param timeout_seconds: int (optional) - An optional timeout applied to each run of this job. A value of `0` means no timeout. + An optional timeout applied to each run of this job. A value of ``0`` means no timeout. :param trigger: :class:`TriggerSettings` (optional) A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or sending an API request to - `runNow`. + ``runNow``. :param usage_policy_id: str (optional) The id of the user specified usage policy to use for this job. If not specified, a default usage - policy may be applied when creating or modifying the job. See `effective_usage_policy_id` for the + policy may be applied when creating or modifying the job. See ``effective_usage_policy_id`` for the usage policy used by this workload. :param webhook_notifications: :class:`WebhookNotifications` (optional) A collection of system notification IDs to notify when runs of this job begin or complete. @@ -263,68 +272,59 @@ paths: The id of the user specified budget policy to use for this job. If not specified, a default budget policy may be applied when creating or modifying the job. See - `effective_budget_policy_id` for the budget policy used by + ``effective_budget_policy_id`` for the budget policy used by this workload. continuous: type: object properties: + maintenance_window: + type: object + properties: + start_hour: + type: integer + description: >- + An integer between 0 and 23 denoting the start hour + for the maintenance window in the 24-hour day. + Platform-initiated maintenance is triggered only + within a one-hour window starting at this hour. This + field is required. + day_of_week: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: >- + The day of week on which maintenance is allowed to + happen. This field is required. + timezone_id: + type: string + description: >- + A Java timezone ID. The maintenance window is + resolved with respect to this timezone. See `Java + TimeZone + `__ + for details. This field is required. + required: + - start_hour + - day_of_week + - timezone_id + description: >- + A recurring weekly time window during which + platform-initiated maintenance is allowed to run for + a continuous job. pause_status: type: string x-enum: - PAUSED - UNPAUSED description: >- - Create a collection of name/value pairs. - - - Example enumeration: - - - >>> class Color(Enum): - - ... RED = 1 - - ... BLUE = 2 - - ... GREEN = 3 - - - Access them by: - - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many - members they have: - - - >>> len(Color) - - 3 - - - >>> list(Color) - - [, , ] - - - Methods can be added to enumerations, and members can - have their own - - attributes -- see the documentation for details. + Indicate whether the continuous execution of the job is + paused or not. Defaults to UNPAUSED. task_retry_mode: type: string x-enum: @@ -336,7 +336,7 @@ paths: description: >- An optional continuous property for this job. The continuous property will ensure that there is always one run executing. - Only one of `schedule` and `continuous` can be used. + Only one of ``schedule`` and ``continuous`` can be used. deployment: type: object properties: @@ -346,13 +346,27 @@ paths: - BUNDLE - SYSTEM_MANAGED description: >- - * `BUNDLE`: The job is managed by Databricks Asset - Bundle. * `SYSTEM_MANAGED`: The job is + - ``BUNDLE``: The job is managed by Databricks Asset + Bundle. - managed by Databricks and is read-only. + - ``SYSTEM_MANAGED``: The job is managed by Databricks + and is read-only. + deployment_id: + type: string + description: >- + ID of the deployment that manages this job. Only set + when ``kind`` is ``BUNDLE``. Used to look up deployment + metadata from the Deployment Metadata service. metadata_file_path: type: string description: Path of the file that contains deployment metadata. + version_id: + type: string + description: >- + ID of the version of the deployment that produced this + job. Only set when ``kind`` is ``BUNDLE``. Identifies a + specific snapshot of the deployment in the Deployment + Metadata service. required: - kind description: Deployment information for jobs managed by external sources. @@ -367,9 +381,9 @@ paths: - EDITABLE - UI_LOCKED description: >- - Edit mode of the job. * `UI_LOCKED`: The job is in a locked - UI state and cannot be modified. * `EDITABLE`: The job is in - an editable state and can be modified. + Edit mode of the job. - ``UI_LOCKED``: The job is in a + locked UI state and cannot be modified. - ``EDITABLE``: The + job is in an editable state and can be modified. email_notifications: type: object properties: @@ -382,9 +396,9 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold specified for - the `RUN_DURATION_SECONDS` metric in the `health` field. - If no rule for the `RUN_DURATION_SECONDS` metric is - specified in the `health` field for the job, + the ``RUN_DURATION_SECONDS`` metric in the ``health`` + field. If no rule for the ``RUN_DURATION_SECONDS`` + metric is specified in the ``health`` field for the job, notifications are not sent. on_failure: type: array @@ -394,10 +408,26 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an - `INTERNAL_ERROR` `life_cycle_state` or a `FAILED`, or - `TIMED_OUT` result_state. If this is not specified on - job creation, reset, or update the list is empty, and + ``INTERNAL_ERROR`` ``life_cycle_state`` or a ``FAILED``, + or ``TIMED_OUT`` result_state. If this is not specified + on job creation, reset, or update the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a continuous + job. on_start: type: array items: @@ -414,11 +444,12 @@ paths: description: >- A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. - Streaming backlog thresholds can be set in the `health` - field using the following metrics: - `STREAMING_BACKLOG_BYTES`, `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the + Streaming backlog thresholds can be set in the + ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. on_success: @@ -428,14 +459,93 @@ paths: description: >- A list of email addresses to be notified when a run successfully completes. A run is considered to have - completed successfully if it ends with a `TERMINATED` - `life_cycle_state` and a `SUCCESS` result_state. If not - specified on job creation, reset, or update, the list is - empty, and notifications are not sent. + completed successfully if it ends with a ``TERMINATED`` + ``life_cycle_state`` and a ``SUCCESS`` result_state. If + not specified on job creation, reset, or update, the + list is empty, and notifications are not sent. description: >- An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted. + environment_variables: + type: array + items: + type: object + properties: + environment_variables_key: + type: string + description: >- + Identifier for this entry. Must be unique within + ``JobSettings.environment_variables``. Tasks reference + it from ``TaskSettings.environment_variables_key``. + files: + type: array + items: + type: string + description: >- + Workspace (``/Workspace/...``) or UC Volumes + (``/Volumes/...``) paths to ``.env`` files. Maximum 5 + files. Files are read, parsed, and merged at task + execution time, not at job creation or update API call + time. File format: each line must be exactly + ``KEY=VALUE``. Keys must match the same regex as + inlined variable names (``^[A-Za-z_][A-Za-z0-9_]*$``); + the value continues to the end of the line. No other + syntax is supported — no comments, no quoted values, + no escape sequences, no variable interpolation. Any + line that does not match the ``KEY=VALUE`` shape fails + the run. Size limits: maximum 1,048,576 bytes (1 MiB) + per file on disk; maximum 131,072 bytes (128 KiB) per + ``KEY=VALUE`` line combined. Caps are enforced at read + time in jobs-runner — files exceeding the per-file + cap, or lines exceeding the per-line cap, fail the + run. On a duplicate key, the later file wins; + ``variables`` override values from any file. Values + may contain {{secrets/scope/key}} references; those + are resolved at task execution time and never + persisted in resolved form. Do not use these files to + store raw secret values; consult `secret management + `__ + for the right way to pass sensitive values. + variables: + type: object + description: >- + Environment variables specified directly as key/value + pairs (as opposed to ``files``, which are read from + ``.env`` file paths). Maximum 100 entries. Each key + must match ``^[A-Za-z_][A-Za-z0-9_]*$`` and be 1 to + 256 characters long. Each value is up to 512 + characters; larger values should be moved into a + ``.env`` file referenced from ``files``. On a + duplicate key, ``variables`` override values from any + file in ``files``. Values may contain + {{secrets/scope/key}} references; those are resolved + at task execution time and never persisted in resolved + form. Do not use this field to store a raw secret + value; consult `secret management + `__ + for the right way to pass sensitive values. + description: >- + A named environment-variable entry, defined once at the + job level and referenced by key from one + or more tasks. Entries live on ``JobSettings.environment_variables``, and tasks select one via + ``TaskSettings.environment_variables_key``. + + Precedence within a single entry (later overrides earlier on duplicate keys): + + files (merged in submitted order; later file in the list wins) └─▶ variables (inline k/v + override file contents) └─▶ runtime (os.environ / sys.env — overrides all, + classic-parity) + + Resolved values are never persisted: ``{{secrets/scope/key}}`` references in ``variables`` or in + ``files`` contents are stored verbatim and resolved at task execution time on the DP . + description: >- + Named environment-variable entries that tasks can reference + by key from ``TaskSettings.environment_variables_key``. Each + entry holds inline ``variables`` plus optional ``.env`` + ``files``. Maximum 10 entries per job. Entries are + independent of one another — there is no cross-entry + merging. environments: type: array items: @@ -464,7 +574,7 @@ paths: description: >- Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs - API 2.1 this value is always set to `"MULTI_TASK"`. + API 2.1 this value is always set to ``"MULTI_TASK"``. git_source: type: object properties: @@ -537,14 +647,14 @@ paths: description: >- Dirty state indicates the job is not fully synced with the job specification in the remote repository. - Possible values are: * `NOT_SYNCED`: The job is not - yet synced with the remote job specification. Import - the remote job specification from UI to make the job - fully synced. * `DISCONNECTED`: The job is temporary - disconnected from the remote job specification and - is allowed for live edit. Import the remote job - specification again from UI to make the job fully - synced. + Possible values are: - ``NOT_SYNCED``: The job is + not yet synced with the remote job specification. + Import the remote job specification from UI to make + the job fully synced. - ``DISCONNECTED``: The job is + temporary disconnected from the remote job + specification and is allowed for live edit. Import + the remote job specification again from UI to make + the job fully synced. required: - job_config_path - import_from_git_branch @@ -565,12 +675,13 @@ paths: An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, - and SQL File tasks. If `git_source` is set, these tasks + and SQL File tasks. If ``git_source`` is set, these tasks retrieve the file from the remote repository by default. - However, this behavior can be overridden by setting `source` - to `WORKSPACE` on the task. Note: dbt and SQL File tasks - support only version-controlled sources. If dbt or SQL File - tasks are used, `git_source` must be defined on the job. + However, this behavior can be overridden by setting + ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL + File tasks support only version-controlled sources. If dbt + or SQL File tasks are used, ``git_source`` must be defined + on the job. health: type: object properties: @@ -592,27 +703,21 @@ paths: evaluated for a particular health rule. - * `RUN_DURATION_SECONDS`: Expected total time for - a run in seconds. * `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data waiting - to be consumed across all streams. This metric + - ``RUN_DURATION_SECONDS``: Expected total time + for a run in seconds. - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate of the - maximum offset lag - - across all streams. This metric is in Public - Preview. * `STREAMING_BACKLOG_SECONDS`: An - estimate - - of the maximum consumer delay across all streams. - This metric is in Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of the + - ``STREAMING_BACKLOG_BYTES``: An estimate of the + maximum bytes of data waiting to be consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate of + the maximum offset lag across all streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of + the maximum consumer delay across all streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate of the maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. This metric is in Public Preview. op: type: string x-enum: @@ -644,6 +749,14 @@ paths: description: >- If new_cluster, a description of a cluster that is created for each task. + serverless_compute_id: + type: string + description: >- + The ID of the serverless compute object to bind this + cluster to. At most one JobCluster per job may set + this field; the rate limit defined on the referenced + serverless compute applies across all tasks bound to + this cluster. required: - job_cluster_key - new_cluster @@ -667,7 +780,7 @@ paths: concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting - this value to `0` causes all new runs to be skipped. + this value to ``0`` causes all new runs to be skipped. name: type: string description: >- @@ -682,11 +795,11 @@ paths: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is skipped. + specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending - notifications to each of the `email_notifications` and - `webhook_notifications` for this job. + notifications to each of the ``email_notifications`` and + ``webhook_notifications`` for this job. parameters: type: array items: @@ -701,6 +814,11 @@ paths: - name - default description: Job-level parameter definitions + parent_path: + type: string + description: >- + Path of the job parent folder in workspace file tree. If + absent, the job doesn't have a workspace object. performance_target: type: string x-enum: @@ -710,9 +828,9 @@ paths: The performance mode on a serverless job. This field determines the level of compute performance or cost-efficiency for the run. The performance target does not - apply to tasks that run on Serverless GPU compute. * - `STANDARD`: Enables cost-efficient execution of serverless - workloads. * `PERFORMANCE_OPTIMIZED`: Prioritizes fast + apply to tasks that run on Serverless GPU compute. - + ``STANDARD``: Enables cost-efficient execution of serverless + workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. queue: @@ -736,7 +854,7 @@ paths: type: string description: >- Application ID of an active service principal. Setting - this field requires the `servicePrincipal/user` role. + this field requires the ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -745,8 +863,9 @@ paths: description: >- The user or service principal that the job runs as, if specified in the request. This field indicates the explicit - configuration of `run_as` for the job. To find the value in - all cases, explicit or implicit, use `run_as_user_name`. + configuration of ``run_as`` for the job. To find the value + in all cases, explicit or implicit, use + ``run_as_user_name``. schedule: type: object properties: @@ -756,15 +875,48 @@ paths: type: string description: >- A Java timezone ID. The schedule for a job is resolved - with respect to this timezone. See [Java TimeZone] for - details. This field is required. [Java TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + with respect to this timezone. See `Java TimeZone + `__ + for details. This field is required. pause_status: type: string x-enum: - PAUSED - UNPAUSED description: Indicate whether this schedule is paused or not. + sql_condition: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to run + the condition query against. + trigger_mode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + Determines how the SQL query result is interpreted + to decide whether the condition fires. Must be set + to a recognized value when provided. When unset on + an existing serialized configuration, the server + preserves the original semantics by interpreting it + as ``QUERY_RETURNS_ROWS``. New configurations should + set this explicitly — explicit + ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + description: >- + SQL condition that must be satisfied before a scheduled + run is triggered. The condition is evaluated after the + cron expression fires and must return a truthy result + for the run to proceed. required: - quartz_cron_expression - timezone_id @@ -772,7 +924,7 @@ paths: An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request - to `runNow`. + to ``runNow``. tags: type: object description: >- @@ -787,11 +939,446 @@ paths: properties: task_key: type: string + agentic_task: + type: object + properties: + context: + type: object + properties: + conversation_ids: + type: array + items: + type: string + description: >- + Optional. Conversation IDs to load as context + for the agent. + instructions: + type: array + items: + type: string + description: >- + Optional. Additional instructions to guide the + agent's behavior. + description: >- + Optional. Context input providing conversation + history and instructions. + genie_code_api: + type: object + properties: {} + description: >- + Inline Genie Code conversation driven by a single + prompt. Mutually exclusive with the supervisor + variants. + goal: + type: string + description: >- + Deprecated. Use ``input`` (field 7) instead. Kept + for backwards compatibility with existing callers; + will be removed in a future revision. + input: + type: string + description: >- + Optional. The user query / task input the agent + must accomplish. Mirrors the OpenAI Responses API + ``input`` field. Replaces the deprecated ``goal`` + field; new callers should populate ``input`` going + forward. + output_schema: + type: object + properties: + properties: + type: object + description: Map of property name to property definition. + description: >- + Optional. JSON-Schema-style declaration of the + structured output the agent should produce. + Replaces the deprecated ``task_output_schema`` + map; new callers should populate ``output_schema`` + going forward. + supervisor_agent: + type: object + properties: + agent_id: + type: string + description: >- + Required. The ID of the supervisor agent + (tile_id). + description: >- + A Supervisor Agent that orchestrates sub-agents + and tools, referenced by tile_id. + supervisor_api: + type: object + properties: + instructions: + type: string + description: >- + Optional. System instructions that guide how + the supervisor routes queries across tools and + synthesizes responses. + model: + type: string + description: >- + Required. Databricks-provided FMAPI model, + e.g. "databricks-claude-sonnet-4-5". + tools: + type: array + items: + type: object + properties: + app: + type: object + properties: + name: + type: string + description: >- + Databricks App tool config. Set when the + discriminator == "app". + catalog: + type: object + properties: + name: + type: string + description: >- + Unity Catalog catalog tool config. Set + when the discriminator == "catalog". + dashboard: + type: object + properties: + dashboard_id: + type: string + description: >- + Lakeview dashboard tool config. Set when + the discriminator == "dashboard". + description: + type: string + description: >- + Optional. Free-form description of the + tool. + genie_space: + type: object + properties: + space_id: + type: string + description: >- + Genie space tool config. Set when the + discriminator == "genie_space". + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + description: >- + Knowledge Assistant tool config. Set + when the discriminator == + "knowledge_assistant". + schema: + type: object + properties: + name: + type: string + description: >- + Unity Catalog schema tool config. Set + when the discriminator == "schema". + serving_endpoint: + type: object + properties: + name: + type: string + description: >- + Model serving endpoint tool config. Set + when the discriminator == + "serving_endpoint". + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: Supervisor Agent ID. + description: >- + Supervisor Agent tool config (a + supervisor agent used as a tool). Set + when the discriminator == + "supervisor_agent". + table: + type: object + properties: + name: + type: string + description: >- + Unity Catalog table tool config. Set + when the discriminator == "table". + tool_type: + type: string + uc_connection: + type: object + properties: + name: + type: string + description: >- + External MCP / UC connection tool + config. Set when the discriminator == + "uc_connection". + uc_function: + type: object + properties: + name: + type: string + description: >- + Unity Catalog function tool config. Set + when the discriminator == "uc_function". + uc_mcp: + type: object + properties: + name: + type: string + description: >- + UC-registered MCP service tool config. + Set when the discriminator == "uc_mcp". + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Vector Search index tool config. Set + when the discriminator == + "vector_search_index". + volume: + type: object + properties: + name: + type: string + description: >- + Unity Catalog volume tool config. Set + when the discriminator == "volume". + web_search: + type: object + properties: {} + description: >- + Web search tool config. Set when the + discriminator == "web_search". + description: >- + Tool configuration for an inlined agentic + task agent. Discriminator selects which + per-asset + config message applies (``genie_space``, ``knowledge_assistant``, ``uc_function``, + ``uc_connection``, ``uc_mcp``, ``app``, ``table``, ``vector_search_index``, ``volume``, + ``catalog``, ``schema``, ``dashboard``, ``supervisor_agent``, ``serving_endpoint``, + ``web_search``); ``description`` is a free-form top-level description; each per-asset config + carries only the asset's identifier. + + The discriminator is exposed under two interchangeable names: ``type`` mirrors the Supervisor + (Responses) API surface, and ``tool_type`` is the AIP-friendly alias for clients where ``type`` + is a reserved keyword (Terraform, etc.). Set exactly one. + description: Optional. The tools the inlined agent can use. + description: >- + Inlined Responses-API supervisor configuration + (model + instructions + tools). Mutually exclusive + with ``supervisor_agent``. + task_output_schema: + type: object + description: >- + Deprecated. Use ``output_schema`` (field 8) + instead. Kept for backwards compatibility with + existing callers; will be removed in a future + revision. + trace_destination: + type: object + properties: + catalog_name: + type: string + description: >- + Unity Catalog catalog name for the trace + table. + experiment_id: + type: string + description: MLflow experiment ID where traces are written. + schema_name: + type: string + description: Unity Catalog schema name for the trace table. + table_prefix: + type: string + description: >- + Table-name prefix under + ``.``. Trace tables are + created as + ``.._<...>``. + description: >- + Optional. Where MLflow traces produced by this + task run should be persisted. When unset, traces + follow the workspace default destination. + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + type: object + properties: + experiment: + type: string + description: >- + MLflow experiment name for this run. If an + experiment with this name already exists under the + calling user, the run is appended to it; otherwise + a new experiment is created. To target a specific + MLflow storage location (for example, when running + as a service principal), set + ``mlflow_experiment_directory``. + deployments: + type: array + items: + type: object + properties: + command_path: + type: string + description: >- + Workspace path of the script to run on each + node in this deployment. Upload the script + to this path and supply the path here. When + the task runs, the file at this path is run + on each node; if it fails, the task fails + with its exit code. Example script contents: + Plain Python: python train.py --epochs 10 + Multi-GPU via accelerate: accelerate launch + train.py --config config.yaml Distributed + via torchrun: torchrun --nproc_per_node=8 + train.py + compute: + type: object + properties: + accelerator_type: + type: string + x-enum: + - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 + - GPU_8xH100 + description: >- + Hardware accelerator type (for example, + ``GPU_1xA10`` or ``GPU_8xH100``). The + number of accelerators per node is + encoded in the enum value — + ``GPU_8xH100`` means 8 H100 GPUs per + node. + accelerator_count: + type: integer + description: >- + Total number of accelerators across all + nodes. Must be a positive multiple of + the per-node accelerator count encoded + in ``accelerator_type``. For example, + ``GPU_8xH100`` with ``accelerator_count: + 16`` allocates 2 nodes (8 GPUs per + node). + required: + - accelerator_type + - accelerator_count + description: >- + Compute resources allocated to each node in + this deployment. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs on + the specified container image instead of the + default Databricks client image. Format: + ``{organization}/{repository}:{tag}`` + name: + type: string + description: >- + Optional human-readable name for this + deployment (for example, ``driver``, + ``worker``, ``param_server``). Used for log + and UI display. Distinct names are + recommended so deployments can be told + apart, but uniqueness is not enforced. + required: + - command_path + - compute + description: >- + DeploymentSpec: configuration for one deployment + within an AiRuntimeTask. Each entry in + ``AiRuntimeTask.deployments`` describes a group of nodes that share the same command and + compute. Many single-program training algorithms use a single entry where every node runs the + same command; role-split workloads (driver + worker, parameter server, separate eval node, etc.) + use multiple entries. + description: >- + Deployment specs for this task. Exactly one + deployment is currently supported (a single entry + where every node runs the same command); this is a + current-Preview constraint. Role-split workloads + (driver + worker, parameter server, separate eval + node, etc.) with multiple entries are the eventual + intent but not yet supported. + code_source_path: + type: string + description: >- + Workspace or UC volume path of the code-source + archive, unpacked on each node and exposed through + ``$CODE_SOURCE``. Set by first-party tooling; not + for direct callers. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom container + image. When set, the task runs on the specified + container image instead of the default Databricks + client image. Format: + ``{organization}/{repository}:{tag}`` + mlflow_experiment_directory: + type: string + description: >- + Optional workspace directory under which the + MLflow experiment named in ``experiment`` is + created. Must start with ``/Workspace``. Set this + when running as a service principal that has no + default user directory; for regular users the + experiment defaults to the user's home directory. + mlflow_run: + type: string + description: >- + Optional display name for the MLflow run created + under ``experiment``. If omitted, MLflow generates + a default name. + parameters: + type: object + description: >- + Optional named parameters passed to each + deployment's command. Keys are parameter names, + values the corresponding arguments (for example, + ``{"epochs": "3", "dataset": + "s3://bucket/train"}``). Values may contain + dynamic references such as + ``{{job.trigger.time.iso_date}}`` or + ``{{tasks..values.}}``, which Jobs + substitutes before execution (see + ``AiRuntimeTaskResolvedValues.parameters`` in + runs.proto). + required: + - experiment + - deployments + description: >- + The task runs a multi-gpu compute workload on + Databricks AI Runtime. Specify the accelerator type + and count, the command to run, and where the + workload's code and MLflow output are stored. alert_task: type: object properties: alert_id: type: string + parameters: + type: object + description: >- + Per-run parameter overrides, keyed by parameter + name, applied onto the alert's stored query + parameters before the query is executed. Only + scalar values are supported. Values may reference + job parameters with ``{{job.parameters.*}}``, + which are resolved before the task runs. An + override whose key does not match a stored + parameter fails the task run. Limited to 10000 + characters when serialized as JSON; keys must be + 1-100 characters and contain only letters, digits, + underscores, dashes, and periods. subscribers: type: array items: @@ -819,11 +1406,14 @@ paths: type: string description: >- The workspace_path is the path to the alert file - in the workspace. The path: * must start with - "/Workspace" * must be a normalized path. User has + in the workspace. The path: - must start with + "/Workspace" - must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert. - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends + notifications to subscribers when the ``alert_task`` + field is present. clean_rooms_notebook_task: type: object properties: @@ -849,10 +1439,10 @@ paths: - clean_room_name - notebook_name description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. [clean - rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ + notebook when the ``clean_rooms_notebook_task`` field + is present. compute: type: object properties: @@ -872,29 +1462,21 @@ paths: - LESS_THAN_OR_EQUAL - NOT_EQUAL description: >- - * `EQUAL_TO`, `NOT_EQUAL` operators perform string - comparison of their operands. This means that - - `“12.0” == “12”` will evaluate to `false`. * - `GREATER_THAN`, `GREATER_THAN_OR_EQUAL`, - - `LESS_THAN`, `LESS_THAN_OR_EQUAL` operators - perform numeric comparison of their operands. - - `“12.0” >= “12”` will evaluate to `true`, `“10.0” - >= “12”` will evaluate to - - `false`. - + - ``EQUAL_TO``, ``NOT_EQUAL`` operators perform + string comparison of their operands. This means + that ``“12.0” == “12”`` will evaluate to ``false``. + - ``GREATER_THAN``, ``GREATER_THAN_OR_EQUAL``, + ``LESS_THAN``, ``LESS_THAN_OR_EQUAL`` operators + perform numeric comparison of their operands. ``“12.0” >= “12”`` will evaluate to + ``true``, ``“10.0” >= “12”`` will evaluate to ``false``. The boolean comparison to task values can be - implemented with operators `EQUAL_TO`, - `NOT_EQUAL`. + implemented with operators ``EQUAL_TO``, - If a task value was set to a boolean value, it - will be serialized to `“true”` or + ``NOT_EQUAL``. If a task value was set to a + boolean value, it will be serialized to - `“false”` for the comparison. + ``“true”`` or ``“false”`` for the comparison. left: type: string description: >- @@ -914,9 +1496,9 @@ paths: description: >- The task evaluates a condition that can be used to control the execution of other tasks when the - `condition_task` field is present. The condition task - does not require a cluster to execute and does not - support retries or notifications. + ``condition_task`` field is present. The condition + task does not require a cluster to execute and does + not support retries or notifications. dashboard_task: type: object properties: @@ -933,14 +1515,14 @@ paths: parameter key. The parameter value format is dependent on the filter type: - For text and single-select filters, provide a single value - (e.g. `"value"`) - For date and datetime filters, - provide the value in ISO 8601 format (e.g. - `"2000-01-01T00:00:00"`) - For multi-select - filters, provide a JSON array of values (e.g. - `"[\"value1\",\"value2\"]"`) - For range and date - range filters, provide a JSON object with `start` - and `end` (e.g. - `"{\"start\":\"1\",\"end\":\"10\"}"`) + (e.g. ``"value"``) - For date and datetime + filters, provide the value in ISO 8601 format + (e.g. ``"2000-01-01T00:00:00"``) - For + multi-select filters, provide a JSON array of + values (e.g. ``"[\"value1\",\"value2\"]"``) - For + range and date range filters, provide a JSON + object with ``start`` and ``end`` (e.g. + ``"{\"start\":\"1\",\"end\":\"10\"}"``) subscription: type: object properties: @@ -962,8 +1544,8 @@ paths: type: string description: >- A snapshot of the dashboard will be sent - to the user's email when the `user_name` - field is present. + to the user's email when the + ``user_name`` field is present. description: >- The list of subscribers to send the snapshot of the dashboard to. @@ -1039,7 +1621,7 @@ paths: description: >- Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If - not provided, the `default` schema is used. + not provided, the ``default`` schema is used. source: type: string x-enum: @@ -1047,15 +1629,15 @@ paths: - WORKSPACE description: >- Optional location type of the project directory. - When set to `WORKSPACE`, the project will be + When set to ``WORKSPACE``, the project will be retrieved from the local Databricks workspace. - When set to `GIT`, the project will be retrieved - from a Git repository defined in `git_source`. If - the value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` otherwise. - * `WORKSPACE`: Project is located in Databricks - workspace. * `GIT`: Project is located in cloud - Git provider. + When set to ``GIT``, the project will be retrieved + from a Git repository defined in ``git_source``. + If the value is empty, the task will use ``GIT`` + if ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Project is located in + Databricks workspace. - ``GIT``: Project is + located in cloud Git provider. warehouse_id: type: string description: >- @@ -1063,12 +1645,12 @@ paths: provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the - `--profiles-dir` command line argument. + ``--profiles-dir`` command line argument. required: - commands description: >- The task runs one or more dbt commands when the - `dbt_task` field is present. The dbt task requires + ``dbt_task`` field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse. depends_on: @@ -1090,9 +1672,9 @@ paths: An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task - will run only if the `run_if` condition is true. The - key is `task_key`, and the value is the name assigned - to the dependent task. + will run only if the ``run_if`` condition is true. The + key is ``task_key``, and the value is the name + assigned to the dependent task. description: type: string description: An optional description for this task. @@ -1116,11 +1698,11 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold specified - for the `RUN_DURATION_SECONDS` metric in the - `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified in the - `health` field for the job, notifications are not - sent. + for the ``RUN_DURATION_SECONDS`` metric in the + ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is specified in + the ``health`` field for the job, notifications + are not sent. on_failure: type: array items: @@ -1129,10 +1711,26 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with - an `INTERNAL_ERROR` `life_cycle_state` or a - `FAILED`, or `TIMED_OUT` result_state. If this is - not specified on job creation, reset, or update + an ``INTERNAL_ERROR`` ``life_cycle_state`` or a + ``FAILED``, or ``TIMED_OUT`` result_state. If this + is not specified on job creation, reset, or update the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a + continuous job. on_start: type: array items: @@ -1150,11 +1748,11 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in - the `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on + the ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. @@ -1166,9 +1764,9 @@ paths: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a - `TERMINATED` `life_cycle_state` and a `SUCCESS` - result_state. If not specified on job creation, - reset, or update, the list is empty, and + ``TERMINATED`` ``life_cycle_state`` and a + ``SUCCESS`` result_state. If not specified on job + creation, reset, or update, the list is empty, and notifications are not sent. description: >- An optional set of email addresses that is notified @@ -1181,6 +1779,16 @@ paths: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry + defined in ``JobSettings.environment_variables``. The + selected entry's variables and file contents are + applied to this task at execution time. Length and + pattern mirror ``environment_key`` so the two + references look identical to customers reading task + settings. existing_cluster_id: type: string description: >- @@ -1212,12 +1820,30 @@ paths: - task description: >- The task executes a nested task for every input - provided when the `for_each_task` field is present. + provided when the ``for_each_task`` field is present. gen_ai_compute_task: type: object properties: dl_runtime_image: type: string + description: Runtime image + client_version: + type: string + description: >- + Version of the client (e.g., sgcli wheel) that + submitted this task. Used by handlers to gate + behavior or reject incompatible versions. + code_source_tar_path: + type: string + description: >- + Optional path to a tarball containing the user's + workspace contents. When set, the entry script + extracts the tarball into the working directory + before running the training script, so the + training script can import sibling modules and + read packaged data files. Must be a workspace path + (e.g. ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. command: type: string description: >- @@ -1236,6 +1862,13 @@ paths: description: GPU type. required: - num_gpus + docker_image_url: + type: string + description: >- + Optional custom Docker container image URL for + running the training script. Format: + organization/repository:tag (e.g., + "pytorch/pytorch:2.0.1") mlflow_experiment_name: type: string description: >- @@ -1243,6 +1876,24 @@ paths: experiment to log the run to. If name is not found, backend will create the mlflow experiment using the name. + mlflow_run_name: + type: string + description: >- + Optional name to assign to the MLflow run created + for this task. If unset, MLflow auto-generates a + name. Used alongside ``mlflow_experiment_name`` to + identify the run in the MLflow UI. + requirements_yaml_path: + type: string + description: >- + Optional path to a requirements.yaml file + describing pip dependencies to install before + running the training script. Consumed by the entry + script; format matches the runtime + requirements.yaml convention used by sgcli. Must + be a workspace path (e.g. + ``/Workspace/Users/...``) or volume; ``dbfs:/`` is + not supported. source: type: string x-enum: @@ -1250,15 +1901,15 @@ paths: - WORKSPACE description: >- Optional location type of the training script. - When set to `WORKSPACE`, the script will be + When set to ``WORKSPACE``, the script will be retrieved from the local Databricks workspace. - When set to `GIT`, the script will be retrieved - from a Git repository defined in `git_source`. If - the value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` otherwise. - * `WORKSPACE`: Script is located in Databricks - workspace. * `GIT`: Script is located in cloud Git - provider. + When set to ``GIT``, the script will be retrieved + from a Git repository defined in ``git_source``. + If the value is empty, the task will use ``GIT`` + if ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Script is located in + Databricks workspace. - ``GIT``: Script is located + in cloud Git provider. training_script_path: type: string description: >- @@ -1266,9 +1917,9 @@ paths: Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the Databricks workspace, - the path must be absolute and begin with `/`. For - files stored in a remote repository, the path must - be relative. This field is required. + the path must be absolute and begin with ``/``. + For files stored in a remote repository, the path + must be relative. This field is required. yaml_parameters: type: string description: >- @@ -1283,6 +1934,23 @@ paths: parameters passed to the training script. required: - dl_runtime_image + description: >- + DEPRECATED — use ``AiRuntimeTask`` for all new BYOT + multi-node GPU workloads (see + ai_runtime_task.proto). ``AiRuntimeTask`` is the only supported BYOT task type for new + workloads; this proto is retained only for AIR CLI (fka SGCLI) pywheel backwards compatibility + and will be removed once the pywheel → databricks-cli migration completes (post- PuPr). + genie_task: + type: object + properties: + configuration_id: + type: string + description: >- + Required. Resource name of the agent task + configuration to run. + required: + - configuration_id + description: Runs a Genie or Genie Code agent task. health: type: object properties: @@ -1304,30 +1972,25 @@ paths: evaluated for a particular health rule. - * `RUN_DURATION_SECONDS`: Expected total - time for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all streams. - This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate of - the maximum offset lag - - across all streams. This metric is in Public - Preview. * `STREAMING_BACKLOG_SECONDS`: An - estimate + - ``RUN_DURATION_SECONDS``: Expected total + time for a run in seconds. + - ``STREAMING_BACKLOG_BYTES``: An estimate + of the maximum bytes of data waiting to be + consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate + of the maximum offset lag across all + streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of the maximum consumer delay across all - streams. This metric is in Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of - the maximum number of outstanding files + streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate + of the maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. This metric is in Public Preview. op: type: string x-enum: @@ -1353,7 +2016,7 @@ paths: type: string description: >- If job_cluster_key, this task is executed reusing the - cluster specified in `job.settings.job_clusters`. + cluster specified in ``job.settings.job_clusters``. libraries: type: string description: >- @@ -1364,10 +2027,10 @@ paths: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be - unsuccessful if it completes with the `FAILED` - result_state or `INTERNAL_ERROR` `life_cycle_state`. - The value `-1` means to retry indefinitely and the - value `0` means to never retry. + unsuccessful if it completes with the ``FAILED`` + result_state or ``INTERNAL_ERROR`` + ``life_cycle_state``. The value ``-1`` means to retry + indefinitely and the value ``0`` means to never retry. min_retry_interval_millis: type: integer description: >- @@ -1392,20 +2055,20 @@ paths: job. If the run is initiated by a call to :method:jobs/run Now with parameters specified, the two parameters maps are merged. If the same - key is specified in `base_parameters` and in - `run-now`, the value from `run-now` is used. Use - [Task parameter variables] to set parameters - containing information about job runs. If the - notebook takes a parameter that is not specified - in the job’s `base_parameters` or the `run-now` - override parameters, the default value from the - notebook is used. Retrieve these parameters in a - notebook using [dbutils.widgets.get]. The JSON - representation of this field cannot exceed 1MB. - [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-widgets + key is specified in ``base_parameters`` and in + ``run-now``, the value from ``run-now`` is used. + Use `Task parameter variables + `__ + to set parameters containing information about job + runs. If the notebook takes a parameter that is + not specified in the job’s ``base_parameters`` or + the ``run-now`` override parameters, the default + value from the notebook is used. Retrieve these + parameters in a notebook using + `dbutils.widgets.get + `__. + The JSON representation of this field cannot + exceed 1MB. source: type: string x-enum: @@ -1413,19 +2076,19 @@ paths: - WORKSPACE description: >- Optional location type of the notebook. When set - to `WORKSPACE`, the notebook will be retrieved + to ``WORKSPACE``, the notebook will be retrieved from the local Databricks workspace. When set to - `GIT`, the notebook will be retrieved from a Git - repository defined in `git_source`. If the value - is empty, the task will use `GIT` if `git_source` - is defined and `WORKSPACE` otherwise. * - `WORKSPACE`: Notebook is located in Databricks - workspace. * `GIT`: Notebook is located in cloud - Git provider. + ``GIT``, the notebook will be retrieved from a Git + repository defined in ``git_source``. If the value + is empty, the task will use ``GIT`` if + ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Notebook is located in + Databricks workspace. - ``GIT``: Notebook is + located in cloud Git provider. warehouse_id: type: string description: >- - Optional `warehouse_id` to run the notebook on a + Optional ``warehouse_id`` to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses. Note that SQL warehouses only support @@ -1434,7 +2097,7 @@ paths: required: - notebook_path description: >- - The task runs a notebook when the `notebook_task` + The task runs a notebook when the ``notebook_task`` field is present. notification_settings: type: object @@ -1445,17 +2108,18 @@ paths: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is canceled. + specified in ``on_failure`` if the run is + canceled. no_alert_for_skipped_runs: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is skipped. + specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` for - this task. + ``email_notifications`` and ``webhook_notifications`` + for this task. pipeline_task: type: object properties: @@ -1464,19 +2128,115 @@ paths: full_refresh: type: boolean description: >- - If true, triggers a full refresh on the delta live - table. + If true, triggers a full refresh on the spark + declarative pipeline. + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + parameters: + type: object + description: >- + Key/value-map of parameters passed to the pipeline + execution. Limited to 10k characters in total. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are + unioned with other selective refresh options + (refresh_selection, full_refresh_selection) to + determine the final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: A list of tables to update without fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints + without clearing data. required: - pipeline_id description: >- The task triggers a pipeline update when the - `pipeline_task` field is present. Only pipelines + ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: type: object properties: connection_resource_name: type: string + incremental_refresh_config: + type: object + properties: + archive_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the archive window. Default: + MONTH + archive_window_periods: + type: integer + description: >- + Number of periods in the archive window (total + data retained). Must be greater than the + refresh window when using the same + granularity. Default: 3 (with YEAR granularity + = 3 years) + detect_data_changes: + type: boolean + description: >- + If true, Power BI will check whether data has + changed in each partition before reimporting. + Partitions with unchanged data are skipped, + reducing refresh cost further. Uses + MAX(datetime_column) as the change detector. + mode: + type: string + x-enum: + - REFRESH_POLICY_MODE_HYBRID + - REFRESH_POLICY_MODE_IMPORT_ONLY + description: >- + The refresh policy mode. Hybrid adds a + real-time DirectQuery partition for the most + recent data; Import uses only batch-imported + partitions. Default: HYBRID + only_refresh_complete_periods: + type: boolean + description: >- + If true, only refresh complete periods (e.g., + skip today's partial data). + refresh_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the refresh window. Default: + DAY + refresh_window_periods: + type: integer + description: >- + Number of periods in the rolling refresh + window. Partitions within this window are + re-imported on each refresh. Default: 1 + description: >- + Incremental refresh policy applied to all IMPORT + mode tables in the model. Windows and mode are + shared; partition columns are set per-table on + PowerBiTable. power_bi_model: type: object properties: @@ -1570,6 +2330,14 @@ paths: properties: catalog: type: string + incremental_refresh_datetime_column: + type: string + description: >- + The datetime column used for incremental + refresh partitioning on this table. e.g., + "order_date", "updated_at" Only applicable + when the task has incremental_refresh_config + set and this table uses IMPORT storage mode. name: type: string description: The table name in Databricks @@ -1583,6 +2351,24 @@ paths: - DUAL - IMPORT description: The Power BI storage mode of the table + table_type: + type: string + x-enum: + - EXTERNAL + - EXTERNAL_SHALLOW_CLONE + - FOREIGN + - MANAGED + - MANAGED_SHALLOW_CLONE + - MATERIALIZED_VIEW + - METRIC_VIEW + - STREAMING_TABLE + - VIEW + description: >- + The Unity Catalog table type of this table. + When set, the server trusts it to classify + the table during metric-view filtering and + skips the server-side lookup. When unset, + the server resolves the type on its own. description: The tables to be exported to Power BI warehouse_id: type: string @@ -1591,7 +2377,25 @@ paths: source description: >- The task triggers a Power BI semantic model update - when the `power_bi_task` field is present. + when the ``power_bi_task`` field is present. + python_operator_task: + type: object + properties: + main: + type: string + parameters: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: >- + An ordered list of task parameters. + TODO(JOBS-30885): Add limits for parameters. + description: The task runs a Python operator task. python_wheel_task: type: object properties: @@ -1603,28 +2407,28 @@ paths: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using - `$packageName.$entryPoint()` + ``$packageName.$entryPoint()`` named_parameters: type: object description: >- Command-line parameters passed to Python wheel - task in the form of `["--name=task", - "--data=dbfs:/path/to/data.json"]`. Leave it empty - if `parameters` is not null. + task in the form of ``["--name=task", + "--data=dbfs:/path/to/data.json"]``. Leave it + empty if ``parameters`` is not null. parameters: type: array items: type: string description: >- Command-line parameters passed to Python wheel - task. Leave it empty if `named_parameters` is not - null. + task. Leave it empty if ``named_parameters`` is + not null. required: - package_name - entry_point description: >- The task runs a Python wheel when the - `python_wheel_task` field is present. + ``python_wheel_task`` field is present. retry_on_timeout: type: boolean description: >- @@ -1643,14 +2447,14 @@ paths: description: >- An optional value specifying the condition determining whether the task is run once its dependencies have - been completed. * `ALL_SUCCESS`: All dependencies have - executed and succeeded * `AT_LEAST_ONE_SUCCESS`: At - least one dependency has succeeded * `NONE_FAILED`: - None of the dependencies have failed and at least one - was executed * `ALL_DONE`: All dependencies have been - completed * `AT_LEAST_ONE_FAILED`: At least one - dependency failed * `ALL_FAILED`: ALl dependencies - have failed + been completed. - ``ALL_SUCCESS``: All dependencies + have executed and succeeded - + ``AT_LEAST_ONE_SUCCESS``: At least one dependency has + succeeded - ``NONE_FAILED``: None of the dependencies + have failed and at least one was executed - + ``ALL_DONE``: All dependencies have been completed - + ``AT_LEAST_ONE_FAILED``: At least one dependency + failed - ``ALL_FAILED``: ALl dependencies have failed run_job_task: type: object properties: @@ -1662,30 +2466,29 @@ paths: type: string description: >- An array of commands to execute for jobs with the - dbt task, for example `"dbt_commands": ["dbt + dbt task, for example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt deps", "dbt seed", "dbt - run"]` ⚠ **Deprecation note** Use [job parameters] - to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + run"]`` ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR - tasks, for example `"jar_params": ["john doe", - "35"]`. The parameters are used to invoke the main - function of the main class specified in the Spark - JAR task. If not specified upon `run-now`, it - defaults to an empty list. jar_params cannot be + tasks, for example ``"jar_params": ["john doe", + "35"]``. The parameters are used to invoke the + main function of the main class specified in the + Spark JAR task. If not specified upon ``run-now``, + it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed - 10,000 bytes. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) cannot + exceed 10,000 bytes. ⚠ **Deprecation note** Use + `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: Job-level parameters used to trigger the job. @@ -1693,27 +2496,54 @@ paths: type: object description: >- A map from keys to values for jobs with notebook - task, for example `"notebook_params": {"name": - "john doe", "age": "35"}`. The map is passed to + task, for example ``"notebook_params": {"name": + "john doe", "age": "35"}``. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. If not specified - upon `run-now`, the triggered run uses the job’s - base parameters. notebook_params cannot be - specified in conjunction with jar_params. ⚠ - **Deprecation note** Use [job parameters] to pass - information down to tasks. The JSON representation - of this field (for example - `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 bytes. - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, the + triggered run uses the job’s base parameters. + notebook_params cannot be specified in conjunction + with jar_params. ⚠ **Deprecation note** Use `job + parameters + `__ + to pass information down to tasks. The JSON + representation of this field (for example + ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 bytes. pipeline_params: type: object properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are + unioned with other selective refresh options + (refresh_selection, full_refresh_selection) to + determine the final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints + without clearing data. description: >- Controls whether the pipeline should perform a full refresh @@ -1725,58 +2555,61 @@ paths: type: string description: >- A list of parameters for jobs with Python tasks, - for example `"python_params": ["john doe", "35"]`. - The parameters are passed to Python file as - command-line parameters. If specified upon - `run-now`, it would overwrite the parameters + for example ``"python_params": ["john doe", + "35"]``. The parameters are passed to Python file + as command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use - [job parameters] to pass information down to - tasks. Important These parameters accept only - Latin characters (ASCII character set). Using - non-ASCII characters returns an error. Examples of - invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII + character set). Using non-ASCII characters returns + an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and + emojis. spark_submit_params: type: array items: type: string description: >- A list of parameters for jobs with spark submit - task, for example `"spark_submit_params": - ["--class", "org.apache.spark.examples.SparkPi"]`. - The parameters are passed to spark-submit script - as command-line parameters. If specified upon - `run-now`, it would overwrite the parameters + task, for example ``"spark_submit_params": + ["--class", + "org.apache.spark.examples.SparkPi"]``. The + parameters are passed to spark-submit script as + command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use - [job parameters] to pass information down to - tasks. Important These parameters accept only - Latin characters (ASCII character set). Using - non-ASCII characters returns an error. Examples of - invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII + character set). Using non-ASCII characters returns + an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and + emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL task, - for example `"sql_params": {"name": "john doe", - "age": "35"}`. The SQL alert task does not support - custom parameters. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + for example ``"sql_params": {"name": "john doe", + "age": "35"}``. The SQL alert task does not + support custom parameters. ⚠ **Deprecation note** + Use `job parameters + `__ + to pass information down to tasks. required: - job_id description: >- - The task triggers another job when the `run_job_task` - field is present. + The task triggers another job when the + ``run_job_task`` field is present. spark_jar_task: type: object properties: @@ -1788,26 +2621,26 @@ paths: The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library. The code - must use `SparkContext.getOrCreate` to obtain a + must use ``SparkContext.getOrCreate`` to obtain a Spark context; otherwise, runs of the job fail. parameters: type: array items: type: string description: >- - Parameters passed to the main method. Use [Task - parameter variables] to set parameters containing - information about job runs. [Task parameter - variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Parameters passed to the main method. Use `Task + parameter variables + `__ + to set parameters containing information about job + runs. run_as_repl: type: boolean description: >- - Deprecated. A value of `false` is no longer + Deprecated. A value of ``false`` is no longer supported. description: >- - The task runs a JAR when the `spark_jar_task` field is - present. + The task runs a JAR when the ``spark_jar_task`` field + is present. spark_python_task: type: object properties: @@ -1819,10 +2652,10 @@ paths: type: string description: >- Command line parameters passed to the Python file. - Use [Task parameter variables] to set parameters - containing information about job runs. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Use `Task parameter variables + `__ + to set parameters containing information about job + runs. source: type: string x-enum: @@ -1830,20 +2663,21 @@ paths: - WORKSPACE description: >- Optional location type of the Python file. When - set to `WORKSPACE` or not specified, the file will - be retrieved from the local Databricks workspace - or cloud location (if the `python_file` has a URI - format). When set to `GIT`, the Python file will - be retrieved from a Git repository defined in - `git_source`. * `WORKSPACE`: The Python file is - located in a Databricks workspace or at a cloud - filesystem URI. * `GIT`: The Python file is - located in a remote Git repository. + set to ``WORKSPACE`` or not specified, the file + will be retrieved from the local Databricks + workspace or cloud location (if the + ``python_file`` has a URI format). When set to + ``GIT``, the Python file will be retrieved from a + Git repository defined in ``git_source``. - + ``WORKSPACE``: The Python file is located in a + Databricks workspace or at a cloud filesystem URI. + - ``GIT``: The Python file is located in a remote + Git repository. required: - python_file description: >- The task runs a Python file when the - `spark_python_task` field is present. + ``spark_python_task`` field is present. spark_submit_task: type: object properties: @@ -1946,15 +2780,16 @@ paths: - WORKSPACE description: >- Optional location type of the SQL file. When - set to `WORKSPACE`, the SQL file will be + set to ``WORKSPACE``, the SQL file will be retrieved from the local Databricks workspace. - When set to `GIT`, the SQL file will be + When set to ``GIT``, the SQL file will be retrieved from a Git repository defined in - `git_source`. If the value is empty, the task - will use `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: SQL file - is located in Databricks workspace. * `GIT`: - SQL file is located in cloud Git provider. + ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - + ``WORKSPACE``: SQL file is located in + Databricks workspace. - ``GIT``: SQL file is + located in cloud Git provider. required: - path description: >- @@ -1981,12 +2816,12 @@ paths: description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the - `sql_task` field is present. + ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job - task. A value of `0` means no timeout. + task. A value of ``0`` means no timeout. webhook_notifications: type: object properties: @@ -2011,7 +2846,37 @@ paths: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 - destinations can be specified for the `on_failure` + destinations can be specified for the + ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance completes + for a continuous job. A maximum of 3 destinations + can be specified for the + ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance starts + for a continuous job. A maximum of 3 destinations + can be specified for the ``on_maintenance_start`` property. on_start: type: array @@ -2025,7 +2890,7 @@ paths: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 - destinations can be specified for the `on_start` + destinations can be specified for the ``on_start`` property. on_streaming_backlog_exceeded: type: array @@ -2040,16 +2905,17 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog - thresholds can be set in the `health` field using - the following metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on + thresholds can be set in the ``health`` field + using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be - specified for the `on_streaming_backlog_exceeded` - property. + specified for the + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -2063,7 +2929,7 @@ paths: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the - `on_success` property. + ``on_success`` property. description: >- A collection of system notification IDs to notify when runs of this task begin or complete. The default @@ -2077,13 +2943,13 @@ paths: :method:jobs/update, :method:jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them using :method:jobs/get. Use - the `next_page_token` field at the object root to determine - if more results are available. + the ``next_page_token`` field at the object root to + determine if more results are available. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job. A value - of `0` means no timeout. + of ``0`` means no timeout. trigger: type: object properties: @@ -2219,12 +3085,46 @@ paths: x-enum: - DAYS - HOURS + - MINUTES - WEEKS description: The unit of time for the interval. required: - interval - unit description: Periodic trigger settings. + sql_condition: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to run + the condition query against. + trigger_mode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + Determines how the SQL query result is interpreted + to decide whether the condition fires. Must be set + to a recognized value when provided. When unset on + an existing serialized configuration, the server + preserves the original semantics by interpreting it + as ``QUERY_RETURNS_ROWS``. New configurations should + set this explicitly — explicit + ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + description: >- + SQL condition that must be satisfied for the trigger to + fire. Can be used in combination with other trigger + types and runs *after* other trigger types conditions + are evaluated. table_update: type: object properties: @@ -2261,14 +3161,14 @@ paths: A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or sending an - API request to `runNow`. + API request to ``runNow``. usage_policy_id: type: string description: >- The id of the user specified usage policy to use for this job. If not specified, a default usage policy may be applied when creating or modifying the job. See - `effective_usage_policy_id` for the usage policy used by + ``effective_usage_policy_id`` for the usage policy used by this workload. webhook_notifications: type: object @@ -2294,7 +3194,35 @@ paths: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be - specified for the `on_failure` property. + specified for the ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call when + platform-initiated maintenance completes for a + continuous job. A maximum of 3 destinations can be + specified for the ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call when + platform-initiated maintenance starts for a continuous + job. A maximum of 3 destinations can be specified for + the ``on_maintenance_start`` property. on_start: type: array items: @@ -2307,7 +3235,7 @@ paths: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be - specified for the `on_start` property. + specified for the ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -2321,14 +3249,15 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the - `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the + ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -2341,7 +3270,7 @@ paths: description: >- An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 - destinations can be specified for the `on_success` + destinations can be specified for the ``on_success`` property. description: >- A collection of system notification IDs to notify when runs @@ -2502,24 +3431,27 @@ paths: elements. A request for a single job will return all properties for that job, and the first 100 elements - of array properties (`tasks`, + of array properties - `job_clusters`, `environments` and `parameters`). Use the - `next_page_token` field to check for more + (``tasks``, ``job_clusters``, ``environments`` and ``parameters``). Use + the ``next_page_token`` field - results and pass its value as the `page_token` in subsequent requests. - If any array properties have + to check for more results and pass its value as the ``page_token`` in + subsequent requests. If any - more than 100 elements, additional results will be returned on - subsequent requests. Arrays without + array properties have more than 100 elements, additional results will be + returned on subsequent - additional results will be empty on later pages. + requests. Arrays without additional results will be empty on later + pages. :param job_id: int The canonical identifier of the job to retrieve information about. This field is required. + :param include_trigger_state: bool (optional) + Flag that indicates that trigger state should be included in the response. :param page_token: str (optional) - Use `next_page_token` returned from the previous GetJob response to request the next page of the + Use ``next_page_token`` returned from the previous GetJob response to request the next page of the job's array properties. :returns: :class:`Job` @@ -2532,14 +3464,22 @@ paths: description: >- The canonical identifier of the job to retrieve information about. This field is required. + - name: include_trigger_state + in: query + required: false + schema: + type: boolean + description: >- + Flag that indicates that trigger state should be included in the + response. - name: page_token in: query required: false schema: type: string description: >- - Use `next_page_token` returned from the previous GetJob response to - request the next page of the job's array properties. + Use ``next_page_token`` returned from the previous GetJob response + to request the next page of the job's array properties. responses: '200': description: Success @@ -2802,18 +3742,19 @@ paths: elements. A request for a single run will return all properties for that run, and the first 100 elements - of array properties (`tasks`, + of array properties - `job_clusters`, `job_parameters` and `repair_history`). Use the - next_page_token field to check for + (``tasks``, ``job_clusters``, ``job_parameters`` and + ``repair_history``). Use the next_page_token - more results and pass its value as the page_token in subsequent - requests. If any array properties have + field to check for more results and pass its value as the page_token in + subsequent requests. If any - more than 100 elements, additional results will be returned on - subsequent requests. Arrays without + array properties have more than 100 elements, additional results will be + returned on subsequent - additional results will be empty on later pages. + requests. Arrays without additional results will be empty on later + pages. :param run_id: int @@ -2823,7 +3764,7 @@ paths: :param include_resolved_values: bool (optional) Whether to include resolved parameter values in the response. :param page_token: str (optional) - Use `next_page_token` returned from the previous GetRun response to request the next page of the + Use ``next_page_token`` returned from the previous GetRun response to request the next page of the run's array properties. :returns: :class:`Run` @@ -2854,8 +3795,8 @@ paths: schema: type: string description: >- - Use `next_page_token` returned from the previous GetRun response to - request the next page of the run's array properties. + Use ``next_page_token`` returned from the previous GetRun response + to request the next page of the run's array properties. responses: '200': description: Success @@ -2887,7 +3828,7 @@ paths: Retrieve the output and metadata of a single task run. When a notebook task returns a value through - the `dbutils.notebook.exit()` call, you can use this endpoint to + the ``dbutils.notebook.exit()`` call, you can use this endpoint to retrieve that value. Databricks restricts this API to returning the first 5 MB of the output. To return @@ -2896,10 +3837,10 @@ paths: job results in a cloud storage service. - This endpoint validates that the __run_id__ parameter is valid and + This endpoint validates that the **run_id** parameter is valid and returns an HTTP status code 400 if - the __run_id__ parameter is invalid. Runs are automatically removed + the **run_id** parameter is invalid. Runs are automatically removed after 60 days. If you to want to reference them beyond 60 days, you must save old run results before they @@ -2955,10 +3896,10 @@ paths: A filter on the list based on the exact (case insensitive) job name. :param offset: int (optional) The offset of the first job to return, relative to the most recently created job. Deprecated since - June 2023. Use `page_token` to iterate through the pages instead. + June 2023. Use ``page_token`` to iterate through the pages instead. :param page_token: str (optional) - Use `next_page_token` or `prev_page_token` returned from the previous request to list the next or - previous page of jobs respectively. + Use ``next_page_token`` or ``prev_page_token`` returned from the previous request to list the next + or previous page of jobs respectively. :returns: Iterator over :class:`BaseJob` parameters: @@ -2992,15 +3933,15 @@ paths: type: integer description: >- The offset of the first job to return, relative to the most recently - created job. Deprecated since June 2023. Use `page_token` to iterate - through the pages instead. + created job. Deprecated since June 2023. Use ``page_token`` to + iterate through the pages instead. - name: page_token in: query required: false schema: type: string description: >- - Use `next_page_token` or `prev_page_token` returned from the + Use ``next_page_token`` or ``prev_page_token`` returned from the previous request to list the next or previous page of jobs respectively. responses: @@ -3032,12 +3973,12 @@ paths: List runs in descending order by start time. :param active_only: bool (optional) - If active_only is `true`, only active runs are included in the results; otherwise, lists both active - and completed runs. An active run is a run in the `QUEUED`, `PENDING`, `RUNNING`, or `TERMINATING`. - This field cannot be `true` when completed_only is `true`. + If active_only is ``true``, only active runs are included in the results; otherwise, lists both + active and completed runs. An active run is a run in the ``QUEUED``, ``PENDING``, ``RUNNING``, or + ``TERMINATING``. This field cannot be ``true`` when completed_only is ``true``. :param completed_only: bool (optional) - If completed_only is `true`, only completed runs are included in the results; otherwise, lists both - active and completed runs. This field cannot be `true` when active_only is `true`. + If completed_only is ``true``, only completed runs are included in the results; otherwise, lists + both active and completed runs. This field cannot be ``true`` when active_only is ``true``. :param expand_tasks: bool (optional) Whether to include task and cluster details in the response. Note that only the first 100 elements will be shown. Use :method:jobs/getrun to paginate through all tasks and clusters. @@ -3048,18 +3989,18 @@ paths: is 20. If a request specifies a limit of 0, the service instead uses the maximum limit. :param offset: int (optional) The offset of the first run to return, relative to the most recent run. Deprecated since June 2023. - Use `page_token` to iterate through the pages instead. + Use ``page_token`` to iterate through the pages instead. :param page_token: str (optional) - Use `next_page_token` or `prev_page_token` returned from the previous request to list the next or - previous page of runs respectively. + Use ``next_page_token`` or ``prev_page_token`` returned from the previous request to list the next + or previous page of runs respectively. :param run_type: :class:`RunType` (optional) The type of runs to return. For a description of run types, see :method:jobs/getRun. :param start_time_from: int (optional) - Show runs that started _at or after_ this value. The value must be a UTC timestamp in milliseconds. - Can be combined with _start_time_to_ to filter by a time range. + Show runs that started *at or after* this value. The value must be a UTC timestamp in milliseconds. + Can be combined with *start_time_to* to filter by a time range. :param start_time_to: int (optional) - Show runs that started _at or before_ this value. The value must be a UTC timestamp in milliseconds. - Can be combined with _start_time_from_ to filter by a time range. + Show runs that started *at or before* this value. The value must be a UTC timestamp in milliseconds. + Can be combined with *start_time_from* to filter by a time range. :returns: Iterator over :class:`BaseRun` parameters: @@ -3069,20 +4010,20 @@ paths: schema: type: boolean description: >- - If active_only is `true`, only active runs are included in the + If active_only is ``true``, only active runs are included in the results; otherwise, lists both active and completed runs. An active - run is a run in the `QUEUED`, `PENDING`, `RUNNING`, or - `TERMINATING`. This field cannot be `true` when completed_only is - `true`. + run is a run in the ``QUEUED``, ``PENDING``, ``RUNNING``, or + ``TERMINATING``. This field cannot be ``true`` when completed_only + is ``true``. - name: completed_only in: query required: false schema: type: boolean description: >- - If completed_only is `true`, only completed runs are included in the - results; otherwise, lists both active and completed runs. This field - cannot be `true` when active_only is `true`. + If completed_only is ``true``, only completed runs are included in + the results; otherwise, lists both active and completed runs. This + field cannot be ``true`` when active_only is ``true``. - name: expand_tasks in: query required: false @@ -3116,15 +4057,15 @@ paths: type: integer description: >- The offset of the first run to return, relative to the most recent - run. Deprecated since June 2023. Use `page_token` to iterate through - the pages instead. + run. Deprecated since June 2023. Use ``page_token`` to iterate + through the pages instead. - name: page_token in: query required: false schema: type: string description: >- - Use `next_page_token` or `prev_page_token` returned from the + Use ``next_page_token`` or ``prev_page_token`` returned from the previous request to list the next or previous page of runs respectively. - name: run_type @@ -3141,8 +4082,8 @@ paths: schema: type: integer description: >- - Show runs that started _at or after_ this value. The value must be a - UTC timestamp in milliseconds. Can be combined with _start_time_to_ + Show runs that started *at or after* this value. The value must be a + UTC timestamp in milliseconds. Can be combined with *start_time_to* to filter by a time range. - name: start_time_to in: query @@ -3150,9 +4091,9 @@ paths: schema: type: integer description: >- - Show runs that started _at or before_ this value. The value must be + Show runs that started *at or before* this value. The value must be a UTC timestamp in milliseconds. Can be combined with - _start_time_from_ to filter by a time range. + *start_time_from* to filter by a time range. responses: '200': description: Success @@ -3191,100 +4132,100 @@ paths: :param run_id: int The job run ID of the run to repair. The run must not be in progress. :param dbt_commands: List[str] (optional) - An array of commands to execute for jobs with the dbt task, for example `"dbt_commands": ["dbt - deps", "dbt seed", "dbt deps", "dbt seed", "dbt run"]` - - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + An array of commands to execute for jobs with the dbt task, for example ``"dbt_commands": ["dbt + deps", "dbt seed", "dbt deps", "dbt seed", "dbt run"]`` - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. :param jar_params: List[str] (optional) - A list of parameters for jobs with Spark JAR tasks, for example `"jar_params": ["john doe", "35"]`. - The parameters are used to invoke the main function of the main class specified in the Spark JAR - task. If not specified upon `run-now`, it defaults to an empty list. jar_params cannot be specified - in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed 10,000 bytes. - - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. - - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + A list of parameters for jobs with Spark JAR tasks, for example ``"jar_params": ["john doe", + "35"]``. The parameters are used to invoke the main function of the main class specified in the + Spark JAR task. If not specified upon ``run-now``, it defaults to an empty list. jar_params cannot + be specified in conjunction with notebook_params. The JSON representation of this field (for example + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 bytes. + + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. :param job_parameters: Dict[str,str] (optional) - Job-level parameters used in the run. for example `"param": "overriding_val"` + Job-level parameters used in the run. for example ``"param": "overriding_val"`` :param latest_repair_id: int (optional) The ID of the latest repair. This parameter is not required when repairing a run for the first time, but must be provided on subsequent requests to repair the same run. :param notebook_params: Dict[str,str] (optional) - A map from keys to values for jobs with notebook task, for example `"notebook_params": {"name": - "john doe", "age": "35"}`. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. + A map from keys to values for jobs with notebook task, for example ``"notebook_params": {"name": + "john doe", "age": "35"}``. The map is passed to the notebook and is accessible through the + `dbutils.widgets.get `__ function. - If not specified upon `run-now`, the triggered run uses the job’s base parameters. + If not specified upon ``run-now``, the triggered run uses the job’s base parameters. notebook_params cannot be specified in conjunction with jar_params. - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. - The JSON representation of this field (for example `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 bytes. - - [dbutils.widgets.get]: https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + The JSON representation of this field (for example ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 bytes. :param performance_target: :class:`PerformanceTarget` (optional) The performance mode on a serverless job. The performance target determines the level of compute performance or cost-efficiency for the run. This field overrides the performance target defined on the job level. - * `STANDARD`: Enables cost-efficient execution of serverless workloads. * `PERFORMANCE_OPTIMIZED`: - Prioritizes fast startup and execution times through rapid scaling and optimized cluster - performance. + - ``STANDARD``: Enables cost-efficient execution of serverless workloads. + - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and + optimized cluster performance. :param pipeline_params: :class:`PipelineParams` (optional) Controls whether the pipeline should perform a full refresh :param python_named_params: Dict[str,str] (optional) :param python_params: List[str] (optional) - A list of parameters for jobs with Python tasks, for example `"python_params": ["john doe", "35"]`. - The parameters are passed to Python file as command-line parameters. If specified upon `run-now`, it - would overwrite the parameters specified in job setting. The JSON representation of this field (for - example `{"python_params":["john doe","35"]}`) cannot exceed 10,000 bytes. + A list of parameters for jobs with Python tasks, for example ``"python_params": ["john doe", + "35"]``. The parameters are passed to Python file as command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of + this field (for example ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. Important These parameters accept only Latin characters (ASCII character set). Using non-ASCII characters returns an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, and emojis. - - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown :param rerun_all_failed_tasks: bool (optional) - If true, repair all failed tasks. Only one of `rerun_tasks` or `rerun_all_failed_tasks` can be used. + If true, repair all failed tasks. Only one of ``rerun_tasks`` or ``rerun_all_failed_tasks`` can be + used. :param rerun_dependent_tasks: bool (optional) - If true, repair all tasks that depend on the tasks in `rerun_tasks`, even if they were previously - successful. Can be also used in combination with `rerun_all_failed_tasks`. + If true, repair all tasks that depend on the tasks in ``rerun_tasks``, even if they were previously + successful. Can be also used in combination with ``rerun_all_failed_tasks``. :param rerun_tasks: List[str] (optional) The task keys of the task runs to repair. :param spark_submit_params: List[str] (optional) - A list of parameters for jobs with spark submit task, for example `"spark_submit_params": - ["--class", "org.apache.spark.examples.SparkPi"]`. The parameters are passed to spark-submit script - as command-line parameters. If specified upon `run-now`, it would overwrite the parameters specified - in job setting. The JSON representation of this field (for example `{"python_params":["john - doe","35"]}`) cannot exceed 10,000 bytes. + A list of parameters for jobs with spark submit task, for example ``"spark_submit_params": + ["--class", "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script + as command-line parameters. If specified upon ``run-now``, it would overwrite the parameters + specified in job setting. The JSON representation of this field (for example + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. Important These parameters accept only Latin characters (ASCII character set). Using non-ASCII characters returns an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, and emojis. - - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown :param sql_params: Dict[str,str] (optional) - A map from keys to values for jobs with SQL task, for example `"sql_params": {"name": "john doe", - "age": "35"}`. The SQL alert task does not support custom parameters. - - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + A map from keys to values for jobs with SQL task, for example ``"sql_params": {"name": "john doe", + "age": "35"}``. The SQL alert task does not support custom parameters. - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. :returns: Long-running operation waiter for :class:`Run`. @@ -3306,32 +4247,32 @@ paths: type: string description: >- An array of commands to execute for jobs with the dbt task, - for example `"dbt_commands": ["dbt deps", "dbt seed", "dbt - deps", "dbt seed", "dbt run"]` ⚠ **Deprecation note** Use - [job parameters] to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + for example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt + deps", "dbt seed", "dbt run"]`` ⚠ **Deprecation note** Use + `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR tasks, for - example `"jar_params": ["john doe", "35"]`. The parameters + example ``"jar_params": ["john doe", "35"]``. The parameters are used to invoke the main function of the main class specified in the Spark JAR task. If not specified upon - `run-now`, it defaults to an empty list. jar_params cannot + ``run-now``, it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed 10,000 - bytes. ⚠ **Deprecation note** Use [job parameters] to pass - information down to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 + bytes. ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: >- - Job-level parameters used in the run. for example `"param": - "overriding_val"` + Job-level parameters used in the run. for example ``"param": + "overriding_val"`` latest_repair_id: type: integer description: >- @@ -3342,19 +4283,19 @@ paths: type: object description: >- A map from keys to values for jobs with notebook task, for - example `"notebook_params": {"name": "john doe", "age": - "35"}`. The map is passed to the notebook and is accessible - through the [dbutils.widgets.get] function. If not specified - upon `run-now`, the triggered run uses the job’s base - parameters. notebook_params cannot be specified in - conjunction with jar_params. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. The JSON - representation of this field (for example - `{"notebook_params":{"name":"john doe","age":"35"}}`) cannot - exceed 10,000 bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + example ``"notebook_params": {"name": "john doe", "age": + "35"}``. The map is passed to the notebook and is accessible + through the `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, the triggered + run uses the job’s base parameters. notebook_params cannot + be specified in conjunction with jar_params. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. The JSON representation + of this field (for example + ``{"notebook_params":{"name":"john doe","age":"35"}}``) + cannot exceed 10,000 bytes. performance_target: type: string x-enum: @@ -3364,9 +4305,9 @@ paths: The performance mode on a serverless job. The performance target determines the level of compute performance or cost-efficiency for the run. This field overrides the - performance target defined on the job level. * `STANDARD`: - Enables cost-efficient execution of serverless workloads. * - `PERFORMANCE_OPTIMIZED`: Prioritizes fast startup and + performance target defined on the job level. - ``STANDARD``: + Enables cost-efficient execution of serverless workloads. - + ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. pipeline_params: @@ -3374,6 +4315,32 @@ paths: properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are unioned + with other selective refresh options (refresh_selection, + full_refresh_selection) to determine the final set of + flows to refresh. + refresh_selection: + type: array + items: + type: string + description: A list of tables to update without fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints without + clearing data. description: Controls whether the pipeline should perform a full refresh python_named_params: type: object @@ -3383,29 +4350,31 @@ paths: type: string description: >- A list of parameters for jobs with Python tasks, for example - `"python_params": ["john doe", "35"]`. The parameters are + ``"python_params": ["john doe", "35"]``. The parameters are passed to Python file as command-line parameters. If - specified upon `run-now`, it would overwrite the parameters - specified in job setting. The JSON representation of this - field (for example `{"python_params":["john doe","35"]}`) - cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. Important - These parameters accept only Latin characters (ASCII - character set). Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + specified upon ``run-now``, it would overwrite the + parameters specified in job setting. The JSON representation + of this field (for example ``{"python_params":["john + doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII character + set). Using non-ASCII characters returns an error. Examples + of invalid, non-ASCII characters are Chinese, Japanese + kanjis, and emojis. rerun_all_failed_tasks: type: boolean description: >- - If true, repair all failed tasks. Only one of `rerun_tasks` - or `rerun_all_failed_tasks` can be used. + If true, repair all failed tasks. Only one of + ``rerun_tasks`` or ``rerun_all_failed_tasks`` can be used. rerun_dependent_tasks: type: boolean description: >- If true, repair all tasks that depend on the tasks in - `rerun_tasks`, even if they were previously successful. Can - be also used in combination with `rerun_all_failed_tasks`. + ``rerun_tasks``, even if they were previously successful. + Can be also used in combination with + ``rerun_all_failed_tasks``. rerun_tasks: type: array items: @@ -3417,28 +4386,29 @@ paths: type: string description: >- A list of parameters for jobs with spark submit task, for - example `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The parameters are + example ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script as command-line parameters. If - specified upon `run-now`, it would overwrite the parameters - specified in job setting. The JSON representation of this - field (for example `{"python_params":["john doe","35"]}`) - cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. Important - These parameters accept only Latin characters (ASCII - character set). Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + specified upon ``run-now``, it would overwrite the + parameters specified in job setting. The JSON representation + of this field (for example ``{"python_params":["john + doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII character + set). Using non-ASCII characters returns an error. Examples + of invalid, non-ASCII characters are Chinese, Japanese + kanjis, and emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL task, for - example `"sql_params": {"name": "john doe", "age": "35"}`. + example ``"sql_params": {"name": "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ - **Deprecation note** Use [job parameters] to pass - information down to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. required: - run_id responses: @@ -3464,12 +4434,12 @@ paths: post: operationId: jobs_reset summary: >- - Overwrite all settings for the given job. Use the [_Update_ + Overwrite all settings for the given job. Use the [*Update* endpoint](:method:jobs/update) to update tags: - jobs description: >- - Overwrite all settings for the given job. Use the [_Update_ + Overwrite all settings for the given job. Use the [*Update* endpoint](:method:jobs/update) to update job settings partially. @@ -3480,8 +4450,8 @@ paths: :param new_settings: :class:`JobSettings` The new settings of the job. These settings completely replace the old settings. - Changes to the field `JobBaseSettings.timeout_seconds` are applied to active runs. Changes to other - fields are applied to future runs only. + Changes to the field ``JobBaseSettings.timeout_seconds`` are applied to active runs. Changes to + other fields are applied to future runs only. requestBody: content: application/json: @@ -3501,63 +4471,54 @@ paths: continuous: type: object properties: + maintenance_window: + type: object + properties: + start_hour: + type: integer + description: >- + An integer between 0 and 23 denoting the start + hour for the maintenance window in the 24-hour + day. Platform-initiated maintenance is triggered + only within a one-hour window starting at this + hour. This field is required. + day_of_week: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: >- + The day of week on which maintenance is allowed + to happen. This field is required. + timezone_id: + type: string + description: >- + A Java timezone ID. The maintenance window is + resolved with respect to this timezone. See + `Java TimeZone + `__ + for details. This field is required. + required: + - start_hour + - day_of_week + - timezone_id + description: >- + A recurring weekly time window during which + platform-initiated maintenance is allowed to run for + a continuous job. pause_status: type: string x-enum: - PAUSED - UNPAUSED description: >- - Create a collection of name/value pairs. - - - Example enumeration: - - - >>> class Color(Enum): - - ... RED = 1 - - ... BLUE = 2 - - ... GREEN = 3 - - - Access them by: - - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many - members they have: - - - >>> len(Color) - - 3 - - - >>> list(Color) - - [, , ] - - - Methods can be added to enumerations, and members - can have their own - - attributes -- see the documentation for details. + Indicate whether the continuous execution of the job + is paused or not. Defaults to UNPAUSED. task_retry_mode: type: string x-enum: @@ -3569,8 +4530,8 @@ paths: description: >- An optional continuous property for this job. The continuous property will ensure that there is always one - run executing. Only one of `schedule` and `continuous` - can be used. + run executing. Only one of ``schedule`` and + ``continuous`` can be used. deployment: type: object properties: @@ -3580,13 +4541,28 @@ paths: - BUNDLE - SYSTEM_MANAGED description: >- - * `BUNDLE`: The job is managed by Databricks Asset - Bundle. * `SYSTEM_MANAGED`: The job is + - ``BUNDLE``: The job is managed by Databricks Asset + Bundle. - managed by Databricks and is read-only. + - ``SYSTEM_MANAGED``: The job is managed by + Databricks and is read-only. + deployment_id: + type: string + description: >- + ID of the deployment that manages this job. Only set + when ``kind`` is ``BUNDLE``. Used to look up + deployment metadata from the Deployment Metadata + service. metadata_file_path: type: string description: Path of the file that contains deployment metadata. + version_id: + type: string + description: >- + ID of the version of the deployment that produced + this job. Only set when ``kind`` is ``BUNDLE``. + Identifies a specific snapshot of the deployment in + the Deployment Metadata service. required: - kind description: >- @@ -3603,8 +4579,8 @@ paths: - EDITABLE - UI_LOCKED description: >- - Edit mode of the job. * `UI_LOCKED`: The job is in a - locked UI state and cannot be modified. * `EDITABLE`: + Edit mode of the job. - ``UI_LOCKED``: The job is in a + locked UI state and cannot be modified. - ``EDITABLE``: The job is in an editable state and can be modified. email_notifications: type: object @@ -3618,10 +4594,10 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold specified - for the `RUN_DURATION_SECONDS` metric in the - `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified in the - `health` field for the job, notifications are not + for the ``RUN_DURATION_SECONDS`` metric in the + ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is specified in the + ``health`` field for the job, notifications are not sent. on_failure: type: array @@ -3631,10 +4607,26 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an - `INTERNAL_ERROR` `life_cycle_state` or a `FAILED`, - or `TIMED_OUT` result_state. If this is not - specified on job creation, reset, or update the list - is empty, and notifications are not sent. + ``INTERNAL_ERROR`` ``life_cycle_state`` or a + ``FAILED``, or ``TIMED_OUT`` result_state. If this + is not specified on job creation, reset, or update + the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a + continuous job. on_start: type: array items: @@ -3652,12 +4644,12 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in - the `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the - 10-minute average of these metrics. If the issue + the ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on + the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. on_success: type: array @@ -3667,14 +4659,100 @@ paths: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a - `TERMINATED` `life_cycle_state` and a `SUCCESS` - result_state. If not specified on job creation, - reset, or update, the list is empty, and + ``TERMINATED`` ``life_cycle_state`` and a + ``SUCCESS`` result_state. If not specified on job + creation, reset, or update, the list is empty, and notifications are not sent. description: >- An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted. + environment_variables: + type: array + items: + type: object + properties: + environment_variables_key: + type: string + description: >- + Identifier for this entry. Must be unique within + ``JobSettings.environment_variables``. Tasks + reference it from + ``TaskSettings.environment_variables_key``. + files: + type: array + items: + type: string + description: >- + Workspace (``/Workspace/...``) or UC Volumes + (``/Volumes/...``) paths to ``.env`` files. + Maximum 5 files. Files are read, parsed, and + merged at task execution time, not at job creation + or update API call time. File format: each line + must be exactly ``KEY=VALUE``. Keys must match the + same regex as inlined variable names + (``^[A-Za-z_][A-Za-z0-9_]*$``); the value + continues to the end of the line. No other syntax + is supported — no comments, no quoted values, no + escape sequences, no variable interpolation. Any + line that does not match the ``KEY=VALUE`` shape + fails the run. Size limits: maximum 1,048,576 + bytes (1 MiB) per file on disk; maximum 131,072 + bytes (128 KiB) per ``KEY=VALUE`` line combined. + Caps are enforced at read time in jobs-runner — + files exceeding the per-file cap, or lines + exceeding the per-line cap, fail the run. On a + duplicate key, the later file wins; ``variables`` + override values from any file. Values may contain + {{secrets/scope/key}} references; those are + resolved at task execution time and never + persisted in resolved form. Do not use these files + to store raw secret values; consult `secret + management + `__ + for the right way to pass sensitive values. + variables: + type: object + description: >- + Environment variables specified directly as + key/value pairs (as opposed to ``files``, which + are read from ``.env`` file paths). Maximum 100 + entries. Each key must match + ``^[A-Za-z_][A-Za-z0-9_]*$`` and be 1 to 256 + characters long. Each value is up to 512 + characters; larger values should be moved into a + ``.env`` file referenced from ``files``. On a + duplicate key, ``variables`` override values from + any file in ``files``. Values may contain + {{secrets/scope/key}} references; those are + resolved at task execution time and never + persisted in resolved form. Do not use this field + to store a raw secret value; consult `secret + management + `__ + for the right way to pass sensitive values. + description: >- + A named environment-variable entry, defined once at + the job level and referenced by key from one + or more tasks. Entries live on ``JobSettings.environment_variables``, and tasks select one via + ``TaskSettings.environment_variables_key``. + + Precedence within a single entry (later overrides earlier on duplicate keys): + + files (merged in submitted order; later file in the list wins) └─▶ variables (inline k/v + override file contents) └─▶ runtime (os.environ / sys.env — overrides all, + classic-parity) + + Resolved values are never persisted: ``{{secrets/scope/key}}`` references in ``variables`` or in + ``files`` contents are stored verbatim and resolved at task execution time on the DP . + description: >- + Named environment-variable entries that tasks can + reference by key from + ``TaskSettings.environment_variables_key``. Each entry + holds inline ``variables`` plus optional ``.env`` + ``files``. Maximum 10 entries per job. Entries are + independent of one another — there is no cross-entry + merging. environments: type: array items: @@ -3703,7 +4781,8 @@ paths: description: >- Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the - Jobs API 2.1 this value is always set to `"MULTI_TASK"`. + Jobs API 2.1 this value is always set to + ``"MULTI_TASK"``. git_source: type: object properties: @@ -3778,11 +4857,11 @@ paths: description: >- Dirty state indicates the job is not fully synced with the job specification in the remote - repository. Possible values are: * `NOT_SYNCED`: - The job is not yet synced with the remote job - specification. Import the remote job - specification from UI to make the job fully - synced. * `DISCONNECTED`: The job is temporary + repository. Possible values are: - + ``NOT_SYNCED``: The job is not yet synced with + the remote job specification. Import the remote + job specification from UI to make the job fully + synced. - ``DISCONNECTED``: The job is temporary disconnected from the remote job specification and is allowed for live edit. Import the remote job specification again from UI to make the job @@ -3807,13 +4886,14 @@ paths: An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, - dbt, Python script, and SQL File tasks. If `git_source` - is set, these tasks retrieve the file from the remote - repository by default. However, this behavior can be - overridden by setting `source` to `WORKSPACE` on the - task. Note: dbt and SQL File tasks support only - version-controlled sources. If dbt or SQL File tasks are - used, `git_source` must be defined on the job. + dbt, Python script, and SQL File tasks. If + ``git_source`` is set, these tasks retrieve the file + from the remote repository by default. However, this + behavior can be overridden by setting ``source`` to + ``WORKSPACE`` on the task. Note: dbt and SQL File tasks + support only version-controlled sources. If dbt or SQL + File tasks are used, ``git_source`` must be defined on + the job. health: type: object properties: @@ -3835,29 +4915,25 @@ paths: evaluated for a particular health rule. - * `RUN_DURATION_SECONDS`: Expected total time - for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all streams. - This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate of - the maximum offset lag - - across all streams. This metric is in Public - Preview. * `STREAMING_BACKLOG_SECONDS`: An - estimate + - ``RUN_DURATION_SECONDS``: Expected total + time for a run in seconds. + - ``STREAMING_BACKLOG_BYTES``: An estimate of + the maximum bytes of data waiting to be + consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate + of the maximum offset lag across all streams. + This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of the maximum consumer delay across all - streams. This metric is in Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of the - maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate of + the maximum number of outstanding files across + all + streams. This metric is in Public Preview. op: type: string x-enum: @@ -3890,6 +4966,14 @@ paths: description: >- If new_cluster, a description of a cluster that is created for each task. + serverless_compute_id: + type: string + description: >- + The ID of the serverless compute object to bind + this cluster to. At most one JobCluster per job + may set this field; the rate limit defined on the + referenced serverless compute applies across all + tasks bound to this cluster. required: - job_cluster_key - new_cluster @@ -3914,7 +4998,7 @@ paths: kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value - to `0` causes all new runs to be skipped. + to ``0`` causes all new runs to be skipped. name: type: string description: >- @@ -3929,12 +5013,12 @@ paths: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is skipped. + specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` for - this job. + ``email_notifications`` and ``webhook_notifications`` + for this job. parameters: type: array items: @@ -3949,6 +5033,11 @@ paths: - name - default description: Job-level parameter definitions + parent_path: + type: string + description: >- + Path of the job parent folder in workspace file tree. If + absent, the job doesn't have a workspace object. performance_target: type: string x-enum: @@ -3958,9 +5047,9 @@ paths: The performance mode on a serverless job. This field determines the level of compute performance or cost-efficiency for the run. The performance target does - not apply to tasks that run on Serverless GPU compute. * - `STANDARD`: Enables cost-efficient execution of - serverless workloads. * `PERFORMANCE_OPTIMIZED`: + not apply to tasks that run on Serverless GPU compute. - + ``STANDARD``: Enables cost-efficient execution of + serverless workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. queue: @@ -3985,7 +5074,7 @@ paths: description: >- Application ID of an active service principal. Setting this field requires the - `servicePrincipal/user` role. + ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -3994,9 +5083,9 @@ paths: description: >- The user or service principal that the job runs as, if specified in the request. This field indicates the - explicit configuration of `run_as` for the job. To find - the value in all cases, explicit or implicit, use - `run_as_user_name`. + explicit configuration of ``run_as`` for the job. To + find the value in all cases, explicit or implicit, use + ``run_as_user_name``. schedule: type: object properties: @@ -4006,16 +5095,50 @@ paths: type: string description: >- A Java timezone ID. The schedule for a job is - resolved with respect to this timezone. See [Java - TimeZone] for details. This field is required. [Java - TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + resolved with respect to this timezone. See `Java + TimeZone + `__ + for details. This field is required. pause_status: type: string x-enum: - PAUSED - UNPAUSED description: Indicate whether this schedule is paused or not. + sql_condition: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to + run the condition query against. + trigger_mode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + Determines how the SQL query result is + interpreted to decide whether the condition + fires. Must be set to a recognized value when + provided. When unset on an existing serialized + configuration, the server preserves the original + semantics by interpreting it as + ``QUERY_RETURNS_ROWS``. New configurations + should set this explicitly — explicit + ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + description: >- + SQL condition that must be satisfied before a + scheduled run is triggered. The condition is + evaluated after the cron expression fires and must + return a truthy result for the run to proceed. required: - quartz_cron_expression - timezone_id @@ -4023,7 +5146,7 @@ paths: An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API - request to `runNow`. + request to ``runNow``. tags: type: object description: >- @@ -4039,11 +5162,461 @@ paths: properties: task_key: type: string + agentic_task: + type: object + properties: + context: + type: object + properties: + conversation_ids: + type: array + items: + type: string + description: >- + Optional. Conversation IDs to load as + context for the agent. + instructions: + type: array + items: + type: string + description: >- + Optional. Additional instructions to guide + the agent's behavior. + description: >- + Optional. Context input providing conversation + history and instructions. + genie_code_api: + type: object + properties: {} + description: >- + Inline Genie Code conversation driven by a + single prompt. Mutually exclusive with the + supervisor variants. + goal: + type: string + description: >- + Deprecated. Use ``input`` (field 7) instead. + Kept for backwards compatibility with existing + callers; will be removed in a future revision. + input: + type: string + description: >- + Optional. The user query / task input the + agent must accomplish. Mirrors the OpenAI + Responses API ``input`` field. Replaces the + deprecated ``goal`` field; new callers should + populate ``input`` going forward. + output_schema: + type: object + properties: + properties: + type: object + description: >- + Map of property name to property + definition. + description: >- + Optional. JSON-Schema-style declaration of the + structured output the agent should produce. + Replaces the deprecated ``task_output_schema`` + map; new callers should populate + ``output_schema`` going forward. + supervisor_agent: + type: object + properties: + agent_id: + type: string + description: >- + Required. The ID of the supervisor agent + (tile_id). + description: >- + A Supervisor Agent that orchestrates + sub-agents and tools, referenced by tile_id. + supervisor_api: + type: object + properties: + instructions: + type: string + description: >- + Optional. System instructions that guide + how the supervisor routes queries across + tools and synthesizes responses. + model: + type: string + description: >- + Required. Databricks-provided FMAPI model, + e.g. "databricks-claude-sonnet-4-5". + tools: + type: array + items: + type: object + properties: + app: + type: object + properties: + name: + type: string + description: >- + Databricks App tool config. Set when the + discriminator == "app". + catalog: + type: object + properties: + name: + type: string + description: >- + Unity Catalog catalog tool config. Set + when the discriminator == "catalog". + dashboard: + type: object + properties: + dashboard_id: + type: string + description: >- + Lakeview dashboard tool config. Set when + the discriminator == "dashboard". + description: + type: string + description: >- + Optional. Free-form description of the + tool. + genie_space: + type: object + properties: + space_id: + type: string + description: >- + Genie space tool config. Set when the + discriminator == "genie_space". + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + description: >- + Knowledge Assistant tool config. Set + when the discriminator == + "knowledge_assistant". + schema: + type: object + properties: + name: + type: string + description: >- + Unity Catalog schema tool config. Set + when the discriminator == "schema". + serving_endpoint: + type: object + properties: + name: + type: string + description: >- + Model serving endpoint tool config. Set + when the discriminator == + "serving_endpoint". + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: Supervisor Agent ID. + description: >- + Supervisor Agent tool config (a + supervisor agent used as a tool). Set + when the discriminator == + "supervisor_agent". + table: + type: object + properties: + name: + type: string + description: >- + Unity Catalog table tool config. Set + when the discriminator == "table". + tool_type: + type: string + uc_connection: + type: object + properties: + name: + type: string + description: >- + External MCP / UC connection tool + config. Set when the discriminator == + "uc_connection". + uc_function: + type: object + properties: + name: + type: string + description: >- + Unity Catalog function tool config. Set + when the discriminator == "uc_function". + uc_mcp: + type: object + properties: + name: + type: string + description: >- + UC-registered MCP service tool config. + Set when the discriminator == "uc_mcp". + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Vector Search index tool config. Set + when the discriminator == + "vector_search_index". + volume: + type: object + properties: + name: + type: string + description: >- + Unity Catalog volume tool config. Set + when the discriminator == "volume". + web_search: + type: object + properties: {} + description: >- + Web search tool config. Set when the + discriminator == "web_search". + description: >- + Tool configuration for an inlined + agentic task agent. Discriminator + selects which per-asset + config message applies (``genie_space``, ``knowledge_assistant``, ``uc_function``, + ``uc_connection``, ``uc_mcp``, ``app``, ``table``, ``vector_search_index``, ``volume``, + ``catalog``, ``schema``, ``dashboard``, ``supervisor_agent``, ``serving_endpoint``, + ``web_search``); ``description`` is a free-form top-level description; each per-asset config + carries only the asset's identifier. + + The discriminator is exposed under two interchangeable names: ``type`` mirrors the Supervisor + (Responses) API surface, and ``tool_type`` is the AIP-friendly alias for clients where ``type`` + is a reserved keyword (Terraform, etc.). Set exactly one. + description: >- + Optional. The tools the inlined agent can + use. + description: >- + Inlined Responses-API supervisor configuration + (model + instructions + tools). Mutually + exclusive with ``supervisor_agent``. + task_output_schema: + type: object + description: >- + Deprecated. Use ``output_schema`` (field 8) + instead. Kept for backwards compatibility with + existing callers; will be removed in a future + revision. + trace_destination: + type: object + properties: + catalog_name: + type: string + description: >- + Unity Catalog catalog name for the trace + table. + experiment_id: + type: string + description: >- + MLflow experiment ID where traces are + written. + schema_name: + type: string + description: >- + Unity Catalog schema name for the trace + table. + table_prefix: + type: string + description: >- + Table-name prefix under + ``.``. Trace tables are + created as + ``.._<...>``. + description: >- + Optional. Where MLflow traces produced by this + task run should be persisted. When unset, + traces follow the workspace default + destination. + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + type: object + properties: + experiment: + type: string + description: >- + MLflow experiment name for this run. If an + experiment with this name already exists under + the calling user, the run is appended to it; + otherwise a new experiment is created. To + target a specific MLflow storage location (for + example, when running as a service principal), + set ``mlflow_experiment_directory``. + deployments: + type: array + items: + type: object + properties: + command_path: + type: string + description: >- + Workspace path of the script to run on + each node in this deployment. Upload the + script to this path and supply the path + here. When the task runs, the file at + this path is run on each node; if it + fails, the task fails with its exit + code. Example script contents: Plain + Python: python train.py --epochs 10 + Multi-GPU via accelerate: accelerate + launch train.py --config config.yaml + Distributed via torchrun: torchrun + --nproc_per_node=8 train.py + compute: + type: object + properties: + accelerator_type: + type: string + x-enum: + - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 + - GPU_8xH100 + description: >- + Hardware accelerator type (for example, + ``GPU_1xA10`` or ``GPU_8xH100``). The + number of accelerators per node is + encoded in the enum value — + ``GPU_8xH100`` means 8 H100 GPUs per + node. + accelerator_count: + type: integer + description: >- + Total number of accelerators across all + nodes. Must be a positive multiple of + the per-node accelerator count encoded + in ``accelerator_type``. For example, + ``GPU_8xH100`` with ``accelerator_count: + 16`` allocates 2 nodes (8 GPUs per + node). + required: + - accelerator_type + - accelerator_count + description: >- + Compute resources allocated to each node + in this deployment. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs + on the specified container image instead + of the default Databricks client image. + Format: + ``{organization}/{repository}:{tag}`` + name: + type: string + description: >- + Optional human-readable name for this + deployment (for example, ``driver``, + ``worker``, ``param_server``). Used for + log and UI display. Distinct names are + recommended so deployments can be told + apart, but uniqueness is not enforced. + required: + - command_path + - compute + description: >- + DeploymentSpec: configuration for one + deployment within an AiRuntimeTask. Each + entry in + ``AiRuntimeTask.deployments`` describes a group of nodes that share the same command and + compute. Many single-program training algorithms use a single entry where every node runs the + same command; role-split workloads (driver + worker, parameter server, separate eval node, etc.) + use multiple entries. + description: >- + Deployment specs for this task. Exactly one + deployment is currently supported (a single + entry where every node runs the same command); + this is a current-Preview constraint. + Role-split workloads (driver + worker, + parameter server, separate eval node, etc.) + with multiple entries are the eventual intent + but not yet supported. + code_source_path: + type: string + description: >- + Workspace or UC volume path of the code-source + archive, unpacked on each node and exposed + through ``$CODE_SOURCE``. Set by first-party + tooling; not for direct callers. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs on + the specified container image instead of the + default Databricks client image. Format: + ``{organization}/{repository}:{tag}`` + mlflow_experiment_directory: + type: string + description: >- + Optional workspace directory under which the + MLflow experiment named in ``experiment`` is + created. Must start with ``/Workspace``. Set + this when running as a service principal that + has no default user directory; for regular + users the experiment defaults to the user's + home directory. + mlflow_run: + type: string + description: >- + Optional display name for the MLflow run + created under ``experiment``. If omitted, + MLflow generates a default name. + parameters: + type: object + description: >- + Optional named parameters passed to each + deployment's command. Keys are parameter + names, values the corresponding arguments (for + example, ``{"epochs": "3", "dataset": + "s3://bucket/train"}``). Values may contain + dynamic references such as + ``{{job.trigger.time.iso_date}}`` or + ``{{tasks..values.}}``, which + Jobs substitutes before execution (see + ``AiRuntimeTaskResolvedValues.parameters`` in + runs.proto). + required: + - experiment + - deployments + description: >- + The task runs a multi-gpu compute workload on + Databricks AI Runtime. Specify the accelerator + type and count, the command to run, and where the + workload's code and MLflow output are stored. alert_task: type: object properties: alert_id: type: string + parameters: + type: object + description: >- + Per-run parameter overrides, keyed by + parameter name, applied onto the alert's + stored query parameters before the query is + executed. Only scalar values are supported. + Values may reference job parameters with + ``{{job.parameters.*}}``, which are resolved + before the task runs. An override whose key + does not match a stored parameter fails the + task run. Limited to 10000 characters when + serialized as JSON; keys must be 1-100 + characters and contain only letters, digits, + underscores, dashes, and periods. subscribers: type: array items: @@ -4073,11 +5646,14 @@ paths: type: string description: >- The workspace_path is the path to the alert - file in the workspace. The path: * must start - with "/Workspace" * must be a normalized path. + file in the workspace. The path: - must start + with "/Workspace" - must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert. - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends + notifications to subscribers when the + ``alert_task`` field is present. clean_rooms_notebook_task: type: object properties: @@ -4103,10 +5679,10 @@ paths: - clean_room_name - notebook_name description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. - [clean rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ + notebook when the ``clean_rooms_notebook_task`` + field is present. compute: type: object properties: @@ -4126,30 +5702,23 @@ paths: - LESS_THAN_OR_EQUAL - NOT_EQUAL description: >- - * `EQUAL_TO`, `NOT_EQUAL` operators perform - string comparison of their operands. This - means that - - `“12.0” == “12”` will evaluate to `false`. * - `GREATER_THAN`, `GREATER_THAN_OR_EQUAL`, - - `LESS_THAN`, `LESS_THAN_OR_EQUAL` operators - perform numeric comparison of their operands. - - `“12.0” >= “12”` will evaluate to `true`, - `“10.0” >= “12”` will evaluate to - - `false`. - + - ``EQUAL_TO``, ``NOT_EQUAL`` operators + perform string comparison of their operands. + This means + that ``“12.0” == “12”`` will evaluate to ``false``. + - ``GREATER_THAN``, ``GREATER_THAN_OR_EQUAL``, + ``LESS_THAN``, ``LESS_THAN_OR_EQUAL`` + operators + perform numeric comparison of their operands. ``“12.0” >= “12”`` will evaluate to + ``true``, ``“10.0” >= “12”`` will evaluate to ``false``. The boolean comparison to task values can be - implemented with operators `EQUAL_TO`, - `NOT_EQUAL`. + implemented with operators ``EQUAL_TO``, - If a task value was set to a boolean value, it - will be serialized to `“true”` or + ``NOT_EQUAL``. If a task value was set to a + boolean value, it will be serialized to - `“false”` for the comparison. + ``“true”`` or ``“false”`` for the comparison. left: type: string description: >- @@ -4169,7 +5738,7 @@ paths: description: >- The task evaluates a condition that can be used to control the execution of other tasks when the - `condition_task` field is present. The condition + ``condition_task`` field is present. The condition task does not require a cluster to execute and does not support retries or notifications. dashboard_task: @@ -4188,14 +5757,14 @@ paths: identifier as the parameter key. The parameter value format is dependent on the filter type: - For text and single-select filters, provide - a single value (e.g. `"value"`) - For date and - datetime filters, provide the value in ISO - 8601 format (e.g. `"2000-01-01T00:00:00"`) - + a single value (e.g. ``"value"``) - For date + and datetime filters, provide the value in ISO + 8601 format (e.g. ``"2000-01-01T00:00:00"``) - For multi-select filters, provide a JSON array - of values (e.g. `"[\"value1\",\"value2\"]"`) - - For range and date range filters, provide a - JSON object with `start` and `end` (e.g. - `"{\"start\":\"1\",\"end\":\"10\"}"`) + of values (e.g. ``"[\"value1\",\"value2\"]"``) + - For range and date range filters, provide a + JSON object with ``start`` and ``end`` (e.g. + ``"{\"start\":\"1\",\"end\":\"10\"}"``) subscription: type: object properties: @@ -4217,8 +5786,8 @@ paths: type: string description: >- A snapshot of the dashboard will be sent - to the user's email when the `user_name` - field is present. + to the user's email when the + ``user_name`` field is present. description: >- The list of subscribers to send the snapshot of the dashboard to. @@ -4296,7 +5865,7 @@ paths: description: >- Optional schema to write to. This parameter is only used when a warehouse_id is also - provided. If not provided, the `default` + provided. If not provided, the ``default`` schema is used. source: type: string @@ -4305,15 +5874,15 @@ paths: - WORKSPACE description: >- Optional location type of the project - directory. When set to `WORKSPACE`, the + directory. When set to ``WORKSPACE``, the project will be retrieved from the local - Databricks workspace. When set to `GIT`, the + Databricks workspace. When set to ``GIT``, the project will be retrieved from a Git - repository defined in `git_source`. If the - value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Project is located - in Databricks workspace. * `GIT`: Project is + repository defined in ``git_source``. If the + value is empty, the task will use ``GIT`` if + ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Project is located + in Databricks workspace. - ``GIT``: Project is located in cloud Git provider. warehouse_id: type: string @@ -4322,15 +5891,15 @@ paths: provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command - basis by using the `--profiles-dir` command + basis by using the ``--profiles-dir`` command line argument. required: - commands description: >- The task runs one or more dbt commands when the - `dbt_task` field is present. The dbt task requires - both Databricks SQL and the ability to use a - serverless or a pro SQL warehouse. + ``dbt_task`` field is present. The dbt task + requires both Databricks SQL and the ability to + use a serverless or a pro SQL warehouse. depends_on: type: array items: @@ -4350,9 +5919,10 @@ paths: An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this - task. The task will run only if the `run_if` - condition is true. The key is `task_key`, and the - value is the name assigned to the dependent task. + task. The task will run only if the ``run_if`` + condition is true. The key is ``task_key``, and + the value is the name assigned to the dependent + task. description: type: string description: An optional description for this task. @@ -4379,10 +5949,10 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold - specified for the `RUN_DURATION_SECONDS` - metric in the `health` field. If no rule for - the `RUN_DURATION_SECONDS` metric is specified - in the `health` field for the job, + specified for the ``RUN_DURATION_SECONDS`` + metric in the ``health`` field. If no rule for + the ``RUN_DURATION_SECONDS`` metric is + specified in the ``health`` field for the job, notifications are not sent. on_failure: type: array @@ -4392,12 +5962,28 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if - it ends with an `INTERNAL_ERROR` - `life_cycle_state` or a `FAILED`, or - `TIMED_OUT` result_state. If this is not + it ends with an ``INTERNAL_ERROR`` + ``life_cycle_state`` or a ``FAILED``, or + ``TIMED_OUT`` result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a + continuous job. on_start: type: array items: @@ -4415,11 +6001,12 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can - be set in the `health` field using the - following metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based + be set in the ``health`` field using the + following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. @@ -4431,10 +6018,11 @@ paths: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if - it ends with a `TERMINATED` `life_cycle_state` - and a `SUCCESS` result_state. If not specified - on job creation, reset, or update, the list is - empty, and notifications are not sent. + it ends with a ``TERMINATED`` + ``life_cycle_state`` and a ``SUCCESS`` + result_state. If not specified on job + creation, reset, or update, the list is empty, + and notifications are not sent. description: >- An optional set of email addresses that is notified when runs of this task begin or complete @@ -4447,6 +6035,16 @@ paths: job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry + defined in ``JobSettings.environment_variables``. + The selected entry's variables and file contents + are applied to this task at execution time. Length + and pattern mirror ``environment_key`` so the two + references look identical to customers reading + task settings. existing_cluster_id: type: string description: >- @@ -4478,13 +6076,32 @@ paths: - task description: >- The task executes a nested task for every input - provided when the `for_each_task` field is + provided when the ``for_each_task`` field is present. gen_ai_compute_task: type: object properties: dl_runtime_image: type: string + description: Runtime image + client_version: + type: string + description: >- + Version of the client (e.g., sgcli wheel) that + submitted this task. Used by handlers to gate + behavior or reject incompatible versions. + code_source_tar_path: + type: string + description: >- + Optional path to a tarball containing the + user's workspace contents. When set, the entry + script extracts the tarball into the working + directory before running the training script, + so the training script can import sibling + modules and read packaged data files. Must be + a workspace path (e.g. + ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. command: type: string description: >- @@ -4503,6 +6120,13 @@ paths: description: GPU type. required: - num_gpus + docker_image_url: + type: string + description: >- + Optional custom Docker container image URL for + running the training script. Format: + organization/repository:tag (e.g., + "pytorch/pytorch:2.0.1") mlflow_experiment_name: type: string description: >- @@ -4510,6 +6134,25 @@ paths: MLflow experiment to log the run to. If name is not found, backend will create the mlflow experiment using the name. + mlflow_run_name: + type: string + description: >- + Optional name to assign to the MLflow run + created for this task. If unset, MLflow + auto-generates a name. Used alongside + ``mlflow_experiment_name`` to identify the run + in the MLflow UI. + requirements_yaml_path: + type: string + description: >- + Optional path to a requirements.yaml file + describing pip dependencies to install before + running the training script. Consumed by the + entry script; format matches the runtime + requirements.yaml convention used by sgcli. + Must be a workspace path (e.g. + ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. source: type: string x-enum: @@ -4517,15 +6160,16 @@ paths: - WORKSPACE description: >- Optional location type of the training script. - When set to `WORKSPACE`, the script will be + When set to ``WORKSPACE``, the script will be retrieved from the local Databricks workspace. - When set to `GIT`, the script will be + When set to ``GIT``, the script will be retrieved from a Git repository defined in - `git_source`. If the value is empty, the task - will use `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: Script - is located in Databricks workspace. * `GIT`: - Script is located in cloud Git provider. + ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - + ``WORKSPACE``: Script is located in Databricks + workspace. - ``GIT``: Script is located in + cloud Git provider. training_script_path: type: string description: >- @@ -4534,7 +6178,7 @@ paths: gcs:/) and workspace paths are supported. For python files stored in the Databricks workspace, the path must be absolute and begin - with `/`. For files stored in a remote + with ``/``. For files stored in a remote repository, the path must be relative. This field is required. yaml_parameters: @@ -4551,6 +6195,23 @@ paths: parameters passed to the training script. required: - dl_runtime_image + description: >- + DEPRECATED — use ``AiRuntimeTask`` for all new + BYOT multi-node GPU workloads (see + ai_runtime_task.proto). ``AiRuntimeTask`` is the only supported BYOT task type for new + workloads; this proto is retained only for AIR CLI (fka SGCLI) pywheel backwards compatibility + and will be removed once the pywheel → databricks-cli migration completes (post- PuPr). + genie_task: + type: object + properties: + configuration_id: + type: string + description: >- + Required. Resource name of the agent task + configuration to run. + required: + - configuration_id + description: Runs a Genie or Genie Code agent task. health: type: object properties: @@ -4573,32 +6234,25 @@ paths: rule. - * `RUN_DURATION_SECONDS`: Expected total - time for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all - streams. This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate - of the maximum offset lag - - across all streams. This metric is in - Public Preview. * - `STREAMING_BACKLOG_SECONDS`: An estimate - - of the maximum consumer delay across all - streams. This metric is in Public - Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate - of the maximum number of outstanding - files across all - - streams. This metric is in Public - Preview. + - ``RUN_DURATION_SECONDS``: Expected + total time for a run in seconds. + + - ``STREAMING_BACKLOG_BYTES``: An + estimate of the maximum bytes of data + waiting to be consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An + estimate of the maximum offset lag + across all streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An + estimate of the maximum consumer delay + across all streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An + estimate of the maximum number of + outstanding files across all + streams. This metric is in Public Preview. op: type: string x-enum: @@ -4625,7 +6279,7 @@ paths: description: >- If job_cluster_key, this task is executed reusing the cluster specified in - `job.settings.job_clusters`. + ``job.settings.job_clusters``. libraries: type: string description: >- @@ -4636,11 +6290,11 @@ paths: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be - unsuccessful if it completes with the `FAILED` - result_state or `INTERNAL_ERROR` - `life_cycle_state`. The value `-1` means to retry - indefinitely and the value `0` means to never - retry. + unsuccessful if it completes with the ``FAILED`` + result_state or ``INTERNAL_ERROR`` + ``life_cycle_state``. The value ``-1`` means to + retry indefinitely and the value ``0`` means to + never retry. min_retry_interval_millis: type: integer description: >- @@ -4666,21 +6320,21 @@ paths: :method:jobs/run Now with parameters specified, the two parameters maps are merged. If the same key is specified in - `base_parameters` and in `run-now`, the value - from `run-now` is used. Use [Task parameter - variables] to set parameters containing - information about job runs. If the notebook - takes a parameter that is not specified in the - job’s `base_parameters` or the `run-now` + ``base_parameters`` and in ``run-now``, the + value from ``run-now`` is used. Use `Task + parameter variables + `__ + to set parameters containing information about + job runs. If the notebook takes a parameter + that is not specified in the job’s + ``base_parameters`` or the ``run-now`` override parameters, the default value from the notebook is used. Retrieve these parameters in a notebook using - [dbutils.widgets.get]. The JSON representation - of this field cannot exceed 1MB. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-widgets + `dbutils.widgets.get + `__. + The JSON representation of this field cannot + exceed 1MB. source: type: string x-enum: @@ -4688,20 +6342,21 @@ paths: - WORKSPACE description: >- Optional location type of the notebook. When - set to `WORKSPACE`, the notebook will be + set to ``WORKSPACE``, the notebook will be retrieved from the local Databricks workspace. - When set to `GIT`, the notebook will be + When set to ``GIT``, the notebook will be retrieved from a Git repository defined in - `git_source`. If the value is empty, the task - will use `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: Notebook - is located in Databricks workspace. * `GIT`: - Notebook is located in cloud Git provider. + ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - + ``WORKSPACE``: Notebook is located in + Databricks workspace. - ``GIT``: Notebook is + located in cloud Git provider. warehouse_id: type: string description: >- - Optional `warehouse_id` to run the notebook on - a SQL warehouse. Classic SQL warehouses are + Optional ``warehouse_id`` to run the notebook + on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses. Note that SQL warehouses only support SQL cells; if the notebook contains @@ -4709,8 +6364,8 @@ paths: required: - notebook_path description: >- - The task runs a notebook when the `notebook_task` - field is present. + The task runs a notebook when the + ``notebook_task`` field is present. notification_settings: type: object properties: @@ -4720,19 +6375,19 @@ paths: type: boolean description: >- If true, do not send notifications to - recipients specified in `on_failure` if the + recipients specified in ``on_failure`` if the run is canceled. no_alert_for_skipped_runs: type: boolean description: >- If true, do not send notifications to - recipients specified in `on_failure` if the + recipients specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` - for this task. + ``email_notifications`` and + ``webhook_notifications`` for this task. pipeline_task: type: object properties: @@ -4741,19 +6396,119 @@ paths: full_refresh: type: boolean description: >- - If true, triggers a full refresh on the delta - live table. + If true, triggers a full refresh on the spark + declarative pipeline. + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + parameters: + type: object + description: >- + Key/value-map of parameters passed to the + pipeline execution. Limited to 10k characters + in total. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are + unioned with other selective refresh options + (refresh_selection, full_refresh_selection) to + determine the final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints + without clearing data. required: - pipeline_id description: >- The task triggers a pipeline update when the - `pipeline_task` field is present. Only pipelines + ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: type: object properties: connection_resource_name: type: string + incremental_refresh_config: + type: object + properties: + archive_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the archive window. + Default: MONTH + archive_window_periods: + type: integer + description: >- + Number of periods in the archive window + (total data retained). Must be greater + than the refresh window when using the + same granularity. Default: 3 (with YEAR + granularity = 3 years) + detect_data_changes: + type: boolean + description: >- + If true, Power BI will check whether data + has changed in each partition before + reimporting. Partitions with unchanged + data are skipped, reducing refresh cost + further. Uses MAX(datetime_column) as the + change detector. + mode: + type: string + x-enum: + - REFRESH_POLICY_MODE_HYBRID + - REFRESH_POLICY_MODE_IMPORT_ONLY + description: >- + The refresh policy mode. Hybrid adds a + real-time DirectQuery partition for the + most recent data; Import uses only + batch-imported partitions. Default: HYBRID + only_refresh_complete_periods: + type: boolean + description: >- + If true, only refresh complete periods + (e.g., skip today's partial data). + refresh_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the refresh window. + Default: DAY + refresh_window_periods: + type: integer + description: >- + Number of periods in the rolling refresh + window. Partitions within this window are + re-imported on each refresh. Default: 1 + description: >- + Incremental refresh policy applied to all + IMPORT mode tables in the model. Windows and + mode are shared; partition columns are set + per-table on PowerBiTable. power_bi_model: type: object properties: @@ -4851,6 +6606,15 @@ paths: properties: catalog: type: string + incremental_refresh_datetime_column: + type: string + description: >- + The datetime column used for incremental + refresh partitioning on this table. + e.g., "order_date", "updated_at" Only + applicable when the task has + incremental_refresh_config set and this + table uses IMPORT storage mode. name: type: string description: The table name in Databricks @@ -4864,6 +6628,25 @@ paths: - DUAL - IMPORT description: The Power BI storage mode of the table + table_type: + type: string + x-enum: + - EXTERNAL + - EXTERNAL_SHALLOW_CLONE + - FOREIGN + - MANAGED + - MANAGED_SHALLOW_CLONE + - MATERIALIZED_VIEW + - METRIC_VIEW + - STREAMING_TABLE + - VIEW + description: >- + The Unity Catalog table type of this + table. When set, the server trusts it to + classify the table during metric-view + filtering and skips the server-side + lookup. When unset, the server resolves + the type on its own. description: The tables to be exported to Power BI warehouse_id: type: string @@ -4872,7 +6655,25 @@ paths: data source description: >- The task triggers a Power BI semantic model update - when the `power_bi_task` field is present. + when the ``power_bi_task`` field is present. + python_operator_task: + type: object + properties: + main: + type: string + parameters: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: >- + An ordered list of task parameters. + TODO(JOBS-30885): Add limits for parameters. + description: The task runs a Python operator task. python_wheel_task: type: object properties: @@ -4884,28 +6685,28 @@ paths: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using - `$packageName.$entryPoint()` + ``$packageName.$entryPoint()`` named_parameters: type: object description: >- Command-line parameters passed to Python wheel - task in the form of `["--name=task", - "--data=dbfs:/path/to/data.json"]`. Leave it - empty if `parameters` is not null. + task in the form of ``["--name=task", + "--data=dbfs:/path/to/data.json"]``. Leave it + empty if ``parameters`` is not null. parameters: type: array items: type: string description: >- Command-line parameters passed to Python wheel - task. Leave it empty if `named_parameters` is - not null. + task. Leave it empty if ``named_parameters`` + is not null. required: - package_name - entry_point description: >- The task runs a Python wheel when the - `python_wheel_task` field is present. + ``python_wheel_task`` field is present. retry_on_timeout: type: boolean description: >- @@ -4924,15 +6725,15 @@ paths: description: >- An optional value specifying the condition determining whether the task is run once its - dependencies have been completed. * `ALL_SUCCESS`: - All dependencies have executed and succeeded * - `AT_LEAST_ONE_SUCCESS`: At least one dependency - has succeeded * `NONE_FAILED`: None of the - dependencies have failed and at least one was - executed * `ALL_DONE`: All dependencies have been - completed * `AT_LEAST_ONE_FAILED`: At least one - dependency failed * `ALL_FAILED`: ALl dependencies - have failed + dependencies have been completed. - + ``ALL_SUCCESS``: All dependencies have executed + and succeeded - ``AT_LEAST_ONE_SUCCESS``: At least + one dependency has succeeded - ``NONE_FAILED``: + None of the dependencies have failed and at least + one was executed - ``ALL_DONE``: All dependencies + have been completed - ``AT_LEAST_ONE_FAILED``: At + least one dependency failed - ``ALL_FAILED``: ALl + dependencies have failed run_job_task: type: object properties: @@ -4944,31 +6745,31 @@ paths: type: string description: >- An array of commands to execute for jobs with - the dbt task, for example `"dbt_commands": + the dbt task, for example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt deps", "dbt - seed", "dbt run"]` ⚠ **Deprecation note** Use - [job parameters] to pass information down to - tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + seed", "dbt run"]`` ⚠ **Deprecation note** Use + `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR - tasks, for example `"jar_params": ["john doe", - "35"]`. The parameters are used to invoke the - main function of the main class specified in - the Spark JAR task. If not specified upon - `run-now`, it defaults to an empty list. - jar_params cannot be specified in conjunction - with notebook_params. The JSON representation - of this field (for example - `{"jar_params":["john doe","35"]}`) cannot + tasks, for example ``"jar_params": ["john + doe", "35"]``. The parameters are used to + invoke the main function of the main class + specified in the Spark JAR task. If not + specified upon ``run-now``, it defaults to an + empty list. jar_params cannot be specified in + conjunction with notebook_params. The JSON + representation of this field (for example + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation note** - Use [job parameters] to pass information down - to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + Use `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: Job-level parameters used to trigger the job. @@ -4976,28 +6777,59 @@ paths: type: object description: >- A map from keys to values for jobs with - notebook task, for example `"notebook_params": - {"name": "john doe", "age": "35"}`. The map is - passed to the notebook and is accessible - through the [dbutils.widgets.get] function. If - not specified upon `run-now`, the triggered - run uses the job’s base parameters. - notebook_params cannot be specified in - conjunction with jar_params. ⚠ **Deprecation - note** Use [job parameters] to pass - information down to tasks. The JSON + notebook task, for example + ``"notebook_params": {"name": "john doe", + "age": "35"}``. The map is passed to the + notebook and is accessible through the + `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, + the triggered run uses the job’s base + parameters. notebook_params cannot be + specified in conjunction with jar_params. ⚠ + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. The JSON representation of this field (for example - `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 - bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 + bytes. pipeline_params: type: object properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update with + fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These + are unioned with other selective refresh + options (refresh_selection, + full_refresh_selection) to determine the + final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset + checkpoints without clearing data. description: >- Controls whether the pipeline should perform a full refresh @@ -5009,22 +6841,22 @@ paths: type: string description: >- A list of parameters for jobs with Python - tasks, for example `"python_params": ["john - doe", "35"]`. The parameters are passed to + tasks, for example ``"python_params": ["john + doe", "35"]``. The parameters are passed to Python file as command-line parameters. If - specified upon `run-now`, it would overwrite + specified upon ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot - exceed 10,000 bytes. ⚠ **Deprecation note** - Use [job parameters] to pass information down - to tasks. Important These parameters accept - only Latin characters (ASCII character set). - Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are - Chinese, Japanese kanjis, and emojis. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"python_params":["john doe","35"]}``) + cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important + These parameters accept only Latin characters + (ASCII character set). Using non-ASCII + characters returns an error. Examples of + invalid, non-ASCII characters are Chinese, + Japanese kanjis, and emojis. spark_submit_params: type: array items: @@ -5032,39 +6864,38 @@ paths: description: >- A list of parameters for jobs with spark submit task, for example - `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The + ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script as command-line parameters. If specified upon - `run-now`, it would overwrite the parameters + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot - exceed 10,000 bytes. ⚠ **Deprecation note** - Use [job parameters] to pass information down - to tasks. Important These parameters accept - only Latin characters (ASCII character set). - Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are - Chinese, Japanese kanjis, and emojis. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"python_params":["john doe","35"]}``) + cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important + These parameters accept only Latin characters + (ASCII character set). Using non-ASCII + characters returns an error. Examples of + invalid, non-ASCII characters are Chinese, + Japanese kanjis, and emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL - task, for example `"sql_params": {"name": - "john doe", "age": "35"}`. The SQL alert task + task, for example ``"sql_params": {"name": + "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ - **Deprecation note** Use [job parameters] to - pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. required: - job_id description: >- The task triggers another job when the - `run_job_task` field is present. + ``run_job_task`` field is present. spark_jar_task: type: object properties: @@ -5076,7 +6907,7 @@ paths: The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library. The - code must use `SparkContext.getOrCreate` to + code must use ``SparkContext.getOrCreate`` to obtain a Spark context; otherwise, runs of the job fail. parameters: @@ -5085,17 +6916,17 @@ paths: type: string description: >- Parameters passed to the main method. Use - [Task parameter variables] to set parameters - containing information about job runs. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + `Task parameter variables + `__ + to set parameters containing information about + job runs. run_as_repl: type: boolean description: >- - Deprecated. A value of `false` is no longer + Deprecated. A value of ``false`` is no longer supported. description: >- - The task runs a JAR when the `spark_jar_task` + The task runs a JAR when the ``spark_jar_task`` field is present. spark_python_task: type: object @@ -5108,10 +6939,10 @@ paths: type: string description: >- Command line parameters passed to the Python - file. Use [Task parameter variables] to set - parameters containing information about job - runs. [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + file. Use `Task parameter variables + `__ + to set parameters containing information about + job runs. source: type: string x-enum: @@ -5119,21 +6950,22 @@ paths: - WORKSPACE description: >- Optional location type of the Python file. - When set to `WORKSPACE` or not specified, the - file will be retrieved from the local + When set to ``WORKSPACE`` or not specified, + the file will be retrieved from the local Databricks workspace or cloud location (if the - `python_file` has a URI format). When set to - `GIT`, the Python file will be retrieved from - a Git repository defined in `git_source`. * - `WORKSPACE`: The Python file is located in a - Databricks workspace or at a cloud filesystem - URI. * `GIT`: The Python file is located in a - remote Git repository. + ``python_file`` has a URI format). When set to + ``GIT``, the Python file will be retrieved + from a Git repository defined in + ``git_source``. - ``WORKSPACE``: The Python + file is located in a Databricks workspace or + at a cloud filesystem URI. - ``GIT``: The + Python file is located in a remote Git + repository. required: - python_file description: >- The task runs a Python file when the - `spark_python_task` field is present. + ``spark_python_task`` field is present. spark_submit_task: type: object properties: @@ -5238,16 +7070,17 @@ paths: - WORKSPACE description: >- Optional location type of the SQL file. - When set to `WORKSPACE`, the SQL file will - be retrieved from the local Databricks - workspace. When set to `GIT`, the SQL file - will be retrieved from a Git repository - defined in `git_source`. If the value is - empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: SQL file is - located in Databricks workspace. * `GIT`: - SQL file is located in cloud Git provider. + When set to ``WORKSPACE``, the SQL file + will be retrieved from the local + Databricks workspace. When set to ``GIT``, + the SQL file will be retrieved from a Git + repository defined in ``git_source``. If + the value is empty, the task will use + ``GIT`` if ``git_source`` is defined and + ``WORKSPACE`` otherwise. - ``WORKSPACE``: + SQL file is located in Databricks + workspace. - ``GIT``: SQL file is located + in cloud Git provider. required: - path description: >- @@ -5274,12 +7107,12 @@ paths: description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the - `sql_task` field is present. + ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this - job task. A value of `0` means no timeout. + job task. A value of ``0`` means no timeout. webhook_notifications: type: object properties: @@ -5292,7 +7125,36 @@ paths: type: string required: - id - on_failure: + on_failure: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when the run fails. A maximum of 3 + destinations can be specified for the + ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance + completes for a continuous job. A maximum of 3 + destinations can be specified for the + ``on_maintenance_complete`` property. + on_maintenance_start: type: array items: type: object @@ -5303,9 +7165,10 @@ paths: - id description: >- An optional list of system notification IDs to - call when the run fails. A maximum of 3 + call when platform-initiated maintenance + starts for a continuous job. A maximum of 3 destinations can be specified for the - `on_failure` property. + ``on_maintenance_start`` property. on_start: type: array items: @@ -5319,7 +7182,7 @@ paths: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the - `on_start` property. + ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -5333,17 +7196,17 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog - thresholds can be set in the `health` field + thresholds can be set in the ``health`` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -5357,7 +7220,7 @@ paths: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for - the `on_success` property. + the ``on_success`` property. description: >- A collection of system notification IDs to notify when runs of this task begin or complete. The @@ -5372,13 +7235,14 @@ paths: :method:jobs/update, :method:jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them using - :method:jobs/get. Use the `next_page_token` field at the - object root to determine if more results are available. + :method:jobs/get. Use the ``next_page_token`` field at + the object root to determine if more results are + available. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job. A - value of `0` means no timeout. + value of ``0`` means no timeout. trigger: type: object properties: @@ -5516,12 +7380,47 @@ paths: x-enum: - DAYS - HOURS + - MINUTES - WEEKS description: The unit of time for the interval. required: - interval - unit description: Periodic trigger settings. + sql_condition: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to + run the condition query against. + trigger_mode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + Determines how the SQL query result is + interpreted to decide whether the condition + fires. Must be set to a recognized value when + provided. When unset on an existing serialized + configuration, the server preserves the original + semantics by interpreting it as + ``QUERY_RETURNS_ROWS``. New configurations + should set this explicitly — explicit + ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + description: >- + SQL condition that must be satisfied for the trigger + to fire. Can be used in combination with other + trigger types and runs *after* other trigger types + conditions are evaluated. table_update: type: object properties: @@ -5558,15 +7457,15 @@ paths: A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or - sending an API request to `runNow`. + sending an API request to ``runNow``. usage_policy_id: type: string description: >- The id of the user specified usage policy to use for this job. If not specified, a default usage policy may be applied when creating or modifying the job. See - `effective_usage_policy_id` for the usage policy used by - this workload. + ``effective_usage_policy_id`` for the usage policy used + by this workload. webhook_notifications: type: object properties: @@ -5591,7 +7490,36 @@ paths: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can - be specified for the `on_failure` property. + be specified for the ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call + when platform-initiated maintenance completes for a + continuous job. A maximum of 3 destinations can be + specified for the ``on_maintenance_complete`` + property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call + when platform-initiated maintenance starts for a + continuous job. A maximum of 3 destinations can be + specified for the ``on_maintenance_start`` property. on_start: type: array items: @@ -5604,7 +7532,7 @@ paths: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can - be specified for the `on_start` property. + be specified for the ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -5618,15 +7546,15 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be - set in the `health` field using the following - metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the - 10-minute average of these metrics. If the issue + set in the ``health`` field using the following + metrics: ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on + the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -5639,7 +7567,7 @@ paths: description: >- An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 - destinations can be specified for the `on_success` + destinations can be specified for the ``on_success`` property. description: >- A collection of system notification IDs to notify when @@ -5647,7 +7575,7 @@ paths: description: >- The new settings of the job. These settings completely replace the old settings. Changes to the field - `JobBaseSettings.timeout_seconds` are applied to active + ``JobBaseSettings.timeout_seconds`` are applied to active runs. Changes to other fields are applied to future runs only. required: @@ -5671,21 +7599,21 @@ paths: /api/2.2/jobs/run-now: post: operationId: jobs_run_now - summary: Run a job and return the `run_id` of the triggered run. + summary: Run a job and return the ``run_id`` of the triggered run. tags: - jobs description: |- - Run a job and return the `run_id` of the triggered run. + Run a job and return the ``run_id`` of the triggered run. :param job_id: int The ID of the job to be executed :param dbt_commands: List[str] (optional) - An array of commands to execute for jobs with the dbt task, for example `"dbt_commands": ["dbt - deps", "dbt seed", "dbt deps", "dbt seed", "dbt run"]` - - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + An array of commands to execute for jobs with the dbt task, for example ``"dbt_commands": ["dbt + deps", "dbt seed", "dbt deps", "dbt seed", "dbt run"]`` - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. :param idempotency_token: str (optional) An optional token to guarantee the idempotency of job run requests. If a run with the provided token already exists, the request does not create a new run but returns the ID of the existing run @@ -5696,91 +7624,94 @@ paths: This token must have at most 64 characters. - For more information, see [How to ensure idempotency for jobs]. - - [How to ensure idempotency for jobs]: https://kb.databricks.com/jobs/jobs-idempotency.html + For more information, see `How to ensure idempotency for jobs + `__. :param jar_params: List[str] (optional) - A list of parameters for jobs with Spark JAR tasks, for example `"jar_params": ["john doe", "35"]`. - The parameters are used to invoke the main function of the main class specified in the Spark JAR - task. If not specified upon `run-now`, it defaults to an empty list. jar_params cannot be specified - in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed 10,000 bytes. - - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. - - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + A list of parameters for jobs with Spark JAR tasks, for example ``"jar_params": ["john doe", + "35"]``. The parameters are used to invoke the main function of the main class specified in the + Spark JAR task. If not specified upon ``run-now``, it defaults to an empty list. jar_params cannot + be specified in conjunction with notebook_params. The JSON representation of this field (for example + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 bytes. + + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. :param job_parameters: Dict[str,str] (optional) - Job-level parameters used in the run. for example `"param": "overriding_val"` + Job-level parameters used in the run. for example ``"param": "overriding_val"`` :param notebook_params: Dict[str,str] (optional) - A map from keys to values for jobs with notebook task, for example `"notebook_params": {"name": - "john doe", "age": "35"}`. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. + A map from keys to values for jobs with notebook task, for example ``"notebook_params": {"name": + "john doe", "age": "35"}``. The map is passed to the notebook and is accessible through the + `dbutils.widgets.get `__ function. - If not specified upon `run-now`, the triggered run uses the job’s base parameters. + If not specified upon ``run-now``, the triggered run uses the job’s base parameters. notebook_params cannot be specified in conjunction with jar_params. - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. - - The JSON representation of this field (for example `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 bytes. + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. - [dbutils.widgets.get]: https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + The JSON representation of this field (for example ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 bytes. :param only: List[str] (optional) A list of task keys to run inside of the job. If this field is not provided, all tasks in the job will be run. + + Prefix a task key with ``+`` to also run its upstream tasks, or suffix it with ``+`` to also run its + downstream tasks. For example, ``+my_task`` runs ``my_task`` and everything upstream of it, + ``my_task+`` runs ``my_task`` and everything downstream of it, and ``+my_task+`` runs both. A task + key with no ``+`` runs only that task. :param performance_target: :class:`PerformanceTarget` (optional) The performance mode on a serverless job. The performance target determines the level of compute performance or cost-efficiency for the run. This field overrides the performance target defined on the job level. - * `STANDARD`: Enables cost-efficient execution of serverless workloads. * `PERFORMANCE_OPTIMIZED`: - Prioritizes fast startup and execution times through rapid scaling and optimized cluster - performance. + - ``STANDARD``: Enables cost-efficient execution of serverless workloads. + - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and + optimized cluster performance. :param pipeline_params: :class:`PipelineParams` (optional) Controls whether the pipeline should perform a full refresh :param python_named_params: Dict[str,str] (optional) :param python_params: List[str] (optional) - A list of parameters for jobs with Python tasks, for example `"python_params": ["john doe", "35"]`. - The parameters are passed to Python file as command-line parameters. If specified upon `run-now`, it - would overwrite the parameters specified in job setting. The JSON representation of this field (for - example `{"python_params":["john doe","35"]}`) cannot exceed 10,000 bytes. + A list of parameters for jobs with Python tasks, for example ``"python_params": ["john doe", + "35"]``. The parameters are passed to Python file as command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of + this field (for example ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. Important These parameters accept only Latin characters (ASCII character set). Using non-ASCII characters returns an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, and emojis. - - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown :param queue: :class:`QueueSettings` (optional) The queue settings of the run. :param spark_submit_params: List[str] (optional) - A list of parameters for jobs with spark submit task, for example `"spark_submit_params": - ["--class", "org.apache.spark.examples.SparkPi"]`. The parameters are passed to spark-submit script - as command-line parameters. If specified upon `run-now`, it would overwrite the parameters specified - in job setting. The JSON representation of this field (for example `{"python_params":["john - doe","35"]}`) cannot exceed 10,000 bytes. + A list of parameters for jobs with spark submit task, for example ``"spark_submit_params": + ["--class", "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script + as command-line parameters. If specified upon ``run-now``, it would overwrite the parameters + specified in job setting. The JSON representation of this field (for example + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. Important These parameters accept only Latin characters (ASCII character set). Using non-ASCII characters returns an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, and emojis. - - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown :param sql_params: Dict[str,str] (optional) - A map from keys to values for jobs with SQL task, for example `"sql_params": {"name": "john doe", - "age": "35"}`. The SQL alert task does not support custom parameters. - - ⚠ **Deprecation note** Use [job parameters] to pass information down to tasks. + A map from keys to values for jobs with SQL task, for example ``"sql_params": {"name": "john doe", + "age": "35"}``. The SQL alert task does not support custom parameters. - [job parameters]: https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ⚠ **Deprecation note** Use `job parameters + `__ to pass information + down to tasks. :returns: Long-running operation waiter for :class:`Run`. @@ -5800,11 +7731,11 @@ paths: type: string description: >- An array of commands to execute for jobs with the dbt task, - for example `"dbt_commands": ["dbt deps", "dbt seed", "dbt - deps", "dbt seed", "dbt run"]` ⚠ **Deprecation note** Use - [job parameters] to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + for example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt + deps", "dbt seed", "dbt run"]`` ⚠ **Deprecation note** Use + `job parameters + `__ + to pass information down to tasks. idempotency_token: type: string description: >- @@ -5816,55 +7747,60 @@ paths: idempotency token, upon failure you can retry until the request succeeds. Databricks guarantees that exactly one run is launched with that idempotency token. This token must - have at most 64 characters. For more information, see [How - to ensure idempotency for jobs]. [How to ensure idempotency - for jobs]: - https://kb.databricks.com/jobs/jobs-idempotency.html + have at most 64 characters. For more information, see `How + to ensure idempotency for jobs + `__. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR tasks, for - example `"jar_params": ["john doe", "35"]`. The parameters + example ``"jar_params": ["john doe", "35"]``. The parameters are used to invoke the main function of the main class specified in the Spark JAR task. If not specified upon - `run-now`, it defaults to an empty list. jar_params cannot + ``run-now``, it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed 10,000 - bytes. ⚠ **Deprecation note** Use [job parameters] to pass - information down to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 + bytes. ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: >- - Job-level parameters used in the run. for example `"param": - "overriding_val"` + Job-level parameters used in the run. for example ``"param": + "overriding_val"`` notebook_params: type: object description: >- A map from keys to values for jobs with notebook task, for - example `"notebook_params": {"name": "john doe", "age": - "35"}`. The map is passed to the notebook and is accessible - through the [dbutils.widgets.get] function. If not specified - upon `run-now`, the triggered run uses the job’s base - parameters. notebook_params cannot be specified in - conjunction with jar_params. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. The JSON - representation of this field (for example - `{"notebook_params":{"name":"john doe","age":"35"}}`) cannot - exceed 10,000 bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + example ``"notebook_params": {"name": "john doe", "age": + "35"}``. The map is passed to the notebook and is accessible + through the `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, the triggered + run uses the job’s base parameters. notebook_params cannot + be specified in conjunction with jar_params. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. The JSON representation + of this field (for example + ``{"notebook_params":{"name":"john doe","age":"35"}}``) + cannot exceed 10,000 bytes. only: type: array items: type: string description: >- A list of task keys to run inside of the job. If this field - is not provided, all tasks in the job will be run. + is not provided, all tasks in the job will be run. Prefix a + task key with ``+`` to also run its upstream tasks, or + suffix it with ``+`` to also run its downstream tasks. For + example, ``+my_task`` runs ``my_task`` and everything + upstream of it, ``my_task+`` runs ``my_task`` and everything + downstream of it, and ``+my_task+`` runs both. A task key + with no ``+`` runs only that task. performance_target: type: string x-enum: @@ -5874,9 +7810,9 @@ paths: The performance mode on a serverless job. The performance target determines the level of compute performance or cost-efficiency for the run. This field overrides the - performance target defined on the job level. * `STANDARD`: - Enables cost-efficient execution of serverless workloads. * - `PERFORMANCE_OPTIMIZED`: Prioritizes fast startup and + performance target defined on the job level. - ``STANDARD``: + Enables cost-efficient execution of serverless workloads. - + ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. pipeline_params: @@ -5884,6 +7820,32 @@ paths: properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are unioned + with other selective refresh options (refresh_selection, + full_refresh_selection) to determine the final set of + flows to refresh. + refresh_selection: + type: array + items: + type: string + description: A list of tables to update without fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints without + clearing data. description: Controls whether the pipeline should perform a full refresh python_named_params: type: object @@ -5893,18 +7855,19 @@ paths: type: string description: >- A list of parameters for jobs with Python tasks, for example - `"python_params": ["john doe", "35"]`. The parameters are + ``"python_params": ["john doe", "35"]``. The parameters are passed to Python file as command-line parameters. If - specified upon `run-now`, it would overwrite the parameters - specified in job setting. The JSON representation of this - field (for example `{"python_params":["john doe","35"]}`) - cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. Important - These parameters accept only Latin characters (ASCII - character set). Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + specified upon ``run-now``, it would overwrite the + parameters specified in job setting. The JSON representation + of this field (for example ``{"python_params":["john + doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII character + set). Using non-ASCII characters returns an error. Examples + of invalid, non-ASCII characters are Chinese, Japanese + kanjis, and emojis. queue: type: object properties: @@ -5919,28 +7882,29 @@ paths: type: string description: >- A list of parameters for jobs with spark submit task, for - example `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The parameters are + example ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script as command-line parameters. If - specified upon `run-now`, it would overwrite the parameters - specified in job setting. The JSON representation of this - field (for example `{"python_params":["john doe","35"]}`) - cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. Important - These parameters accept only Latin characters (ASCII - character set). Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + specified upon ``run-now``, it would overwrite the + parameters specified in job setting. The JSON representation + of this field (for example ``{"python_params":["john + doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII character + set). Using non-ASCII characters returns an error. Examples + of invalid, non-ASCII characters are Chinese, Japanese + kanjis, and emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL task, for - example `"sql_params": {"name": "john doe", "age": "35"}`. + example ``"sql_params": {"name": "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ - **Deprecation note** Use [job parameters] to pass - information down to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. required: - job_id responses: @@ -5975,9 +7939,9 @@ paths: directly without creating a job. Runs submitted using this endpoint don’t display in the UI. Use the - `jobs/runs/get` API to check the + ``jobs/runs/get`` API to check - run state after the job is submitted. + the run state after the job is submitted. **Important:** Jobs submitted using this endpoint are not saved as a @@ -5989,10 +7953,10 @@ paths: serverless compute in case of failure. If your job fails, you may want to use classic compute to - specify the compute needs for the job. Alternatively, use the `POST - /jobs/create` and `POST + specify the compute needs for the job. Alternatively, use the ``POST + /jobs/create`` and ``POST - /jobs/run-now` endpoints to create and run a saved job. + /jobs/run-now`` endpoints to create and run a saved job. :param access_control_list: List[:class:`JobAccessControlRequest`] @@ -6003,17 +7967,23 @@ paths: will be not be attributed to any budget policy. :param email_notifications: :class:`JobEmailNotifications` (optional) An optional set of email addresses notified when the run begins or completes. + :param environment_variables: List[:class:`JobEnvironmentVariables`] + (optional) + Named environment-variable entries that tasks of this one-time run can reference by key from + ``RunTaskSettings.environment_variables_key``. Each entry holds inline ``variables`` plus optional + ``.env`` ``files``. Handled identically to ``JobSettings.environment_variables``. Maximum 10 + entries. Entries are independent of one another — there is no cross-entry merging. :param environments: List[:class:`JobEnvironment`] (optional) A list of task execution environment specifications that can be referenced by tasks of this run. :param git_source: :class:`GitSource` (optional) An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. - If `git_source` is set, these tasks retrieve the file from the remote repository by default. - However, this behavior can be overridden by setting `source` to `WORKSPACE` on the task. + If ``git_source`` is set, these tasks retrieve the file from the remote repository by default. + However, this behavior can be overridden by setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are - used, `git_source` must be defined on the job. + used, ``git_source`` must be defined on the job. :param health: :class:`JobsHealthRules` (optional) :param idempotency_token: str (optional) @@ -6026,24 +7996,35 @@ paths: This token must have at most 64 characters. - For more information, see [How to ensure idempotency for jobs]. - - [How to ensure idempotency for jobs]: https://kb.databricks.com/jobs/jobs-idempotency.html + For more information, see `How to ensure idempotency for jobs + `__. :param notification_settings: :class:`JobNotificationSettings` (optional) Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` for this run. + ``email_notifications`` and ``webhook_notifications`` for this run. + :param performance_target: :class:`PerformanceTarget` (optional) + The performance mode on a serverless one-time run. This field determines the level of compute + performance or cost-efficiency for the run. The performance target does not apply to tasks that run + on Serverless GPU compute. + + - ``STANDARD``: Enables cost-efficient execution of serverless workloads. + - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and + optimized cluster performance. :param queue: :class:`QueueSettings` (optional) The queue settings of the one-time run. :param run_as: :class:`JobRunAs` (optional) Specifies the user or service principal that the job runs as. If not specified, the job runs as the user who submits the request. :param run_name: str (optional) - An optional name for the run. The default value is `Untitled`. + An optional name for the run. The default value is ``Untitled``. + :param tags: Dict[str,str] (optional) + A map of tags associated with the one-time run, handled identically to job-level tags. The tags are + forwarded to the cluster as cluster tags for the run's compute, attributed in billing, and surfaced + in system tables. A maximum of 25 tags can be added. :param tasks: List[:class:`SubmitTask`] (optional) :param timeout_seconds: int (optional) - An optional timeout applied to each run of this job. A value of `0` means no timeout. + An optional timeout applied to each run of this job. A value of ``0`` means no timeout. :param usage_policy_id: str (optional) The user specified id of the usage policy to use for this one-time run. If not specified, a default usage policy may be applied when creating or modifying the job. @@ -6099,9 +8080,9 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold specified for - the `RUN_DURATION_SECONDS` metric in the `health` field. - If no rule for the `RUN_DURATION_SECONDS` metric is - specified in the `health` field for the job, + the ``RUN_DURATION_SECONDS`` metric in the ``health`` + field. If no rule for the ``RUN_DURATION_SECONDS`` + metric is specified in the ``health`` field for the job, notifications are not sent. on_failure: type: array @@ -6111,10 +8092,26 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an - `INTERNAL_ERROR` `life_cycle_state` or a `FAILED`, or - `TIMED_OUT` result_state. If this is not specified on - job creation, reset, or update the list is empty, and + ``INTERNAL_ERROR`` ``life_cycle_state`` or a ``FAILED``, + or ``TIMED_OUT`` result_state. If this is not specified + on job creation, reset, or update the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a continuous + job. on_start: type: array items: @@ -6131,11 +8128,12 @@ paths: description: >- A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. - Streaming backlog thresholds can be set in the `health` - field using the following metrics: - `STREAMING_BACKLOG_BYTES`, `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the + Streaming backlog thresholds can be set in the + ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. on_success: @@ -6145,13 +8143,94 @@ paths: description: >- A list of email addresses to be notified when a run successfully completes. A run is considered to have - completed successfully if it ends with a `TERMINATED` - `life_cycle_state` and a `SUCCESS` result_state. If not - specified on job creation, reset, or update, the list is - empty, and notifications are not sent. + completed successfully if it ends with a ``TERMINATED`` + ``life_cycle_state`` and a ``SUCCESS`` result_state. If + not specified on job creation, reset, or update, the + list is empty, and notifications are not sent. description: >- An optional set of email addresses notified when the run begins or completes. + environment_variables: + type: array + items: + type: object + properties: + environment_variables_key: + type: string + description: >- + Identifier for this entry. Must be unique within + ``JobSettings.environment_variables``. Tasks reference + it from ``TaskSettings.environment_variables_key``. + files: + type: array + items: + type: string + description: >- + Workspace (``/Workspace/...``) or UC Volumes + (``/Volumes/...``) paths to ``.env`` files. Maximum 5 + files. Files are read, parsed, and merged at task + execution time, not at job creation or update API call + time. File format: each line must be exactly + ``KEY=VALUE``. Keys must match the same regex as + inlined variable names (``^[A-Za-z_][A-Za-z0-9_]*$``); + the value continues to the end of the line. No other + syntax is supported — no comments, no quoted values, + no escape sequences, no variable interpolation. Any + line that does not match the ``KEY=VALUE`` shape fails + the run. Size limits: maximum 1,048,576 bytes (1 MiB) + per file on disk; maximum 131,072 bytes (128 KiB) per + ``KEY=VALUE`` line combined. Caps are enforced at read + time in jobs-runner — files exceeding the per-file + cap, or lines exceeding the per-line cap, fail the + run. On a duplicate key, the later file wins; + ``variables`` override values from any file. Values + may contain {{secrets/scope/key}} references; those + are resolved at task execution time and never + persisted in resolved form. Do not use these files to + store raw secret values; consult `secret management + `__ + for the right way to pass sensitive values. + variables: + type: object + description: >- + Environment variables specified directly as key/value + pairs (as opposed to ``files``, which are read from + ``.env`` file paths). Maximum 100 entries. Each key + must match ``^[A-Za-z_][A-Za-z0-9_]*$`` and be 1 to + 256 characters long. Each value is up to 512 + characters; larger values should be moved into a + ``.env`` file referenced from ``files``. On a + duplicate key, ``variables`` override values from any + file in ``files``. Values may contain + {{secrets/scope/key}} references; those are resolved + at task execution time and never persisted in resolved + form. Do not use this field to store a raw secret + value; consult `secret management + `__ + for the right way to pass sensitive values. + description: >- + A named environment-variable entry, defined once at the + job level and referenced by key from one + or more tasks. Entries live on ``JobSettings.environment_variables``, and tasks select one via + ``TaskSettings.environment_variables_key``. + + Precedence within a single entry (later overrides earlier on duplicate keys): + + files (merged in submitted order; later file in the list wins) └─▶ variables (inline k/v + override file contents) └─▶ runtime (os.environ / sys.env — overrides all, + classic-parity) + + Resolved values are never persisted: ``{{secrets/scope/key}}`` references in ``variables`` or in + ``files`` contents are stored verbatim and resolved at task execution time on the DP . + description: >- + Named environment-variable entries that tasks of this + one-time run can reference by key from + ``RunTaskSettings.environment_variables_key``. Each entry + holds inline ``variables`` plus optional ``.env`` ``files``. + Handled identically to + ``JobSettings.environment_variables``. Maximum 10 entries. + Entries are independent of one another — there is no + cross-entry merging. environments: type: array items: @@ -6238,14 +8317,14 @@ paths: description: >- Dirty state indicates the job is not fully synced with the job specification in the remote repository. - Possible values are: * `NOT_SYNCED`: The job is not - yet synced with the remote job specification. Import - the remote job specification from UI to make the job - fully synced. * `DISCONNECTED`: The job is temporary - disconnected from the remote job specification and - is allowed for live edit. Import the remote job - specification again from UI to make the job fully - synced. + Possible values are: - ``NOT_SYNCED``: The job is + not yet synced with the remote job specification. + Import the remote job specification from UI to make + the job fully synced. - ``DISCONNECTED``: The job is + temporary disconnected from the remote job + specification and is allowed for live edit. Import + the remote job specification again from UI to make + the job fully synced. required: - job_config_path - import_from_git_branch @@ -6266,12 +8345,13 @@ paths: An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, - and SQL File tasks. If `git_source` is set, these tasks + and SQL File tasks. If ``git_source`` is set, these tasks retrieve the file from the remote repository by default. - However, this behavior can be overridden by setting `source` - to `WORKSPACE` on the task. Note: dbt and SQL File tasks - support only version-controlled sources. If dbt or SQL File - tasks are used, `git_source` must be defined on the job. + However, this behavior can be overridden by setting + ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL + File tasks support only version-controlled sources. If dbt + or SQL File tasks are used, ``git_source`` must be defined + on the job. health: type: object properties: @@ -6293,27 +8373,21 @@ paths: evaluated for a particular health rule. - * `RUN_DURATION_SECONDS`: Expected total time for - a run in seconds. * `STREAMING_BACKLOG_BYTES`: + - ``RUN_DURATION_SECONDS``: Expected total time + for a run in seconds. - An estimate of the maximum bytes of data waiting - to be consumed across all streams. This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate of the - maximum offset lag - - across all streams. This metric is in Public - Preview. * `STREAMING_BACKLOG_SECONDS`: An - estimate - - of the maximum consumer delay across all streams. - This metric is in Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of the + - ``STREAMING_BACKLOG_BYTES``: An estimate of the + maximum bytes of data waiting to be consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate of + the maximum offset lag across all streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of + the maximum consumer delay across all streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate of the maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. This metric is in Public Preview. op: type: string x-enum: @@ -6345,9 +8419,8 @@ paths: retry until the request succeeds. Databricks guarantees that exactly one run is launched with that idempotency token. This token must have at most 64 characters. For more - information, see [How to ensure idempotency for jobs]. [How - to ensure idempotency for jobs]: - https://kb.databricks.com/jobs/jobs-idempotency.html + information, see `How to ensure idempotency for jobs + `__. notification_settings: type: object properties: @@ -6357,11 +8430,25 @@ paths: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is skipped. + specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending - notifications to each of the `email_notifications` and - `webhook_notifications` for this run. + notifications to each of the ``email_notifications`` and + ``webhook_notifications`` for this run. + performance_target: + type: string + x-enum: + - PERFORMANCE_OPTIMIZED + - STANDARD + description: >- + The performance mode on a serverless one-time run. This + field determines the level of compute performance or + cost-efficiency for the run. The performance target does not + apply to tasks that run on Serverless GPU compute. - + ``STANDARD``: Enables cost-efficient execution of serverless + workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast + startup and execution times through rapid scaling and + optimized cluster performance. queue: type: object properties: @@ -6383,7 +8470,7 @@ paths: type: string description: >- Application ID of an active service principal. Setting - this field requires the `servicePrincipal/user` role. + this field requires the ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -6397,7 +8484,15 @@ paths: type: string description: >- An optional name for the run. The default value is - `Untitled`. + ``Untitled``. + tags: + type: object + description: >- + A map of tags associated with the one-time run, handled + identically to job-level tags. The tags are forwarded to the + cluster as cluster tags for the run's compute, attributed in + billing, and surfaced in system tables. A maximum of 25 tags + can be added. tasks: type: array items: @@ -6405,11 +8500,446 @@ paths: properties: task_key: type: string + agentic_task: + type: object + properties: + context: + type: object + properties: + conversation_ids: + type: array + items: + type: string + description: >- + Optional. Conversation IDs to load as context + for the agent. + instructions: + type: array + items: + type: string + description: >- + Optional. Additional instructions to guide the + agent's behavior. + description: >- + Optional. Context input providing conversation + history and instructions. + genie_code_api: + type: object + properties: {} + description: >- + Inline Genie Code conversation driven by a single + prompt. Mutually exclusive with the supervisor + variants. + goal: + type: string + description: >- + Deprecated. Use ``input`` (field 7) instead. Kept + for backwards compatibility with existing callers; + will be removed in a future revision. + input: + type: string + description: >- + Optional. The user query / task input the agent + must accomplish. Mirrors the OpenAI Responses API + ``input`` field. Replaces the deprecated ``goal`` + field; new callers should populate ``input`` going + forward. + output_schema: + type: object + properties: + properties: + type: object + description: Map of property name to property definition. + description: >- + Optional. JSON-Schema-style declaration of the + structured output the agent should produce. + Replaces the deprecated ``task_output_schema`` + map; new callers should populate ``output_schema`` + going forward. + supervisor_agent: + type: object + properties: + agent_id: + type: string + description: >- + Required. The ID of the supervisor agent + (tile_id). + description: >- + A Supervisor Agent that orchestrates sub-agents + and tools, referenced by tile_id. + supervisor_api: + type: object + properties: + instructions: + type: string + description: >- + Optional. System instructions that guide how + the supervisor routes queries across tools and + synthesizes responses. + model: + type: string + description: >- + Required. Databricks-provided FMAPI model, + e.g. "databricks-claude-sonnet-4-5". + tools: + type: array + items: + type: object + properties: + app: + type: object + properties: + name: + type: string + description: >- + Databricks App tool config. Set when the + discriminator == "app". + catalog: + type: object + properties: + name: + type: string + description: >- + Unity Catalog catalog tool config. Set + when the discriminator == "catalog". + dashboard: + type: object + properties: + dashboard_id: + type: string + description: >- + Lakeview dashboard tool config. Set when + the discriminator == "dashboard". + description: + type: string + description: >- + Optional. Free-form description of the + tool. + genie_space: + type: object + properties: + space_id: + type: string + description: >- + Genie space tool config. Set when the + discriminator == "genie_space". + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + description: >- + Knowledge Assistant tool config. Set + when the discriminator == + "knowledge_assistant". + schema: + type: object + properties: + name: + type: string + description: >- + Unity Catalog schema tool config. Set + when the discriminator == "schema". + serving_endpoint: + type: object + properties: + name: + type: string + description: >- + Model serving endpoint tool config. Set + when the discriminator == + "serving_endpoint". + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: Supervisor Agent ID. + description: >- + Supervisor Agent tool config (a + supervisor agent used as a tool). Set + when the discriminator == + "supervisor_agent". + table: + type: object + properties: + name: + type: string + description: >- + Unity Catalog table tool config. Set + when the discriminator == "table". + tool_type: + type: string + uc_connection: + type: object + properties: + name: + type: string + description: >- + External MCP / UC connection tool + config. Set when the discriminator == + "uc_connection". + uc_function: + type: object + properties: + name: + type: string + description: >- + Unity Catalog function tool config. Set + when the discriminator == "uc_function". + uc_mcp: + type: object + properties: + name: + type: string + description: >- + UC-registered MCP service tool config. + Set when the discriminator == "uc_mcp". + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Vector Search index tool config. Set + when the discriminator == + "vector_search_index". + volume: + type: object + properties: + name: + type: string + description: >- + Unity Catalog volume tool config. Set + when the discriminator == "volume". + web_search: + type: object + properties: {} + description: >- + Web search tool config. Set when the + discriminator == "web_search". + description: >- + Tool configuration for an inlined agentic + task agent. Discriminator selects which + per-asset + config message applies (``genie_space``, ``knowledge_assistant``, ``uc_function``, + ``uc_connection``, ``uc_mcp``, ``app``, ``table``, ``vector_search_index``, ``volume``, + ``catalog``, ``schema``, ``dashboard``, ``supervisor_agent``, ``serving_endpoint``, + ``web_search``); ``description`` is a free-form top-level description; each per-asset config + carries only the asset's identifier. + + The discriminator is exposed under two interchangeable names: ``type`` mirrors the Supervisor + (Responses) API surface, and ``tool_type`` is the AIP-friendly alias for clients where ``type`` + is a reserved keyword (Terraform, etc.). Set exactly one. + description: Optional. The tools the inlined agent can use. + description: >- + Inlined Responses-API supervisor configuration + (model + instructions + tools). Mutually exclusive + with ``supervisor_agent``. + task_output_schema: + type: object + description: >- + Deprecated. Use ``output_schema`` (field 8) + instead. Kept for backwards compatibility with + existing callers; will be removed in a future + revision. + trace_destination: + type: object + properties: + catalog_name: + type: string + description: >- + Unity Catalog catalog name for the trace + table. + experiment_id: + type: string + description: MLflow experiment ID where traces are written. + schema_name: + type: string + description: Unity Catalog schema name for the trace table. + table_prefix: + type: string + description: >- + Table-name prefix under + ``.``. Trace tables are + created as + ``.._<...>``. + description: >- + Optional. Where MLflow traces produced by this + task run should be persisted. When unset, traces + follow the workspace default destination. + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + type: object + properties: + experiment: + type: string + description: >- + MLflow experiment name for this run. If an + experiment with this name already exists under the + calling user, the run is appended to it; otherwise + a new experiment is created. To target a specific + MLflow storage location (for example, when running + as a service principal), set + ``mlflow_experiment_directory``. + deployments: + type: array + items: + type: object + properties: + command_path: + type: string + description: >- + Workspace path of the script to run on each + node in this deployment. Upload the script + to this path and supply the path here. When + the task runs, the file at this path is run + on each node; if it fails, the task fails + with its exit code. Example script contents: + Plain Python: python train.py --epochs 10 + Multi-GPU via accelerate: accelerate launch + train.py --config config.yaml Distributed + via torchrun: torchrun --nproc_per_node=8 + train.py + compute: + type: object + properties: + accelerator_type: + type: string + x-enum: + - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 + - GPU_8xH100 + description: >- + Hardware accelerator type (for example, + ``GPU_1xA10`` or ``GPU_8xH100``). The + number of accelerators per node is + encoded in the enum value — + ``GPU_8xH100`` means 8 H100 GPUs per + node. + accelerator_count: + type: integer + description: >- + Total number of accelerators across all + nodes. Must be a positive multiple of + the per-node accelerator count encoded + in ``accelerator_type``. For example, + ``GPU_8xH100`` with ``accelerator_count: + 16`` allocates 2 nodes (8 GPUs per + node). + required: + - accelerator_type + - accelerator_count + description: >- + Compute resources allocated to each node in + this deployment. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs on + the specified container image instead of the + default Databricks client image. Format: + ``{organization}/{repository}:{tag}`` + name: + type: string + description: >- + Optional human-readable name for this + deployment (for example, ``driver``, + ``worker``, ``param_server``). Used for log + and UI display. Distinct names are + recommended so deployments can be told + apart, but uniqueness is not enforced. + required: + - command_path + - compute + description: >- + DeploymentSpec: configuration for one deployment + within an AiRuntimeTask. Each entry in + ``AiRuntimeTask.deployments`` describes a group of nodes that share the same command and + compute. Many single-program training algorithms use a single entry where every node runs the + same command; role-split workloads (driver + worker, parameter server, separate eval node, etc.) + use multiple entries. + description: >- + Deployment specs for this task. Exactly one + deployment is currently supported (a single entry + where every node runs the same command); this is a + current-Preview constraint. Role-split workloads + (driver + worker, parameter server, separate eval + node, etc.) with multiple entries are the eventual + intent but not yet supported. + code_source_path: + type: string + description: >- + Workspace or UC volume path of the code-source + archive, unpacked on each node and exposed through + ``$CODE_SOURCE``. Set by first-party tooling; not + for direct callers. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom container + image. When set, the task runs on the specified + container image instead of the default Databricks + client image. Format: + ``{organization}/{repository}:{tag}`` + mlflow_experiment_directory: + type: string + description: >- + Optional workspace directory under which the + MLflow experiment named in ``experiment`` is + created. Must start with ``/Workspace``. Set this + when running as a service principal that has no + default user directory; for regular users the + experiment defaults to the user's home directory. + mlflow_run: + type: string + description: >- + Optional display name for the MLflow run created + under ``experiment``. If omitted, MLflow generates + a default name. + parameters: + type: object + description: >- + Optional named parameters passed to each + deployment's command. Keys are parameter names, + values the corresponding arguments (for example, + ``{"epochs": "3", "dataset": + "s3://bucket/train"}``). Values may contain + dynamic references such as + ``{{job.trigger.time.iso_date}}`` or + ``{{tasks..values.}}``, which Jobs + substitutes before execution (see + ``AiRuntimeTaskResolvedValues.parameters`` in + runs.proto). + required: + - experiment + - deployments + description: >- + The task runs a multi-gpu compute workload on + Databricks AI Runtime. Specify the accelerator type + and count, the command to run, and where the + workload's code and MLflow output are stored. alert_task: type: object properties: alert_id: type: string + parameters: + type: object + description: >- + Per-run parameter overrides, keyed by parameter + name, applied onto the alert's stored query + parameters before the query is executed. Only + scalar values are supported. Values may reference + job parameters with ``{{job.parameters.*}}``, + which are resolved before the task runs. An + override whose key does not match a stored + parameter fails the task run. Limited to 10000 + characters when serialized as JSON; keys must be + 1-100 characters and contain only letters, digits, + underscores, dashes, and periods. subscribers: type: array items: @@ -6437,11 +8967,14 @@ paths: type: string description: >- The workspace_path is the path to the alert file - in the workspace. The path: * must start with - "/Workspace" * must be a normalized path. User has + in the workspace. The path: - must start with + "/Workspace" - must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert. - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends + notifications to subscribers when the ``alert_task`` + field is present. clean_rooms_notebook_task: type: object properties: @@ -6467,10 +9000,10 @@ paths: - clean_room_name - notebook_name description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. [clean - rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ + notebook when the ``clean_rooms_notebook_task`` field + is present. compute: type: object properties: @@ -6490,29 +9023,21 @@ paths: - LESS_THAN_OR_EQUAL - NOT_EQUAL description: >- - * `EQUAL_TO`, `NOT_EQUAL` operators perform string - comparison of their operands. This means that - - `“12.0” == “12”` will evaluate to `false`. * - `GREATER_THAN`, `GREATER_THAN_OR_EQUAL`, - - `LESS_THAN`, `LESS_THAN_OR_EQUAL` operators - perform numeric comparison of their operands. - - `“12.0” >= “12”` will evaluate to `true`, `“10.0” - >= “12”` will evaluate to - - `false`. - + - ``EQUAL_TO``, ``NOT_EQUAL`` operators perform + string comparison of their operands. This means + that ``“12.0” == “12”`` will evaluate to ``false``. + - ``GREATER_THAN``, ``GREATER_THAN_OR_EQUAL``, + ``LESS_THAN``, ``LESS_THAN_OR_EQUAL`` operators + perform numeric comparison of their operands. ``“12.0” >= “12”`` will evaluate to + ``true``, ``“10.0” >= “12”`` will evaluate to ``false``. The boolean comparison to task values can be - implemented with operators `EQUAL_TO`, - `NOT_EQUAL`. + implemented with operators ``EQUAL_TO``, - If a task value was set to a boolean value, it - will be serialized to `“true”` or + ``NOT_EQUAL``. If a task value was set to a + boolean value, it will be serialized to - `“false”` for the comparison. + ``“true”`` or ``“false”`` for the comparison. left: type: string description: >- @@ -6532,9 +9057,9 @@ paths: description: >- The task evaluates a condition that can be used to control the execution of other tasks when the - `condition_task` field is present. The condition task - does not require a cluster to execute and does not - support retries or notifications. + ``condition_task`` field is present. The condition + task does not require a cluster to execute and does + not support retries or notifications. dashboard_task: type: object properties: @@ -6551,14 +9076,14 @@ paths: parameter key. The parameter value format is dependent on the filter type: - For text and single-select filters, provide a single value - (e.g. `"value"`) - For date and datetime filters, - provide the value in ISO 8601 format (e.g. - `"2000-01-01T00:00:00"`) - For multi-select - filters, provide a JSON array of values (e.g. - `"[\"value1\",\"value2\"]"`) - For range and date - range filters, provide a JSON object with `start` - and `end` (e.g. - `"{\"start\":\"1\",\"end\":\"10\"}"`) + (e.g. ``"value"``) - For date and datetime + filters, provide the value in ISO 8601 format + (e.g. ``"2000-01-01T00:00:00"``) - For + multi-select filters, provide a JSON array of + values (e.g. ``"[\"value1\",\"value2\"]"``) - For + range and date range filters, provide a JSON + object with ``start`` and ``end`` (e.g. + ``"{\"start\":\"1\",\"end\":\"10\"}"``) subscription: type: object properties: @@ -6580,8 +9105,8 @@ paths: type: string description: >- A snapshot of the dashboard will be sent - to the user's email when the `user_name` - field is present. + to the user's email when the + ``user_name`` field is present. description: >- The list of subscribers to send the snapshot of the dashboard to. @@ -6657,7 +9182,7 @@ paths: description: >- Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If - not provided, the `default` schema is used. + not provided, the ``default`` schema is used. source: type: string x-enum: @@ -6665,15 +9190,15 @@ paths: - WORKSPACE description: >- Optional location type of the project directory. - When set to `WORKSPACE`, the project will be + When set to ``WORKSPACE``, the project will be retrieved from the local Databricks workspace. - When set to `GIT`, the project will be retrieved - from a Git repository defined in `git_source`. If - the value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` otherwise. - * `WORKSPACE`: Project is located in Databricks - workspace. * `GIT`: Project is located in cloud - Git provider. + When set to ``GIT``, the project will be retrieved + from a Git repository defined in ``git_source``. + If the value is empty, the task will use ``GIT`` + if ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Project is located in + Databricks workspace. - ``GIT``: Project is + located in cloud Git provider. warehouse_id: type: string description: >- @@ -6681,12 +9206,12 @@ paths: provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the - `--profiles-dir` command line argument. + ``--profiles-dir`` command line argument. required: - commands description: >- The task runs one or more dbt commands when the - `dbt_task` field is present. The dbt task requires + ``dbt_task`` field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse. depends_on: @@ -6708,7 +9233,7 @@ paths: An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully before executing this task. - The key is `task_key`, and the value is the name + The key is ``task_key``, and the value is the name assigned to the dependent task. description: type: string @@ -6716,6 +9241,11 @@ paths: disable_auto_optimization: type: boolean description: An option to disable auto optimization in serverless + disabled: + type: boolean + description: >- + An optional flag to disable the task. If set to true, + the task will not run even if it is part of a job. email_notifications: type: object properties: @@ -6728,11 +9258,11 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold specified - for the `RUN_DURATION_SECONDS` metric in the - `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified in the - `health` field for the job, notifications are not - sent. + for the ``RUN_DURATION_SECONDS`` metric in the + ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is specified in + the ``health`` field for the job, notifications + are not sent. on_failure: type: array items: @@ -6741,10 +9271,26 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with - an `INTERNAL_ERROR` `life_cycle_state` or a - `FAILED`, or `TIMED_OUT` result_state. If this is - not specified on job creation, reset, or update + an ``INTERNAL_ERROR`` ``life_cycle_state`` or a + ``FAILED``, or ``TIMED_OUT`` result_state. If this + is not specified on job creation, reset, or update the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a + continuous job. on_start: type: array items: @@ -6762,11 +9308,11 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in - the `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on + the ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. @@ -6778,9 +9324,9 @@ paths: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a - `TERMINATED` `life_cycle_state` and a `SUCCESS` - result_state. If not specified on job creation, - reset, or update, the list is empty, and + ``TERMINATED`` ``life_cycle_state`` and a + ``SUCCESS`` result_state. If not specified on job + creation, reset, or update, the list is empty, and notifications are not sent. description: >- An optional set of email addresses notified when the @@ -6792,6 +9338,16 @@ paths: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry + defined in ``RunSettings.environment_variables``. The + selected entry's variables and file contents are + applied to this task at execution time. Length and + pattern mirror ``environment_key`` so the two + references look identical to customers reading task + settings. existing_cluster_id: type: string description: >- @@ -6811,11 +9367,475 @@ paths: properties: task_key: type: string + agentic_task: + type: object + properties: + context: + type: object + properties: + conversation_ids: + type: array + items: + type: string + description: >- + Optional. Conversation IDs to load as + context for the agent. + instructions: + type: array + items: + type: string + description: >- + Optional. Additional instructions to + guide the agent's behavior. + description: >- + Optional. Context input providing + conversation history and instructions. + genie_code_api: + type: object + properties: {} + description: >- + Inline Genie Code conversation driven by a + single prompt. Mutually exclusive with the + supervisor variants. + goal: + type: string + description: >- + Deprecated. Use ``input`` (field 7) + instead. Kept for backwards compatibility + with existing callers; will be removed in + a future revision. + input: + type: string + description: >- + Optional. The user query / task input the + agent must accomplish. Mirrors the OpenAI + Responses API ``input`` field. Replaces + the deprecated ``goal`` field; new callers + should populate ``input`` going forward. + output_schema: + type: object + properties: + properties: + type: object + description: >- + Map of property name to property + definition. + description: >- + Optional. JSON-Schema-style declaration of + the structured output the agent should + produce. Replaces the deprecated + ``task_output_schema`` map; new callers + should populate ``output_schema`` going + forward. + supervisor_agent: + type: object + properties: + agent_id: + type: string + description: >- + Required. The ID of the supervisor agent + (tile_id). + description: >- + A Supervisor Agent that orchestrates + sub-agents and tools, referenced by + tile_id. + supervisor_api: + type: object + properties: + instructions: + type: string + description: >- + Optional. System instructions that guide + how the supervisor routes queries across + tools and synthesizes responses. + model: + type: string + description: >- + Required. Databricks-provided FMAPI + model, e.g. + "databricks-claude-sonnet-4-5". + tools: + type: array + items: + type: object + properties: + app: + type: object + properties: + name: + type: string + description: >- + Databricks App tool config. Set when the + discriminator == "app". + catalog: + type: object + properties: + name: + type: string + description: >- + Unity Catalog catalog tool config. Set + when the discriminator == "catalog". + dashboard: + type: object + properties: + dashboard_id: + type: string + description: >- + Lakeview dashboard tool config. Set when + the discriminator == "dashboard". + description: + type: string + description: >- + Optional. Free-form description of the + tool. + genie_space: + type: object + properties: + space_id: + type: string + description: >- + Genie space tool config. Set when the + discriminator == "genie_space". + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + description: >- + Knowledge Assistant tool config. Set + when the discriminator == + "knowledge_assistant". + schema: + type: object + properties: + name: + type: string + description: >- + Unity Catalog schema tool config. Set + when the discriminator == "schema". + serving_endpoint: + type: object + properties: + name: + type: string + description: >- + Model serving endpoint tool config. Set + when the discriminator == + "serving_endpoint". + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: Supervisor Agent ID. + description: >- + Supervisor Agent tool config (a + supervisor agent used as a tool). Set + when the discriminator == + "supervisor_agent". + table: + type: object + properties: + name: + type: string + description: >- + Unity Catalog table tool config. Set + when the discriminator == "table". + tool_type: + type: string + uc_connection: + type: object + properties: + name: + type: string + description: >- + External MCP / UC connection tool + config. Set when the discriminator == + "uc_connection". + uc_function: + type: object + properties: + name: + type: string + description: >- + Unity Catalog function tool config. Set + when the discriminator == "uc_function". + uc_mcp: + type: object + properties: + name: + type: string + description: >- + UC-registered MCP service tool config. + Set when the discriminator == "uc_mcp". + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Vector Search index tool config. Set + when the discriminator == + "vector_search_index". + volume: + type: object + properties: + name: + type: string + description: >- + Unity Catalog volume tool config. Set + when the discriminator == "volume". + web_search: + type: object + properties: {} + description: >- + Web search tool config. Set when the + discriminator == "web_search". + description: >- + Tool configuration for an inlined + agentic task agent. Discriminator + selects which per-asset + config message applies (``genie_space``, ``knowledge_assistant``, ``uc_function``, + ``uc_connection``, ``uc_mcp``, ``app``, ``table``, ``vector_search_index``, ``volume``, + ``catalog``, ``schema``, ``dashboard``, ``supervisor_agent``, ``serving_endpoint``, + ``web_search``); ``description`` is a free-form top-level description; each per-asset config + carries only the asset's identifier. + + The discriminator is exposed under two interchangeable names: ``type`` mirrors the Supervisor + (Responses) API surface, and ``tool_type`` is the AIP-friendly alias for clients where ``type`` + is a reserved keyword (Terraform, etc.). Set exactly one. + description: >- + Optional. The tools the inlined agent + can use. + description: >- + Inlined Responses-API supervisor + configuration (model + instructions + + tools). Mutually exclusive with + ``supervisor_agent``. + task_output_schema: + type: object + description: >- + Deprecated. Use ``output_schema`` (field + 8) instead. Kept for backwards + compatibility with existing callers; will + be removed in a future revision. + trace_destination: + type: object + properties: + catalog_name: + type: string + description: >- + Unity Catalog catalog name for the trace + table. + experiment_id: + type: string + description: >- + MLflow experiment ID where traces are + written. + schema_name: + type: string + description: >- + Unity Catalog schema name for the trace + table. + table_prefix: + type: string + description: >- + Table-name prefix under + ``.``. Trace tables are + created as + ``.._<...>``. + description: >- + Optional. Where MLflow traces produced by + this task run should be persisted. When + unset, traces follow the workspace default + destination. + description: >- + Agentic Task for job-based multi-agent + execution + ai_runtime_task: + type: object + properties: + experiment: + type: string + description: >- + MLflow experiment name for this run. If an + experiment with this name already exists + under the calling user, the run is + appended to it; otherwise a new experiment + is created. To target a specific MLflow + storage location (for example, when + running as a service principal), set + ``mlflow_experiment_directory``. + deployments: + type: array + items: + type: object + properties: + command_path: + type: string + description: >- + Workspace path of the script to run on + each node in this deployment. Upload the + script to this path and supply the path + here. When the task runs, the file at + this path is run on each node; if it + fails, the task fails with its exit + code. Example script contents: Plain + Python: python train.py --epochs 10 + Multi-GPU via accelerate: accelerate + launch train.py --config config.yaml + Distributed via torchrun: torchrun + --nproc_per_node=8 train.py + compute: + type: object + properties: + accelerator_type: + type: string + x-enum: + - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 + - GPU_8xH100 + description: >- + Hardware accelerator type (for example, + ``GPU_1xA10`` or ``GPU_8xH100``). The + number of accelerators per node is + encoded in the enum value — + ``GPU_8xH100`` means 8 H100 GPUs per + node. + accelerator_count: + type: integer + description: >- + Total number of accelerators across all + nodes. Must be a positive multiple of + the per-node accelerator count encoded + in ``accelerator_type``. For example, + ``GPU_8xH100`` with ``accelerator_count: + 16`` allocates 2 nodes (8 GPUs per + node). + required: + - accelerator_type + - accelerator_count + description: >- + Compute resources allocated to each node + in this deployment. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs + on the specified container image instead + of the default Databricks client image. + Format: + ``{organization}/{repository}:{tag}`` + name: + type: string + description: >- + Optional human-readable name for this + deployment (for example, ``driver``, + ``worker``, ``param_server``). Used for + log and UI display. Distinct names are + recommended so deployments can be told + apart, but uniqueness is not enforced. + required: + - command_path + - compute + description: >- + DeploymentSpec: configuration for one + deployment within an AiRuntimeTask. Each + entry in + ``AiRuntimeTask.deployments`` describes a group of nodes that share the same command and + compute. Many single-program training algorithms use a single entry where every node runs the + same command; role-split workloads (driver + worker, parameter server, separate eval node, etc.) + use multiple entries. + description: >- + Deployment specs for this task. Exactly + one deployment is currently supported (a + single entry where every node runs the + same command); this is a current-Preview + constraint. Role-split workloads (driver + + worker, parameter server, separate eval + node, etc.) with multiple entries are the + eventual intent but not yet supported. + code_source_path: + type: string + description: >- + Workspace or UC volume path of the + code-source archive, unpacked on each node + and exposed through ``$CODE_SOURCE``. Set + by first-party tooling; not for direct + callers. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs + on the specified container image instead + of the default Databricks client image. + Format: + ``{organization}/{repository}:{tag}`` + mlflow_experiment_directory: + type: string + description: >- + Optional workspace directory under which + the MLflow experiment named in + ``experiment`` is created. Must start with + ``/Workspace``. Set this when running as a + service principal that has no default user + directory; for regular users the + experiment defaults to the user's home + directory. + mlflow_run: + type: string + description: >- + Optional display name for the MLflow run + created under ``experiment``. If omitted, + MLflow generates a default name. + parameters: + type: object + description: >- + Optional named parameters passed to each + deployment's command. Keys are parameter + names, values the corresponding arguments + (for example, ``{"epochs": "3", "dataset": + "s3://bucket/train"}``). Values may + contain dynamic references such as + ``{{job.trigger.time.iso_date}}`` or + ``{{tasks..values.}}``, + which Jobs substitutes before execution + (see + ``AiRuntimeTaskResolvedValues.parameters`` + in runs.proto). + required: + - experiment + - deployments + description: >- + The task runs a multi-gpu compute workload on + Databricks AI Runtime. Specify the accelerator + type and count, the command to run, and where + the workload's code and MLflow output are + stored. alert_task: type: object properties: alert_id: type: string + parameters: + type: object + description: >- + Per-run parameter overrides, keyed by + parameter name, applied onto the alert's + stored query parameters before the query + is executed. Only scalar values are + supported. Values may reference job + parameters with ``{{job.parameters.*}}``, + which are resolved before the task runs. + An override whose key does not match a + stored parameter fails the task run. + Limited to 10000 characters when + serialized as JSON; keys must be 1-100 + characters and contain only letters, + digits, underscores, dashes, and periods. subscribers: type: array items: @@ -6845,12 +9865,15 @@ paths: type: string description: >- The workspace_path is the path to the - alert file in the workspace. The path: * - must start with "/Workspace" * must be a + alert file in the workspace. The path: - + must start with "/Workspace" - must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert. - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and + sends notifications to subscribers when the + ``alert_task`` field is present. clean_rooms_notebook_task: type: object properties: @@ -6879,10 +9902,11 @@ paths: - clean_room_name - notebook_name description: >- - The task runs a [clean rooms] notebook when - the `clean_rooms_notebook_task` field is - present. [clean rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ + notebook when the + ``clean_rooms_notebook_task`` field is + present. compute: type: object properties: @@ -6902,32 +9926,25 @@ paths: - LESS_THAN_OR_EQUAL - NOT_EQUAL description: >- - * `EQUAL_TO`, `NOT_EQUAL` operators + - ``EQUAL_TO``, ``NOT_EQUAL`` operators perform string comparison of their - operands. This means that - - `“12.0” == “12”` will evaluate to `false`. - * `GREATER_THAN`, `GREATER_THAN_OR_EQUAL`, - - `LESS_THAN`, `LESS_THAN_OR_EQUAL` - operators perform numeric comparison of - their operands. - - `“12.0” >= “12”` will evaluate to `true`, - `“10.0” >= “12”` will evaluate to - - `false`. - + operands. This means + that ``“12.0” == “12”`` will evaluate to ``false``. + - ``GREATER_THAN``, + ``GREATER_THAN_OR_EQUAL``, ``LESS_THAN``, + ``LESS_THAN_OR_EQUAL`` operators + perform numeric comparison of their operands. ``“12.0” >= “12”`` will evaluate to + ``true``, ``“10.0” >= “12”`` will evaluate to ``false``. The boolean comparison to task values can - be implemented with operators `EQUAL_TO`, - `NOT_EQUAL`. + be implemented with operators + ``EQUAL_TO``, - If a task value was set to a boolean - value, it will be serialized to `“true”` - or + ``NOT_EQUAL``. If a task value was set to + a boolean value, it will be serialized to - `“false”` for the comparison. + ``“true”`` or ``“false”`` for the + comparison. left: type: string description: >- @@ -6947,7 +9964,7 @@ paths: description: >- The task evaluates a condition that can be used to control the execution of other tasks - when the `condition_task` field is present. + when the ``condition_task`` field is present. The condition task does not require a cluster to execute and does not support retries or notifications. @@ -6970,16 +9987,16 @@ paths: key. The parameter value format is dependent on the filter type: - For text and single-select filters, provide a - single value (e.g. `"value"`) - For date + single value (e.g. ``"value"``) - For date and datetime filters, provide the value in ISO 8601 format (e.g. - `"2000-01-01T00:00:00"`) - For + ``"2000-01-01T00:00:00"``) - For multi-select filters, provide a JSON array of values (e.g. - `"[\"value1\",\"value2\"]"`) - For range + ``"[\"value1\",\"value2\"]"``) - For range and date range filters, provide a JSON - object with `start` and `end` (e.g. - `"{\"start\":\"1\",\"end\":\"10\"}"`) + object with ``start`` and ``end`` (e.g. + ``"{\"start\":\"1\",\"end\":\"10\"}"``) subscription: type: object properties: @@ -7001,8 +10018,8 @@ paths: type: string description: >- A snapshot of the dashboard will be sent - to the user's email when the `user_name` - field is present. + to the user's email when the + ``user_name`` field is present. description: >- The list of subscribers to send the snapshot of the dashboard to. @@ -7083,7 +10100,7 @@ paths: Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the - `default` schema is used. + ``default`` schema is used. source: type: string x-enum: @@ -7091,16 +10108,17 @@ paths: - WORKSPACE description: >- Optional location type of the project - directory. When set to `WORKSPACE`, the + directory. When set to ``WORKSPACE``, the project will be retrieved from the local - Databricks workspace. When set to `GIT`, + Databricks workspace. When set to ``GIT``, the project will be retrieved from a Git - repository defined in `git_source`. If the - value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Project is - located in Databricks workspace. * `GIT`: - Project is located in cloud Git provider. + repository defined in ``git_source``. If + the value is empty, the task will use + ``GIT`` if ``git_source`` is defined and + ``WORKSPACE`` otherwise. - ``WORKSPACE``: + Project is located in Databricks + workspace. - ``GIT``: Project is located + in cloud Git provider. warehouse_id: type: string description: >- @@ -7109,14 +10127,15 @@ paths: provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the - `--profiles-dir` command line argument. + ``--profiles-dir`` command line argument. required: - commands description: >- The task runs one or more dbt commands when - the `dbt_task` field is present. The dbt task - requires both Databricks SQL and the ability - to use a serverless or a pro SQL warehouse. + the ``dbt_task`` field is present. The dbt + task requires both Databricks SQL and the + ability to use a serverless or a pro SQL + warehouse. depends_on: type: array items: @@ -7138,9 +10157,9 @@ paths: dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if - the `run_if` condition is true. The key is - `task_key`, and the value is the name assigned - to the dependent task. + the ``run_if`` condition is true. The key is + ``task_key``, and the value is the name + assigned to the dependent task. description: type: string description: An optional description for this task. @@ -7168,11 +10187,11 @@ paths: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the - `RUN_DURATION_SECONDS` metric in the - `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified - in the `health` field for the job, - notifications are not sent. + ``RUN_DURATION_SECONDS`` metric in the + ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is + specified in the ``health`` field for the + job, notifications are not sent. on_failure: type: array items: @@ -7182,11 +10201,27 @@ paths: when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an - `INTERNAL_ERROR` `life_cycle_state` or a - `FAILED`, or `TIMED_OUT` result_state. If - this is not specified on job creation, - reset, or update the list is empty, and - notifications are not sent. + ``INTERNAL_ERROR`` ``life_cycle_state`` or + a ``FAILED``, or ``TIMED_OUT`` + result_state. If this is not specified on + job creation, reset, or update the list is + empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes + for a continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for + a continuous job. on_start: type: array items: @@ -7204,12 +10239,12 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog - thresholds can be set in the `health` + thresholds can be set in the ``health`` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. @@ -7222,8 +10257,8 @@ paths: when a run successfully completes. A run is considered to have completed successfully if it ends with a - `TERMINATED` `life_cycle_state` and a - `SUCCESS` result_state. If not specified + ``TERMINATED`` ``life_cycle_state`` and a + ``SUCCESS`` result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent. @@ -7240,6 +10275,17 @@ paths: a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` + entry defined in + ``JobSettings.environment_variables``. The + selected entry's variables and file contents + are applied to this task at execution time. + Length and pattern mirror ``environment_key`` + so the two references look identical to + customers reading task settings. existing_cluster_id: type: string description: >- @@ -7254,13 +10300,33 @@ paths: $ref: '#/components/schemas/ForEachTask' description: >- The task executes a nested task for every - input provided when the `for_each_task` field - is present. + input provided when the ``for_each_task`` + field is present. gen_ai_compute_task: type: object properties: dl_runtime_image: type: string + description: Runtime image + client_version: + type: string + description: >- + Version of the client (e.g., sgcli wheel) + that submitted this task. Used by handlers + to gate behavior or reject incompatible + versions. + code_source_tar_path: + type: string + description: >- + Optional path to a tarball containing the + user's workspace contents. When set, the + entry script extracts the tarball into the + working directory before running the + training script, so the training script + can import sibling modules and read + packaged data files. Must be a workspace + path (e.g. ``/Workspace/Users/...``) or + volume; ``dbfs:/`` is not supported. command: type: string description: >- @@ -7279,6 +10345,13 @@ paths: description: GPU type. required: - num_gpus + docker_image_url: + type: string + description: >- + Optional custom Docker container image URL + for running the training script. Format: + organization/repository:tag (e.g., + "pytorch/pytorch:2.0.1") mlflow_experiment_name: type: string description: >- @@ -7286,6 +10359,26 @@ paths: MLflow experiment to log the run to. If name is not found, backend will create the mlflow experiment using the name. + mlflow_run_name: + type: string + description: >- + Optional name to assign to the MLflow run + created for this task. If unset, MLflow + auto-generates a name. Used alongside + ``mlflow_experiment_name`` to identify the + run in the MLflow UI. + requirements_yaml_path: + type: string + description: >- + Optional path to a requirements.yaml file + describing pip dependencies to install + before running the training script. + Consumed by the entry script; format + matches the runtime requirements.yaml + convention used by sgcli. Must be a + workspace path (e.g. + ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. source: type: string x-enum: @@ -7293,16 +10386,17 @@ paths: - WORKSPACE description: >- Optional location type of the training - script. When set to `WORKSPACE`, the + script. When set to ``WORKSPACE``, the script will be retrieved from the local - Databricks workspace. When set to `GIT`, + Databricks workspace. When set to ``GIT``, the script will be retrieved from a Git - repository defined in `git_source`. If the - value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Script is - located in Databricks workspace. * `GIT`: - Script is located in cloud Git provider. + repository defined in ``git_source``. If + the value is empty, the task will use + ``GIT`` if ``git_source`` is defined and + ``WORKSPACE`` otherwise. - ``WORKSPACE``: + Script is located in Databricks workspace. + - ``GIT``: Script is located in cloud Git + provider. training_script_path: type: string description: >- @@ -7311,7 +10405,7 @@ paths: s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the Databricks workspace, the path must be - absolute and begin with `/`. For files + absolute and begin with ``/``. For files stored in a remote repository, the path must be relative. This field is required. yaml_parameters: @@ -7330,6 +10424,23 @@ paths: script. required: - dl_runtime_image + description: >- + DEPRECATED — use ``AiRuntimeTask`` for all new + BYOT multi-node GPU workloads (see + ai_runtime_task.proto). ``AiRuntimeTask`` is the only supported BYOT task type for new + workloads; this proto is retained only for AIR CLI (fka SGCLI) pywheel backwards compatibility + and will be removed once the pywheel → databricks-cli migration completes (post- PuPr). + genie_task: + type: object + properties: + configuration_id: + type: string + description: >- + Required. Resource name of the agent task + configuration to run. + required: + - configuration_id + description: Runs a Genie or Genie Code agent task. health: type: object properties: @@ -7352,32 +10463,25 @@ paths: rule. - * `RUN_DURATION_SECONDS`: Expected total - time for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all - streams. This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate - of the maximum offset lag - - across all streams. This metric is in - Public Preview. * - `STREAMING_BACKLOG_SECONDS`: An estimate - - of the maximum consumer delay across all - streams. This metric is in Public - Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate - of the maximum number of outstanding - files across all - - streams. This metric is in Public - Preview. + - ``RUN_DURATION_SECONDS``: Expected + total time for a run in seconds. + + - ``STREAMING_BACKLOG_BYTES``: An + estimate of the maximum bytes of data + waiting to be consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An + estimate of the maximum offset lag + across all streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An + estimate of the maximum consumer delay + across all streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An + estimate of the maximum number of + outstanding files across all + streams. This metric is in Public Preview. op: type: string x-enum: @@ -7404,7 +10508,7 @@ paths: description: >- If job_cluster_key, this task is executed reusing the cluster specified in - `job.settings.job_clusters`. + ``job.settings.job_clusters``. libraries: type: string description: >- @@ -7416,11 +10520,11 @@ paths: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be - unsuccessful if it completes with the `FAILED` - result_state or `INTERNAL_ERROR` - `life_cycle_state`. The value `-1` means to - retry indefinitely and the value `0` means to - never retry. + unsuccessful if it completes with the + ``FAILED`` result_state or ``INTERNAL_ERROR`` + ``life_cycle_state``. The value ``-1`` means + to retry indefinitely and the value ``0`` + means to never retry. min_retry_interval_millis: type: integer description: >- @@ -7447,22 +10551,21 @@ paths: call to :method:jobs/run Now with parameters specified, the two parameters maps are merged. If the same key is - specified in `base_parameters` and in - `run-now`, the value from `run-now` is - used. Use [Task parameter variables] to - set parameters containing information + specified in ``base_parameters`` and in + ``run-now``, the value from ``run-now`` is + used. Use `Task parameter variables + `__ + to set parameters containing information about job runs. If the notebook takes a parameter that is not specified in the - job’s `base_parameters` or the `run-now` - override parameters, the default value - from the notebook is used. Retrieve these - parameters in a notebook using - [dbutils.widgets.get]. The JSON - representation of this field cannot exceed - 1MB. [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-widgets + job’s ``base_parameters`` or the + ``run-now`` override parameters, the + default value from the notebook is used. + Retrieve these parameters in a notebook + using `dbutils.widgets.get + `__. + The JSON representation of this field + cannot exceed 1MB. source: type: string x-enum: @@ -7470,20 +10573,21 @@ paths: - WORKSPACE description: >- Optional location type of the notebook. - When set to `WORKSPACE`, the notebook will - be retrieved from the local Databricks - workspace. When set to `GIT`, the notebook - will be retrieved from a Git repository - defined in `git_source`. If the value is - empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Notebook is - located in Databricks workspace. * `GIT`: - Notebook is located in cloud Git provider. + When set to ``WORKSPACE``, the notebook + will be retrieved from the local + Databricks workspace. When set to ``GIT``, + the notebook will be retrieved from a Git + repository defined in ``git_source``. If + the value is empty, the task will use + ``GIT`` if ``git_source`` is defined and + ``WORKSPACE`` otherwise. - ``WORKSPACE``: + Notebook is located in Databricks + workspace. - ``GIT``: Notebook is located + in cloud Git provider. warehouse_id: type: string description: >- - Optional `warehouse_id` to run the + Optional ``warehouse_id`` to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses. Note @@ -7494,7 +10598,7 @@ paths: - notebook_path description: >- The task runs a notebook when the - `notebook_task` field is present. + ``notebook_task`` field is present. notification_settings: type: object properties: @@ -7504,19 +10608,19 @@ paths: type: boolean description: >- If true, do not send notifications to - recipients specified in `on_failure` if + recipients specified in ``on_failure`` if the run is canceled. no_alert_for_skipped_runs: type: boolean description: >- If true, do not send notifications to - recipients specified in `on_failure` if + recipients specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and - `webhook_notifications` for this task. + ``email_notifications`` and + ``webhook_notifications`` for this task. pipeline_task: type: object properties: @@ -7526,12 +10630,49 @@ paths: type: boolean description: >- If true, triggers a full refresh on the - delta live table. + spark declarative pipeline. + full_refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update with + fullRefresh. + parameters: + type: object + description: >- + Key/value-map of parameters passed to the + pipeline execution. Limited to 10k + characters in total. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These + are unioned with other selective refresh + options (refresh_selection, + full_refresh_selection) to determine the + final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset + checkpoints without clearing data. required: - pipeline_id description: >- The task triggers a pipeline update when the - `pipeline_task` field is present. Only + ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: @@ -7539,6 +10680,75 @@ paths: properties: connection_resource_name: type: string + incremental_refresh_config: + type: object + properties: + archive_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the archive window. + Default: MONTH + archive_window_periods: + type: integer + description: >- + Number of periods in the archive window + (total data retained). Must be greater + than the refresh window when using the + same granularity. Default: 3 (with YEAR + granularity = 3 years) + detect_data_changes: + type: boolean + description: >- + If true, Power BI will check whether + data has changed in each partition + before reimporting. Partitions with + unchanged data are skipped, reducing + refresh cost further. Uses + MAX(datetime_column) as the change + detector. + mode: + type: string + x-enum: + - REFRESH_POLICY_MODE_HYBRID + - REFRESH_POLICY_MODE_IMPORT_ONLY + description: >- + The refresh policy mode. Hybrid adds a + real-time DirectQuery partition for the + most recent data; Import uses only + batch-imported partitions. Default: + HYBRID + only_refresh_complete_periods: + type: boolean + description: >- + If true, only refresh complete periods + (e.g., skip today's partial data). + refresh_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the refresh window. + Default: DAY + refresh_window_periods: + type: integer + description: >- + Number of periods in the rolling refresh + window. Partitions within this window + are re-imported on each refresh. + Default: 1 + description: >- + Incremental refresh policy applied to all + IMPORT mode tables in the model. Windows + and mode are shared; partition columns are + set per-table on PowerBiTable. power_bi_model: type: object properties: @@ -7636,6 +10846,15 @@ paths: properties: catalog: type: string + incremental_refresh_datetime_column: + type: string + description: >- + The datetime column used for incremental + refresh partitioning on this table. + e.g., "order_date", "updated_at" Only + applicable when the task has + incremental_refresh_config set and this + table uses IMPORT storage mode. name: type: string description: The table name in Databricks @@ -7649,6 +10868,25 @@ paths: - DUAL - IMPORT description: The Power BI storage mode of the table + table_type: + type: string + x-enum: + - EXTERNAL + - EXTERNAL_SHALLOW_CLONE + - FOREIGN + - MANAGED + - MANAGED_SHALLOW_CLONE + - MATERIALIZED_VIEW + - METRIC_VIEW + - STREAMING_TABLE + - VIEW + description: >- + The Unity Catalog table type of this + table. When set, the server trusts it to + classify the table during metric-view + filtering and skips the server-side + lookup. When unset, the server resolves + the type on its own. description: The tables to be exported to Power BI warehouse_id: type: string @@ -7657,8 +10895,27 @@ paths: BI data source description: >- The task triggers a Power BI semantic model - update when the `power_bi_task` field is + update when the ``power_bi_task`` field is present. + python_operator_task: + type: object + properties: + main: + type: string + parameters: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: >- + An ordered list of task parameters. + TODO(JOBS-30885): Add limits for + parameters. + description: The task runs a Python operator task. python_wheel_task: type: object properties: @@ -7671,14 +10928,15 @@ paths: exist in the metadata of the package it executes the function from the package directly using - `$packageName.$entryPoint()` + ``$packageName.$entryPoint()`` named_parameters: type: object description: >- Command-line parameters passed to Python - wheel task in the form of `["--name=task", - "--data=dbfs:/path/to/data.json"]`. Leave - it empty if `parameters` is not null. + wheel task in the form of + ``["--name=task", + "--data=dbfs:/path/to/data.json"]``. Leave + it empty if ``parameters`` is not null. parameters: type: array items: @@ -7686,13 +10944,13 @@ paths: description: >- Command-line parameters passed to Python wheel task. Leave it empty if - `named_parameters` is not null. + ``named_parameters`` is not null. required: - package_name - entry_point description: >- The task runs a Python wheel when the - `python_wheel_task` field is present. + ``python_wheel_task`` field is present. retry_on_timeout: type: boolean description: >- @@ -7711,15 +10969,16 @@ paths: description: >- An optional value specifying the condition determining whether the task is run once its - dependencies have been completed. * - `ALL_SUCCESS`: All dependencies have executed - and succeeded * `AT_LEAST_ONE_SUCCESS`: At - least one dependency has succeeded * - `NONE_FAILED`: None of the dependencies have - failed and at least one was executed * - `ALL_DONE`: All dependencies have been - completed * `AT_LEAST_ONE_FAILED`: At least - one dependency failed * `ALL_FAILED`: ALl + dependencies have been completed. - + ``ALL_SUCCESS``: All dependencies have + executed and succeeded - + ``AT_LEAST_ONE_SUCCESS``: At least one + dependency has succeeded - ``NONE_FAILED``: + None of the dependencies have failed and at + least one was executed - ``ALL_DONE``: All + dependencies have been completed - + ``AT_LEAST_ONE_FAILED``: At least one + dependency failed - ``ALL_FAILED``: ALl dependencies have failed run_job_task: type: object @@ -7733,33 +10992,31 @@ paths: description: >- An array of commands to execute for jobs with the dbt task, for example - `"dbt_commands": ["dbt deps", "dbt seed", - "dbt deps", "dbt seed", "dbt run"]` ⚠ - **Deprecation note** Use [job parameters] - to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``"dbt_commands": ["dbt deps", "dbt seed", + "dbt deps", "dbt seed", "dbt run"]`` ⚠ + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark - JAR tasks, for example `"jar_params": - ["john doe", "35"]`. The parameters are + JAR tasks, for example ``"jar_params": + ["john doe", "35"]``. The parameters are used to invoke the main function of the main class specified in the Spark JAR - task. If not specified upon `run-now`, it - defaults to an empty list. jar_params + task. If not specified upon ``run-now``, + it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot - exceed 10,000 bytes. ⚠ **Deprecation - note** Use [job parameters] to pass - information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) + cannot exceed 10,000 bytes. ⚠ + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: >- @@ -7770,28 +11027,59 @@ paths: description: >- A map from keys to values for jobs with notebook task, for example - `"notebook_params": {"name": "john doe", - "age": "35"}`. The map is passed to the + ``"notebook_params": {"name": "john doe", + "age": "35"}``. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. If not - specified upon `run-now`, the triggered - run uses the job’s base parameters. - notebook_params cannot be specified in - conjunction with jar_params. ⚠ - **Deprecation note** Use [job parameters] + `dbutils.widgets.get + `__ + function. If not specified upon + ``run-now``, the triggered run uses the + job’s base parameters. notebook_params + cannot be specified in conjunction with + jar_params. ⚠ **Deprecation note** Use + `job parameters + `__ to pass information down to tasks. The JSON representation of this field (for - example `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 - bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + example ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 + bytes. pipeline_params: type: object properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update with + fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These + are unioned with other selective refresh + options (refresh_selection, + full_refresh_selection) to determine the + final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset + checkpoints without clearing data. description: >- Controls whether the pipeline should perform a full refresh @@ -7803,24 +11091,24 @@ paths: type: string description: >- A list of parameters for jobs with Python - tasks, for example `"python_params": - ["john doe", "35"]`. The parameters are + tasks, for example ``"python_params": + ["john doe", "35"]``. The parameters are passed to Python file as command-line - parameters. If specified upon `run-now`, + parameters. If specified upon ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ - **Deprecation note** Use [job parameters] + **Deprecation note** Use `job parameters + `__ to pass information down to tasks. Important These parameters accept only Latin characters (ASCII character set). Using non-ASCII characters returns an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, - and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + and emojis. spark_submit_params: type: array items: @@ -7828,41 +11116,41 @@ paths: description: >- A list of parameters for jobs with spark submit task, for example - `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The - parameters are passed to spark-submit + ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. + The parameters are passed to spark-submit script as command-line parameters. If - specified upon `run-now`, it would + specified upon ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ - **Deprecation note** Use [job parameters] + **Deprecation note** Use `job parameters + `__ to pass information down to tasks. Important These parameters accept only Latin characters (ASCII character set). Using non-ASCII characters returns an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, - and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + and emojis. sql_params: type: object description: >- A map from keys to values for jobs with - SQL task, for example `"sql_params": - {"name": "john doe", "age": "35"}`. The + SQL task, for example ``"sql_params": + {"name": "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ **Deprecation note** Use - [job parameters] to pass information down - to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. required: - job_id description: >- The task triggers another job when the - `run_job_task` field is present. + ``run_job_task`` field is present. spark_jar_task: type: object properties: @@ -7875,7 +11163,7 @@ paths: main method to be executed. This class must be contained in a JAR provided as a library. The code must use - `SparkContext.getOrCreate` to obtain a + ``SparkContext.getOrCreate`` to obtain a Spark context; otherwise, runs of the job fail. parameters: @@ -7884,18 +11172,18 @@ paths: type: string description: >- Parameters passed to the main method. Use - [Task parameter variables] to set - parameters containing information about - job runs. [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + `Task parameter variables + `__ + to set parameters containing information + about job runs. run_as_repl: type: boolean description: >- - Deprecated. A value of `false` is no + Deprecated. A value of ``false`` is no longer supported. description: >- - The task runs a JAR when the `spark_jar_task` - field is present. + The task runs a JAR when the + ``spark_jar_task`` field is present. spark_python_task: type: object properties: @@ -7907,11 +11195,10 @@ paths: type: string description: >- Command line parameters passed to the - Python file. Use [Task parameter - variables] to set parameters containing - information about job runs. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Python file. Use `Task parameter variables + `__ + to set parameters containing information + about job runs. source: type: string x-enum: @@ -7919,22 +11206,22 @@ paths: - WORKSPACE description: >- Optional location type of the Python file. - When set to `WORKSPACE` or not specified, - the file will be retrieved from the local - Databricks workspace or cloud location (if - the `python_file` has a URI format). When - set to `GIT`, the Python file will be - retrieved from a Git repository defined in - `git_source`. * `WORKSPACE`: The Python - file is located in a Databricks workspace - or at a cloud filesystem URI. * `GIT`: The - Python file is located in a remote Git - repository. + When set to ``WORKSPACE`` or not + specified, the file will be retrieved from + the local Databricks workspace or cloud + location (if the ``python_file`` has a URI + format). When set to ``GIT``, the Python + file will be retrieved from a Git + repository defined in ``git_source``. - + ``WORKSPACE``: The Python file is located + in a Databricks workspace or at a cloud + filesystem URI. - ``GIT``: The Python file + is located in a remote Git repository. required: - python_file description: >- The task runs a Python file when the - `spark_python_task` field is present. + ``spark_python_task`` field is present. spark_submit_task: type: object properties: @@ -8039,17 +11326,18 @@ paths: - WORKSPACE description: >- Optional location type of the SQL file. - When set to `WORKSPACE`, the SQL file + When set to ``WORKSPACE``, the SQL file will be retrieved from the local - Databricks workspace. When set to `GIT`, - the SQL file will be retrieved from a - Git repository defined in `git_source`. - If the value is empty, the task will use - `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: - SQL file is located in Databricks - workspace. * `GIT`: SQL file is located - in cloud Git provider. + Databricks workspace. When set to + ``GIT``, the SQL file will be retrieved + from a Git repository defined in + ``git_source``. If the value is empty, + the task will use ``GIT`` if + ``git_source`` is defined and + ``WORKSPACE`` otherwise. - + ``WORKSPACE``: SQL file is located in + Databricks workspace. - ``GIT``: SQL + file is located in cloud Git provider. required: - path description: >- @@ -8076,13 +11364,13 @@ paths: description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL - dashboard when the `sql_task` field is + dashboard when the ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of - this job task. A value of `0` means no + this job task. A value of ``0`` means no timeout. webhook_notifications: type: object @@ -8109,7 +11397,38 @@ paths: An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the - `on_failure` property. + ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification + IDs to call when platform-initiated + maintenance completes for a continuous + job. A maximum of 3 destinations can be + specified for the + ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification + IDs to call when platform-initiated + maintenance starts for a continuous job. A + maximum of 3 destinations can be specified + for the ``on_maintenance_start`` property. on_start: type: array items: @@ -8123,7 +11442,7 @@ paths: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the - `on_start` property. + ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -8138,17 +11457,18 @@ paths: IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in - the `health` field using the following - metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is + the ``health`` field using the following + metrics: ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` + property. on_success: type: array items: @@ -8162,7 +11482,7 @@ paths: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations - can be specified for the `on_success` + can be specified for the ``on_success`` property. description: >- A collection of system notification IDs to @@ -8186,12 +11506,30 @@ paths: - task description: >- The task executes a nested task for every input - provided when the `for_each_task` field is present. + provided when the ``for_each_task`` field is present. gen_ai_compute_task: type: object properties: dl_runtime_image: type: string + description: Runtime image + client_version: + type: string + description: >- + Version of the client (e.g., sgcli wheel) that + submitted this task. Used by handlers to gate + behavior or reject incompatible versions. + code_source_tar_path: + type: string + description: >- + Optional path to a tarball containing the user's + workspace contents. When set, the entry script + extracts the tarball into the working directory + before running the training script, so the + training script can import sibling modules and + read packaged data files. Must be a workspace path + (e.g. ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. command: type: string description: >- @@ -8210,6 +11548,13 @@ paths: description: GPU type. required: - num_gpus + docker_image_url: + type: string + description: >- + Optional custom Docker container image URL for + running the training script. Format: + organization/repository:tag (e.g., + "pytorch/pytorch:2.0.1") mlflow_experiment_name: type: string description: >- @@ -8217,6 +11562,24 @@ paths: experiment to log the run to. If name is not found, backend will create the mlflow experiment using the name. + mlflow_run_name: + type: string + description: >- + Optional name to assign to the MLflow run created + for this task. If unset, MLflow auto-generates a + name. Used alongside ``mlflow_experiment_name`` to + identify the run in the MLflow UI. + requirements_yaml_path: + type: string + description: >- + Optional path to a requirements.yaml file + describing pip dependencies to install before + running the training script. Consumed by the entry + script; format matches the runtime + requirements.yaml convention used by sgcli. Must + be a workspace path (e.g. + ``/Workspace/Users/...``) or volume; ``dbfs:/`` is + not supported. source: type: string x-enum: @@ -8224,15 +11587,15 @@ paths: - WORKSPACE description: >- Optional location type of the training script. - When set to `WORKSPACE`, the script will be + When set to ``WORKSPACE``, the script will be retrieved from the local Databricks workspace. - When set to `GIT`, the script will be retrieved - from a Git repository defined in `git_source`. If - the value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` otherwise. - * `WORKSPACE`: Script is located in Databricks - workspace. * `GIT`: Script is located in cloud Git - provider. + When set to ``GIT``, the script will be retrieved + from a Git repository defined in ``git_source``. + If the value is empty, the task will use ``GIT`` + if ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Script is located in + Databricks workspace. - ``GIT``: Script is located + in cloud Git provider. training_script_path: type: string description: >- @@ -8240,9 +11603,9 @@ paths: Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the Databricks workspace, - the path must be absolute and begin with `/`. For - files stored in a remote repository, the path must - be relative. This field is required. + the path must be absolute and begin with ``/``. + For files stored in a remote repository, the path + must be relative. This field is required. yaml_parameters: type: string description: >- @@ -8257,6 +11620,23 @@ paths: parameters passed to the training script. required: - dl_runtime_image + description: >- + DEPRECATED — use ``AiRuntimeTask`` for all new BYOT + multi-node GPU workloads (see + ai_runtime_task.proto). ``AiRuntimeTask`` is the only supported BYOT task type for new + workloads; this proto is retained only for AIR CLI (fka SGCLI) pywheel backwards compatibility + and will be removed once the pywheel → databricks-cli migration completes (post- PuPr). + genie_task: + type: object + properties: + configuration_id: + type: string + description: >- + Required. Resource name of the agent task + configuration to run. + required: + - configuration_id + description: Runs a Genie or Genie Code agent task. health: type: object properties: @@ -8278,30 +11658,25 @@ paths: evaluated for a particular health rule. - * `RUN_DURATION_SECONDS`: Expected total - time for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all streams. - This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate of - the maximum offset lag - - across all streams. This metric is in Public - Preview. * `STREAMING_BACKLOG_SECONDS`: An - estimate + - ``RUN_DURATION_SECONDS``: Expected total + time for a run in seconds. + - ``STREAMING_BACKLOG_BYTES``: An estimate + of the maximum bytes of data waiting to be + consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate + of the maximum offset lag across all + streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of the maximum consumer delay across all - streams. This metric is in Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of - the maximum number of outstanding files + streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate + of the maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. This metric is in Public Preview. op: type: string x-enum: @@ -8333,10 +11708,10 @@ paths: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be - unsuccessful if it completes with the `FAILED` - result_state or `INTERNAL_ERROR` `life_cycle_state`. - The value `-1` means to retry indefinitely and the - value `0` means to never retry. + unsuccessful if it completes with the ``FAILED`` + result_state or ``INTERNAL_ERROR`` + ``life_cycle_state``. The value ``-1`` means to retry + indefinitely and the value ``0`` means to never retry. min_retry_interval_millis: type: integer description: >- @@ -8361,20 +11736,20 @@ paths: job. If the run is initiated by a call to :method:jobs/run Now with parameters specified, the two parameters maps are merged. If the same - key is specified in `base_parameters` and in - `run-now`, the value from `run-now` is used. Use - [Task parameter variables] to set parameters - containing information about job runs. If the - notebook takes a parameter that is not specified - in the job’s `base_parameters` or the `run-now` - override parameters, the default value from the - notebook is used. Retrieve these parameters in a - notebook using [dbutils.widgets.get]. The JSON - representation of this field cannot exceed 1MB. - [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-widgets + key is specified in ``base_parameters`` and in + ``run-now``, the value from ``run-now`` is used. + Use `Task parameter variables + `__ + to set parameters containing information about job + runs. If the notebook takes a parameter that is + not specified in the job’s ``base_parameters`` or + the ``run-now`` override parameters, the default + value from the notebook is used. Retrieve these + parameters in a notebook using + `dbutils.widgets.get + `__. + The JSON representation of this field cannot + exceed 1MB. source: type: string x-enum: @@ -8382,19 +11757,19 @@ paths: - WORKSPACE description: >- Optional location type of the notebook. When set - to `WORKSPACE`, the notebook will be retrieved + to ``WORKSPACE``, the notebook will be retrieved from the local Databricks workspace. When set to - `GIT`, the notebook will be retrieved from a Git - repository defined in `git_source`. If the value - is empty, the task will use `GIT` if `git_source` - is defined and `WORKSPACE` otherwise. * - `WORKSPACE`: Notebook is located in Databricks - workspace. * `GIT`: Notebook is located in cloud - Git provider. + ``GIT``, the notebook will be retrieved from a Git + repository defined in ``git_source``. If the value + is empty, the task will use ``GIT`` if + ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Notebook is located in + Databricks workspace. - ``GIT``: Notebook is + located in cloud Git provider. warehouse_id: type: string description: >- - Optional `warehouse_id` to run the notebook on a + Optional ``warehouse_id`` to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses. Note that SQL warehouses only support @@ -8403,7 +11778,7 @@ paths: required: - notebook_path description: >- - The task runs a notebook when the `notebook_task` + The task runs a notebook when the ``notebook_task`` field is present. notification_settings: type: object @@ -8414,17 +11789,18 @@ paths: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is canceled. + specified in ``on_failure`` if the run is + canceled. no_alert_for_skipped_runs: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is skipped. + specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` for - this task run. + ``email_notifications`` and ``webhook_notifications`` + for this task run. pipeline_task: type: object properties: @@ -8433,19 +11809,115 @@ paths: full_refresh: type: boolean description: >- - If true, triggers a full refresh on the delta live - table. + If true, triggers a full refresh on the spark + declarative pipeline. + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + parameters: + type: object + description: >- + Key/value-map of parameters passed to the pipeline + execution. Limited to 10k characters in total. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are + unioned with other selective refresh options + (refresh_selection, full_refresh_selection) to + determine the final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: A list of tables to update without fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints + without clearing data. required: - pipeline_id description: >- The task triggers a pipeline update when the - `pipeline_task` field is present. Only pipelines + ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: type: object properties: connection_resource_name: type: string + incremental_refresh_config: + type: object + properties: + archive_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the archive window. Default: + MONTH + archive_window_periods: + type: integer + description: >- + Number of periods in the archive window (total + data retained). Must be greater than the + refresh window when using the same + granularity. Default: 3 (with YEAR granularity + = 3 years) + detect_data_changes: + type: boolean + description: >- + If true, Power BI will check whether data has + changed in each partition before reimporting. + Partitions with unchanged data are skipped, + reducing refresh cost further. Uses + MAX(datetime_column) as the change detector. + mode: + type: string + x-enum: + - REFRESH_POLICY_MODE_HYBRID + - REFRESH_POLICY_MODE_IMPORT_ONLY + description: >- + The refresh policy mode. Hybrid adds a + real-time DirectQuery partition for the most + recent data; Import uses only batch-imported + partitions. Default: HYBRID + only_refresh_complete_periods: + type: boolean + description: >- + If true, only refresh complete periods (e.g., + skip today's partial data). + refresh_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the refresh window. Default: + DAY + refresh_window_periods: + type: integer + description: >- + Number of periods in the rolling refresh + window. Partitions within this window are + re-imported on each refresh. Default: 1 + description: >- + Incremental refresh policy applied to all IMPORT + mode tables in the model. Windows and mode are + shared; partition columns are set per-table on + PowerBiTable. power_bi_model: type: object properties: @@ -8539,6 +12011,14 @@ paths: properties: catalog: type: string + incremental_refresh_datetime_column: + type: string + description: >- + The datetime column used for incremental + refresh partitioning on this table. e.g., + "order_date", "updated_at" Only applicable + when the task has incremental_refresh_config + set and this table uses IMPORT storage mode. name: type: string description: The table name in Databricks @@ -8552,6 +12032,24 @@ paths: - DUAL - IMPORT description: The Power BI storage mode of the table + table_type: + type: string + x-enum: + - EXTERNAL + - EXTERNAL_SHALLOW_CLONE + - FOREIGN + - MANAGED + - MANAGED_SHALLOW_CLONE + - MATERIALIZED_VIEW + - METRIC_VIEW + - STREAMING_TABLE + - VIEW + description: >- + The Unity Catalog table type of this table. + When set, the server trusts it to classify + the table during metric-view filtering and + skips the server-side lookup. When unset, + the server resolves the type on its own. description: The tables to be exported to Power BI warehouse_id: type: string @@ -8560,7 +12058,25 @@ paths: source description: >- The task triggers a Power BI semantic model update - when the `power_bi_task` field is present. + when the ``power_bi_task`` field is present. + python_operator_task: + type: object + properties: + main: + type: string + parameters: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: >- + An ordered list of task parameters. + TODO(JOBS-30885): Add limits for parameters. + description: The task runs a Python operator task. python_wheel_task: type: object properties: @@ -8572,28 +12088,28 @@ paths: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using - `$packageName.$entryPoint()` + ``$packageName.$entryPoint()`` named_parameters: type: object description: >- Command-line parameters passed to Python wheel - task in the form of `["--name=task", - "--data=dbfs:/path/to/data.json"]`. Leave it empty - if `parameters` is not null. + task in the form of ``["--name=task", + "--data=dbfs:/path/to/data.json"]``. Leave it + empty if ``parameters`` is not null. parameters: type: array items: type: string description: >- Command-line parameters passed to Python wheel - task. Leave it empty if `named_parameters` is not - null. + task. Leave it empty if ``named_parameters`` is + not null. required: - package_name - entry_point description: >- The task runs a Python wheel when the - `python_wheel_task` field is present. + ``python_wheel_task`` field is present. retry_on_timeout: type: boolean description: >- @@ -8613,8 +12129,8 @@ paths: An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. When omitted, - defaults to `ALL_SUCCESS`. See :method:jobs/create for - a list of possible values. + defaults to ``ALL_SUCCESS``. See :method:jobs/create + for a list of possible values. run_job_task: type: object properties: @@ -8626,30 +12142,29 @@ paths: type: string description: >- An array of commands to execute for jobs with the - dbt task, for example `"dbt_commands": ["dbt + dbt task, for example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt deps", "dbt seed", "dbt - run"]` ⚠ **Deprecation note** Use [job parameters] - to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + run"]`` ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR - tasks, for example `"jar_params": ["john doe", - "35"]`. The parameters are used to invoke the main - function of the main class specified in the Spark - JAR task. If not specified upon `run-now`, it - defaults to an empty list. jar_params cannot be + tasks, for example ``"jar_params": ["john doe", + "35"]``. The parameters are used to invoke the + main function of the main class specified in the + Spark JAR task. If not specified upon ``run-now``, + it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed - 10,000 bytes. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) cannot + exceed 10,000 bytes. ⚠ **Deprecation note** Use + `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: Job-level parameters used to trigger the job. @@ -8657,27 +12172,54 @@ paths: type: object description: >- A map from keys to values for jobs with notebook - task, for example `"notebook_params": {"name": - "john doe", "age": "35"}`. The map is passed to + task, for example ``"notebook_params": {"name": + "john doe", "age": "35"}``. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. If not specified - upon `run-now`, the triggered run uses the job’s - base parameters. notebook_params cannot be - specified in conjunction with jar_params. ⚠ - **Deprecation note** Use [job parameters] to pass - information down to tasks. The JSON representation - of this field (for example - `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 bytes. - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, the + triggered run uses the job’s base parameters. + notebook_params cannot be specified in conjunction + with jar_params. ⚠ **Deprecation note** Use `job + parameters + `__ + to pass information down to tasks. The JSON + representation of this field (for example + ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 bytes. pipeline_params: type: object properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are + unioned with other selective refresh options + (refresh_selection, full_refresh_selection) to + determine the final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints + without clearing data. description: >- Controls whether the pipeline should perform a full refresh @@ -8689,58 +12231,61 @@ paths: type: string description: >- A list of parameters for jobs with Python tasks, - for example `"python_params": ["john doe", "35"]`. - The parameters are passed to Python file as - command-line parameters. If specified upon - `run-now`, it would overwrite the parameters + for example ``"python_params": ["john doe", + "35"]``. The parameters are passed to Python file + as command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use - [job parameters] to pass information down to - tasks. Important These parameters accept only - Latin characters (ASCII character set). Using - non-ASCII characters returns an error. Examples of - invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII + character set). Using non-ASCII characters returns + an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and + emojis. spark_submit_params: type: array items: type: string description: >- A list of parameters for jobs with spark submit - task, for example `"spark_submit_params": - ["--class", "org.apache.spark.examples.SparkPi"]`. - The parameters are passed to spark-submit script - as command-line parameters. If specified upon - `run-now`, it would overwrite the parameters + task, for example ``"spark_submit_params": + ["--class", + "org.apache.spark.examples.SparkPi"]``. The + parameters are passed to spark-submit script as + command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use - [job parameters] to pass information down to - tasks. Important These parameters accept only - Latin characters (ASCII character set). Using - non-ASCII characters returns an error. Examples of - invalid, non-ASCII characters are Chinese, - Japanese kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. Important These + parameters accept only Latin characters (ASCII + character set). Using non-ASCII characters returns + an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and + emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL task, - for example `"sql_params": {"name": "john doe", - "age": "35"}`. The SQL alert task does not support - custom parameters. ⚠ **Deprecation note** Use [job - parameters] to pass information down to tasks. - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + for example ``"sql_params": {"name": "john doe", + "age": "35"}``. The SQL alert task does not + support custom parameters. ⚠ **Deprecation note** + Use `job parameters + `__ + to pass information down to tasks. required: - job_id description: >- - The task triggers another job when the `run_job_task` - field is present. + The task triggers another job when the + ``run_job_task`` field is present. spark_jar_task: type: object properties: @@ -8752,26 +12297,26 @@ paths: The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library. The code - must use `SparkContext.getOrCreate` to obtain a + must use ``SparkContext.getOrCreate`` to obtain a Spark context; otherwise, runs of the job fail. parameters: type: array items: type: string description: >- - Parameters passed to the main method. Use [Task - parameter variables] to set parameters containing - information about job runs. [Task parameter - variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Parameters passed to the main method. Use `Task + parameter variables + `__ + to set parameters containing information about job + runs. run_as_repl: type: boolean description: >- - Deprecated. A value of `false` is no longer + Deprecated. A value of ``false`` is no longer supported. description: >- - The task runs a JAR when the `spark_jar_task` field is - present. + The task runs a JAR when the ``spark_jar_task`` field + is present. spark_python_task: type: object properties: @@ -8783,10 +12328,10 @@ paths: type: string description: >- Command line parameters passed to the Python file. - Use [Task parameter variables] to set parameters - containing information about job runs. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Use `Task parameter variables + `__ + to set parameters containing information about job + runs. source: type: string x-enum: @@ -8794,20 +12339,21 @@ paths: - WORKSPACE description: >- Optional location type of the Python file. When - set to `WORKSPACE` or not specified, the file will - be retrieved from the local Databricks workspace - or cloud location (if the `python_file` has a URI - format). When set to `GIT`, the Python file will - be retrieved from a Git repository defined in - `git_source`. * `WORKSPACE`: The Python file is - located in a Databricks workspace or at a cloud - filesystem URI. * `GIT`: The Python file is - located in a remote Git repository. + set to ``WORKSPACE`` or not specified, the file + will be retrieved from the local Databricks + workspace or cloud location (if the + ``python_file`` has a URI format). When set to + ``GIT``, the Python file will be retrieved from a + Git repository defined in ``git_source``. - + ``WORKSPACE``: The Python file is located in a + Databricks workspace or at a cloud filesystem URI. + - ``GIT``: The Python file is located in a remote + Git repository. required: - python_file description: >- The task runs a Python file when the - `spark_python_task` field is present. + ``spark_python_task`` field is present. spark_submit_task: type: object properties: @@ -8910,15 +12456,16 @@ paths: - WORKSPACE description: >- Optional location type of the SQL file. When - set to `WORKSPACE`, the SQL file will be + set to ``WORKSPACE``, the SQL file will be retrieved from the local Databricks workspace. - When set to `GIT`, the SQL file will be + When set to ``GIT``, the SQL file will be retrieved from a Git repository defined in - `git_source`. If the value is empty, the task - will use `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: SQL file - is located in Databricks workspace. * `GIT`: - SQL file is located in cloud Git provider. + ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - + ``WORKSPACE``: SQL file is located in + Databricks workspace. - ``GIT``: SQL file is + located in cloud Git provider. required: - path description: >- @@ -8945,12 +12492,12 @@ paths: description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the - `sql_task` field is present. + ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job - task. A value of `0` means no timeout. + task. A value of ``0`` means no timeout. webhook_notifications: type: object properties: @@ -8975,7 +12522,37 @@ paths: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 - destinations can be specified for the `on_failure` + destinations can be specified for the + ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance completes + for a continuous job. A maximum of 3 destinations + can be specified for the + ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance starts + for a continuous job. A maximum of 3 destinations + can be specified for the ``on_maintenance_start`` property. on_start: type: array @@ -8989,7 +12566,7 @@ paths: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 - destinations can be specified for the `on_start` + destinations can be specified for the ``on_start`` property. on_streaming_backlog_exceeded: type: array @@ -9004,16 +12581,17 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog - thresholds can be set in the `health` field using - the following metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on + thresholds can be set in the ``health`` field + using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be - specified for the `on_streaming_backlog_exceeded` - property. + specified for the + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -9027,7 +12605,7 @@ paths: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the - `on_success` property. + ``on_success`` property. description: >- A collection of system notification IDs to notify when the run begins or completes. The default behavior is @@ -9039,7 +12617,7 @@ paths: type: integer description: >- An optional timeout applied to each run of this job. A value - of `0` means no timeout. + of ``0`` means no timeout. usage_policy_id: type: string description: >- @@ -9070,7 +12648,35 @@ paths: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be - specified for the `on_failure` property. + specified for the ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call when + platform-initiated maintenance completes for a + continuous job. A maximum of 3 destinations can be + specified for the ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call when + platform-initiated maintenance starts for a continuous + job. A maximum of 3 destinations can be specified for + the ``on_maintenance_start`` property. on_start: type: array items: @@ -9083,7 +12689,7 @@ paths: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be - specified for the `on_start` property. + specified for the ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -9097,14 +12703,15 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the - `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the + ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -9117,7 +12724,7 @@ paths: description: >- An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 - destinations can be specified for the `on_success` + destinations can be specified for the ``on_success`` property. description: >- A collection of system notification IDs to notify when the @@ -9146,12 +12753,12 @@ paths: operationId: jobs_update summary: >- Add, update, or remove specific settings of an existing job. Use the - [_Reset_ + [*Reset* tags: - jobs description: >- Add, update, or remove specific settings of an existing job. Use the - [_Reset_ + [*Reset* endpoint](:method:jobs/reset) to overwrite all job settings. @@ -9160,17 +12767,17 @@ paths: The canonical identifier of the job to update. This field is required. :param fields_to_remove: List[str] (optional) Remove top-level fields in the job settings. Removing nested fields is not supported, except for - tasks and job clusters (`tasks/task_1`). This field is optional. + tasks and job clusters (``tasks/task_1``). This field is optional. :param new_settings: :class:`JobSettings` (optional) The new settings for the job. - Top-level fields specified in `new_settings` are completely replaced, except for arrays which are + Top-level fields specified in ``new_settings`` are completely replaced, except for arrays which are merged. That is, new and existing entries are completely replaced based on the respective key - fields, i.e. `task_key` or `job_cluster_key`, while previous entries are kept. + fields, i.e. ``task_key`` or ``job_cluster_key``, while previous entries are kept. Partially updating nested fields is not supported. - Changes to the field `JobSettings.timeout_seconds` are applied to active runs. Changes to other + Changes to the field ``JobSettings.timeout_seconds`` are applied to active runs. Changes to other fields are applied to future runs only. requestBody: content: @@ -9182,80 +12789,71 @@ paths: type: integer description: >- The canonical identifier of the job to update. This field is - required. - fields_to_remove: - type: array - items: - type: string - description: >- - Remove top-level fields in the job settings. Removing nested - fields is not supported, except for tasks and job clusters - (`tasks/task_1`). This field is optional. - new_settings: - type: object - properties: - budget_policy_id: - type: string - continuous: - type: object - properties: - pause_status: - type: string - x-enum: - - PAUSED - - UNPAUSED - description: >- - Create a collection of name/value pairs. - - - Example enumeration: - - - >>> class Color(Enum): - - ... RED = 1 - - ... BLUE = 2 - - ... GREEN = 3 - - - Access them by: - - - - attribute access: - - >>> Color.RED - - - - value lookup: - - >>> Color(1) - - - - name lookup: - - >>> Color['RED'] - - - Enumerations can be iterated over, and know how many - members they have: - - - >>> len(Color) - - 3 - - - >>> list(Color) - - [, , ] - - - Methods can be added to enumerations, and members - can have their own - - attributes -- see the documentation for details. + required. + fields_to_remove: + type: array + items: + type: string + description: >- + Remove top-level fields in the job settings. Removing nested + fields is not supported, except for tasks and job clusters + (``tasks/task_1``). This field is optional. + new_settings: + type: object + properties: + budget_policy_id: + type: string + continuous: + type: object + properties: + maintenance_window: + type: object + properties: + start_hour: + type: integer + description: >- + An integer between 0 and 23 denoting the start + hour for the maintenance window in the 24-hour + day. Platform-initiated maintenance is triggered + only within a one-hour window starting at this + hour. This field is required. + day_of_week: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: >- + The day of week on which maintenance is allowed + to happen. This field is required. + timezone_id: + type: string + description: >- + A Java timezone ID. The maintenance window is + resolved with respect to this timezone. See + `Java TimeZone + `__ + for details. This field is required. + required: + - start_hour + - day_of_week + - timezone_id + description: >- + A recurring weekly time window during which + platform-initiated maintenance is allowed to run for + a continuous job. + pause_status: + type: string + x-enum: + - PAUSED + - UNPAUSED + description: >- + Indicate whether the continuous execution of the job + is paused or not. Defaults to UNPAUSED. task_retry_mode: type: string x-enum: @@ -9267,8 +12865,8 @@ paths: description: >- An optional continuous property for this job. The continuous property will ensure that there is always one - run executing. Only one of `schedule` and `continuous` - can be used. + run executing. Only one of ``schedule`` and + ``continuous`` can be used. deployment: type: object properties: @@ -9278,13 +12876,28 @@ paths: - BUNDLE - SYSTEM_MANAGED description: >- - * `BUNDLE`: The job is managed by Databricks Asset - Bundle. * `SYSTEM_MANAGED`: The job is + - ``BUNDLE``: The job is managed by Databricks Asset + Bundle. - managed by Databricks and is read-only. + - ``SYSTEM_MANAGED``: The job is managed by + Databricks and is read-only. + deployment_id: + type: string + description: >- + ID of the deployment that manages this job. Only set + when ``kind`` is ``BUNDLE``. Used to look up + deployment metadata from the Deployment Metadata + service. metadata_file_path: type: string description: Path of the file that contains deployment metadata. + version_id: + type: string + description: >- + ID of the version of the deployment that produced + this job. Only set when ``kind`` is ``BUNDLE``. + Identifies a specific snapshot of the deployment in + the Deployment Metadata service. required: - kind description: >- @@ -9301,8 +12914,8 @@ paths: - EDITABLE - UI_LOCKED description: >- - Edit mode of the job. * `UI_LOCKED`: The job is in a - locked UI state and cannot be modified. * `EDITABLE`: + Edit mode of the job. - ``UI_LOCKED``: The job is in a + locked UI state and cannot be modified. - ``EDITABLE``: The job is in an editable state and can be modified. email_notifications: type: object @@ -9316,10 +12929,10 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold specified - for the `RUN_DURATION_SECONDS` metric in the - `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified in the - `health` field for the job, notifications are not + for the ``RUN_DURATION_SECONDS`` metric in the + ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is specified in the + ``health`` field for the job, notifications are not sent. on_failure: type: array @@ -9329,10 +12942,26 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an - `INTERNAL_ERROR` `life_cycle_state` or a `FAILED`, - or `TIMED_OUT` result_state. If this is not - specified on job creation, reset, or update the list - is empty, and notifications are not sent. + ``INTERNAL_ERROR`` ``life_cycle_state`` or a + ``FAILED``, or ``TIMED_OUT`` result_state. If this + is not specified on job creation, reset, or update + the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a + continuous job. on_start: type: array items: @@ -9350,12 +12979,12 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in - the `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the - 10-minute average of these metrics. If the issue + the ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on + the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. on_success: type: array @@ -9365,14 +12994,100 @@ paths: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a - `TERMINATED` `life_cycle_state` and a `SUCCESS` - result_state. If not specified on job creation, - reset, or update, the list is empty, and + ``TERMINATED`` ``life_cycle_state`` and a + ``SUCCESS`` result_state. If not specified on job + creation, reset, or update, the list is empty, and notifications are not sent. description: >- An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted. + environment_variables: + type: array + items: + type: object + properties: + environment_variables_key: + type: string + description: >- + Identifier for this entry. Must be unique within + ``JobSettings.environment_variables``. Tasks + reference it from + ``TaskSettings.environment_variables_key``. + files: + type: array + items: + type: string + description: >- + Workspace (``/Workspace/...``) or UC Volumes + (``/Volumes/...``) paths to ``.env`` files. + Maximum 5 files. Files are read, parsed, and + merged at task execution time, not at job creation + or update API call time. File format: each line + must be exactly ``KEY=VALUE``. Keys must match the + same regex as inlined variable names + (``^[A-Za-z_][A-Za-z0-9_]*$``); the value + continues to the end of the line. No other syntax + is supported — no comments, no quoted values, no + escape sequences, no variable interpolation. Any + line that does not match the ``KEY=VALUE`` shape + fails the run. Size limits: maximum 1,048,576 + bytes (1 MiB) per file on disk; maximum 131,072 + bytes (128 KiB) per ``KEY=VALUE`` line combined. + Caps are enforced at read time in jobs-runner — + files exceeding the per-file cap, or lines + exceeding the per-line cap, fail the run. On a + duplicate key, the later file wins; ``variables`` + override values from any file. Values may contain + {{secrets/scope/key}} references; those are + resolved at task execution time and never + persisted in resolved form. Do not use these files + to store raw secret values; consult `secret + management + `__ + for the right way to pass sensitive values. + variables: + type: object + description: >- + Environment variables specified directly as + key/value pairs (as opposed to ``files``, which + are read from ``.env`` file paths). Maximum 100 + entries. Each key must match + ``^[A-Za-z_][A-Za-z0-9_]*$`` and be 1 to 256 + characters long. Each value is up to 512 + characters; larger values should be moved into a + ``.env`` file referenced from ``files``. On a + duplicate key, ``variables`` override values from + any file in ``files``. Values may contain + {{secrets/scope/key}} references; those are + resolved at task execution time and never + persisted in resolved form. Do not use this field + to store a raw secret value; consult `secret + management + `__ + for the right way to pass sensitive values. + description: >- + A named environment-variable entry, defined once at + the job level and referenced by key from one + or more tasks. Entries live on ``JobSettings.environment_variables``, and tasks select one via + ``TaskSettings.environment_variables_key``. + + Precedence within a single entry (later overrides earlier on duplicate keys): + + files (merged in submitted order; later file in the list wins) └─▶ variables (inline k/v + override file contents) └─▶ runtime (os.environ / sys.env — overrides all, + classic-parity) + + Resolved values are never persisted: ``{{secrets/scope/key}}`` references in ``variables`` or in + ``files`` contents are stored verbatim and resolved at task execution time on the DP . + description: >- + Named environment-variable entries that tasks can + reference by key from + ``TaskSettings.environment_variables_key``. Each entry + holds inline ``variables`` plus optional ``.env`` + ``files``. Maximum 10 entries per job. Entries are + independent of one another — there is no cross-entry + merging. environments: type: array items: @@ -9401,7 +13116,8 @@ paths: description: >- Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the - Jobs API 2.1 this value is always set to `"MULTI_TASK"`. + Jobs API 2.1 this value is always set to + ``"MULTI_TASK"``. git_source: type: object properties: @@ -9476,11 +13192,11 @@ paths: description: >- Dirty state indicates the job is not fully synced with the job specification in the remote - repository. Possible values are: * `NOT_SYNCED`: - The job is not yet synced with the remote job - specification. Import the remote job - specification from UI to make the job fully - synced. * `DISCONNECTED`: The job is temporary + repository. Possible values are: - + ``NOT_SYNCED``: The job is not yet synced with + the remote job specification. Import the remote + job specification from UI to make the job fully + synced. - ``DISCONNECTED``: The job is temporary disconnected from the remote job specification and is allowed for live edit. Import the remote job specification again from UI to make the job @@ -9505,13 +13221,14 @@ paths: An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, - dbt, Python script, and SQL File tasks. If `git_source` - is set, these tasks retrieve the file from the remote - repository by default. However, this behavior can be - overridden by setting `source` to `WORKSPACE` on the - task. Note: dbt and SQL File tasks support only - version-controlled sources. If dbt or SQL File tasks are - used, `git_source` must be defined on the job. + dbt, Python script, and SQL File tasks. If + ``git_source`` is set, these tasks retrieve the file + from the remote repository by default. However, this + behavior can be overridden by setting ``source`` to + ``WORKSPACE`` on the task. Note: dbt and SQL File tasks + support only version-controlled sources. If dbt or SQL + File tasks are used, ``git_source`` must be defined on + the job. health: type: object properties: @@ -9533,29 +13250,25 @@ paths: evaluated for a particular health rule. - * `RUN_DURATION_SECONDS`: Expected total time - for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all streams. - This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate of - the maximum offset lag - - across all streams. This metric is in Public - Preview. * `STREAMING_BACKLOG_SECONDS`: An - estimate + - ``RUN_DURATION_SECONDS``: Expected total + time for a run in seconds. + - ``STREAMING_BACKLOG_BYTES``: An estimate of + the maximum bytes of data waiting to be + consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate + of the maximum offset lag across all streams. + This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of the maximum consumer delay across all - streams. This metric is in Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of the - maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate of + the maximum number of outstanding files across + all + streams. This metric is in Public Preview. op: type: string x-enum: @@ -9588,6 +13301,14 @@ paths: description: >- If new_cluster, a description of a cluster that is created for each task. + serverless_compute_id: + type: string + description: >- + The ID of the serverless compute object to bind + this cluster to. At most one JobCluster per job + may set this field; the rate limit defined on the + referenced serverless compute applies across all + tasks bound to this cluster. required: - job_cluster_key - new_cluster @@ -9612,7 +13333,7 @@ paths: kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value - to `0` causes all new runs to be skipped. + to ``0`` causes all new runs to be skipped. name: type: string description: >- @@ -9627,12 +13348,12 @@ paths: type: boolean description: >- If true, do not send notifications to recipients - specified in `on_failure` if the run is skipped. + specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` for - this job. + ``email_notifications`` and ``webhook_notifications`` + for this job. parameters: type: array items: @@ -9647,6 +13368,11 @@ paths: - name - default description: Job-level parameter definitions + parent_path: + type: string + description: >- + Path of the job parent folder in workspace file tree. If + absent, the job doesn't have a workspace object. performance_target: type: string x-enum: @@ -9656,9 +13382,9 @@ paths: The performance mode on a serverless job. This field determines the level of compute performance or cost-efficiency for the run. The performance target does - not apply to tasks that run on Serverless GPU compute. * - `STANDARD`: Enables cost-efficient execution of - serverless workloads. * `PERFORMANCE_OPTIMIZED`: + not apply to tasks that run on Serverless GPU compute. - + ``STANDARD``: Enables cost-efficient execution of + serverless workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. queue: @@ -9683,7 +13409,7 @@ paths: description: >- Application ID of an active service principal. Setting this field requires the - `servicePrincipal/user` role. + ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -9692,9 +13418,9 @@ paths: description: >- The user or service principal that the job runs as, if specified in the request. This field indicates the - explicit configuration of `run_as` for the job. To find - the value in all cases, explicit or implicit, use - `run_as_user_name`. + explicit configuration of ``run_as`` for the job. To + find the value in all cases, explicit or implicit, use + ``run_as_user_name``. schedule: type: object properties: @@ -9704,16 +13430,50 @@ paths: type: string description: >- A Java timezone ID. The schedule for a job is - resolved with respect to this timezone. See [Java - TimeZone] for details. This field is required. [Java - TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + resolved with respect to this timezone. See `Java + TimeZone + `__ + for details. This field is required. pause_status: type: string x-enum: - PAUSED - UNPAUSED description: Indicate whether this schedule is paused or not. + sql_condition: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to + run the condition query against. + trigger_mode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + Determines how the SQL query result is + interpreted to decide whether the condition + fires. Must be set to a recognized value when + provided. When unset on an existing serialized + configuration, the server preserves the original + semantics by interpreting it as + ``QUERY_RETURNS_ROWS``. New configurations + should set this explicitly — explicit + ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + description: >- + SQL condition that must be satisfied before a + scheduled run is triggered. The condition is + evaluated after the cron expression fires and must + return a truthy result for the run to proceed. required: - quartz_cron_expression - timezone_id @@ -9721,7 +13481,7 @@ paths: An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API - request to `runNow`. + request to ``runNow``. tags: type: object description: >- @@ -9737,11 +13497,461 @@ paths: properties: task_key: type: string + agentic_task: + type: object + properties: + context: + type: object + properties: + conversation_ids: + type: array + items: + type: string + description: >- + Optional. Conversation IDs to load as + context for the agent. + instructions: + type: array + items: + type: string + description: >- + Optional. Additional instructions to guide + the agent's behavior. + description: >- + Optional. Context input providing conversation + history and instructions. + genie_code_api: + type: object + properties: {} + description: >- + Inline Genie Code conversation driven by a + single prompt. Mutually exclusive with the + supervisor variants. + goal: + type: string + description: >- + Deprecated. Use ``input`` (field 7) instead. + Kept for backwards compatibility with existing + callers; will be removed in a future revision. + input: + type: string + description: >- + Optional. The user query / task input the + agent must accomplish. Mirrors the OpenAI + Responses API ``input`` field. Replaces the + deprecated ``goal`` field; new callers should + populate ``input`` going forward. + output_schema: + type: object + properties: + properties: + type: object + description: >- + Map of property name to property + definition. + description: >- + Optional. JSON-Schema-style declaration of the + structured output the agent should produce. + Replaces the deprecated ``task_output_schema`` + map; new callers should populate + ``output_schema`` going forward. + supervisor_agent: + type: object + properties: + agent_id: + type: string + description: >- + Required. The ID of the supervisor agent + (tile_id). + description: >- + A Supervisor Agent that orchestrates + sub-agents and tools, referenced by tile_id. + supervisor_api: + type: object + properties: + instructions: + type: string + description: >- + Optional. System instructions that guide + how the supervisor routes queries across + tools and synthesizes responses. + model: + type: string + description: >- + Required. Databricks-provided FMAPI model, + e.g. "databricks-claude-sonnet-4-5". + tools: + type: array + items: + type: object + properties: + app: + type: object + properties: + name: + type: string + description: >- + Databricks App tool config. Set when the + discriminator == "app". + catalog: + type: object + properties: + name: + type: string + description: >- + Unity Catalog catalog tool config. Set + when the discriminator == "catalog". + dashboard: + type: object + properties: + dashboard_id: + type: string + description: >- + Lakeview dashboard tool config. Set when + the discriminator == "dashboard". + description: + type: string + description: >- + Optional. Free-form description of the + tool. + genie_space: + type: object + properties: + space_id: + type: string + description: >- + Genie space tool config. Set when the + discriminator == "genie_space". + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + description: >- + Knowledge Assistant tool config. Set + when the discriminator == + "knowledge_assistant". + schema: + type: object + properties: + name: + type: string + description: >- + Unity Catalog schema tool config. Set + when the discriminator == "schema". + serving_endpoint: + type: object + properties: + name: + type: string + description: >- + Model serving endpoint tool config. Set + when the discriminator == + "serving_endpoint". + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: Supervisor Agent ID. + description: >- + Supervisor Agent tool config (a + supervisor agent used as a tool). Set + when the discriminator == + "supervisor_agent". + table: + type: object + properties: + name: + type: string + description: >- + Unity Catalog table tool config. Set + when the discriminator == "table". + tool_type: + type: string + uc_connection: + type: object + properties: + name: + type: string + description: >- + External MCP / UC connection tool + config. Set when the discriminator == + "uc_connection". + uc_function: + type: object + properties: + name: + type: string + description: >- + Unity Catalog function tool config. Set + when the discriminator == "uc_function". + uc_mcp: + type: object + properties: + name: + type: string + description: >- + UC-registered MCP service tool config. + Set when the discriminator == "uc_mcp". + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Vector Search index tool config. Set + when the discriminator == + "vector_search_index". + volume: + type: object + properties: + name: + type: string + description: >- + Unity Catalog volume tool config. Set + when the discriminator == "volume". + web_search: + type: object + properties: {} + description: >- + Web search tool config. Set when the + discriminator == "web_search". + description: >- + Tool configuration for an inlined + agentic task agent. Discriminator + selects which per-asset + config message applies (``genie_space``, ``knowledge_assistant``, ``uc_function``, + ``uc_connection``, ``uc_mcp``, ``app``, ``table``, ``vector_search_index``, ``volume``, + ``catalog``, ``schema``, ``dashboard``, ``supervisor_agent``, ``serving_endpoint``, + ``web_search``); ``description`` is a free-form top-level description; each per-asset config + carries only the asset's identifier. + + The discriminator is exposed under two interchangeable names: ``type`` mirrors the Supervisor + (Responses) API surface, and ``tool_type`` is the AIP-friendly alias for clients where ``type`` + is a reserved keyword (Terraform, etc.). Set exactly one. + description: >- + Optional. The tools the inlined agent can + use. + description: >- + Inlined Responses-API supervisor configuration + (model + instructions + tools). Mutually + exclusive with ``supervisor_agent``. + task_output_schema: + type: object + description: >- + Deprecated. Use ``output_schema`` (field 8) + instead. Kept for backwards compatibility with + existing callers; will be removed in a future + revision. + trace_destination: + type: object + properties: + catalog_name: + type: string + description: >- + Unity Catalog catalog name for the trace + table. + experiment_id: + type: string + description: >- + MLflow experiment ID where traces are + written. + schema_name: + type: string + description: >- + Unity Catalog schema name for the trace + table. + table_prefix: + type: string + description: >- + Table-name prefix under + ``.``. Trace tables are + created as + ``.._<...>``. + description: >- + Optional. Where MLflow traces produced by this + task run should be persisted. When unset, + traces follow the workspace default + destination. + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + type: object + properties: + experiment: + type: string + description: >- + MLflow experiment name for this run. If an + experiment with this name already exists under + the calling user, the run is appended to it; + otherwise a new experiment is created. To + target a specific MLflow storage location (for + example, when running as a service principal), + set ``mlflow_experiment_directory``. + deployments: + type: array + items: + type: object + properties: + command_path: + type: string + description: >- + Workspace path of the script to run on + each node in this deployment. Upload the + script to this path and supply the path + here. When the task runs, the file at + this path is run on each node; if it + fails, the task fails with its exit + code. Example script contents: Plain + Python: python train.py --epochs 10 + Multi-GPU via accelerate: accelerate + launch train.py --config config.yaml + Distributed via torchrun: torchrun + --nproc_per_node=8 train.py + compute: + type: object + properties: + accelerator_type: + type: string + x-enum: + - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 + - GPU_8xH100 + description: >- + Hardware accelerator type (for example, + ``GPU_1xA10`` or ``GPU_8xH100``). The + number of accelerators per node is + encoded in the enum value — + ``GPU_8xH100`` means 8 H100 GPUs per + node. + accelerator_count: + type: integer + description: >- + Total number of accelerators across all + nodes. Must be a positive multiple of + the per-node accelerator count encoded + in ``accelerator_type``. For example, + ``GPU_8xH100`` with ``accelerator_count: + 16`` allocates 2 nodes (8 GPUs per + node). + required: + - accelerator_type + - accelerator_count + description: >- + Compute resources allocated to each node + in this deployment. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs + on the specified container image instead + of the default Databricks client image. + Format: + ``{organization}/{repository}:{tag}`` + name: + type: string + description: >- + Optional human-readable name for this + deployment (for example, ``driver``, + ``worker``, ``param_server``). Used for + log and UI display. Distinct names are + recommended so deployments can be told + apart, but uniqueness is not enforced. + required: + - command_path + - compute + description: >- + DeploymentSpec: configuration for one + deployment within an AiRuntimeTask. Each + entry in + ``AiRuntimeTask.deployments`` describes a group of nodes that share the same command and + compute. Many single-program training algorithms use a single entry where every node runs the + same command; role-split workloads (driver + worker, parameter server, separate eval node, etc.) + use multiple entries. + description: >- + Deployment specs for this task. Exactly one + deployment is currently supported (a single + entry where every node runs the same command); + this is a current-Preview constraint. + Role-split workloads (driver + worker, + parameter server, separate eval node, etc.) + with multiple entries are the eventual intent + but not yet supported. + code_source_path: + type: string + description: >- + Workspace or UC volume path of the code-source + archive, unpacked on each node and exposed + through ``$CODE_SOURCE``. Set by first-party + tooling; not for direct callers. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom + container image. When set, the task runs on + the specified container image instead of the + default Databricks client image. Format: + ``{organization}/{repository}:{tag}`` + mlflow_experiment_directory: + type: string + description: >- + Optional workspace directory under which the + MLflow experiment named in ``experiment`` is + created. Must start with ``/Workspace``. Set + this when running as a service principal that + has no default user directory; for regular + users the experiment defaults to the user's + home directory. + mlflow_run: + type: string + description: >- + Optional display name for the MLflow run + created under ``experiment``. If omitted, + MLflow generates a default name. + parameters: + type: object + description: >- + Optional named parameters passed to each + deployment's command. Keys are parameter + names, values the corresponding arguments (for + example, ``{"epochs": "3", "dataset": + "s3://bucket/train"}``). Values may contain + dynamic references such as + ``{{job.trigger.time.iso_date}}`` or + ``{{tasks..values.}}``, which + Jobs substitutes before execution (see + ``AiRuntimeTaskResolvedValues.parameters`` in + runs.proto). + required: + - experiment + - deployments + description: >- + The task runs a multi-gpu compute workload on + Databricks AI Runtime. Specify the accelerator + type and count, the command to run, and where the + workload's code and MLflow output are stored. alert_task: type: object properties: alert_id: type: string + parameters: + type: object + description: >- + Per-run parameter overrides, keyed by + parameter name, applied onto the alert's + stored query parameters before the query is + executed. Only scalar values are supported. + Values may reference job parameters with + ``{{job.parameters.*}}``, which are resolved + before the task runs. An override whose key + does not match a stored parameter fails the + task run. Limited to 10000 characters when + serialized as JSON; keys must be 1-100 + characters and contain only letters, digits, + underscores, dashes, and periods. subscribers: type: array items: @@ -9771,11 +13981,14 @@ paths: type: string description: >- The workspace_path is the path to the alert - file in the workspace. The path: * must start - with "/Workspace" * must be a normalized path. + file in the workspace. The path: - must start + with "/Workspace" - must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert. - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends + notifications to subscribers when the + ``alert_task`` field is present. clean_rooms_notebook_task: type: object properties: @@ -9801,10 +14014,10 @@ paths: - clean_room_name - notebook_name description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. - [clean rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ + notebook when the ``clean_rooms_notebook_task`` + field is present. compute: type: object properties: @@ -9824,30 +14037,23 @@ paths: - LESS_THAN_OR_EQUAL - NOT_EQUAL description: >- - * `EQUAL_TO`, `NOT_EQUAL` operators perform - string comparison of their operands. This - means that - - `“12.0” == “12”` will evaluate to `false`. * - `GREATER_THAN`, `GREATER_THAN_OR_EQUAL`, - - `LESS_THAN`, `LESS_THAN_OR_EQUAL` operators - perform numeric comparison of their operands. - - `“12.0” >= “12”` will evaluate to `true`, - `“10.0” >= “12”` will evaluate to - - `false`. - + - ``EQUAL_TO``, ``NOT_EQUAL`` operators + perform string comparison of their operands. + This means + that ``“12.0” == “12”`` will evaluate to ``false``. + - ``GREATER_THAN``, ``GREATER_THAN_OR_EQUAL``, + ``LESS_THAN``, ``LESS_THAN_OR_EQUAL`` + operators + perform numeric comparison of their operands. ``“12.0” >= “12”`` will evaluate to + ``true``, ``“10.0” >= “12”`` will evaluate to ``false``. The boolean comparison to task values can be - implemented with operators `EQUAL_TO`, - `NOT_EQUAL`. + implemented with operators ``EQUAL_TO``, - If a task value was set to a boolean value, it - will be serialized to `“true”` or + ``NOT_EQUAL``. If a task value was set to a + boolean value, it will be serialized to - `“false”` for the comparison. + ``“true”`` or ``“false”`` for the comparison. left: type: string description: >- @@ -9867,7 +14073,7 @@ paths: description: >- The task evaluates a condition that can be used to control the execution of other tasks when the - `condition_task` field is present. The condition + ``condition_task`` field is present. The condition task does not require a cluster to execute and does not support retries or notifications. dashboard_task: @@ -9886,14 +14092,14 @@ paths: identifier as the parameter key. The parameter value format is dependent on the filter type: - For text and single-select filters, provide - a single value (e.g. `"value"`) - For date and - datetime filters, provide the value in ISO - 8601 format (e.g. `"2000-01-01T00:00:00"`) - + a single value (e.g. ``"value"``) - For date + and datetime filters, provide the value in ISO + 8601 format (e.g. ``"2000-01-01T00:00:00"``) - For multi-select filters, provide a JSON array - of values (e.g. `"[\"value1\",\"value2\"]"`) - - For range and date range filters, provide a - JSON object with `start` and `end` (e.g. - `"{\"start\":\"1\",\"end\":\"10\"}"`) + of values (e.g. ``"[\"value1\",\"value2\"]"``) + - For range and date range filters, provide a + JSON object with ``start`` and ``end`` (e.g. + ``"{\"start\":\"1\",\"end\":\"10\"}"``) subscription: type: object properties: @@ -9915,8 +14121,8 @@ paths: type: string description: >- A snapshot of the dashboard will be sent - to the user's email when the `user_name` - field is present. + to the user's email when the + ``user_name`` field is present. description: >- The list of subscribers to send the snapshot of the dashboard to. @@ -9994,7 +14200,7 @@ paths: description: >- Optional schema to write to. This parameter is only used when a warehouse_id is also - provided. If not provided, the `default` + provided. If not provided, the ``default`` schema is used. source: type: string @@ -10003,15 +14209,15 @@ paths: - WORKSPACE description: >- Optional location type of the project - directory. When set to `WORKSPACE`, the + directory. When set to ``WORKSPACE``, the project will be retrieved from the local - Databricks workspace. When set to `GIT`, the + Databricks workspace. When set to ``GIT``, the project will be retrieved from a Git - repository defined in `git_source`. If the - value is empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Project is located - in Databricks workspace. * `GIT`: Project is + repository defined in ``git_source``. If the + value is empty, the task will use ``GIT`` if + ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Project is located + in Databricks workspace. - ``GIT``: Project is located in cloud Git provider. warehouse_id: type: string @@ -10020,15 +14226,15 @@ paths: provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command - basis by using the `--profiles-dir` command + basis by using the ``--profiles-dir`` command line argument. required: - commands description: >- The task runs one or more dbt commands when the - `dbt_task` field is present. The dbt task requires - both Databricks SQL and the ability to use a - serverless or a pro SQL warehouse. + ``dbt_task`` field is present. The dbt task + requires both Databricks SQL and the ability to + use a serverless or a pro SQL warehouse. depends_on: type: array items: @@ -10048,9 +14254,10 @@ paths: An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this - task. The task will run only if the `run_if` - condition is true. The key is `task_key`, and the - value is the name assigned to the dependent task. + task. The task will run only if the ``run_if`` + condition is true. The key is ``task_key``, and + the value is the name assigned to the dependent + task. description: type: string description: An optional description for this task. @@ -10077,10 +14284,10 @@ paths: description: >- A list of email addresses to be notified when the duration of a run exceeds the threshold - specified for the `RUN_DURATION_SECONDS` - metric in the `health` field. If no rule for - the `RUN_DURATION_SECONDS` metric is specified - in the `health` field for the job, + specified for the ``RUN_DURATION_SECONDS`` + metric in the ``health`` field. If no rule for + the ``RUN_DURATION_SECONDS`` metric is + specified in the ``health`` field for the job, notifications are not sent. on_failure: type: array @@ -10090,12 +14297,28 @@ paths: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if - it ends with an `INTERNAL_ERROR` - `life_cycle_state` or a `FAILED`, or - `TIMED_OUT` result_state. If this is not + it ends with an ``INTERNAL_ERROR`` + ``life_cycle_state`` or a ``FAILED``, or + ``TIMED_OUT`` result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance completes for a + continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when + platform-initiated maintenance starts for a + continuous job. on_start: type: array items: @@ -10113,11 +14336,12 @@ paths: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can - be set in the `health` field using the - following metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based + be set in the ``health`` field using the + following metrics: + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. @@ -10129,10 +14353,11 @@ paths: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if - it ends with a `TERMINATED` `life_cycle_state` - and a `SUCCESS` result_state. If not specified - on job creation, reset, or update, the list is - empty, and notifications are not sent. + it ends with a ``TERMINATED`` + ``life_cycle_state`` and a ``SUCCESS`` + result_state. If not specified on job + creation, reset, or update, the list is empty, + and notifications are not sent. description: >- An optional set of email addresses that is notified when runs of this task begin or complete @@ -10145,6 +14370,16 @@ paths: job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry + defined in ``JobSettings.environment_variables``. + The selected entry's variables and file contents + are applied to this task at execution time. Length + and pattern mirror ``environment_key`` so the two + references look identical to customers reading + task settings. existing_cluster_id: type: string description: >- @@ -10176,13 +14411,32 @@ paths: - task description: >- The task executes a nested task for every input - provided when the `for_each_task` field is + provided when the ``for_each_task`` field is present. gen_ai_compute_task: type: object properties: dl_runtime_image: type: string + description: Runtime image + client_version: + type: string + description: >- + Version of the client (e.g., sgcli wheel) that + submitted this task. Used by handlers to gate + behavior or reject incompatible versions. + code_source_tar_path: + type: string + description: >- + Optional path to a tarball containing the + user's workspace contents. When set, the entry + script extracts the tarball into the working + directory before running the training script, + so the training script can import sibling + modules and read packaged data files. Must be + a workspace path (e.g. + ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. command: type: string description: >- @@ -10201,6 +14455,13 @@ paths: description: GPU type. required: - num_gpus + docker_image_url: + type: string + description: >- + Optional custom Docker container image URL for + running the training script. Format: + organization/repository:tag (e.g., + "pytorch/pytorch:2.0.1") mlflow_experiment_name: type: string description: >- @@ -10208,6 +14469,25 @@ paths: MLflow experiment to log the run to. If name is not found, backend will create the mlflow experiment using the name. + mlflow_run_name: + type: string + description: >- + Optional name to assign to the MLflow run + created for this task. If unset, MLflow + auto-generates a name. Used alongside + ``mlflow_experiment_name`` to identify the run + in the MLflow UI. + requirements_yaml_path: + type: string + description: >- + Optional path to a requirements.yaml file + describing pip dependencies to install before + running the training script. Consumed by the + entry script; format matches the runtime + requirements.yaml convention used by sgcli. + Must be a workspace path (e.g. + ``/Workspace/Users/...``) or volume; + ``dbfs:/`` is not supported. source: type: string x-enum: @@ -10215,15 +14495,16 @@ paths: - WORKSPACE description: >- Optional location type of the training script. - When set to `WORKSPACE`, the script will be + When set to ``WORKSPACE``, the script will be retrieved from the local Databricks workspace. - When set to `GIT`, the script will be + When set to ``GIT``, the script will be retrieved from a Git repository defined in - `git_source`. If the value is empty, the task - will use `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: Script - is located in Databricks workspace. * `GIT`: - Script is located in cloud Git provider. + ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - + ``WORKSPACE``: Script is located in Databricks + workspace. - ``GIT``: Script is located in + cloud Git provider. training_script_path: type: string description: >- @@ -10232,7 +14513,7 @@ paths: gcs:/) and workspace paths are supported. For python files stored in the Databricks workspace, the path must be absolute and begin - with `/`. For files stored in a remote + with ``/``. For files stored in a remote repository, the path must be relative. This field is required. yaml_parameters: @@ -10249,6 +14530,23 @@ paths: parameters passed to the training script. required: - dl_runtime_image + description: >- + DEPRECATED — use ``AiRuntimeTask`` for all new + BYOT multi-node GPU workloads (see + ai_runtime_task.proto). ``AiRuntimeTask`` is the only supported BYOT task type for new + workloads; this proto is retained only for AIR CLI (fka SGCLI) pywheel backwards compatibility + and will be removed once the pywheel → databricks-cli migration completes (post- PuPr). + genie_task: + type: object + properties: + configuration_id: + type: string + description: >- + Required. Resource name of the agent task + configuration to run. + required: + - configuration_id + description: Runs a Genie or Genie Code agent task. health: type: object properties: @@ -10271,32 +14569,25 @@ paths: rule. - * `RUN_DURATION_SECONDS`: Expected total - time for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data - waiting to be consumed across all - streams. This metric - - is in Public Preview. * - `STREAMING_BACKLOG_RECORDS`: An estimate - of the maximum offset lag - - across all streams. This metric is in - Public Preview. * - `STREAMING_BACKLOG_SECONDS`: An estimate - - of the maximum consumer delay across all - streams. This metric is in Public - Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate - of the maximum number of outstanding - files across all - - streams. This metric is in Public - Preview. + - ``RUN_DURATION_SECONDS``: Expected + total time for a run in seconds. + + - ``STREAMING_BACKLOG_BYTES``: An + estimate of the maximum bytes of data + waiting to be consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An + estimate of the maximum offset lag + across all streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An + estimate of the maximum consumer delay + across all streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An + estimate of the maximum number of + outstanding files across all + streams. This metric is in Public Preview. op: type: string x-enum: @@ -10323,7 +14614,7 @@ paths: description: >- If job_cluster_key, this task is executed reusing the cluster specified in - `job.settings.job_clusters`. + ``job.settings.job_clusters``. libraries: type: string description: >- @@ -10334,11 +14625,11 @@ paths: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be - unsuccessful if it completes with the `FAILED` - result_state or `INTERNAL_ERROR` - `life_cycle_state`. The value `-1` means to retry - indefinitely and the value `0` means to never - retry. + unsuccessful if it completes with the ``FAILED`` + result_state or ``INTERNAL_ERROR`` + ``life_cycle_state``. The value ``-1`` means to + retry indefinitely and the value ``0`` means to + never retry. min_retry_interval_millis: type: integer description: >- @@ -10364,21 +14655,21 @@ paths: :method:jobs/run Now with parameters specified, the two parameters maps are merged. If the same key is specified in - `base_parameters` and in `run-now`, the value - from `run-now` is used. Use [Task parameter - variables] to set parameters containing - information about job runs. If the notebook - takes a parameter that is not specified in the - job’s `base_parameters` or the `run-now` + ``base_parameters`` and in ``run-now``, the + value from ``run-now`` is used. Use `Task + parameter variables + `__ + to set parameters containing information about + job runs. If the notebook takes a parameter + that is not specified in the job’s + ``base_parameters`` or the ``run-now`` override parameters, the default value from the notebook is used. Retrieve these parameters in a notebook using - [dbutils.widgets.get]. The JSON representation - of this field cannot exceed 1MB. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-widgets + `dbutils.widgets.get + `__. + The JSON representation of this field cannot + exceed 1MB. source: type: string x-enum: @@ -10386,20 +14677,21 @@ paths: - WORKSPACE description: >- Optional location type of the notebook. When - set to `WORKSPACE`, the notebook will be + set to ``WORKSPACE``, the notebook will be retrieved from the local Databricks workspace. - When set to `GIT`, the notebook will be + When set to ``GIT``, the notebook will be retrieved from a Git repository defined in - `git_source`. If the value is empty, the task - will use `GIT` if `git_source` is defined and - `WORKSPACE` otherwise. * `WORKSPACE`: Notebook - is located in Databricks workspace. * `GIT`: - Notebook is located in cloud Git provider. + ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - + ``WORKSPACE``: Notebook is located in + Databricks workspace. - ``GIT``: Notebook is + located in cloud Git provider. warehouse_id: type: string description: >- - Optional `warehouse_id` to run the notebook on - a SQL warehouse. Classic SQL warehouses are + Optional ``warehouse_id`` to run the notebook + on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses. Note that SQL warehouses only support SQL cells; if the notebook contains @@ -10407,8 +14699,8 @@ paths: required: - notebook_path description: >- - The task runs a notebook when the `notebook_task` - field is present. + The task runs a notebook when the + ``notebook_task`` field is present. notification_settings: type: object properties: @@ -10418,19 +14710,19 @@ paths: type: boolean description: >- If true, do not send notifications to - recipients specified in `on_failure` if the + recipients specified in ``on_failure`` if the run is canceled. no_alert_for_skipped_runs: type: boolean description: >- If true, do not send notifications to - recipients specified in `on_failure` if the + recipients specified in ``on_failure`` if the run is skipped. description: >- Optional notification settings that are used when sending notifications to each of the - `email_notifications` and `webhook_notifications` - for this task. + ``email_notifications`` and + ``webhook_notifications`` for this task. pipeline_task: type: object properties: @@ -10439,19 +14731,119 @@ paths: full_refresh: type: boolean description: >- - If true, triggers a full refresh on the delta - live table. + If true, triggers a full refresh on the spark + declarative pipeline. + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + parameters: + type: object + description: >- + Key/value-map of parameters passed to the + pipeline execution. Limited to 10k characters + in total. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are + unioned with other selective refresh options + (refresh_selection, full_refresh_selection) to + determine the final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints + without clearing data. required: - pipeline_id description: >- The task triggers a pipeline update when the - `pipeline_task` field is present. Only pipelines + ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: type: object properties: connection_resource_name: type: string + incremental_refresh_config: + type: object + properties: + archive_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the archive window. + Default: MONTH + archive_window_periods: + type: integer + description: >- + Number of periods in the archive window + (total data retained). Must be greater + than the refresh window when using the + same granularity. Default: 3 (with YEAR + granularity = 3 years) + detect_data_changes: + type: boolean + description: >- + If true, Power BI will check whether data + has changed in each partition before + reimporting. Partitions with unchanged + data are skipped, reducing refresh cost + further. Uses MAX(datetime_column) as the + change detector. + mode: + type: string + x-enum: + - REFRESH_POLICY_MODE_HYBRID + - REFRESH_POLICY_MODE_IMPORT_ONLY + description: >- + The refresh policy mode. Hybrid adds a + real-time DirectQuery partition for the + most recent data; Import uses only + batch-imported partitions. Default: HYBRID + only_refresh_complete_periods: + type: boolean + description: >- + If true, only refresh complete periods + (e.g., skip today's partial data). + refresh_window_granularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: >- + Granularity for the refresh window. + Default: DAY + refresh_window_periods: + type: integer + description: >- + Number of periods in the rolling refresh + window. Partitions within this window are + re-imported on each refresh. Default: 1 + description: >- + Incremental refresh policy applied to all + IMPORT mode tables in the model. Windows and + mode are shared; partition columns are set + per-table on PowerBiTable. power_bi_model: type: object properties: @@ -10549,6 +14941,15 @@ paths: properties: catalog: type: string + incremental_refresh_datetime_column: + type: string + description: >- + The datetime column used for incremental + refresh partitioning on this table. + e.g., "order_date", "updated_at" Only + applicable when the task has + incremental_refresh_config set and this + table uses IMPORT storage mode. name: type: string description: The table name in Databricks @@ -10562,6 +14963,25 @@ paths: - DUAL - IMPORT description: The Power BI storage mode of the table + table_type: + type: string + x-enum: + - EXTERNAL + - EXTERNAL_SHALLOW_CLONE + - FOREIGN + - MANAGED + - MANAGED_SHALLOW_CLONE + - MATERIALIZED_VIEW + - METRIC_VIEW + - STREAMING_TABLE + - VIEW + description: >- + The Unity Catalog table type of this + table. When set, the server trusts it to + classify the table during metric-view + filtering and skips the server-side + lookup. When unset, the server resolves + the type on its own. description: The tables to be exported to Power BI warehouse_id: type: string @@ -10570,7 +14990,25 @@ paths: data source description: >- The task triggers a Power BI semantic model update - when the `power_bi_task` field is present. + when the ``power_bi_task`` field is present. + python_operator_task: + type: object + properties: + main: + type: string + parameters: + type: array + items: + type: object + properties: + name: + type: string + value: + type: string + description: >- + An ordered list of task parameters. + TODO(JOBS-30885): Add limits for parameters. + description: The task runs a Python operator task. python_wheel_task: type: object properties: @@ -10582,28 +15020,28 @@ paths: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using - `$packageName.$entryPoint()` + ``$packageName.$entryPoint()`` named_parameters: type: object description: >- Command-line parameters passed to Python wheel - task in the form of `["--name=task", - "--data=dbfs:/path/to/data.json"]`. Leave it - empty if `parameters` is not null. + task in the form of ``["--name=task", + "--data=dbfs:/path/to/data.json"]``. Leave it + empty if ``parameters`` is not null. parameters: type: array items: type: string description: >- Command-line parameters passed to Python wheel - task. Leave it empty if `named_parameters` is - not null. + task. Leave it empty if ``named_parameters`` + is not null. required: - package_name - entry_point description: >- The task runs a Python wheel when the - `python_wheel_task` field is present. + ``python_wheel_task`` field is present. retry_on_timeout: type: boolean description: >- @@ -10622,15 +15060,15 @@ paths: description: >- An optional value specifying the condition determining whether the task is run once its - dependencies have been completed. * `ALL_SUCCESS`: - All dependencies have executed and succeeded * - `AT_LEAST_ONE_SUCCESS`: At least one dependency - has succeeded * `NONE_FAILED`: None of the - dependencies have failed and at least one was - executed * `ALL_DONE`: All dependencies have been - completed * `AT_LEAST_ONE_FAILED`: At least one - dependency failed * `ALL_FAILED`: ALl dependencies - have failed + dependencies have been completed. - + ``ALL_SUCCESS``: All dependencies have executed + and succeeded - ``AT_LEAST_ONE_SUCCESS``: At least + one dependency has succeeded - ``NONE_FAILED``: + None of the dependencies have failed and at least + one was executed - ``ALL_DONE``: All dependencies + have been completed - ``AT_LEAST_ONE_FAILED``: At + least one dependency failed - ``ALL_FAILED``: ALl + dependencies have failed run_job_task: type: object properties: @@ -10642,31 +15080,31 @@ paths: type: string description: >- An array of commands to execute for jobs with - the dbt task, for example `"dbt_commands": + the dbt task, for example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt deps", "dbt - seed", "dbt run"]` ⚠ **Deprecation note** Use - [job parameters] to pass information down to - tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + seed", "dbt run"]`` ⚠ **Deprecation note** Use + `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR - tasks, for example `"jar_params": ["john doe", - "35"]`. The parameters are used to invoke the - main function of the main class specified in - the Spark JAR task. If not specified upon - `run-now`, it defaults to an empty list. - jar_params cannot be specified in conjunction - with notebook_params. The JSON representation - of this field (for example - `{"jar_params":["john doe","35"]}`) cannot + tasks, for example ``"jar_params": ["john + doe", "35"]``. The parameters are used to + invoke the main function of the main class + specified in the Spark JAR task. If not + specified upon ``run-now``, it defaults to an + empty list. jar_params cannot be specified in + conjunction with notebook_params. The JSON + representation of this field (for example + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ **Deprecation note** - Use [job parameters] to pass information down - to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + Use `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: Job-level parameters used to trigger the job. @@ -10674,28 +15112,59 @@ paths: type: object description: >- A map from keys to values for jobs with - notebook task, for example `"notebook_params": - {"name": "john doe", "age": "35"}`. The map is - passed to the notebook and is accessible - through the [dbutils.widgets.get] function. If - not specified upon `run-now`, the triggered - run uses the job’s base parameters. - notebook_params cannot be specified in - conjunction with jar_params. ⚠ **Deprecation - note** Use [job parameters] to pass - information down to tasks. The JSON + notebook task, for example + ``"notebook_params": {"name": "john doe", + "age": "35"}``. The map is passed to the + notebook and is accessible through the + `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, + the triggered run uses the job’s base + parameters. notebook_params cannot be + specified in conjunction with jar_params. ⚠ + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. The JSON representation of this field (for example - `{"notebook_params":{"name":"john - doe","age":"35"}}`) cannot exceed 10,000 - bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html - [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 + bytes. pipeline_params: type: object properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update with + fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These + are unioned with other selective refresh + options (refresh_selection, + full_refresh_selection) to determine the + final set of flows to refresh. + refresh_selection: + type: array + items: + type: string + description: >- + A list of tables to update without + fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset + checkpoints without clearing data. description: >- Controls whether the pipeline should perform a full refresh @@ -10707,22 +15176,22 @@ paths: type: string description: >- A list of parameters for jobs with Python - tasks, for example `"python_params": ["john - doe", "35"]`. The parameters are passed to + tasks, for example ``"python_params": ["john + doe", "35"]``. The parameters are passed to Python file as command-line parameters. If - specified upon `run-now`, it would overwrite + specified upon ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot - exceed 10,000 bytes. ⚠ **Deprecation note** - Use [job parameters] to pass information down - to tasks. Important These parameters accept - only Latin characters (ASCII character set). - Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are - Chinese, Japanese kanjis, and emojis. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"python_params":["john doe","35"]}``) + cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important + These parameters accept only Latin characters + (ASCII character set). Using non-ASCII + characters returns an error. Examples of + invalid, non-ASCII characters are Chinese, + Japanese kanjis, and emojis. spark_submit_params: type: array items: @@ -10730,39 +15199,38 @@ paths: description: >- A list of parameters for jobs with spark submit task, for example - `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The + ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script as command-line parameters. If specified upon - `run-now`, it would overwrite the parameters + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot - exceed 10,000 bytes. ⚠ **Deprecation note** - Use [job parameters] to pass information down - to tasks. Important These parameters accept - only Latin characters (ASCII character set). - Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are - Chinese, Japanese kanjis, and emojis. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"python_params":["john doe","35"]}``) + cannot exceed 10,000 bytes. ⚠ **Deprecation + note** Use `job parameters + `__ + to pass information down to tasks. Important + These parameters accept only Latin characters + (ASCII character set). Using non-ASCII + characters returns an error. Examples of + invalid, non-ASCII characters are Chinese, + Japanese kanjis, and emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL - task, for example `"sql_params": {"name": - "john doe", "age": "35"}`. The SQL alert task + task, for example ``"sql_params": {"name": + "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ - **Deprecation note** Use [job parameters] to - pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. required: - job_id description: >- The task triggers another job when the - `run_job_task` field is present. + ``run_job_task`` field is present. spark_jar_task: type: object properties: @@ -10774,7 +15242,7 @@ paths: The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library. The - code must use `SparkContext.getOrCreate` to + code must use ``SparkContext.getOrCreate`` to obtain a Spark context; otherwise, runs of the job fail. parameters: @@ -10783,17 +15251,17 @@ paths: type: string description: >- Parameters passed to the main method. Use - [Task parameter variables] to set parameters - containing information about job runs. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + `Task parameter variables + `__ + to set parameters containing information about + job runs. run_as_repl: type: boolean description: >- - Deprecated. A value of `false` is no longer + Deprecated. A value of ``false`` is no longer supported. description: >- - The task runs a JAR when the `spark_jar_task` + The task runs a JAR when the ``spark_jar_task`` field is present. spark_python_task: type: object @@ -10806,10 +15274,10 @@ paths: type: string description: >- Command line parameters passed to the Python - file. Use [Task parameter variables] to set - parameters containing information about job - runs. [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + file. Use `Task parameter variables + `__ + to set parameters containing information about + job runs. source: type: string x-enum: @@ -10817,21 +15285,22 @@ paths: - WORKSPACE description: >- Optional location type of the Python file. - When set to `WORKSPACE` or not specified, the - file will be retrieved from the local + When set to ``WORKSPACE`` or not specified, + the file will be retrieved from the local Databricks workspace or cloud location (if the - `python_file` has a URI format). When set to - `GIT`, the Python file will be retrieved from - a Git repository defined in `git_source`. * - `WORKSPACE`: The Python file is located in a - Databricks workspace or at a cloud filesystem - URI. * `GIT`: The Python file is located in a - remote Git repository. + ``python_file`` has a URI format). When set to + ``GIT``, the Python file will be retrieved + from a Git repository defined in + ``git_source``. - ``WORKSPACE``: The Python + file is located in a Databricks workspace or + at a cloud filesystem URI. - ``GIT``: The + Python file is located in a remote Git + repository. required: - python_file description: >- The task runs a Python file when the - `spark_python_task` field is present. + ``spark_python_task`` field is present. spark_submit_task: type: object properties: @@ -10936,16 +15405,17 @@ paths: - WORKSPACE description: >- Optional location type of the SQL file. - When set to `WORKSPACE`, the SQL file will - be retrieved from the local Databricks - workspace. When set to `GIT`, the SQL file - will be retrieved from a Git repository - defined in `git_source`. If the value is - empty, the task will use `GIT` if - `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: SQL file is - located in Databricks workspace. * `GIT`: - SQL file is located in cloud Git provider. + When set to ``WORKSPACE``, the SQL file + will be retrieved from the local + Databricks workspace. When set to ``GIT``, + the SQL file will be retrieved from a Git + repository defined in ``git_source``. If + the value is empty, the task will use + ``GIT`` if ``git_source`` is defined and + ``WORKSPACE`` otherwise. - ``WORKSPACE``: + SQL file is located in Databricks + workspace. - ``GIT``: SQL file is located + in cloud Git provider. required: - path description: >- @@ -10972,12 +15442,12 @@ paths: description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the - `sql_task` field is present. + ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this - job task. A value of `0` means no timeout. + job task. A value of ``0`` means no timeout. webhook_notifications: type: object properties: @@ -11003,7 +15473,37 @@ paths: An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the - `on_failure` property. + ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance + completes for a continuous job. A maximum of 3 + destinations can be specified for the + ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to + call when platform-initiated maintenance + starts for a continuous job. A maximum of 3 + destinations can be specified for the + ``on_maintenance_start`` property. on_start: type: array items: @@ -11017,7 +15517,7 @@ paths: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the - `on_start` property. + ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -11031,17 +15531,17 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog - thresholds can be set in the `health` field + thresholds can be set in the ``health`` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based + ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -11055,7 +15555,7 @@ paths: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for - the `on_success` property. + the ``on_success`` property. description: >- A collection of system notification IDs to notify when runs of this task begin or complete. The @@ -11070,13 +15570,14 @@ paths: :method:jobs/update, :method:jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them using - :method:jobs/get. Use the `next_page_token` field at the - object root to determine if more results are available. + :method:jobs/get. Use the ``next_page_token`` field at + the object root to determine if more results are + available. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job. A - value of `0` means no timeout. + value of ``0`` means no timeout. trigger: type: object properties: @@ -11214,12 +15715,47 @@ paths: x-enum: - DAYS - HOURS + - MINUTES - WEEKS description: The unit of time for the interval. required: - interval - unit description: Periodic trigger settings. + sql_condition: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to + run the condition query against. + trigger_mode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + Determines how the SQL query result is + interpreted to decide whether the condition + fires. Must be set to a recognized value when + provided. When unset on an existing serialized + configuration, the server preserves the original + semantics by interpreting it as + ``QUERY_RETURNS_ROWS``. New configurations + should set this explicitly — explicit + ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + description: >- + SQL condition that must be satisfied for the trigger + to fire. Can be used in combination with other + trigger types and runs *after* other trigger types + conditions are evaluated. table_update: type: object properties: @@ -11256,15 +15792,15 @@ paths: A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or - sending an API request to `runNow`. + sending an API request to ``runNow``. usage_policy_id: type: string description: >- The id of the user specified usage policy to use for this job. If not specified, a default usage policy may be applied when creating or modifying the job. See - `effective_usage_policy_id` for the usage policy used by - this workload. + ``effective_usage_policy_id`` for the usage policy used + by this workload. webhook_notifications: type: object properties: @@ -11289,7 +15825,36 @@ paths: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can - be specified for the `on_failure` property. + be specified for the ``on_failure`` property. + on_maintenance_complete: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call + when platform-initiated maintenance completes for a + continuous job. A maximum of 3 destinations can be + specified for the ``on_maintenance_complete`` + property. + on_maintenance_start: + type: array + items: + type: object + properties: + id: + type: string + required: + - id + description: >- + An optional list of system notification IDs to call + when platform-initiated maintenance starts for a + continuous job. A maximum of 3 destinations can be + specified for the ``on_maintenance_start`` property. on_start: type: array items: @@ -11302,7 +15867,7 @@ paths: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can - be specified for the `on_start` property. + be specified for the ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -11316,15 +15881,15 @@ paths: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be - set in the `health` field using the following - metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the - 10-minute average of these metrics. If the issue + set in the ``health`` field using the following + metrics: ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on + the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the - `on_streaming_backlog_exceeded` property. + ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -11337,19 +15902,19 @@ paths: description: >- An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 - destinations can be specified for the `on_success` + destinations can be specified for the ``on_success`` property. description: >- A collection of system notification IDs to notify when runs of this job begin or complete. description: >- The new settings for the job. Top-level fields specified in - `new_settings` are completely replaced, except for arrays + ``new_settings`` are completely replaced, except for arrays which are merged. That is, new and existing entries are completely replaced based on the respective key fields, i.e. - `task_key` or `job_cluster_key`, while previous entries are - kept. Partially updating nested fields is not supported. - Changes to the field `JobSettings.timeout_seconds` are + ``task_key`` or ``job_cluster_key``, while previous entries + are kept. Partially updating nested fields is not supported. + Changes to the field ``JobSettings.timeout_seconds`` are applied to active runs. Changes to other fields are applied to future runs only. required: @@ -11374,13 +15939,13 @@ paths: operationId: policy_compliance_for_jobs_enforce_compliance summary: >- Updates a job so the job clusters that are created when running the job - (specified in `new_cluster`) + (specified in ``new_cluster``) tags: - jobs - policy_compliance_for_jobs description: >- Updates a job so the job clusters that are created when running the job - (specified in `new_cluster`) + (specified in ``new_cluster``) are compliant with the current versions of their respective cluster policies. All-purpose clusters @@ -11504,7 +16069,7 @@ paths: further constrain the maximum number of results returned in a single page. :param page_token: str (optional) A page token that can be used to navigate to the next page or previous page as returned by - `next_page_token` or `prev_page_token`. + ``next_page_token`` or ``prev_page_token``. :returns: Iterator over :class:`JobCompliance` parameters: @@ -11520,43 +16085,360 @@ paths: schema: type: integer description: >- - Use this field to specify the maximum number of results to be - returned by the server. The server may further constrain the maximum - number of results returned in a single page. - - name: page_token - in: query - required: false - schema: - type: string + Use this field to specify the maximum number of results to be + returned by the server. The server may further constrain the maximum + number of results returned in a single page. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token that can be used to navigate to the next page or + previous page as returned by ``next_page_token`` or + ``prev_page_token``. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListJobComplianceForPolicyResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PolicyComplianceForJobsAPI +components: + schemas: + AgentTool: + type: object + properties: + app: + $ref: '#/components/schemas/AppTool' + description: Databricks App tool config. Set when the discriminator == "app". + catalog: + $ref: '#/components/schemas/CatalogTool' + description: >- + Unity Catalog catalog tool config. Set when the discriminator == + "catalog". + dashboard: + $ref: '#/components/schemas/DashboardTool' + description: >- + Lakeview dashboard tool config. Set when the discriminator == + "dashboard". + description: + type: string + description: Optional. Free-form description of the tool. + genie_space: + $ref: '#/components/schemas/GenieSpaceTool' + description: >- + Genie space tool config. Set when the discriminator == + "genie_space". + knowledge_assistant: + $ref: '#/components/schemas/KnowledgeAssistantTool' + description: >- + Knowledge Assistant tool config. Set when the discriminator == + "knowledge_assistant". + schema: + $ref: '#/components/schemas/SchemaTool' + description: >- + Unity Catalog schema tool config. Set when the discriminator == + "schema". + serving_endpoint: + $ref: '#/components/schemas/ServingEndpointTool' + description: >- + Model serving endpoint tool config. Set when the discriminator == + "serving_endpoint". + supervisor_agent: + $ref: '#/components/schemas/SupervisorAgentTool' + description: >- + Supervisor Agent tool config (a supervisor agent used as a tool). + Set when the discriminator == "supervisor_agent". + table: + $ref: '#/components/schemas/TableTool' + description: >- + Unity Catalog table tool config. Set when the discriminator == + "table". + tool_type: + type: string + uc_connection: + $ref: '#/components/schemas/UcConnectionTool' + description: >- + External MCP / UC connection tool config. Set when the discriminator + == "uc_connection". + uc_function: + $ref: '#/components/schemas/UcFunctionTool' + description: >- + Unity Catalog function tool config. Set when the discriminator == + "uc_function". + uc_mcp: + $ref: '#/components/schemas/UcMcpTool' + description: >- + UC-registered MCP service tool config. Set when the discriminator == + "uc_mcp". + vector_search_index: + $ref: '#/components/schemas/VectorSearchIndexTool' + description: >- + Vector Search index tool config. Set when the discriminator == + "vector_search_index". + volume: + $ref: '#/components/schemas/VolumeTool' + description: >- + Unity Catalog volume tool config. Set when the discriminator == + "volume". + web_search: + $ref: '#/components/schemas/WebSearchTool' + description: Web search tool config. Set when the discriminator == "web_search". + description: >- + Tool configuration for an inlined agentic task agent. Discriminator + selects which per-asset + config message applies (``genie_space``, ``knowledge_assistant``, ``uc_function``, + ``uc_connection``, ``uc_mcp``, ``app``, ``table``, ``vector_search_index``, ``volume``, + ``catalog``, ``schema``, ``dashboard``, ``supervisor_agent``, ``serving_endpoint``, + ``web_search``); ``description`` is a free-form top-level description; each per-asset config + carries only the asset's identifier. + + The discriminator is exposed under two interchangeable names: ``type`` mirrors the Supervisor + (Responses) API surface, and ``tool_type`` is the AIP-friendly alias for clients where ``type`` + is a reserved keyword (Terraform, etc.). Set exactly one. + AgenticTask: + type: object + properties: + context: + $ref: '#/components/schemas/AgenticTaskContext' + description: >- + Optional. Context input providing conversation history and + instructions. + genie_code_api: + $ref: '#/components/schemas/GenieCodeApi' + description: >- + Inline Genie Code conversation driven by a single prompt. Mutually + exclusive with the supervisor variants. + goal: + type: string + description: >- + Deprecated. Use ``input`` (field 7) instead. Kept for backwards + compatibility with existing callers; will be removed in a future + revision. + input: + type: string + description: >- + Optional. The user query / task input the agent must accomplish. + Mirrors the OpenAI Responses API ``input`` field. Replaces the + deprecated ``goal`` field; new callers should populate ``input`` + going forward. + output_schema: + $ref: '#/components/schemas/TaskOutputSchema' + description: >- + Optional. JSON-Schema-style declaration of the structured output the + agent should produce. Replaces the deprecated ``task_output_schema`` + map; new callers should populate ``output_schema`` going forward. + supervisor_agent: + $ref: '#/components/schemas/SupervisorAgent' + description: >- + A Supervisor Agent that orchestrates sub-agents and tools, + referenced by tile_id. + supervisor_api: + $ref: '#/components/schemas/SupervisorApi' + description: >- + Inlined Responses-API supervisor configuration (model + instructions + + tools). Mutually exclusive with ``supervisor_agent``. + task_output_schema: + type: object + description: >- + Deprecated. Use ``output_schema`` (field 8) instead. Kept for + backwards compatibility with existing callers; will be removed in a + future revision. + trace_destination: + $ref: '#/components/schemas/TraceDestination' + description: >- + Optional. Where MLflow traces produced by this task run should be + persisted. When unset, traces follow the workspace default + destination. + description: >- + Defines an agentic task configuration for job-based execution. Three + configuration paths via + ``oneof agent``: + + - Reference a pre-registered Supervisor Agent tile via ``supervisor_agent.agent_id``. + - Provide an inlined Responses-API-style config via ``supervisor_api``. + - Run an inline Genie Code prompt via ``genie_code_api``. + AgenticTaskContext: + type: object + properties: + conversation_ids: + type: array + items: + type: string + description: Optional. Conversation IDs to load as context for the agent. + instructions: + type: array + items: + type: string + description: Optional. Additional instructions to guide the agent's behavior. + description: >- + Context input for an agentic task, providing conversation history and + additional instructions to + guide the agent. + AgenticTaskOutput: + type: object + properties: + conversation_id: + type: string + description: The conversation ID generated during this task execution. + created_at: + type: number + description: Deprecated. Run timestamps are exposed on the surrounding job-run. + genie_code: + $ref: '#/components/schemas/GenieCodeOutput' + description: Genie Code execution output (conversation notebook reference). + id: + type: string + description: >- + Deprecated. Run-level identity already lives on the surrounding + ``RunOutput``; ``SupervisorOutput.response_id`` is the canonical + handle for the structured response. + response: + type: string + description: >- + Deprecated. The final text response is now surfaced in + ``task_output["response"]``; new callers should read it from there. + status: + type: string + description: >- + Deprecated. Run lifecycle state is exposed on the surrounding + job-run. + supervisor: + $ref: '#/components/schemas/SupervisorOutput' + description: Supervisor (tile or inlined) execution output. + task_output: + type: object + description: >- + Custom output values from the agent. When the caller specified a + ``task_output_schema`` / ``output_schema``, this carries those + user-defined keys. When no schema is specified, this carries default + supervisor keys (``response``, ``truncated``, ...). + description: Output returned after an agentic task completes. + AiRuntimeTask: + type: object + properties: + experiment: + type: string + description: >- + MLflow experiment name for this run. If an experiment with this name + already exists under the calling user, the run is appended to it; + otherwise a new experiment is created. To target a specific MLflow + storage location (for example, when running as a service principal), + set ``mlflow_experiment_directory``. + deployments: + type: array + items: + $ref: '#/components/schemas/DeploymentSpec' + description: >- + Deployment specs for this task. Exactly one deployment is currently + supported (a single entry where every node runs the same command); + this is a current-Preview constraint. Role-split workloads (driver + + worker, parameter server, separate eval node, etc.) with multiple + entries are the eventual intent but not yet supported. + code_source_path: + type: string + description: >- + Workspace or UC volume path of the code-source archive, unpacked on + each node and exposed through ``$CODE_SOURCE``. Set by first-party + tooling; not for direct callers. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom container image. When set, + the task runs on the specified container image instead of the + default Databricks client image. Format: + ``{organization}/{repository}:{tag}`` + mlflow_experiment_directory: + type: string + description: >- + Optional workspace directory under which the MLflow experiment named + in ``experiment`` is created. Must start with ``/Workspace``. Set + this when running as a service principal that has no default user + directory; for regular users the experiment defaults to the user's + home directory. + mlflow_run: + type: string + description: >- + Optional display name for the MLflow run created under + ``experiment``. If omitted, MLflow generates a default name. + parameters: + type: object description: >- - A page token that can be used to navigate to the next page or - previous page as returned by `next_page_token` or `prev_page_token`. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListJobComplianceForPolicyResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PolicyComplianceForJobsAPI -components: - schemas: + Optional named parameters passed to each deployment's command. Keys + are parameter names, values the corresponding arguments (for + example, ``{"epochs": "3", "dataset": "s3://bucket/train"}``). + Values may contain dynamic references such as + ``{{job.trigger.time.iso_date}}`` or + ``{{tasks..values.}}``, which Jobs substitutes + before execution (see ``AiRuntimeTaskResolvedValues.parameters`` in + runs.proto). + required: + - experiment + - deployments + description: >- + AiRuntimeTask: multi-node GPU compute task definition for Databricks AI + Runtime workloads. + + Jobs-framework-level concepts (retries, per-task timeout, idempotency token, usage/budget + policy, permissions) live on the surrounding TaskSettings / run-submit request and are + intentionally NOT duplicated here. Users compose ``ai_runtime_task`` with the standard Jobs/DABs + task wrapper to get those. + AiRuntimeTaskOutput: + type: object + properties: + mlflow_experiment_id: + type: string + description: >- + MLflow experiment ID the run was logged to. Use it to look up the + experiment in MLflow APIs or the workspace MLflow UI. + mlflow_run_id: + type: string + description: >- + MLflow run ID for this task execution. Use it to look up the run in + MLflow APIs or the workspace MLflow UI. + status_message: + type: string + description: >- + Human-readable status message for this run, suitable for display to + the user (for example, that the run is still waiting for GPU + compute). Set by the server only when there is something to surface; + empty otherwise. + description: >- + AiRuntimeTaskOutput: output identifiers for an AiRuntimeTask run — the + MLflow experiment and + run IDs the task wrote to. + + Run lifecycle and termination status are not on this message; they live on the surrounding + ``RunTask.status`` field (see ``runs.proto:RunTask.status``). AlertTask: type: object properties: alert_id: type: string + parameters: + type: object + description: >- + Per-run parameter overrides, keyed by parameter name, applied onto + the alert's stored query parameters before the query is executed. + Only scalar values are supported. Values may reference job + parameters with ``{{job.parameters.*}}``, which are resolved before + the task runs. An override whose key does not match a stored + parameter fails the task run. Limited to 10000 characters when + serialized as JSON; keys must be 1-100 characters and contain only + letters, digits, underscores, dashes, and periods. subscribers: type: array items: @@ -11574,7 +16456,7 @@ components: type: string description: >- The workspace_path is the path to the alert file in the workspace. - The path: * must start with "/Workspace" * must be a normalized + The path: - must start with "/Workspace" - must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert. AlertTaskOutput: @@ -11594,6 +16476,11 @@ components: Represents a subscriber that will receive alert notifications. A subscriber can be either a user (via email) or a notification destination (via destination_id). + AppTool: + type: object + properties: + name: + type: string BaseJob: type: object properties: @@ -11610,7 +16497,7 @@ components: The id of the budget policy used by this job for cost attribution purposes. This may be set through (in order of precedence): 1. Budget admins through the account or workspace console 2. Jobs UI in - the job details page and Jobs API using `budget_policy_id` 3. + the job details page and Jobs API using ``budget_policy_id`` 3. Inferred default based on accessible budget policies of the run_as identity on job creation or modification. effective_usage_policy_id: @@ -11621,18 +16508,24 @@ components: has_more: type: boolean description: >- - Indicates if the job has more array properties (`tasks`, - `job_clusters`) that are not shown. They can be accessed via + Indicates if the job has more array properties (``tasks``, + ``job_clusters``) that are not shown. They can be accessed via :method:jobs/get endpoint. It is only relevant for API 2.2 - :method:jobs/list requests with `expand_tasks=true`. + :method:jobs/list requests with ``expand_tasks=true``. job_id: type: integer description: The canonical identifier for this job. + path: + type: string + description: >- + Path of the job object in workspace file tree, including file + extension. If absent, the job doesn't have a workspace object. + Example: /Workspace/user@example.com/my_project/my_job.job.json settings: $ref: '#/components/schemas/JobSettings' description: >- Settings for this job and all of its runs. These settings can be - updated using the `resetJob` method. + updated using the ``resetJob`` method. trigger_state: $ref: '#/components/schemas/TriggerStateProto' description: State of the trigger associated with the job. @@ -11646,10 +16539,10 @@ components: description: >- The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum - of the `setup_duration`, `execution_duration`, and the - `cleanup_duration`. The `cleanup_duration` field is set to 0 for + of the ``setup_duration``, ``execution_duration``, and the + ``cleanup_duration``. The ``cleanup_duration`` field is set to 0 for multitask job runs. The total duration of a multitask job run is the - value of the `run_duration` field. + value of the ``run_duration`` field. cluster_instance: $ref: '#/components/schemas/ClusterInstance' description: >- @@ -11666,6 +16559,13 @@ components: description: >- The creator user name. This field won’t be included in the response if the user has already been deleted. + deployment_id: + type: string + description: >- + ID of the deployment that produced the job when this run was + created. Used to look up deployment metadata from the Deployment + Metadata service. Only set for job runs of jobs with a ``BUNDLE`` + deployment. description: type: string description: Description of the run @@ -11675,8 +16575,8 @@ components: The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported - by the job type. * `STANDARD`: Enables cost-efficient execution of - serverless workloads. * `PERFORMANCE_OPTIMIZED`: Prioritizes fast + by the job type. - ``STANDARD``: Enables cost-efficient execution of + serverless workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. effective_usage_policy_id: @@ -11690,34 +16590,45 @@ components: The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running. + environment_variables: + type: array + items: + $ref: '#/components/schemas/JobEnvironmentVariables' + description: >- + Snapshot of ``JobSettings.environment_variables`` as it was at run + launch — the full list of named env-var profiles the job defined. + Per-profile resolved contents are not preserved here; only the + customer-defined ``variables`` and ``files`` paths. To find which + profile a given task ran with, look at + ``RunTaskSettings.environment_variables_key``. execution_duration: type: integer description: >- The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is - the sum of the `setup_duration`, `execution_duration`, and the - `cleanup_duration`. The `execution_duration` field is set to 0 for - multitask job runs. The total duration of a multitask job run is the - value of the `run_duration` field. + the sum of the ``setup_duration``, ``execution_duration``, and the + ``cleanup_duration``. The ``execution_duration`` field is set to 0 + for multitask job runs. The total duration of a multitask job run is + the value of the ``run_duration`` field. git_source: $ref: '#/components/schemas/GitSource' description: >- An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If - `git_source` is set, these tasks retrieve the file from the remote + ``git_source`` is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by - setting `source` to `WORKSPACE` on the task. Note: dbt and SQL File - tasks support only version-controlled sources. If dbt or SQL File - tasks are used, `git_source` must be defined on the job. + setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL + File tasks support only version-controlled sources. If dbt or SQL + File tasks are used, ``git_source`` must be defined on the job. has_more: type: boolean description: >- - Indicates if the run has more array properties (`tasks`, - `job_clusters`) that are not shown. They can be accessed via + Indicates if the run has more array properties (``tasks``, + ``job_clusters``) that are not shown. They can be accessed via :method:jobs/getrun endpoint. It is only relevant for API 2.2 - :method:jobs/listruns requests with `expand_tasks=true`. + :method:jobs/listruns requests with ``expand_tasks=true``. job_clusters: type: array items: @@ -11747,7 +16658,7 @@ components: type: integer description: >- A unique identifier for this job run. This is set to the same value - as `run_id`. + as ``run_id``. original_attempt_run_id: type: integer description: >- @@ -11796,11 +16707,11 @@ components: The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The - duration of a task run is the sum of the `setup_duration`, - `execution_duration`, and the `cleanup_duration`. The - `setup_duration` field is set to 0 for multitask job runs. The total - duration of a multitask job run is the value of the `run_duration` - field. + duration of a task run is the sum of the ``setup_duration``, + ``execution_duration``, and the ``cleanup_duration``. The + ``setup_duration`` field is set to 0 for multitask job runs. The + total duration of a multitask job run is the value of the + ``run_duration`` field. start_time: type: integer description: >- @@ -11811,7 +16722,7 @@ components: issued. state: $ref: '#/components/schemas/RunState' - description: Deprecated. Please use the `status` field instead. + description: Deprecated. Please use the ``status`` field instead. status: $ref: '#/components/schemas/RunStatus' tasks: @@ -11820,14 +16731,27 @@ components: $ref: '#/components/schemas/RunTask' description: >- The list of tasks performed by the run. Each task has its own - `run_id` which you can use to call `JobsGetOutput` to retrieve the - run resutls. If more than 100 tasks are available, you can paginate - through them using :method:jobs/getrun. Use the `next_page_token` - field at the object root to determine if more results are available. + ``run_id`` which you can use to call ``JobsGetOutput`` to retrieve + the run results. If more than 100 tasks are available, you can + paginate through them using :method:jobs/getrun. Use the + ``next_page_token`` field at the object root to determine if more + results are available. trigger: $ref: '#/components/schemas/TriggerType' trigger_info: $ref: '#/components/schemas/TriggerInfo' + version_id: + type: string + description: >- + ID of the deployment version that produced the job when this run was + created. Identifies a specific snapshot of the deployment in the + Deployment Metadata service. Only set for job runs of jobs with a + ``BUNDLE`` deployment. + CatalogTool: + type: object + properties: + name: + type: string CleanRoomTaskRunState: type: object properties: @@ -11882,6 +16806,11 @@ components: description: >- Information on how to access the output schema for the clean room run + shared_output_schema_info: + $ref: '#/components/schemas/OutputSchemaInfo' + description: >- + Information on how to access the shared output schema for the clean + room run ClusterInstance: type: object properties: @@ -11893,7 +16822,7 @@ components: The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to - `/#setting/sparkui/$cluster_id/$spark_context_id`. The Spark UI + ``/#setting/sparkui/$cluster_id/$spark_context_id``. The Spark UI continues to be available after the run has completed. The response won’t include this field if the identifier is not available yet. ClusterSpec: @@ -11905,7 +16834,7 @@ components: type: string description: >- If job_cluster_key, this task is executed reusing the cluster - specified in `job.settings.job_clusters`. + specified in ``job.settings.job_clusters``. libraries: type: string description: >- @@ -11934,6 +16863,29 @@ components: description: GPU type. required: - num_gpus + ComputeSpec: + type: object + properties: + accelerator_type: + $ref: '#/components/schemas/ComputeSpecAcceleratorType' + description: >- + Hardware accelerator type (for example, ``GPU_1xA10`` or + ``GPU_8xH100``). The number of accelerators per node is encoded in + the enum value — ``GPU_8xH100`` means 8 H100 GPUs per node. + accelerator_count: + type: integer + description: >- + Total number of accelerators across all nodes. Must be a positive + multiple of the per-node accelerator count encoded in + ``accelerator_type``. For example, ``GPU_8xH100`` with + ``accelerator_count: 16`` allocates 2 nodes (8 GPUs per node). + required: + - accelerator_type + - accelerator_count + description: >- + ComputeSpec: compute configuration — accelerator type and total + accelerator count across all + nodes. ConditionTask: type: object properties: @@ -11956,8 +16908,13 @@ components: Continuous: type: object properties: + maintenance_window: + $ref: '#/components/schemas/MaintenanceWindow' pause_status: $ref: '#/components/schemas/PauseStatus' + description: >- + Indicate whether the continuous execution of the job is paused or + not. Defaults to UNPAUSED. task_retry_mode: $ref: '#/components/schemas/TaskRetryMode' description: >- @@ -11979,12 +16936,18 @@ components: type: string description: >- A Java timezone ID. The schedule for a job is resolved with respect - to this timezone. See [Java TimeZone] for details. This field is - required. [Java TimeZone]: - https://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html + to this timezone. See `Java TimeZone + `__ + for details. This field is required. pause_status: $ref: '#/components/schemas/PauseStatus' description: Indicate whether this schedule is paused or not. + sql_condition: + $ref: '#/components/schemas/SqlConditionConfiguration' + description: >- + SQL condition that must be satisfied before a scheduled run is + triggered. The condition is evaluated after the cron expression + fires and must return a truthy result for the run to proceed. required: - quartz_cron_expression - timezone_id @@ -12011,12 +16974,12 @@ components: dashboard filters that have a matching URL identifier as the parameter key. The parameter value format is dependent on the filter type: - For text and single-select filters, provide a single value - (e.g. `"value"`) - For date and datetime filters, provide the value - in ISO 8601 format (e.g. `"2000-01-01T00:00:00"`) - For multi-select - filters, provide a JSON array of values (e.g. - `"[\"value1\",\"value2\"]"`) - For range and date range filters, - provide a JSON object with `start` and `end` (e.g. - `"{\"start\":\"1\",\"end\":\"10\"}"`) + (e.g. ``"value"``) - For date and datetime filters, provide the + value in ISO 8601 format (e.g. ``"2000-01-01T00:00:00"``) - For + multi-select filters, provide a JSON array of values (e.g. + ``"[\"value1\",\"value2\"]"``) - For range and date range filters, + provide a JSON object with ``start`` and ``end`` (e.g. + ``"{\"start\":\"1\",\"end\":\"10\"}"``) subscription: $ref: '#/components/schemas/Subscription' description: >- @@ -12036,6 +16999,11 @@ components: type: array items: $ref: '#/components/schemas/DashboardPageSnapshot' + DashboardTool: + type: object + properties: + dashboard_id: + type: string DbtCloudJobRunStep: type: object properties: @@ -12177,27 +17145,68 @@ components: type: string description: >- Optional schema to write to. This parameter is only used when a - warehouse_id is also provided. If not provided, the `default` schema - is used. + warehouse_id is also provided. If not provided, the ``default`` + schema is used. source: $ref: '#/components/schemas/Source' description: >- Optional location type of the project directory. When set to - `WORKSPACE`, the project will be retrieved from the local Databricks - workspace. When set to `GIT`, the project will be retrieved from a - Git repository defined in `git_source`. If the value is empty, the - task will use `GIT` if `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Project is located in Databricks - workspace. * `GIT`: Project is located in cloud Git provider. + ``WORKSPACE``, the project will be retrieved from the local + Databricks workspace. When set to ``GIT``, the project will be + retrieved from a Git repository defined in ``git_source``. If the + value is empty, the task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - ``WORKSPACE``: Project is + located in Databricks workspace. - ``GIT``: Project is located in + cloud Git provider. warehouse_id: type: string description: >- ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the - `--profiles-dir` command line argument. + ``--profiles-dir`` command line argument. required: - commands + DeploymentSpec: + type: object + properties: + command_path: + type: string + description: >- + Workspace path of the script to run on each node in this deployment. + Upload the script to this path and supply the path here. When the + task runs, the file at this path is run on each node; if it fails, + the task fails with its exit code. Example script contents: Plain + Python: python train.py --epochs 10 Multi-GPU via accelerate: + accelerate launch train.py --config config.yaml Distributed via + torchrun: torchrun --nproc_per_node=8 train.py + compute: + $ref: '#/components/schemas/ComputeSpec' + description: Compute resources allocated to each node in this deployment. + docker_image_url: + type: string + description: >- + Optional Docker image URL for a custom container image. When set, + the task runs on the specified container image instead of the + default Databricks client image. Format: + ``{organization}/{repository}:{tag}`` + name: + type: string + description: >- + Optional human-readable name for this deployment (for example, + ``driver``, ``worker``, ``param_server``). Used for log and UI + display. Distinct names are recommended so deployments can be told + apart, but uniqueness is not enforced. + required: + - command_path + - compute + description: >- + DeploymentSpec: configuration for one deployment within an + AiRuntimeTask. Each entry in + ``AiRuntimeTask.deployments`` describes a group of nodes that share the same command and + compute. Many single-program training algorithms use a single entry where every node runs the + same command; role-split workloads (driver + worker, parameter server, separate eval node, etc.) + use multiple entries. EnforcePolicyComplianceForJobResponseJobClusterSettingsChange: type: object properties: @@ -12323,7 +17332,7 @@ components: type: integer error_message: type: string - description: Describes the error message occured during the iterations. + description: Describes the error message occurred during the iterations. termination_category: type: string description: Describes the termination reason for the error message. @@ -12352,34 +17361,70 @@ components: properties: dl_runtime_image: type: string + description: Runtime image + client_version: + type: string + description: >- + Version of the client (e.g., sgcli wheel) that submitted this task. + Used by handlers to gate behavior or reject incompatible versions. + code_source_tar_path: + type: string + description: >- + Optional path to a tarball containing the user's workspace contents. + When set, the entry script extracts the tarball into the working + directory before running the training script, so the training script + can import sibling modules and read packaged data files. Must be a + workspace path (e.g. ``/Workspace/Users/...``) or volume; ``dbfs:/`` + is not supported. command: type: string description: Command launcher to run the actual script, e.g. bash, python etc. compute: $ref: '#/components/schemas/ComputeConfig' + docker_image_url: + type: string + description: >- + Optional custom Docker container image URL for running the training + script. Format: organization/repository:tag (e.g., + "pytorch/pytorch:2.0.1") mlflow_experiment_name: type: string description: >- Optional string containing the name of the MLflow experiment to log the run to. If name is not found, backend will create the mlflow experiment using the name. + mlflow_run_name: + type: string + description: >- + Optional name to assign to the MLflow run created for this task. If + unset, MLflow auto-generates a name. Used alongside + ``mlflow_experiment_name`` to identify the run in the MLflow UI. + requirements_yaml_path: + type: string + description: >- + Optional path to a requirements.yaml file describing pip + dependencies to install before running the training script. Consumed + by the entry script; format matches the runtime requirements.yaml + convention used by sgcli. Must be a workspace path (e.g. + ``/Workspace/Users/...``) or volume; ``dbfs:/`` is not supported. source: $ref: '#/components/schemas/Source' description: >- Optional location type of the training script. When set to - `WORKSPACE`, the script will be retrieved from the local Databricks - workspace. When set to `GIT`, the script will be retrieved from a - Git repository defined in `git_source`. If the value is empty, the - task will use `GIT` if `git_source` is defined and `WORKSPACE` - otherwise. * `WORKSPACE`: Script is located in Databricks workspace. - * `GIT`: Script is located in cloud Git provider. + ``WORKSPACE``, the script will be retrieved from the local + Databricks workspace. When set to ``GIT``, the script will be + retrieved from a Git repository defined in ``git_source``. If the + value is empty, the task will use ``GIT`` if ``git_source`` is + defined and ``WORKSPACE`` otherwise. - ``WORKSPACE``: Script is + located in Databricks workspace. - ``GIT``: Script is located in + cloud Git provider. training_script_path: type: string description: >- The training script file path to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the Databricks workspace, the path must - be absolute and begin with `/`. For files stored in a remote + be absolute and begin with ``/``. For files stored in a remote repository, the path must be relative. This field is required. yaml_parameters: type: string @@ -12394,6 +17439,60 @@ components: the training script. required: - dl_runtime_image + description: >- + DEPRECATED — use ``AiRuntimeTask`` for all new BYOT multi-node GPU + workloads (see + ai_runtime_task.proto). ``AiRuntimeTask`` is the only supported BYOT task type for new + workloads; this proto is retained only for AIR CLI (fka SGCLI) pywheel backwards compatibility + and will be removed once the pywheel → databricks-cli migration completes (post- PuPr). + GenieCodeApi: + type: object + properties: {} + description: >- + Marks the agentic task as a Genie Code conversation. The user query is + carried in + ``AgenticTask.input``; this message is a discriminator-only variant in ``AgenticTask.agent`` and + currently has no per-conversation configuration of its own. + GenieCodeOutput: + type: object + properties: + thread_id: + type: string + description: >- + Identifier of the Genie Code conversation thread for this run. In + Genie Code, threads are backed by workspace notebooks, so this value + also identifies the notebook a UI can deep-link to for the + conversation transcript. + description: >- + Output specific to the ``GenieCodeApi`` execution path of an agentic + task. + GenieSpaceTool: + type: object + properties: + space_id: + type: string + GenieTask: + type: object + properties: + configuration_id: + type: string + description: Required. Resource name of the agent task configuration to run. + required: + - configuration_id + description: >- + GenieTask runs a Genie or Genie Code agent task. The job task is a thin + reference to a + separately stored agent task configuration; all agent configuration (prompt, MCP connections, + referenced assets) lives on that object. + GenieTaskOutput: + type: object + properties: + conversation_id: + type: string + description: >- + The conversation ID of the agent run, used to retrieve the full + conversation history and results. + description: GenieTaskOutput is the output produced by a single Genie task run. GetJobPermissionLevelsResponse: type: object properties: @@ -12471,11 +17570,55 @@ components: source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. - If `git_source` is set, these tasks retrieve the file from the remote repository by default. - However, this behavior can be overridden by setting `source` to `WORKSPACE` on the task. + If ``git_source`` is set, these tasks retrieve the file from the remote repository by default. + However, this behavior can be overridden by setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks - are used, `git_source` must be defined on the job. + are used, ``git_source`` must be defined on the job. + IncrementalRefreshConfig: + type: object + properties: + archive_window_granularity: + $ref: '#/components/schemas/RefreshGranularity' + description: 'Granularity for the archive window. Default: MONTH' + archive_window_periods: + type: integer + description: >- + Number of periods in the archive window (total data retained). Must + be greater than the refresh window when using the same granularity. + Default: 3 (with YEAR granularity = 3 years) + detect_data_changes: + type: boolean + description: >- + If true, Power BI will check whether data has changed in each + partition before reimporting. Partitions with unchanged data are + skipped, reducing refresh cost further. Uses MAX(datetime_column) as + the change detector. + mode: + $ref: '#/components/schemas/RefreshPolicyMode' + description: >- + The refresh policy mode. Hybrid adds a real-time DirectQuery + partition for the most recent data; Import uses only batch-imported + partitions. Default: HYBRID + only_refresh_complete_periods: + type: boolean + description: >- + If true, only refresh complete periods (e.g., skip today's partial + data). + refresh_window_granularity: + $ref: '#/components/schemas/RefreshGranularity' + description: 'Granularity for the refresh window. Default: DAY' + refresh_window_periods: + type: integer + description: >- + Number of periods in the rolling refresh window. Partitions within + this window are re-imported on each refresh. Default: 1 + description: >- + Configuration for Power BI incremental refresh applied to all IMPORT + mode tables. When set, + IMPORT tables with incremental_refresh_datetime_column will use date-based partitioning for + incremental imports instead of full refreshes. The refresh windows and mode are shared across + all tables; the partition column is specified per-table on PowerBiTable. Job: type: object properties: @@ -12495,7 +17638,7 @@ components: The id of the budget policy used by this job for cost attribution purposes. This may be set through (in order of precedence): 1. Budget admins through the account or workspace console 2. Jobs UI in - the job details page and Jobs API using `budget_policy_id` 3. + the job details page and Jobs API using ``budget_policy_id`` 3. Inferred default based on accessible budget policies of the run_as identity on job creation or modification. effective_usage_policy_id: @@ -12506,31 +17649,37 @@ components: has_more: type: boolean description: >- - Indicates if the job has more array properties (`tasks`, - `job_clusters`) that are not shown. They can be accessed via + Indicates if the job has more array properties (``tasks``, + ``job_clusters``) that are not shown. They can be accessed via :method:jobs/get endpoint. It is only relevant for API 2.2 - :method:jobs/list requests with `expand_tasks=true`. + :method:jobs/list requests with ``expand_tasks=true``. job_id: type: integer description: The canonical identifier for this job. next_page_token: type: string description: A token that can be used to list the next page of array properties. + path: + type: string + description: >- + Path of the job object in workspace file tree, including file + extension. If absent, the job doesn't have a workspace object. + Example: /Workspace/user@example.com/my_project/my_job.job.json run_as_user_name: type: string description: >- The email of an active workspace user or the application ID of a service principal that the job runs as. This value can be changed by - setting the `run_as` field when creating or updating a job. By - default, `run_as_user_name` is based on the current job settings and - is set to the creator of the job if job access control is disabled - or to the user with the `is_owner` permission if job access control - is enabled. + setting the ``run_as`` field when creating or updating a job. By + default, ``run_as_user_name`` is based on the current job settings + and is set to the creator of the job if job access control is + disabled or to the user with the ``is_owner`` permission if job + access control is enabled. settings: $ref: '#/components/schemas/JobSettings' description: >- Settings for this job and all of its runs. These settings can be - updated using the `resetJob` method. + updated using the ``resetJob`` method. trigger_state: $ref: '#/components/schemas/TriggerStateProto' description: State of the trigger associated with the job. @@ -12577,6 +17726,13 @@ components: description: >- If new_cluster, a description of a cluster that is created for each task. + serverless_compute_id: + type: string + description: >- + The ID of the serverless compute object to bind this cluster to. At + most one JobCluster per job may set this field; the rate limit + defined on the referenced serverless compute applies across all + tasks bound to this cluster. required: - job_cluster_key - new_cluster @@ -12605,9 +17761,21 @@ components: properties: kind: $ref: '#/components/schemas/JobDeploymentKind' + deployment_id: + type: string + description: >- + ID of the deployment that manages this job. Only set when ``kind`` + is ``BUNDLE``. Used to look up deployment metadata from the + Deployment Metadata service. metadata_file_path: type: string description: Path of the file that contains deployment metadata. + version_id: + type: string + description: >- + ID of the version of the deployment that produced this job. Only set + when ``kind`` is ``BUNDLE``. Identifies a specific snapshot of the + deployment in the Deployment Metadata service. required: - kind JobEmailNotifications: @@ -12621,10 +17789,10 @@ components: type: string description: >- A list of email addresses to be notified when the duration of a run - exceeds the threshold specified for the `RUN_DURATION_SECONDS` - metric in the `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified in the `health` field for - the job, notifications are not sent. + exceeds the threshold specified for the ``RUN_DURATION_SECONDS`` + metric in the ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is specified in the ``health`` field + for the job, notifications are not sent. on_failure: type: array items: @@ -12632,10 +17800,24 @@ components: description: >- A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if - it ends with an `INTERNAL_ERROR` `life_cycle_state` or a `FAILED`, - or `TIMED_OUT` result_state. If this is not specified on job - creation, reset, or update the list is empty, and notifications are - not sent. + it ends with an ``INTERNAL_ERROR`` ``life_cycle_state`` or a + ``FAILED``, or ``TIMED_OUT`` result_state. If this is not specified + on job creation, reset, or update the list is empty, and + notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when platform-initiated + maintenance completes for a continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when platform-initiated + maintenance starts for a continuous job. on_start: type: array items: @@ -12651,11 +17833,11 @@ components: description: >- A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds - can be set in the `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or `STREAMING_BACKLOG_FILES`. Alerting - is based on the 10-minute average of these metrics. If the issue - persists, notifications are resent every 30 minutes. + can be set in the ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or ``STREAMING_BACKLOG_FILES``. + Alerting is based on the 10-minute average of these metrics. If the + issue persists, notifications are resent every 30 minutes. on_success: type: array items: @@ -12663,7 +17845,7 @@ components: description: >- A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it - ends with a `TERMINATED` `life_cycle_state` and a `SUCCESS` + ends with a ``TERMINATED`` ``life_cycle_state`` and a ``SUCCESS`` result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent. JobEnvironment: @@ -12675,6 +17857,69 @@ components: type: string required: - environment_key + JobEnvironmentVariables: + type: object + properties: + environment_variables_key: + type: string + description: >- + Identifier for this entry. Must be unique within + ``JobSettings.environment_variables``. Tasks reference it from + ``TaskSettings.environment_variables_key``. + files: + type: array + items: + type: string + description: >- + Workspace (``/Workspace/...``) or UC Volumes (``/Volumes/...``) + paths to ``.env`` files. Maximum 5 files. Files are read, parsed, + and merged at task execution time, not at job creation or update API + call time. File format: each line must be exactly ``KEY=VALUE``. + Keys must match the same regex as inlined variable names + (``^[A-Za-z_][A-Za-z0-9_]*$``); the value continues to the end of + the line. No other syntax is supported — no comments, no quoted + values, no escape sequences, no variable interpolation. Any line + that does not match the ``KEY=VALUE`` shape fails the run. Size + limits: maximum 1,048,576 bytes (1 MiB) per file on disk; maximum + 131,072 bytes (128 KiB) per ``KEY=VALUE`` line combined. Caps are + enforced at read time in jobs-runner — files exceeding the per-file + cap, or lines exceeding the per-line cap, fail the run. On a + duplicate key, the later file wins; ``variables`` override values + from any file. Values may contain {{secrets/scope/key}} references; + those are resolved at task execution time and never persisted in + resolved form. Do not use these files to store raw secret values; + consult `secret management + `__ for the + right way to pass sensitive values. + variables: + type: object + description: >- + Environment variables specified directly as key/value pairs (as + opposed to ``files``, which are read from ``.env`` file paths). + Maximum 100 entries. Each key must match + ``^[A-Za-z_][A-Za-z0-9_]*$`` and be 1 to 256 characters long. Each + value is up to 512 characters; larger values should be moved into a + ``.env`` file referenced from ``files``. On a duplicate key, + ``variables`` override values from any file in ``files``. Values may + contain {{secrets/scope/key}} references; those are resolved at task + execution time and never persisted in resolved form. Do not use this + field to store a raw secret value; consult `secret management + `__ for the + right way to pass sensitive values. + description: >- + A named environment-variable entry, defined once at the job level and + referenced by key from one + or more tasks. Entries live on ``JobSettings.environment_variables``, and tasks select one via + ``TaskSettings.environment_variables_key``. + + Precedence within a single entry (later overrides earlier on duplicate keys): + + files (merged in submitted order; later file in the list wins) └─▶ variables (inline k/v + override file contents) └─▶ runtime (os.environ / sys.env — overrides all, + classic-parity) + + Resolved values are never persisted: ``{{secrets/scope/key}}`` references in ``variables`` or in + ``files`` contents are stored verbatim and resolved at task execution time on the DP . JobNotificationSettings: type: object properties: @@ -12684,7 +17929,7 @@ components: type: boolean description: >- If true, do not send notifications to recipients specified in - `on_failure` if the run is skipped. + ``on_failure`` if the run is skipped. JobParameter: type: object properties: @@ -12748,7 +17993,7 @@ components: type: string description: >- Application ID of an active service principal. Setting this field - requires the `servicePrincipal/user` role. + requires the ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -12759,7 +18004,8 @@ components: runs as. If not specified, the job runs as the user who created the job. - Either `user_name` or `service_principal_name` should be specified. If not, an error is thrown. + Either ``user_name`` or ``service_principal_name`` should be specified. If not, an error is + thrown. JobSettings: type: object properties: @@ -12770,7 +18016,7 @@ components: description: >- An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only - one of `schedule` and `continuous` can be used. + one of ``schedule`` and ``continuous`` can be used. deployment: $ref: '#/components/schemas/JobDeployment' description: Deployment information for jobs managed by external sources. @@ -12782,14 +18028,24 @@ components: edit_mode: $ref: '#/components/schemas/JobEditMode' description: >- - Edit mode of the job. * `UI_LOCKED`: The job is in a locked UI state - and cannot be modified. * `EDITABLE`: The job is in an editable - state and can be modified. + Edit mode of the job. - ``UI_LOCKED``: The job is in a locked UI + state and cannot be modified. - ``EDITABLE``: The job is in an + editable state and can be modified. email_notifications: $ref: '#/components/schemas/JobEmailNotifications' description: >- An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted. + environment_variables: + type: array + items: + $ref: '#/components/schemas/JobEnvironmentVariables' + description: >- + Named environment-variable entries that tasks can reference by key + from ``TaskSettings.environment_variables_key``. Each entry holds + inline ``variables`` plus optional ``.env`` ``files``. Maximum 10 + entries per job. Entries are independent of one another — there is + no cross-entry merging. environments: type: array items: @@ -12807,18 +18063,18 @@ components: description: >- Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs API 2.1 this value is - always set to `"MULTI_TASK"`. + always set to ``"MULTI_TASK"``. git_source: $ref: '#/components/schemas/GitSource' description: >- An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If - `git_source` is set, these tasks retrieve the file from the remote + ``git_source`` is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by - setting `source` to `WORKSPACE` on the task. Note: dbt and SQL File - tasks support only version-controlled sources. If dbt or SQL File - tasks are used, `git_source` must be defined on the job. + setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL + File tasks support only version-controlled sources. If dbt or SQL + File tasks are used, ``git_source`` must be defined on the job. health: $ref: '#/components/schemas/JobsHealthRules' job_clusters: @@ -12842,8 +18098,8 @@ components: are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value - cannot exceed 1000. Setting this value to `0` causes all new runs to - be skipped. + cannot exceed 1000. Setting this value to ``0`` causes all new runs + to be skipped. name: type: string description: >- @@ -12853,21 +18109,26 @@ components: $ref: '#/components/schemas/JobNotificationSettings' description: >- Optional notification settings that are used when sending - notifications to each of the `email_notifications` and - `webhook_notifications` for this job. + notifications to each of the ``email_notifications`` and + ``webhook_notifications`` for this job. parameters: type: array items: $ref: '#/components/schemas/JobParameterDefinition' description: Job-level parameter definitions + parent_path: + type: string + description: >- + Path of the job parent folder in workspace file tree. If absent, the + job doesn't have a workspace object. performance_target: $ref: '#/components/schemas/PerformanceTarget' description: >- The performance mode on a serverless job. This field determines the level of compute performance or cost-efficiency for the run. The performance target does not apply to tasks that run on Serverless - GPU compute. * `STANDARD`: Enables cost-efficient execution of - serverless workloads. * `PERFORMANCE_OPTIMIZED`: Prioritizes fast + GPU compute. - ``STANDARD``: Enables cost-efficient execution of + serverless workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. queue: @@ -12878,14 +18139,14 @@ components: description: >- The user or service principal that the job runs as, if specified in the request. This field indicates the explicit configuration of - `run_as` for the job. To find the value in all cases, explicit or - implicit, use `run_as_user_name`. + ``run_as`` for the job. To find the value in all cases, explicit or + implicit, use ``run_as_user_name``. schedule: $ref: '#/components/schemas/CronSchedule' description: >- An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the - Jobs UI or sending an API request to `runNow`. + Jobs UI or sending an API request to ``runNow``. tags: type: object description: >- @@ -12903,27 +18164,27 @@ components: (:method:jobs/create, :method:jobs/reset, :method:jobs/update, :method:jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them using - :method:jobs/get. Use the `next_page_token` field at the object root - to determine if more results are available. + :method:jobs/get. Use the ``next_page_token`` field at the object + root to determine if more results are available. timeout_seconds: type: integer description: >- - An optional timeout applied to each run of this job. A value of `0` - means no timeout. + An optional timeout applied to each run of this job. A value of + ``0`` means no timeout. trigger: $ref: '#/components/schemas/TriggerSettings' description: >- A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or sending an API request to - `runNow`. + ``runNow``. usage_policy_id: type: string description: >- The id of the user specified usage policy to use for this job. If not specified, a default usage policy may be applied when creating - or modifying the job. See `effective_usage_policy_id` for the usage - policy used by this workload. + or modifying the job. See ``effective_usage_policy_id`` for the + usage policy used by this workload. webhook_notifications: $ref: '#/components/schemas/WebhookNotifications' description: >- @@ -12942,10 +18203,10 @@ components: $ref: '#/components/schemas/JobSourceDirtyState' description: >- Dirty state indicates the job is not fully synced with the job - specification in the remote repository. Possible values are: * - `NOT_SYNCED`: The job is not yet synced with the remote job + specification in the remote repository. Possible values are: - + ``NOT_SYNCED``: The job is not yet synced with the remote job specification. Import the remote job specification from UI to make - the job fully synced. * `DISCONNECTED`: The job is temporary + the job fully synced. - ``DISCONNECTED``: The job is temporary disconnected from the remote job specification and is allowed for live edit. Import the remote job specification again from UI to make the job fully synced. @@ -12979,6 +18240,11 @@ components: items: $ref: '#/components/schemas/JobsHealthRule' description: An optional set of health rules that can be defined for this job. + KnowledgeAssistantTool: + type: object + properties: + knowledge_assistant_id: + type: string ListJobComplianceForPolicyResponse: type: object properties: @@ -13050,6 +18316,36 @@ components: A list of runs, from most recently started to least. Only included in the response if there are runs to list. description: List of runs was retrieved successfully. + MaintenanceWindow: + type: object + properties: + start_hour: + type: integer + description: >- + An integer between 0 and 23 denoting the start hour for the + maintenance window in the 24-hour day. Platform-initiated + maintenance is triggered only within a one-hour window starting at + this hour. This field is required. + day_of_week: + $ref: '#/components/schemas/DayOfWeek' + description: >- + The day of week on which maintenance is allowed to happen. This + field is required. + timezone_id: + type: string + description: >- + A Java timezone ID. The maintenance window is resolved with respect + to this timezone. See `Java TimeZone + `__ + for details. This field is required. + required: + - start_hour + - day_of_week + - timezone_id + description: >- + A recurring weekly time window during which platform-initiated + maintenance is allowed to run for + a continuous job. ModelTriggerConfiguration: type: object properties: @@ -13103,31 +18399,30 @@ components: Base parameters to be used for each run of this job. If the run is initiated by a call to :method:jobs/run Now with parameters specified, the two parameters maps are merged. If the same key is - specified in `base_parameters` and in `run-now`, the value from - `run-now` is used. Use [Task parameter variables] to set parameters - containing information about job runs. If the notebook takes a - parameter that is not specified in the job’s `base_parameters` or - the `run-now` override parameters, the default value from the - notebook is used. Retrieve these parameters in a notebook using - [dbutils.widgets.get]. The JSON representation of this field cannot - exceed 1MB. [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables - [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-widgets + specified in ``base_parameters`` and in ``run-now``, the value from + ``run-now`` is used. Use `Task parameter variables + `__ to + set parameters containing information about job runs. If the + notebook takes a parameter that is not specified in the job’s + ``base_parameters`` or the ``run-now`` override parameters, the + default value from the notebook is used. Retrieve these parameters + in a notebook using `dbutils.widgets.get + `__. + The JSON representation of this field cannot exceed 1MB. source: $ref: '#/components/schemas/Source' description: >- - Optional location type of the notebook. When set to `WORKSPACE`, the - notebook will be retrieved from the local Databricks workspace. When - set to `GIT`, the notebook will be retrieved from a Git repository - defined in `git_source`. If the value is empty, the task will use - `GIT` if `git_source` is defined and `WORKSPACE` otherwise. * - `WORKSPACE`: Notebook is located in Databricks workspace. * `GIT`: - Notebook is located in cloud Git provider. + Optional location type of the notebook. When set to ``WORKSPACE``, + the notebook will be retrieved from the local Databricks workspace. + When set to ``GIT``, the notebook will be retrieved from a Git + repository defined in ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: Notebook is located in Databricks + workspace. - ``GIT``: Notebook is located in cloud Git provider. warehouse_id: type: string description: >- - Optional `warehouse_id` to run the notebook on a SQL warehouse. + Optional ``warehouse_id`` to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses. Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail. @@ -13165,6 +18460,32 @@ components: properties: full_refresh: type: boolean + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are unioned with other + selective refresh options (refresh_selection, + full_refresh_selection) to determine the final set of flows to + refresh. + refresh_selection: + type: array + items: + type: string + description: A list of tables to update without fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints without clearing + data. PipelineTask: type: object properties: @@ -13172,7 +18493,38 @@ components: type: string full_refresh: type: boolean - description: If true, triggers a full refresh on the delta live table. + description: If true, triggers a full refresh on the spark declarative pipeline. + full_refresh_selection: + type: array + items: + type: string + description: A list of tables to update with fullRefresh. + parameters: + type: object + description: >- + Key/value-map of parameters passed to the pipeline execution. + Limited to 10k characters in total. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are unioned with other + selective refresh options (refresh_selection, + full_refresh_selection) to determine the final set of flows to + refresh. + refresh_selection: + type: array + items: + type: string + description: A list of tables to update without fullRefresh. + reset_checkpoint_selection: + type: array + items: + type: string + description: >- + A list of streaming flows to reset checkpoints without clearing + data. required: - pipeline_id PowerBiModel: @@ -13197,6 +18549,13 @@ components: properties: catalog: type: string + incremental_refresh_datetime_column: + type: string + description: >- + The datetime column used for incremental refresh partitioning on + this table. e.g., "order_date", "updated_at" Only applicable when + the task has incremental_refresh_config set and this table uses + IMPORT storage mode. name: type: string description: The table name in Databricks @@ -13206,11 +18565,24 @@ components: storage_mode: $ref: '#/components/schemas/StorageMode' description: The Power BI storage mode of the table + table_type: + $ref: '#/components/schemas/PowerBiTableTableType' + description: >- + The Unity Catalog table type of this table. When set, the server + trusts it to classify the table during metric-view filtering and + skips the server-side lookup. When unset, the server resolves the + type on its own. PowerBiTask: type: object properties: connection_resource_name: type: string + incremental_refresh_config: + $ref: '#/components/schemas/IncrementalRefreshConfig' + description: >- + Incremental refresh policy applied to all IMPORT mode tables in the + model. Windows and mode are shared; partition columns are set + per-table on PowerBiTable. power_bi_model: $ref: '#/components/schemas/PowerBiModel' description: The semantic model to update @@ -13225,6 +18597,25 @@ components: warehouse_id: type: string description: The SQL warehouse ID to use as the Power BI data source + PythonOperatorTask: + type: object + properties: + main: + type: string + parameters: + type: array + items: + $ref: '#/components/schemas/PythonOperatorTaskParameter' + description: >- + An ordered list of task parameters. TODO(JOBS-30885): Add limits for + parameters. + PythonOperatorTaskParameter: + type: object + properties: + name: + type: string + value: + type: string PythonWheelTask: type: object properties: @@ -13235,20 +18626,20 @@ components: description: >- Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using - `$packageName.$entryPoint()` + ``$packageName.$entryPoint()`` named_parameters: type: object description: >- Command-line parameters passed to Python wheel task in the form of - `["--name=task", "--data=dbfs:/path/to/data.json"]`. Leave it empty - if `parameters` is not null. + ``["--name=task", "--data=dbfs:/path/to/data.json"]``. Leave it + empty if ``parameters`` is not null. parameters: type: array items: type: string description: >- Command-line parameters passed to Python wheel task. Leave it empty - if `named_parameters` is not null. + if ``named_parameters`` is not null. required: - package_name - entry_point @@ -13281,13 +18672,13 @@ components: type: integer description: >- The ID of the repair. Only returned for the items that represent a - repair in `repair_history`. + repair in ``repair_history``. start_time: type: integer description: The start time of the (repaired) run. state: $ref: '#/components/schemas/RunState' - description: Deprecated. Please use the `status` field instead. + description: Deprecated. Please use the ``status`` field instead. status: $ref: '#/components/schemas/RunStatus' task_run_ids: @@ -13309,7 +18700,7 @@ components: type: integer description: >- The ID of the repair. Must be provided in subsequent repairs using - the `latest_repair_id` field to ensure sequential repairs. + the ``latest_repair_id`` field to ensure sequential repairs. description: Run repair was initiated. ResolvedConditionTaskValues: type: object @@ -13335,6 +18726,11 @@ components: properties: parameters: type: object + ResolvedPipelineTaskValues: + type: object + properties: + parameters: + type: object ResolvedPythonWheelTaskValues: type: object properties: @@ -13361,12 +18757,24 @@ components: ResolvedValues: type: object properties: + agentic_task: + $ref: '#/components/schemas/ResolvedValuesAgenticTaskResolvedValues' + ai_runtime_task: + $ref: '#/components/schemas/ResolvedValuesAiRuntimeTaskResolvedValues' + description: >- + Resolved values for an AI Runtime task — env_vars with + ``{{tasks..values.}}`` references substituted to concrete + values before submission to the training service. + alert_task: + $ref: '#/components/schemas/ResolvedValuesAlertTaskResolvedValues' condition_task: $ref: '#/components/schemas/ResolvedConditionTaskValues' dbt_task: $ref: '#/components/schemas/ResolvedDbtTaskValues' notebook_task: $ref: '#/components/schemas/ResolvedNotebookTaskValues' + pipeline_task: + $ref: '#/components/schemas/ResolvedPipelineTaskValues' python_wheel_task: $ref: '#/components/schemas/ResolvedPythonWheelTaskValues' run_job_task: @@ -13381,6 +18789,29 @@ components: $ref: '#/components/schemas/ResolvedStringParamsValues' sql_task: $ref: '#/components/schemas/ResolvedParamPairValues' + ResolvedValuesAgenticTaskResolvedValues: + type: object + properties: + input: + type: string + description: The fully-resolved ``input`` prompt sent to the agent. + description: >- + Resolved values for an agentic task: the ``input`` prompt with parameter + references such as + ``{{tasks..values.}}`` replaced by the concrete values produced by upstream + tasks. + ResolvedValuesAiRuntimeTaskResolvedValues: + type: object + properties: {} + description: >- + Resolved values for an AiRuntimeTask after dynamic-value substitution, + so Jobs can expand + ``{{tasks..values.}}`` references before submission. + ResolvedValuesAlertTaskResolvedValues: + type: object + properties: + parameters: + type: object Run: type: object properties: @@ -13389,20 +18820,21 @@ components: description: >- The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial - run attempt fails, and the job has a retry policy (`max_retries` > - 0), subsequent runs are created with an `original_attempt_run_id` of - the original attempt’s ID and an incrementing `attempt_number`. Runs - are retried only until they succeed, and the maximum - `attempt_number` is the same as the `max_retries` value for the job. + run attempt fails, and the job has a retry policy (``max_retries`` > + 0), subsequent runs are created with an ``original_attempt_run_id`` + of the original attempt’s ID and an incrementing ``attempt_number``. + Runs are retried only until they succeed, and the maximum + ``attempt_number`` is the same as the ``max_retries`` value for the + job. cleanup_duration: type: integer description: >- The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum - of the `setup_duration`, `execution_duration`, and the - `cleanup_duration`. The `cleanup_duration` field is set to 0 for + of the ``setup_duration``, ``execution_duration``, and the + ``cleanup_duration``. The ``cleanup_duration`` field is set to 0 for multitask job runs. The total duration of a multitask job run is the - value of the `run_duration` field. + value of the ``run_duration`` field. cluster_instance: $ref: '#/components/schemas/ClusterInstance' description: >- @@ -13419,6 +18851,13 @@ components: description: >- The creator user name. This field won’t be included in the response if the user has already been deleted. + deployment_id: + type: string + description: >- + ID of the deployment that produced the job when this run was + created. Used to look up deployment metadata from the Deployment + Metadata service. Only set for job runs of jobs with a ``BUNDLE`` + deployment. description: type: string description: Description of the run @@ -13428,8 +18867,8 @@ components: The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported - by the job type. * `STANDARD`: Enables cost-efficient execution of - serverless workloads. * `PERFORMANCE_OPTIMIZED`: Prioritizes fast + by the job type. - ``STANDARD``: Enables cost-efficient execution of + serverless workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. effective_usage_policy_id: @@ -13443,34 +18882,45 @@ components: The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running. + environment_variables: + type: array + items: + $ref: '#/components/schemas/JobEnvironmentVariables' + description: >- + Snapshot of ``JobSettings.environment_variables`` as it was at run + launch — the full list of named env-var profiles the job defined. + Per-profile resolved contents are not preserved here; only the + customer-defined ``variables`` and ``files`` paths. To find which + profile a given task ran with, look at + ``RunTaskSettings.environment_variables_key``. execution_duration: type: integer description: >- The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is - the sum of the `setup_duration`, `execution_duration`, and the - `cleanup_duration`. The `execution_duration` field is set to 0 for - multitask job runs. The total duration of a multitask job run is the - value of the `run_duration` field. + the sum of the ``setup_duration``, ``execution_duration``, and the + ``cleanup_duration``. The ``execution_duration`` field is set to 0 + for multitask job runs. The total duration of a multitask job run is + the value of the ``run_duration`` field. git_source: $ref: '#/components/schemas/GitSource' description: >- An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If - `git_source` is set, these tasks retrieve the file from the remote + ``git_source`` is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by - setting `source` to `WORKSPACE` on the task. Note: dbt and SQL File - tasks support only version-controlled sources. If dbt or SQL File - tasks are used, `git_source` must be defined on the job. + setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL + File tasks support only version-controlled sources. If dbt or SQL + File tasks are used, ``git_source`` must be defined on the job. has_more: type: boolean description: >- - Indicates if the run has more array properties (`tasks`, - `job_clusters`) that are not shown. They can be accessed via + Indicates if the run has more array properties (``tasks``, + ``job_clusters``) that are not shown. They can be accessed via :method:jobs/getrun endpoint. It is only relevant for API 2.2 - :method:jobs/listruns requests with `expand_tasks=true`. + :method:jobs/listruns requests with ``expand_tasks=true``. iterations: type: array items: @@ -13510,7 +18960,7 @@ components: type: integer description: >- A unique identifier for this job run. This is set to the same value - as `run_id`. + as ``run_id``. original_attempt_run_id: type: integer description: >- @@ -13559,11 +19009,11 @@ components: The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The - duration of a task run is the sum of the `setup_duration`, - `execution_duration`, and the `cleanup_duration`. The - `setup_duration` field is set to 0 for multitask job runs. The total - duration of a multitask job run is the value of the `run_duration` - field. + duration of a task run is the sum of the ``setup_duration``, + ``execution_duration``, and the ``cleanup_duration``. The + ``setup_duration`` field is set to 0 for multitask job runs. The + total duration of a multitask job run is the value of the + ``run_duration`` field. start_time: type: integer description: >- @@ -13574,7 +19024,7 @@ components: issued. state: $ref: '#/components/schemas/RunState' - description: Deprecated. Please use the `status` field instead. + description: Deprecated. Please use the ``status`` field instead. status: $ref: '#/components/schemas/RunStatus' tasks: @@ -13583,14 +19033,22 @@ components: $ref: '#/components/schemas/RunTask' description: >- The list of tasks performed by the run. Each task has its own - `run_id` which you can use to call `JobsGetOutput` to retrieve the - run resutls. If more than 100 tasks are available, you can paginate - through them using :method:jobs/getrun. Use the `next_page_token` - field at the object root to determine if more results are available. + ``run_id`` which you can use to call ``JobsGetOutput`` to retrieve + the run results. If more than 100 tasks are available, you can + paginate through them using :method:jobs/getrun. Use the + ``next_page_token`` field at the object root to determine if more + results are available. trigger: $ref: '#/components/schemas/TriggerType' trigger_info: $ref: '#/components/schemas/TriggerInfo' + version_id: + type: string + description: >- + ID of the deployment version that produced the job when this run was + created. Identifies a specific snapshot of the deployment in the + Deployment Metadata service. Only set for job runs of jobs with a + ``BUNDLE`` deployment. description: Run was retrieved successfully RunConditionTask: type: object @@ -13611,7 +19069,7 @@ components: type: string description: >- The condition expression evaluation result. Filled in if the task - was successfully completed. Can be `"true"` or `"false"` + was successfully completed. Can be ``"true"`` or ``"false"`` required: - op - left @@ -13636,7 +19094,7 @@ components: $ref: '#/components/schemas/ForEachStats' description: >- Read only field. Populated for GetRun and ListRuns RPC calls and - stores the execution stats of an For each task + stores the execution stats of a ``For each`` task. required: - inputs - task @@ -13656,25 +19114,25 @@ components: type: string description: >- An array of commands to execute for jobs with the dbt task, for - example `"dbt_commands": ["dbt deps", "dbt seed", "dbt deps", "dbt - seed", "dbt run"]` ⚠ **Deprecation note** Use [job parameters] to - pass information down to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + example ``"dbt_commands": ["dbt deps", "dbt seed", "dbt deps", "dbt + seed", "dbt run"]`` ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. jar_params: type: array items: type: string description: >- A list of parameters for jobs with Spark JAR tasks, for example - `"jar_params": ["john doe", "35"]`. The parameters are used to + ``"jar_params": ["john doe", "35"]``. The parameters are used to invoke the main function of the main class specified in the Spark - JAR task. If not specified upon `run-now`, it defaults to an empty + JAR task. If not specified upon ``run-now``, it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed 10,000 bytes. ⚠ - **Deprecation note** Use [job parameters] to pass information down - to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. job_parameters: type: object description: Job-level parameters used to trigger the job. @@ -13682,18 +19140,18 @@ components: type: object description: >- A map from keys to values for jobs with notebook task, for example - `"notebook_params": {"name": "john doe", "age": "35"}`. The map is + ``"notebook_params": {"name": "john doe", "age": "35"}``. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. If not specified upon `run-now`, the - triggered run uses the job’s base parameters. notebook_params cannot - be specified in conjunction with jar_params. ⚠ **Deprecation note** - Use [job parameters] to pass information down to tasks. The JSON - representation of this field (for example - `{"notebook_params":{"name":"john doe","age":"35"}}`) cannot exceed - 10,000 bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, the triggered run uses + the job’s base parameters. notebook_params cannot be specified in + conjunction with jar_params. ⚠ **Deprecation note** Use `job + parameters + `__ + to pass information down to tasks. The JSON representation of this + field (for example ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 bytes. pipeline_params: $ref: '#/components/schemas/PipelineParams' description: Controls whether the pipeline should perform a full refresh @@ -13705,44 +19163,44 @@ components: type: string description: >- A list of parameters for jobs with Python tasks, for example - `"python_params": ["john doe", "35"]`. The parameters are passed to - Python file as command-line parameters. If specified upon `run-now`, - it would overwrite the parameters specified in job setting. The JSON - representation of this field (for example `{"python_params":["john - doe","35"]}`) cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use - [job parameters] to pass information down to tasks. Important These - parameters accept only Latin characters (ASCII character set). Using - non-ASCII characters returns an error. Examples of invalid, - non-ASCII characters are Chinese, Japanese kanjis, and emojis. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``"python_params": ["john doe", "35"]``. The parameters are passed + to Python file as command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job + setting. The JSON representation of this field (for example + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. + ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. Important These parameters accept + only Latin characters (ASCII character set). Using non-ASCII + characters returns an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and emojis. spark_submit_params: type: array items: type: string description: >- A list of parameters for jobs with spark submit task, for example - `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The parameters are passed to + ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script as command-line parameters. If specified upon - `run-now`, it would overwrite the parameters specified in job + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot exceed 10,000 bytes. ⚠ - **Deprecation note** Use [job parameters] to pass information down - to tasks. Important These parameters accept only Latin characters - (ASCII character set). Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are Chinese, Japanese - kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. + ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. Important These parameters accept + only Latin characters (ASCII character set). Using non-ASCII + characters returns an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL task, for example - `"sql_params": {"name": "john doe", "age": "35"}`. The SQL alert + ``"sql_params": {"name": "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ **Deprecation note** Use - [job parameters] to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. required: - job_id RunNowResponse: @@ -13752,7 +19210,7 @@ components: type: integer description: >- A unique identifier for this job run. This is set to the same value - as `run_id`. + as ``run_id``. run_id: type: integer description: The globally unique ID of the newly triggered run. @@ -13760,6 +19218,18 @@ components: RunOutput: type: object properties: + agentic_task_output: + $ref: '#/components/schemas/AgenticTaskOutput' + description: The output of an agentic task, if available + ai_runtime_task_output: + $ref: '#/components/schemas/AiRuntimeTaskOutput' + description: >- + The output of an AiRuntimeTask, if available — MLflow identifiers, + artifact paths, and per-replica allocated compute. Run lifecycle / + termination status lives on the surrounding framework + ``RunTask.status`` (``runs.proto:RunTask.status`` of type + ``RunStatus``), not on this output. See + ``tasks/genai/ai_runtime_task.proto:AiRuntimeTaskOutput``. alert_output: $ref: '#/components/schemas/AlertTaskOutput' description: The output of an alert task, if available @@ -13789,6 +19259,9 @@ components: description: >- If there was an error executing the run, this field contains any available stack traces. + genie_task_output: + $ref: '#/components/schemas/GenieTaskOutput' + description: The output of a Genie task, if available info: type: string logs: @@ -13810,12 +19283,12 @@ components: description: >- The output of a notebook task, if available. A notebook task that terminates (either successfully or with a failure) without calling - `dbutils.notebook.exit()` is considered to have an empty output. + ``dbutils.notebook.exit()`` is considered to have an empty output. This field is set but its result value is empty. Databricks restricts this API to return the first 5 MB of the output. To return - a larger result, use the [ClusterLogConf] field to configure log - storage for the job cluster. [ClusterLogConf]: - https://docs.databricks.com/dev-tools/api/latest/clusters.html#clusterlogconf + a larger result, use the `ClusterLogConf + `__ + field to configure log storage for the job cluster. run_job_output: $ref: '#/components/schemas/RunJobOutput' description: The output of a run job task, if available @@ -13836,31 +19309,31 @@ components: type: string description: >- A list of parameters for jobs with Spark JAR tasks, for example - `"jar_params": ["john doe", "35"]`. The parameters are used to + ``"jar_params": ["john doe", "35"]``. The parameters are used to invoke the main function of the main class specified in the Spark - JAR task. If not specified upon `run-now`, it defaults to an empty + JAR task. If not specified upon ``run-now``, it defaults to an empty list. jar_params cannot be specified in conjunction with notebook_params. The JSON representation of this field (for example - `{"jar_params":["john doe","35"]}`) cannot exceed 10,000 bytes. ⚠ - **Deprecation note** Use [job parameters] to pass information down - to tasks. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"jar_params":["john doe","35"]}``) cannot exceed 10,000 bytes. ⚠ + **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. notebook_params: type: object description: >- A map from keys to values for jobs with notebook task, for example - `"notebook_params": {"name": "john doe", "age": "35"}`. The map is + ``"notebook_params": {"name": "john doe", "age": "35"}``. The map is passed to the notebook and is accessible through the - [dbutils.widgets.get] function. If not specified upon `run-now`, the - triggered run uses the job’s base parameters. notebook_params cannot - be specified in conjunction with jar_params. ⚠ **Deprecation note** - Use [job parameters] to pass information down to tasks. The JSON - representation of this field (for example - `{"notebook_params":{"name":"john doe","age":"35"}}`) cannot exceed - 10,000 bytes. [dbutils.widgets.get]: - https://docs.databricks.com/dev-tools/databricks-utils.html [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `dbutils.widgets.get + `__ + function. If not specified upon ``run-now``, the triggered run uses + the job’s base parameters. notebook_params cannot be specified in + conjunction with jar_params. ⚠ **Deprecation note** Use `job + parameters + `__ + to pass information down to tasks. The JSON representation of this + field (for example ``{"notebook_params":{"name":"john + doe","age":"35"}}``) cannot exceed 10,000 bytes. pipeline_params: $ref: '#/components/schemas/PipelineParams' description: Controls whether the pipeline should perform a full refresh @@ -13872,44 +19345,44 @@ components: type: string description: >- A list of parameters for jobs with Python tasks, for example - `"python_params": ["john doe", "35"]`. The parameters are passed to - Python file as command-line parameters. If specified upon `run-now`, - it would overwrite the parameters specified in job setting. The JSON - representation of this field (for example `{"python_params":["john - doe","35"]}`) cannot exceed 10,000 bytes. ⚠ **Deprecation note** Use - [job parameters] to pass information down to tasks. Important These - parameters accept only Latin characters (ASCII character set). Using - non-ASCII characters returns an error. Examples of invalid, - non-ASCII characters are Chinese, Japanese kanjis, and emojis. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``"python_params": ["john doe", "35"]``. The parameters are passed + to Python file as command-line parameters. If specified upon + ``run-now``, it would overwrite the parameters specified in job + setting. The JSON representation of this field (for example + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. + ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. Important These parameters accept + only Latin characters (ASCII character set). Using non-ASCII + characters returns an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and emojis. spark_submit_params: type: array items: type: string description: >- A list of parameters for jobs with spark submit task, for example - `"spark_submit_params": ["--class", - "org.apache.spark.examples.SparkPi"]`. The parameters are passed to + ``"spark_submit_params": ["--class", + "org.apache.spark.examples.SparkPi"]``. The parameters are passed to spark-submit script as command-line parameters. If specified upon - `run-now`, it would overwrite the parameters specified in job + ``run-now``, it would overwrite the parameters specified in job setting. The JSON representation of this field (for example - `{"python_params":["john doe","35"]}`) cannot exceed 10,000 bytes. ⚠ - **Deprecation note** Use [job parameters] to pass information down - to tasks. Important These parameters accept only Latin characters - (ASCII character set). Using non-ASCII characters returns an error. - Examples of invalid, non-ASCII characters are Chinese, Japanese - kanjis, and emojis. [job parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + ``{"python_params":["john doe","35"]}``) cannot exceed 10,000 bytes. + ⚠ **Deprecation note** Use `job parameters + `__ + to pass information down to tasks. Important These parameters accept + only Latin characters (ASCII character set). Using non-ASCII + characters returns an error. Examples of invalid, non-ASCII + characters are Chinese, Japanese kanjis, and emojis. sql_params: type: object description: >- A map from keys to values for jobs with SQL task, for example - `"sql_params": {"name": "john doe", "age": "35"}`. The SQL alert + ``"sql_params": {"name": "john doe", "age": "35"}``. The SQL alert task does not support custom parameters. ⚠ **Deprecation note** Use - [job parameters] to pass information down to tasks. [job - parameters]: - https://docs.databricks.com/jobs/job-parameters.html#job-parameter-pushdown + `job parameters + `__ + to pass information down to tasks. RunState: type: object properties: @@ -13963,34 +19436,46 @@ components: from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset. + agentic_task: + $ref: '#/components/schemas/AgenticTask' + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + $ref: '#/components/schemas/AiRuntimeTask' + description: >- + The task runs a multi-gpu compute workload on Databricks AI Runtime. + Specify the accelerator type and count, the command to run, and + where the workload's code and MLflow output are stored. alert_task: $ref: '#/components/schemas/AlertTask' - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends notifications to + subscribers when the ``alert_task`` field is present. attempt_number: type: integer description: >- The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial - run attempt fails, and the job has a retry policy (`max_retries` > - 0), subsequent runs are created with an `original_attempt_run_id` of - the original attempt’s ID and an incrementing `attempt_number`. Runs - are retried only until they succeed, and the maximum - `attempt_number` is the same as the `max_retries` value for the job. + run attempt fails, and the job has a retry policy (``max_retries`` > + 0), subsequent runs are created with an ``original_attempt_run_id`` + of the original attempt’s ID and an incrementing ``attempt_number``. + Runs are retried only until they succeed, and the maximum + ``attempt_number`` is the same as the ``max_retries`` value for the + job. clean_rooms_notebook_task: $ref: '#/components/schemas/CleanRoomsNotebookTask' description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. [clean rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ notebook + when the ``clean_rooms_notebook_task`` field is present. cleanup_duration: type: integer description: >- The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum - of the `setup_duration`, `execution_duration`, and the - `cleanup_duration`. The `cleanup_duration` field is set to 0 for + of the ``setup_duration``, ``execution_duration``, and the + ``cleanup_duration``. The ``cleanup_duration`` field is set to 0 for multitask job runs. The total duration of a multitask job run is the - value of the `run_duration` field. + value of the ``run_duration`` field. cluster_instance: $ref: '#/components/schemas/ClusterInstance' description: >- @@ -14004,9 +19489,9 @@ components: $ref: '#/components/schemas/RunConditionTask' description: >- The task evaluates a condition that can be used to control the - execution of other tasks when the `condition_task` field is present. - The condition task does not require a cluster to execute and does - not support retries or notifications. + execution of other tasks when the ``condition_task`` field is + present. The condition task does not require a cluster to execute + and does not support retries or notifications. dashboard_task: $ref: '#/components/schemas/DashboardTask' description: The task refreshes a dashboard and sends a snapshot to subscribers. @@ -14020,9 +19505,9 @@ components: dbt_task: $ref: '#/components/schemas/DbtTask' description: >- - The task runs one or more dbt commands when the `dbt_task` field is - present. The dbt task requires both Databricks SQL and the ability - to use a serverless or a pro SQL warehouse. + The task runs one or more dbt commands when the ``dbt_task`` field + is present. The dbt task requires both Databricks SQL and the + ability to use a serverless or a pro SQL warehouse. depends_on: type: array items: @@ -14030,22 +19515,27 @@ components: description: >- An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully - before executing this task. The key is `task_key`, and the value is - the name assigned to the dependent task. + before executing this task. The key is ``task_key``, and the value + is the name assigned to the dependent task. description: type: string description: An optional description for this task. disable_auto_optimization: type: boolean description: An option to disable auto optimization in serverless + disabled: + type: boolean + description: >- + An optional flag to disable the task. If set to true, the task will + not run even if it is part of a job. effective_performance_target: $ref: '#/components/schemas/PerformanceTarget' description: >- The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported - by the job type. * `STANDARD`: Enables cost-efficient execution of - serverless workloads. * `PERFORMANCE_OPTIMIZED`: Prioritizes fast + by the job type. - ``STANDARD``: Enables cost-efficient execution of + serverless workloads. - ``PERFORMANCE_OPTIMIZED``: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance. email_notifications: @@ -14065,16 +19555,24 @@ components: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry defined in + ``RunSettings.environment_variables``. The selected entry's + variables and file contents are applied to this task at execution + time. Length and pattern mirror ``environment_key`` so the two + references look identical to customers reading task settings. execution_duration: type: integer description: >- The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is - the sum of the `setup_duration`, `execution_duration`, and the - `cleanup_duration`. The `execution_duration` field is set to 0 for - multitask job runs. The total duration of a multitask job run is the - value of the `run_duration` field. + the sum of the ``setup_duration``, ``execution_duration``, and the + ``cleanup_duration``. The ``execution_duration`` field is set to 0 + for multitask job runs. The total duration of a multitask job run is + the value of the ``run_duration`` field. existing_cluster_id: type: string description: >- @@ -14087,25 +19585,28 @@ components: $ref: '#/components/schemas/RunForEachTask' description: >- The task executes a nested task for every input provided when the - `for_each_task` field is present. + ``for_each_task`` field is present. gen_ai_compute_task: $ref: '#/components/schemas/GenAiComputeTask' + genie_task: + $ref: '#/components/schemas/GenieTask' + description: Runs a Genie or Genie Code agent task. git_source: $ref: '#/components/schemas/GitSource' description: >- An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If - `git_source` is set, these tasks retrieve the file from the remote + ``git_source`` is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by - setting `source` to `WORKSPACE` on the task. Note: dbt and SQL File - tasks support only version-controlled sources. If dbt or SQL File - tasks are used, `git_source` must be defined on the job. + setting ``source`` to ``WORKSPACE`` on the task. Note: dbt and SQL + File tasks support only version-controlled sources. If dbt or SQL + File tasks are used, ``git_source`` must be defined on the job. job_cluster_key: type: string description: >- If job_cluster_key, this task is executed reusing the cluster - specified in `job.settings.job_clusters`. + specified in ``job.settings.job_clusters``. libraries: type: string description: >- @@ -14116,9 +19617,9 @@ components: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the - `FAILED` result_state or `INTERNAL_ERROR` `life_cycle_state`. The - value `-1` means to retry indefinitely and the value `0` means to - never retry. + ``FAILED`` result_state or ``INTERNAL_ERROR`` ``life_cycle_state``. + The value ``-1`` means to retry indefinitely and the value ``0`` + means to never retry. min_retry_interval_millis: type: integer description: >- @@ -14132,28 +19633,33 @@ components: each run. notebook_task: $ref: '#/components/schemas/NotebookTask' - description: The task runs a notebook when the `notebook_task` field is present. + description: >- + The task runs a notebook when the ``notebook_task`` field is + present. notification_settings: $ref: '#/components/schemas/TaskNotificationSettings' description: >- Optional notification settings that are used when sending - notifications to each of the `email_notifications` and - `webhook_notifications` for this task run. + notifications to each of the ``email_notifications`` and + ``webhook_notifications`` for this task run. pipeline_task: $ref: '#/components/schemas/PipelineTask' description: >- - The task triggers a pipeline update when the `pipeline_task` field + The task triggers a pipeline update when the ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: $ref: '#/components/schemas/PowerBiTask' description: >- The task triggers a Power BI semantic model update when the - `power_bi_task` field is present. + ``power_bi_task`` field is present. + python_operator_task: + $ref: '#/components/schemas/PythonOperatorTask' + description: The task runs a Python operator task. python_wheel_task: $ref: '#/components/schemas/PythonWheelTask' description: >- - The task runs a Python wheel when the `python_wheel_task` field is + The task runs a Python wheel when the ``python_wheel_task`` field is present. queue_duration: type: integer @@ -14179,12 +19685,12 @@ components: description: >- An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. - When omitted, defaults to `ALL_SUCCESS`. See :method:jobs/create for - a list of possible values. + When omitted, defaults to ``ALL_SUCCESS``. See :method:jobs/create + for a list of possible values. run_job_task: $ref: '#/components/schemas/RunJobTask' description: >- - The task triggers another job when the `run_job_task` field is + The task triggers another job when the ``run_job_task`` field is present. run_page_url: type: string @@ -14194,18 +19700,18 @@ components: The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The - duration of a task run is the sum of the `setup_duration`, - `execution_duration`, and the `cleanup_duration`. The - `setup_duration` field is set to 0 for multitask job runs. The total - duration of a multitask job run is the value of the `run_duration` - field. + duration of a task run is the sum of the ``setup_duration``, + ``execution_duration``, and the ``cleanup_duration``. The + ``setup_duration`` field is set to 0 for multitask job runs. The + total duration of a multitask job run is the value of the + ``run_duration`` field. spark_jar_task: $ref: '#/components/schemas/SparkJarTask' - description: The task runs a JAR when the `spark_jar_task` field is present. + description: The task runs a JAR when the ``spark_jar_task`` field is present. spark_python_task: $ref: '#/components/schemas/SparkPythonTask' description: >- - The task runs a Python file when the `spark_python_task` field is + The task runs a Python file when the ``spark_python_task`` field is present. spark_submit_task: $ref: '#/components/schemas/SparkSubmitTask' @@ -14218,7 +19724,7 @@ components: $ref: '#/components/schemas/SqlTask' description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a - legacy SQL dashboard when the `sql_task` field is present. + legacy SQL dashboard when the ``sql_task`` field is present. start_time: type: integer description: >- @@ -14229,14 +19735,14 @@ components: issued. state: $ref: '#/components/schemas/RunState' - description: Deprecated. Please use the `status` field instead. + description: Deprecated. Please use the ``status`` field instead. status: $ref: '#/components/schemas/RunStatus' timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job task. A value of - `0` means no timeout. + ``0`` means no timeout. webhook_notifications: $ref: '#/components/schemas/WebhookNotifications' description: >- @@ -14246,6 +19752,16 @@ components: required: - task_key description: Used when outputting a child run, in GetRun or ListRuns. + SchemaTool: + type: object + properties: + name: + type: string + ServingEndpointTool: + type: object + properties: + name: + type: string SparkJarTask: type: object properties: @@ -14256,20 +19772,19 @@ components: description: >- The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a - library. The code must use `SparkContext.getOrCreate` to obtain a + library. The code must use ``SparkContext.getOrCreate`` to obtain a Spark context; otherwise, runs of the job fail. parameters: type: array items: type: string description: >- - Parameters passed to the main method. Use [Task parameter variables] - to set parameters containing information about job runs. [Task - parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Parameters passed to the main method. Use `Task parameter variables + `__ to + set parameters containing information about job runs. run_as_repl: type: boolean - description: Deprecated. A value of `false` is no longer supported. + description: Deprecated. A value of ``false`` is no longer supported. SparkPythonTask: type: object properties: @@ -14280,21 +19795,21 @@ components: items: type: string description: >- - Command line parameters passed to the Python file. Use [Task - parameter variables] to set parameters containing information about - job runs. [Task parameter variables]: - https://docs.databricks.com/jobs.html#parameter-variables + Command line parameters passed to the Python file. Use `Task + parameter variables + `__ to + set parameters containing information about job runs. source: $ref: '#/components/schemas/Source' description: >- - Optional location type of the Python file. When set to `WORKSPACE` + Optional location type of the Python file. When set to ``WORKSPACE`` or not specified, the file will be retrieved from the local - Databricks workspace or cloud location (if the `python_file` has a - URI format). When set to `GIT`, the Python file will be retrieved - from a Git repository defined in `git_source`. * `WORKSPACE`: The - Python file is located in a Databricks workspace or at a cloud - filesystem URI. * `GIT`: The Python file is located in a remote Git - repository. + Databricks workspace or cloud location (if the ``python_file`` has a + URI format). When set to ``GIT``, the Python file will be retrieved + from a Git repository defined in ``git_source``. - ``WORKSPACE``: + The Python file is located in a Databricks workspace or at a cloud + filesystem URI. - ``GIT``: The Python file is located in a remote + Git repository. required: - python_file SparkSubmitTask: @@ -14332,6 +19847,69 @@ components: warehouse_id: type: string description: The canonical identifier of the SQL warehouse. + SqlConditionConfiguration: + type: object + properties: + sql_query_id: + type: string + warehouse_id: + type: string + description: >- + The canonical identifier of the SQL warehouse to run the condition + query against. + trigger_mode: + $ref: '#/components/schemas/SqlConditionTriggerMode' + description: >- + Determines how the SQL query result is interpreted to decide whether + the condition fires. Must be set to a recognized value when + provided. When unset on an existing serialized configuration, the + server preserves the original semantics by interpreting it as + ``QUERY_RETURNS_ROWS``. New configurations should set this + explicitly — explicit ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED`` is + rejected at validation. + required: + - sql_query_id + - warehouse_id + SqlConditionRunInfoDetails: + type: object + properties: + condition_evaluation_satisfied: + type: boolean + description: >- + Whether the last condition evaluation was satisfied (query returned + truthy result). + condition_evaluation_sql_session_id: + type: string + description: >- + The ID of the SQL session, used by the UI to track session context. + Set for the QUERY_RETURNS_ROWS trigger mode. + condition_evaluation_sql_statement_id: + type: string + description: >- + The SQL statement ID of the condition evaluation, set when the + condition is evaluated by running a single SQL statement (the + RESULT_VALUE_CHANGES trigger mode). The UI uses it to link to the + query execution details. + description: >- + SQL condition evaluation details captured at the time the run was + triggered + SqlConditionState: + type: object + properties: + latest_condition_evaluation_satisfied: + type: boolean + latest_condition_evaluation_sql_session_id: + type: string + description: >- + The ID of the SQL session, used by UI to track session context. + Populated for QUERY_RETURNS_ROWS, which executes the query through + Redash. + latest_condition_evaluation_sql_statement_id: + type: string + description: >- + The SEA statement ID of the SQL statement executed for the latest + condition evaluation. Populated for RESULT_VALUE_CHANGES, which + executes the query through the SQL execution API. SqlDashboardOutput: type: object properties: @@ -14480,13 +20058,13 @@ components: source: $ref: '#/components/schemas/Source' description: >- - Optional location type of the SQL file. When set to `WORKSPACE`, the - SQL file will be retrieved from the local Databricks workspace. When - set to `GIT`, the SQL file will be retrieved from a Git repository - defined in `git_source`. If the value is empty, the task will use - `GIT` if `git_source` is defined and `WORKSPACE` otherwise. * - `WORKSPACE`: SQL file is located in Databricks workspace. * `GIT`: - SQL file is located in cloud Git provider. + Optional location type of the SQL file. When set to ``WORKSPACE``, + the SQL file will be retrieved from the local Databricks workspace. + When set to ``GIT``, the SQL file will be retrieved from a Git + repository defined in ``git_source``. If the value is empty, the + task will use ``GIT`` if ``git_source`` is defined and ``WORKSPACE`` + otherwise. - ``WORKSPACE``: SQL file is located in Databricks + workspace. - ``GIT``: SQL file is located in cloud Git provider. required: - path SqlTaskQuery: @@ -14519,15 +20097,26 @@ components: properties: task_key: type: string + agentic_task: + $ref: '#/components/schemas/AgenticTask' + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + $ref: '#/components/schemas/AiRuntimeTask' + description: >- + The task runs a multi-gpu compute workload on Databricks AI Runtime. + Specify the accelerator type and count, the command to run, and + where the workload's code and MLflow output are stored. alert_task: $ref: '#/components/schemas/AlertTask' - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends notifications to + subscribers when the ``alert_task`` field is present. clean_rooms_notebook_task: $ref: '#/components/schemas/CleanRoomsNotebookTask' description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. [clean rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ notebook + when the ``clean_rooms_notebook_task`` field is present. compute: $ref: '#/components/schemas/Compute' description: Task level compute configuration. @@ -14535,9 +20124,9 @@ components: $ref: '#/components/schemas/ConditionTask' description: >- The task evaluates a condition that can be used to control the - execution of other tasks when the `condition_task` field is present. - The condition task does not require a cluster to execute and does - not support retries or notifications. + execution of other tasks when the ``condition_task`` field is + present. The condition task does not require a cluster to execute + and does not support retries or notifications. dashboard_task: $ref: '#/components/schemas/DashboardTask' description: The task refreshes a dashboard and sends a snapshot to subscribers. @@ -14551,9 +20140,9 @@ components: dbt_task: $ref: '#/components/schemas/DbtTask' description: >- - The task runs one or more dbt commands when the `dbt_task` field is - present. The dbt task requires both Databricks SQL and the ability - to use a serverless or a pro SQL warehouse. + The task runs one or more dbt commands when the ``dbt_task`` field + is present. The dbt task requires both Databricks SQL and the + ability to use a serverless or a pro SQL warehouse. depends_on: type: array items: @@ -14561,14 +20150,19 @@ components: description: >- An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully - before executing this task. The key is `task_key`, and the value is - the name assigned to the dependent task. + before executing this task. The key is ``task_key``, and the value + is the name assigned to the dependent task. description: type: string description: An optional description for this task. disable_auto_optimization: type: boolean description: An option to disable auto optimization in serverless + disabled: + type: boolean + description: >- + An optional flag to disable the task. If set to true, the task will + not run even if it is part of a job. email_notifications: $ref: '#/components/schemas/JobEmailNotifications' description: >- @@ -14580,6 +20174,14 @@ components: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry defined in + ``RunSettings.environment_variables``. The selected entry's + variables and file contents are applied to this task at execution + time. Length and pattern mirror ``environment_key`` so the two + references look identical to customers reading task settings. existing_cluster_id: type: string description: >- @@ -14592,9 +20194,12 @@ components: $ref: '#/components/schemas/ForEachTask' description: >- The task executes a nested task for every input provided when the - `for_each_task` field is present. + ``for_each_task`` field is present. gen_ai_compute_task: $ref: '#/components/schemas/GenAiComputeTask' + genie_task: + $ref: '#/components/schemas/GenieTask' + description: Runs a Genie or Genie Code agent task. health: $ref: '#/components/schemas/JobsHealthRules' libraries: @@ -14607,9 +20212,9 @@ components: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the - `FAILED` result_state or `INTERNAL_ERROR` `life_cycle_state`. The - value `-1` means to retry indefinitely and the value `0` means to - never retry. + ``FAILED`` result_state or ``INTERNAL_ERROR`` ``life_cycle_state``. + The value ``-1`` means to retry indefinitely and the value ``0`` + means to never retry. min_retry_interval_millis: type: integer description: >- @@ -14623,28 +20228,33 @@ components: each run. notebook_task: $ref: '#/components/schemas/NotebookTask' - description: The task runs a notebook when the `notebook_task` field is present. + description: >- + The task runs a notebook when the ``notebook_task`` field is + present. notification_settings: $ref: '#/components/schemas/TaskNotificationSettings' description: >- Optional notification settings that are used when sending - notifications to each of the `email_notifications` and - `webhook_notifications` for this task run. + notifications to each of the ``email_notifications`` and + ``webhook_notifications`` for this task run. pipeline_task: $ref: '#/components/schemas/PipelineTask' description: >- - The task triggers a pipeline update when the `pipeline_task` field + The task triggers a pipeline update when the ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: $ref: '#/components/schemas/PowerBiTask' description: >- The task triggers a Power BI semantic model update when the - `power_bi_task` field is present. + ``power_bi_task`` field is present. + python_operator_task: + $ref: '#/components/schemas/PythonOperatorTask' + description: The task runs a Python operator task. python_wheel_task: $ref: '#/components/schemas/PythonWheelTask' description: >- - The task runs a Python wheel when the `python_wheel_task` field is + The task runs a Python wheel when the ``python_wheel_task`` field is present. retry_on_timeout: type: boolean @@ -14656,20 +20266,20 @@ components: description: >- An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. - When omitted, defaults to `ALL_SUCCESS`. See :method:jobs/create for - a list of possible values. + When omitted, defaults to ``ALL_SUCCESS``. See :method:jobs/create + for a list of possible values. run_job_task: $ref: '#/components/schemas/RunJobTask' description: >- - The task triggers another job when the `run_job_task` field is + The task triggers another job when the ``run_job_task`` field is present. spark_jar_task: $ref: '#/components/schemas/SparkJarTask' - description: The task runs a JAR when the `spark_jar_task` field is present. + description: The task runs a JAR when the ``spark_jar_task`` field is present. spark_python_task: $ref: '#/components/schemas/SparkPythonTask' description: >- - The task runs a Python file when the `spark_python_task` field is + The task runs a Python file when the ``spark_python_task`` field is present. spark_submit_task: $ref: '#/components/schemas/SparkSubmitTask' @@ -14682,12 +20292,12 @@ components: $ref: '#/components/schemas/SqlTask' description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a - legacy SQL dashboard when the `sql_task` field is present. + legacy SQL dashboard when the ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job task. A value of - `0` means no timeout. + ``0`` means no timeout. webhook_notifications: $ref: '#/components/schemas/WebhookNotifications' description: >- @@ -14718,7 +20328,57 @@ components: type: string description: >- A snapshot of the dashboard will be sent to the user's email when - the `user_name` field is present. + the ``user_name`` field is present. + SupervisorAgent: + type: object + properties: + agent_id: + type: string + description: Required. The ID of the supervisor agent (tile_id). + description: Configuration for a Supervisor Agent referenced by tile_id. + SupervisorAgentTool: + type: object + properties: + supervisor_agent_id: + type: string + description: Supervisor Agent ID. + description: Supervisor Agent referenced by another supervisor as a tool. + SupervisorApi: + type: object + properties: + instructions: + type: string + description: >- + Optional. System instructions that guide how the supervisor routes + queries across tools and synthesizes responses. + model: + type: string + description: >- + Required. Databricks-provided FMAPI model, e.g. + "databricks-claude-sonnet-4-5". + tools: + type: array + items: + $ref: '#/components/schemas/AgentTool' + description: Optional. The tools the inlined agent can use. + description: >- + Inlined Responses-API supervisor configuration. Used as a oneof variant + inside + ``AgenticTask.agent`` to define an agent without referencing a pre-registered Supervisor Agent + tile. + SupervisorOutput: + type: object + properties: + response_id: + type: string + description: >- + The Responses-API ``response.id`` produced by the supervisor. Use + this ID with the Responses API to fetch the full structured response + (assistant messages, function calls, function-call outputs). + description: >- + Output specific to the ``SupervisorAgent`` / ``SupervisorApi`` execution + path of an agentic + task. TableState: type: object properties: @@ -14728,7 +20388,12 @@ components: type: string description: >- Full table name of the table to monitor, e.g. - `mycatalog.myschema.mytable` + ``mycatalog.myschema.mytable`` + TableTool: + type: object + properties: + name: + type: string TableTriggerState: type: object properties: @@ -14769,15 +20434,26 @@ components: properties: task_key: type: string + agentic_task: + $ref: '#/components/schemas/AgenticTask' + description: Agentic Task for job-based multi-agent execution + ai_runtime_task: + $ref: '#/components/schemas/AiRuntimeTask' + description: >- + The task runs a multi-gpu compute workload on Databricks AI Runtime. + Specify the accelerator type and count, the command to run, and + where the workload's code and MLflow output are stored. alert_task: $ref: '#/components/schemas/AlertTask' - description: New alert v2 task + description: >- + The task evaluates a Databricks alert and sends notifications to + subscribers when the ``alert_task`` field is present. clean_rooms_notebook_task: $ref: '#/components/schemas/CleanRoomsNotebookTask' description: >- - The task runs a [clean rooms] notebook when the - `clean_rooms_notebook_task` field is present. [clean rooms]: - https://docs.databricks.com/clean-rooms/index.html + The task runs a `clean rooms + `__ notebook + when the ``clean_rooms_notebook_task`` field is present. compute: $ref: '#/components/schemas/Compute' description: Task level compute configuration. @@ -14785,9 +20461,9 @@ components: $ref: '#/components/schemas/ConditionTask' description: >- The task evaluates a condition that can be used to control the - execution of other tasks when the `condition_task` field is present. - The condition task does not require a cluster to execute and does - not support retries or notifications. + execution of other tasks when the ``condition_task`` field is + present. The condition task does not require a cluster to execute + and does not support retries or notifications. dashboard_task: $ref: '#/components/schemas/DashboardTask' description: The task refreshes a dashboard and sends a snapshot to subscribers. @@ -14801,9 +20477,9 @@ components: dbt_task: $ref: '#/components/schemas/DbtTask' description: >- - The task runs one or more dbt commands when the `dbt_task` field is - present. The dbt task requires both Databricks SQL and the ability - to use a serverless or a pro SQL warehouse. + The task runs one or more dbt commands when the ``dbt_task`` field + is present. The dbt task requires both Databricks SQL and the + ability to use a serverless or a pro SQL warehouse. depends_on: type: array items: @@ -14811,9 +20487,9 @@ components: description: >- An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before - executing this task. The task will run only if the `run_if` - condition is true. The key is `task_key`, and the value is the name - assigned to the dependent task. + executing this task. The task will run only if the ``run_if`` + condition is true. The key is ``task_key``, and the value is the + name assigned to the dependent task. description: type: string description: An optional description for this task. @@ -14837,6 +20513,14 @@ components: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute. + environment_variables_key: + type: string + description: >- + Reference to a ``JobEnvironmentVariables`` entry defined in + ``JobSettings.environment_variables``. The selected entry's + variables and file contents are applied to this task at execution + time. Length and pattern mirror ``environment_key`` so the two + references look identical to customers reading task settings. existing_cluster_id: type: string description: >- @@ -14849,16 +20533,19 @@ components: $ref: '#/components/schemas/ForEachTask' description: >- The task executes a nested task for every input provided when the - `for_each_task` field is present. + ``for_each_task`` field is present. gen_ai_compute_task: $ref: '#/components/schemas/GenAiComputeTask' + genie_task: + $ref: '#/components/schemas/GenieTask' + description: Runs a Genie or Genie Code agent task. health: $ref: '#/components/schemas/JobsHealthRules' job_cluster_key: type: string description: >- If job_cluster_key, this task is executed reusing the cluster - specified in `job.settings.job_clusters`. + specified in ``job.settings.job_clusters``. libraries: type: string description: >- @@ -14869,9 +20556,9 @@ components: description: >- An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the - `FAILED` result_state or `INTERNAL_ERROR` `life_cycle_state`. The - value `-1` means to retry indefinitely and the value `0` means to - never retry. + ``FAILED`` result_state or ``INTERNAL_ERROR`` ``life_cycle_state``. + The value ``-1`` means to retry indefinitely and the value ``0`` + means to never retry. min_retry_interval_millis: type: integer description: >- @@ -14885,28 +20572,33 @@ components: each run. notebook_task: $ref: '#/components/schemas/NotebookTask' - description: The task runs a notebook when the `notebook_task` field is present. + description: >- + The task runs a notebook when the ``notebook_task`` field is + present. notification_settings: $ref: '#/components/schemas/TaskNotificationSettings' description: >- Optional notification settings that are used when sending - notifications to each of the `email_notifications` and - `webhook_notifications` for this task. + notifications to each of the ``email_notifications`` and + ``webhook_notifications`` for this task. pipeline_task: $ref: '#/components/schemas/PipelineTask' description: >- - The task triggers a pipeline update when the `pipeline_task` field + The task triggers a pipeline update when the ``pipeline_task`` field is present. Only pipelines configured to use triggered more are supported. power_bi_task: $ref: '#/components/schemas/PowerBiTask' description: >- The task triggers a Power BI semantic model update when the - `power_bi_task` field is present. + ``power_bi_task`` field is present. + python_operator_task: + $ref: '#/components/schemas/PythonOperatorTask' + description: The task runs a Python operator task. python_wheel_task: $ref: '#/components/schemas/PythonWheelTask' description: >- - The task runs a Python wheel when the `python_wheel_task` field is + The task runs a Python wheel when the ``python_wheel_task`` field is present. retry_on_timeout: type: boolean @@ -14917,25 +20609,25 @@ components: $ref: '#/components/schemas/RunIf' description: >- An optional value specifying the condition determining whether the - task is run once its dependencies have been completed. * - `ALL_SUCCESS`: All dependencies have executed and succeeded * - `AT_LEAST_ONE_SUCCESS`: At least one dependency has succeeded * - `NONE_FAILED`: None of the dependencies have failed and at least one - was executed * `ALL_DONE`: All dependencies have been completed * - `AT_LEAST_ONE_FAILED`: At least one dependency failed * - `ALL_FAILED`: ALl dependencies have failed + task is run once its dependencies have been completed. - + ``ALL_SUCCESS``: All dependencies have executed and succeeded - + ``AT_LEAST_ONE_SUCCESS``: At least one dependency has succeeded - + ``NONE_FAILED``: None of the dependencies have failed and at least + one was executed - ``ALL_DONE``: All dependencies have been + completed - ``AT_LEAST_ONE_FAILED``: At least one dependency failed + - ``ALL_FAILED``: ALl dependencies have failed run_job_task: $ref: '#/components/schemas/RunJobTask' description: >- - The task triggers another job when the `run_job_task` field is + The task triggers another job when the ``run_job_task`` field is present. spark_jar_task: $ref: '#/components/schemas/SparkJarTask' - description: The task runs a JAR when the `spark_jar_task` field is present. + description: The task runs a JAR when the ``spark_jar_task`` field is present. spark_python_task: $ref: '#/components/schemas/SparkPythonTask' description: >- - The task runs a Python file when the `spark_python_task` field is + The task runs a Python file when the ``spark_python_task`` field is present. spark_submit_task: $ref: '#/components/schemas/SparkSubmitTask' @@ -14948,12 +20640,12 @@ components: $ref: '#/components/schemas/SqlTask' description: >- The task runs a SQL query or file, or it refreshes a SQL alert or a - legacy SQL dashboard when the `sql_task` field is present. + legacy SQL dashboard when the ``sql_task`` field is present. timeout_seconds: type: integer description: >- An optional timeout applied to each run of this job task. A value of - `0` means no timeout. + ``0`` means no timeout. webhook_notifications: $ref: '#/components/schemas/WebhookNotifications' description: >- @@ -14985,10 +20677,10 @@ components: type: string description: >- A list of email addresses to be notified when the duration of a run - exceeds the threshold specified for the `RUN_DURATION_SECONDS` - metric in the `health` field. If no rule for the - `RUN_DURATION_SECONDS` metric is specified in the `health` field for - the job, notifications are not sent. + exceeds the threshold specified for the ``RUN_DURATION_SECONDS`` + metric in the ``health`` field. If no rule for the + ``RUN_DURATION_SECONDS`` metric is specified in the ``health`` field + for the job, notifications are not sent. on_failure: type: array items: @@ -14996,10 +20688,24 @@ components: description: >- A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if - it ends with an `INTERNAL_ERROR` `life_cycle_state` or a `FAILED`, - or `TIMED_OUT` result_state. If this is not specified on job - creation, reset, or update the list is empty, and notifications are - not sent. + it ends with an ``INTERNAL_ERROR`` ``life_cycle_state`` or a + ``FAILED``, or ``TIMED_OUT`` result_state. If this is not specified + on job creation, reset, or update the list is empty, and + notifications are not sent. + on_maintenance_complete: + type: array + items: + type: string + description: >- + A list of email addresses to notify when platform-initiated + maintenance completes for a continuous job. + on_maintenance_start: + type: array + items: + type: string + description: >- + A list of email addresses to notify when platform-initiated + maintenance starts for a continuous job. on_start: type: array items: @@ -15015,11 +20721,11 @@ components: description: >- A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds - can be set in the `health` field using the following metrics: - `STREAMING_BACKLOG_BYTES`, `STREAMING_BACKLOG_RECORDS`, - `STREAMING_BACKLOG_SECONDS`, or `STREAMING_BACKLOG_FILES`. Alerting - is based on the 10-minute average of these metrics. If the issue - persists, notifications are resent every 30 minutes. + can be set in the ``health`` field using the following metrics: + ``STREAMING_BACKLOG_BYTES``, ``STREAMING_BACKLOG_RECORDS``, + ``STREAMING_BACKLOG_SECONDS``, or ``STREAMING_BACKLOG_FILES``. + Alerting is based on the 10-minute average of these metrics. If the + issue persists, notifications are resent every 30 minutes. on_success: type: array items: @@ -15027,7 +20733,7 @@ components: description: >- A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it - ends with a `TERMINATED` `life_cycle_state` and a `SUCCESS` + ends with a ``TERMINATED`` ``life_cycle_state`` and a ``SUCCESS`` result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent. TaskNotificationSettings: @@ -15039,12 +20745,26 @@ components: type: boolean description: >- If true, do not send notifications to recipients specified in - `on_failure` if the run is canceled. + ``on_failure`` if the run is canceled. no_alert_for_skipped_runs: type: boolean description: >- If true, do not send notifications to recipients specified in - `on_failure` if the run is skipped. + ``on_failure`` if the run is skipped. + TaskOutputSchema: + type: object + properties: + properties: + type: object + description: Map of property name to property definition. + description: >- + Single-layer JSON-Schema-style declaration of the structured output the + agent should produce. + TaskOutputSchemaProperty: + type: object + properties: + description: + type: string TerminationDetails: type: object properties: @@ -15057,12 +20777,37 @@ components: unstructured and the format might change. type: $ref: '#/components/schemas/TerminationTypeType' + TraceDestination: + type: object + properties: + catalog_name: + type: string + description: Unity Catalog catalog name for the trace table. + experiment_id: + type: string + description: MLflow experiment ID where traces are written. + schema_name: + type: string + description: Unity Catalog schema name for the trace table. + table_prefix: + type: string + description: >- + Table-name prefix under ``.``. Trace tables are + created as ``.._<...>``. + description: >- + Where MLflow traces produced by an agentic task run should be persisted. + Traces are written to + the given MLflow experiment, with their source data landing in a UC table at + ``.....``. TriggerInfo: type: object properties: run_id: type: integer description: The run id of the Run Job task run + sql_condition: + $ref: '#/components/schemas/SqlConditionRunInfoDetails' + description: SQL condition evaluation details for this run description: Additional details about what triggered the run TriggerSettings: type: object @@ -15077,6 +20822,12 @@ components: periodic: $ref: '#/components/schemas/PeriodicTriggerConfiguration' description: Periodic trigger settings. + sql_condition: + $ref: '#/components/schemas/SqlConditionConfiguration' + description: >- + SQL condition that must be satisfied for the trigger to fire. Can be + used in combination with other trigger types and runs *after* other + trigger types conditions are evaluated. table_update: $ref: '#/components/schemas/TableUpdateTriggerConfiguration' TriggerStateProto: @@ -15084,8 +20835,40 @@ components: properties: file_arrival: $ref: '#/components/schemas/FileArrivalTriggerState' + pause_status: + $ref: '#/components/schemas/PauseStatus' + description: >- + Whether this trigger is paused or not. For continuous schedules, it + can differ from the configured pause_status whenever a paused + continuous job is kickstarted by an operation other than an update, + such as a run-now. + sql_condition: + $ref: '#/components/schemas/SqlConditionState' + description: >- + State for SQL condition evaluation, can coexist with other trigger + states. table: $ref: '#/components/schemas/TableTriggerState' + UcConnectionTool: + type: object + properties: + name: + type: string + UcFunctionTool: + type: object + properties: + name: + type: string + UcMcpTool: + type: object + properties: + name: + type: string + VectorSearchIndexTool: + type: object + properties: + name: + type: string ViewItem: type: object properties: @@ -15100,6 +20883,15 @@ components: type: $ref: '#/components/schemas/ViewType' description: Type of the view item. + VolumeTool: + type: object + properties: + name: + type: string + WebSearchTool: + type: object + properties: {} + description: Built-in web-search tool. No per-asset configuration today. Webhook: type: object properties: @@ -15121,7 +20913,25 @@ components: description: >- An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the - `on_failure` property. + ``on_failure`` property. + on_maintenance_complete: + type: array + items: + $ref: '#/components/schemas/Webhook' + description: >- + An optional list of system notification IDs to call when + platform-initiated maintenance completes for a continuous job. A + maximum of 3 destinations can be specified for the + ``on_maintenance_complete`` property. + on_maintenance_start: + type: array + items: + $ref: '#/components/schemas/Webhook' + description: >- + An optional list of system notification IDs to call when + platform-initiated maintenance starts for a continuous job. A + maximum of 3 destinations can be specified for the + ``on_maintenance_start`` property. on_start: type: array items: @@ -15129,7 +20939,7 @@ components: description: >- An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the - `on_start` property. + ``on_start`` property. on_streaming_backlog_exceeded: type: array items: @@ -15137,13 +20947,13 @@ components: description: >- An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming - backlog thresholds can be set in the `health` field using the - following metrics: `STREAMING_BACKLOG_BYTES`, - `STREAMING_BACKLOG_RECORDS`, `STREAMING_BACKLOG_SECONDS`, or - `STREAMING_BACKLOG_FILES`. Alerting is based on the 10-minute + backlog thresholds can be set in the ``health`` field using the + following metrics: ``STREAMING_BACKLOG_BYTES``, + ``STREAMING_BACKLOG_RECORDS``, ``STREAMING_BACKLOG_SECONDS``, or + ``STREAMING_BACKLOG_FILES``. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be - specified for the `on_streaming_backlog_exceeded` property. + specified for the ``on_streaming_backlog_exceeded`` property. on_success: type: array items: @@ -15151,7 +20961,7 @@ components: description: >- An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified - for the `on_success` property. + for the ``on_success`` property. WidgetErrorDetail: type: object properties: @@ -15248,6 +21058,18 @@ components: original definition to avoid cyclic dependency. + ComputeSpecAcceleratorType: + type: string + x-enum: + - GPU_1xA10 + - GPU_1xH100 + - GPU_8xB300 + - GPU_8xH100 + description: >- + Hardware accelerator type for the AiRuntime workload. Per-node + accelerator count is encoded in + + the value name (e.g. ``GPU_8xH100`` means 8 H100s per node). Condition: type: string x-enum: @@ -15300,28 +21122,32 @@ components: - LESS_THAN_OR_EQUAL - NOT_EQUAL description: >- - * `EQUAL_TO`, `NOT_EQUAL` operators perform string comparison of their - operands. This means that - - `“12.0” == “12”` will evaluate to `false`. * `GREATER_THAN`, - `GREATER_THAN_OR_EQUAL`, - - `LESS_THAN`, `LESS_THAN_OR_EQUAL` operators perform numeric comparison - of their operands. - - `“12.0” >= “12”` will evaluate to `true`, `“10.0” >= “12”` will evaluate - to - - `false`. - + - ``EQUAL_TO``, ``NOT_EQUAL`` operators perform string comparison of + their operands. This means + that ``“12.0” == “12”`` will evaluate to ``false``. + - ``GREATER_THAN``, ``GREATER_THAN_OR_EQUAL``, ``LESS_THAN``, + ``LESS_THAN_OR_EQUAL`` operators + perform numeric comparison of their operands. ``“12.0” >= “12”`` will evaluate to + ``true``, ``“10.0” >= “12”`` will evaluate to ``false``. The boolean comparison to task values can be implemented with operators - `EQUAL_TO`, `NOT_EQUAL`. + ``EQUAL_TO``, - If a task value was set to a boolean value, it will be serialized to - `“true”` or + ``NOT_EQUAL``. If a task value was set to a boolean value, it will be + serialized to - `“false”` for the comparison. + ``“true”`` or ``“false”`` for the comparison. + DayOfWeek: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: Days of week that can be referenced by Jobs scheduling settings. DbtPlatformRunStatus: type: string x-enum: @@ -15429,24 +21255,19 @@ components: x-enum: - BUNDLE - SYSTEM_MANAGED - description: >- - * `BUNDLE`: The job is managed by Databricks Asset Bundle. * - `SYSTEM_MANAGED`: The job is - - managed by Databricks and is read-only. + description: |- + - ``BUNDLE``: The job is managed by Databricks Asset Bundle. + - ``SYSTEM_MANAGED``: The job is managed by Databricks and is read-only. JobEditMode: type: string x-enum: - EDITABLE - UI_LOCKED - description: >- + description: |- Edit mode of the job. - - * `UI_LOCKED`: The job is in a locked UI state and cannot be modified. * - `EDITABLE`: The job is - - in an editable state and can be modified. + - ``UI_LOCKED``: The job is in a locked UI state and cannot be modified. + - ``EDITABLE``: The job is in an editable state and can be modified. JobPermissionLevel: type: string x-enum: @@ -15467,19 +21288,16 @@ components: repository. - Possible values are: * `NOT_SYNCED`: The job is not yet synced with the - remote job - - specification. Import the remote job specification from UI to make the - job fully synced. * + Possible values are: - `DISCONNECTED`: The job is temporary disconnected from the remote job - specification and is - allowed for live edit. Import the remote job specification again from UI - to make the job fully - - synced. + - ``NOT_SYNCED``: The job is not yet synced with the remote job + specification. Import the remote + job specification from UI to make the job fully synced. + - ``DISCONNECTED``: The job is temporary disconnected from the remote + job specification and is + allowed for live edit. Import the remote job specification again from UI to make the job fully + synced. JobsHealthMetric: type: string x-enum: @@ -15493,25 +21311,20 @@ components: health rule. - * `RUN_DURATION_SECONDS`: Expected total time for a run in seconds. * - `STREAMING_BACKLOG_BYTES`: - - An estimate of the maximum bytes of data waiting to be consumed across - all streams. This metric - - is in Public Preview. * `STREAMING_BACKLOG_RECORDS`: An estimate of the - maximum offset lag - - across all streams. This metric is in Public Preview. * - `STREAMING_BACKLOG_SECONDS`: An estimate + - ``RUN_DURATION_SECONDS``: Expected total time for a run in seconds. - of the maximum consumer delay across all streams. This metric is in - Public Preview. * - - `STREAMING_BACKLOG_FILES`: An estimate of the maximum number of + - ``STREAMING_BACKLOG_BYTES``: An estimate of the maximum bytes of data + waiting to be consumed + across all streams. This metric is in Public Preview. + - ``STREAMING_BACKLOG_RECORDS``: An estimate of the maximum offset lag + across all streams. This + metric is in Public Preview. + - ``STREAMING_BACKLOG_SECONDS``: An estimate of the maximum consumer + delay across all streams. + This metric is in Public Preview. + - ``STREAMING_BACKLOG_FILES``: An estimate of the maximum number of outstanding files across all - - streams. This metric is in Public Preview. + streams. This metric is in Public Preview. JobsHealthOperator: type: string x-enum: @@ -15623,6 +21436,7 @@ components: x-enum: - DAYS - HOURS + - MINUTES - WEEKS description: |- Create a collection of name/value pairs. @@ -15661,6 +21475,23 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + PowerBiTableTableType: + type: string + x-enum: + - EXTERNAL + - EXTERNAL_SHALLOW_CLONE + - FOREIGN + - MANAGED + - MANAGED_SHALLOW_CLONE + - MATERIALIZED_VIEW + - METRIC_VIEW + - STREAMING_TABLE + - VIEW + description: >- + The Unity Catalog table type of a table. Values mirror the Unity Catalog + TableType enum so + + classification stays aligned across the two APIs. QueueDetailsCodeCode: type: string x-enum: @@ -15668,18 +21499,32 @@ components: - ACTIVE_RUN_JOB_TASKS_LIMIT_REACHED - MAX_CONCURRENT_RUNS_REACHED description: >- - The reason for queuing the run. * `ACTIVE_RUNS_LIMIT_REACHED`: The run - was queued due to - - reaching the workspace limit of active task runs. * - `MAX_CONCURRENT_RUNS_REACHED`: The run was - - queued due to reaching the per-job limit of concurrent job runs. * - - `ACTIVE_RUN_JOB_TASKS_LIMIT_REACHED`: The run was queued due to reaching - the workspace limit of - - active run job tasks. + The reason for queuing the run. + + + - ``ACTIVE_RUNS_LIMIT_REACHED``: The run was queued due to reaching the + workspace limit of + active task runs. + - ``MAX_CONCURRENT_RUNS_REACHED``: The run was queued due to reaching + the per-job limit of + concurrent job runs. + - ``ACTIVE_RUN_JOB_TASKS_LIMIT_REACHED``: The run was queued due to + reaching the workspace limit + of active run job tasks. + RefreshGranularity: + type: string + x-enum: + - REFRESH_GRANULARITY_DAY + - REFRESH_GRANULARITY_MONTH + - REFRESH_GRANULARITY_QUARTER + - REFRESH_GRANULARITY_YEAR + description: Granularity for incremental refresh windows. + RefreshPolicyMode: + type: string + x-enum: + - REFRESH_POLICY_MODE_HYBRID + - REFRESH_POLICY_MODE_IMPORT_ONLY + description: The refresh policy mode for incremental refresh. RepairHistoryItemType: type: string x-enum: @@ -15702,22 +21547,24 @@ components: task should be run once its dependencies have been completed. When omitted, defaults to - `ALL_SUCCESS`. + ``ALL_SUCCESS``. + + + Possible values are: - Possible values are: * `ALL_SUCCESS`: All dependencies have executed and - succeeded * + - ``ALL_SUCCESS``: All dependencies have executed and succeeded - `AT_LEAST_ONE_SUCCESS`: At least one dependency has succeeded * - `NONE_FAILED`: None of the + - ``AT_LEAST_ONE_SUCCESS``: At least one dependency has succeeded - dependencies have failed and at least one was executed * `ALL_DONE`: All - dependencies have been + - ``NONE_FAILED``: None of the dependencies have failed and at least one + was executed - completed * `AT_LEAST_ONE_FAILED`: At least one dependency failed * - `ALL_FAILED`: ALl + - ``ALL_DONE``: All dependencies have been completed - dependencies have failed + - ``AT_LEAST_ONE_FAILED``: At least one dependency failed + + - ``ALL_FAILED``: ALl dependencies have failed RunLifeCycleState: type: string x-enum: @@ -15731,38 +21578,32 @@ components: - TERMINATING - WAITING_FOR_RETRY description: >- - A value indicating the run's lifecycle state. The possible values are: * - `QUEUED`: The run is - - queued. * `PENDING`: The run is waiting to be executed while the cluster - and execution context - - are being prepared. * `RUNNING`: The task of this run is being executed. - * `TERMINATING`: The - - task of this run has completed, and the cluster and execution context - are being cleaned up. * - - `TERMINATED`: The task of this run has completed, and the cluster and - execution context have - - been cleaned up. This state is terminal. * `SKIPPED`: This run was - aborted because a previous - - run of the same job was already active. This state is terminal. * - `INTERNAL_ERROR`: An - - exceptional state that indicates a failure in the Jobs service, such as - network failure over a - - long period. If a run on a new cluster ends in the `INTERNAL_ERROR` - state, the Jobs service - - terminates the cluster as soon as possible. This state is terminal. * - `BLOCKED`: The run is - - blocked on an upstream dependency. * `WAITING_FOR_RETRY`: The run is - waiting for a retry. + A value indicating the run's lifecycle state. The possible values are: + + + - ``QUEUED``: The run is queued. + + - ``PENDING``: The run is waiting to be executed while the cluster and + execution context are + being prepared. + - ``RUNNING``: The task of this run is being executed. + + - ``TERMINATING``: The task of this run has completed, and the cluster + and execution context are + being cleaned up. + - ``TERMINATED``: The task of this run has completed, and the cluster + and execution context have + been cleaned up. This state is terminal. + - ``SKIPPED``: This run was aborted because a previous run of the same + job was already active. + This state is terminal. + - ``INTERNAL_ERROR``: An exceptional state that indicates a failure in + the Jobs service, such as + network failure over a long period. If a run on a new cluster ends in the ``INTERNAL_ERROR`` + state, the Jobs service terminates the cluster as soon as possible. This state is terminal. + - ``BLOCKED``: The run is blocked on an upstream dependency. + + - ``WAITING_FOR_RETRY``: The run is waiting for a retry. RunLifecycleStateV2State: type: string x-enum: @@ -15788,31 +21629,34 @@ components: - UPSTREAM_CANCELED - UPSTREAM_FAILED description: >- - A value indicating the run's result. The possible values are: * - `SUCCESS`: The task completed + A value indicating the run's result. The possible values are: + - successfully. * `FAILED`: The task completed with an error. * - `TIMEDOUT`: The run was stopped + - ``SUCCESS``: The task completed successfully. - after reaching the timeout. * `CANCELED`: The run was canceled at user - request. * + - ``FAILED``: The task completed with an error. - `MAXIMUM_CONCURRENT_RUNS_REACHED`: The run was skipped because the - maximum concurrent runs were + - ``TIMEDOUT``: The run was stopped after reaching the timeout. - reached. * `EXCLUDED`: The run was skipped because the necessary - conditions were not met. * + - ``CANCELED``: The run was canceled at user request. - `SUCCESS_WITH_FAILURES`: The job run completed successfully with some - failures; leaf tasks were + - ``MAXIMUM_CONCURRENT_RUNS_REACHED``: The run was skipped because the + maximum concurrent runs + were reached. + - ``EXCLUDED``: The run was skipped because the necessary conditions + were not met. - successful. * `UPSTREAM_FAILED`: The run was skipped because of an - upstream failure. * + - ``SUCCESS_WITH_FAILURES``: The job run completed successfully with + some failures; leaf tasks + were successful. + - ``UPSTREAM_FAILED``: The run was skipped because of an upstream + failure. - `UPSTREAM_CANCELED`: The run was skipped because an upstream task was - canceled. * `DISABLED`: + - ``UPSTREAM_CANCELED``: The run was skipped because an upstream task + was canceled. - The run was skipped because it was disabled explicitly by the user. + - ``DISABLED``: The run was skipped because it was disabled explicitly + by the user. RunType: type: string x-enum: @@ -15820,51 +21664,69 @@ components: - SUBMIT_RUN - WORKFLOW_RUN description: >- - The type of a run. * `JOB_RUN`: Normal job run. A run created with - :method:jobs/runNow. * + The type of a run. - `WORKFLOW_RUN`: Workflow run. A run created with [dbutils.notebook.run]. - * `SUBMIT_RUN`: Submit - run. A run created with :method:jobs/submit. + - ``JOB_RUN``: Normal job run. A run created with :method:jobs/runNow. - - [dbutils.notebook.run]: - https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-workflow + - ``WORKFLOW_RUN``: Workflow run. A run created with + `dbutils.notebook.run + `__. + - ``SUBMIT_RUN``: Submit run. A run created with :method:jobs/submit. Source: type: string x-enum: - GIT - WORKSPACE description: >- - Optional location type of the SQL file. When set to `WORKSPACE`, the SQL - file will be retrieved from the local Databricks workspace. When set - to `GIT`, the SQL file will be retrieved from a + Optional location type of the SQL file. When set to ``WORKSPACE``, the + SQL file will be + + retrieved from the local Databricks workspace. When set to ``GIT``, the + SQL file will be - Git repository defined in `git_source`. If the value is empty, the task - will use `GIT` if + retrieved from a Git repository defined in ``git_source``. If the value + is empty, the task will - `git_source` is defined and `WORKSPACE` otherwise. + use ``GIT`` if ``git_source`` is defined and ``WORKSPACE`` otherwise. - * `WORKSPACE`: SQL file is located in Databricks workspace. * `GIT`: SQL - file is located in + - ``WORKSPACE``: SQL file is located in Databricks workspace. - cloud Git provider. + - ``GIT``: SQL file is located in cloud Git provider. SqlAlertState: type: string x-enum: - OK - TRIGGERED - UNKNOWN - description: >- + description: |- The state of the SQL alert. - - * UNKNOWN: alert yet to be evaluated * OK: alert evaluated and did not - fulfill trigger - - conditions * TRIGGERED: alert evaluated and fulfilled trigger conditions + - UNKNOWN: alert yet to be evaluated + - OK: alert evaluated and did not fulfill trigger conditions + - TRIGGERED: alert evaluated and fulfilled trigger conditions + SqlConditionTriggerMode: + type: string + x-enum: + - QUERY_RETURNS_ROWS + - RESULT_VALUE_CHANGES + description: >- + The strategy used to evaluate a SQL condition trigger against a query + result set. + + + - ``SQL_CONDITION_TRIGGER_MODE_UNSPECIFIED``: Sentinel zero-value. Not a + valid input — the + validator rejects this when sent explicitly. Internally treated as ``QUERY_RETURNS_ROWS`` when + reading legacy data that predates this field. + - ``QUERY_RETURNS_ROWS``: Fires whenever the result set has at least one + row. Zero rows means + the condition is not met. This is the original SQL condition behavior. + - ``RESULT_VALUE_CHANGES``: Fires whenever the query's single result + value differs from the + previous evaluation. The first evaluation always fires. Queries must return exactly one cell + (one row, one column). SqlDashboardWidgetOutputStatus: type: string x-enum: @@ -15959,24 +21821,25 @@ components: - NEVER - ON_FAILURE description: >- - task retry mode of the continuous job * NEVER: The failed task will not - be retried. * + task retry mode of the continuous job - ON_FAILURE: Retry a failed task if at least one other task in the job is - still running its first - attempt. When this condition is no longer met or the retry limit is - reached, the job run is + - NEVER: The failed task will not be retried. - cancelled and a new run is started. + - ON_FAILURE: Retry a failed task if at least one other task in the job + is still running its + first attempt. When this condition is no longer met or the retry limit is reached, the job run + is cancelled and a new run is started. TerminationCodeCode: type: string x-enum: + - BREAKING_CHANGE - BUDGET_POLICY_LIMIT_EXCEEDED - CANCELED - CLOUD_FAILURE - CLUSTER_ERROR - CLUSTER_REQUEST_LIMIT_EXCEEDED + - CLUSTER_TERMINATED_BY_USER - DISABLED - DRIVER_ERROR - FEATURE_DISABLED @@ -16001,113 +21864,91 @@ components: The code indicates why the run was terminated. Additional codes might be introduced in future - releases. * `SUCCESS`: The run was completed successfully. * - `SUCCESS_WITH_FAILURES`: The run - - was completed successfully but some child runs failed. * - `USER_CANCELED`: The run was - - successfully canceled during execution by a user. * `CANCELED`: The run - was canceled during - - execution by the Databricks platform; for example, if the maximum run - duration was exceeded. * - - `SKIPPED`: Run was never executed, for example, if the upstream task run - failed, the dependency - - type condition was not met, or there were no material tasks to execute. - * `INTERNAL_ERROR`: The - - run encountered an unexpected error. Refer to the state message for - further details. * - - `DRIVER_ERROR`: The run encountered an error while communicating with - the Spark Driver. * - - `CLUSTER_ERROR`: The run failed due to a cluster error. Refer to the - state message for further - - details. * `REPOSITORY_CHECKOUT_FAILED`: Failed to complete the checkout - due to an error when - - communicating with the third party service. * `INVALID_CLUSTER_REQUEST`: - The run failed because - - it issued an invalid request to start the cluster. * - `WORKSPACE_RUN_LIMIT_EXCEEDED`: The - - workspace has reached the quota for the maximum number of concurrent - active runs. Consider - - scheduling the runs over a larger time frame. * `FEATURE_DISABLED`: The - run failed because it - - tried to access a feature unavailable for the workspace. * - `CLUSTER_REQUEST_LIMIT_EXCEEDED`: The - - number of cluster creation, start, and upsize requests have exceeded the - allotted rate limit. - - Consider spreading the run execution over a larger time frame. * - `STORAGE_ACCESS_ERROR`: The run - - failed due to an error when accessing the customer blob storage. Refer - to the state message for - - further details. * `RUN_EXECUTION_ERROR`: The run was completed with - task failures. For more - - details, refer to the state message or run output. * - `UNAUTHORIZED_ERROR`: The run failed due to - - a permission issue while accessing a resource. Refer to the state - message for further details. * - - `LIBRARY_INSTALLATION_ERROR`: The run failed while installing the - user-requested library. Refer - - to the state message for further details. The causes might include, but - are not limited to: The - - provided library is invalid, there are insufficient permissions to - install the library, and so - - forth. * `MAX_CONCURRENT_RUNS_EXCEEDED`: The scheduled run exceeds the - limit of maximum - - concurrent runs set for the job. * `MAX_SPARK_CONTEXTS_EXCEEDED`: The - run is scheduled on a - - cluster that has already reached the maximum number of contexts it is - configured to create. See: - - [Link]. * `RESOURCE_NOT_FOUND`: A resource necessary for run execution - does not exist. Refer to - - the state message for further details. * `INVALID_RUN_CONFIGURATION`: - The run failed due to an - - invalid configuration. Refer to the state message for further details. * - `CLOUD_FAILURE`: The - - run failed due to a cloud provider issue. Refer to the state message for - further details. * - - `MAX_JOB_QUEUE_SIZE_EXCEEDED`: The run was skipped due to reaching the - job level queue size - - limit. * `DISABLED`: The run was never executed because it was disabled + releases. + + + - ``SUCCESS``: The run was completed successfully. + + - ``SUCCESS_WITH_FAILURES``: The run was completed successfully but some + child runs failed. + + - ``USER_CANCELED``: The run was successfully canceled during execution + by a user. + + - ``CANCELED``: The run was canceled during execution by the Databricks + platform; for example, + if the maximum run duration was exceeded. + - ``SKIPPED``: Run was never executed, for example, if the upstream task + run failed, the + dependency type condition was not met, or there were no material tasks to execute. + - ``INTERNAL_ERROR``: The run encountered an unexpected error. Refer to + the state message for + further details. + - ``DRIVER_ERROR``: The run encountered an error while communicating + with the Spark Driver. + + - ``CLUSTER_ERROR``: The run failed due to a cluster error. Refer to the + state message for + further details. + - ``REPOSITORY_CHECKOUT_FAILED``: Failed to complete the checkout due to + an error when + communicating with the third party service. + - ``INVALID_CLUSTER_REQUEST``: The run failed because it issued an + invalid request to start the + cluster. + - ``WORKSPACE_RUN_LIMIT_EXCEEDED``: The workspace has reached the quota + for the maximum number + of concurrent active runs. Consider scheduling the runs over a larger time frame. + - ``FEATURE_DISABLED``: The run failed because it tried to access a + feature unavailable for the + workspace. + - ``CLUSTER_REQUEST_LIMIT_EXCEEDED``: The number of cluster creation, + start, and upsize requests + have exceeded the allotted rate limit. Consider spreading the run execution over a larger time + frame. + - ``STORAGE_ACCESS_ERROR``: The run failed due to an error when + accessing the customer blob + storage. Refer to the state message for further details. + - ``RUN_EXECUTION_ERROR``: The run was completed with task failures. For + more details, refer to + the state message or run output. + - ``UNAUTHORIZED_ERROR``: The run failed due to a permission issue while + accessing a resource. + Refer to the state message for further details. + - ``LIBRARY_INSTALLATION_ERROR``: The run failed while installing the + user-requested library. + Refer to the state message for further details. The causes might include, but are not limited + to: The provided library is invalid, there are insufficient permissions to install the + library, and so forth. + - ``MAX_CONCURRENT_RUNS_EXCEEDED``: The scheduled run exceeds the limit + of maximum concurrent + runs set for the job. + - ``MAX_SPARK_CONTEXTS_EXCEEDED``: The run is scheduled on a cluster + that has already reached + the maximum number of contexts it is configured to create. See: `Link + `__. + - ``RESOURCE_NOT_FOUND``: A resource necessary for run execution does + not exist. Refer to the + state message for further details. + - ``INVALID_RUN_CONFIGURATION``: The run failed due to an invalid + configuration. Refer to the + state message for further details. + - ``CLOUD_FAILURE``: The run failed due to a cloud provider issue. Refer + to the state message + for further details. + - ``MAX_JOB_QUEUE_SIZE_EXCEEDED``: The run was skipped due to reaching + the job level queue size + limit. + - ``DISABLED``: The run was never executed because it was disabled explicitly by the user. - * `BREAKING_CHANGE`: Run failed because of an intentional breaking - change in Spark, but it will - - be retried with a mitigation config. - - - [Link]: - https://kb.databricks.com/en_US/notebooks/too-many-execution-contexts-are-open-right-now + - ``BREAKING_CHANGE``: Run failed because of an intentional breaking + change in Spark, but it + will be retried with a mitigation config. + - ``CLUSTER_TERMINATED_BY_USER``: The run failed because the externally + managed cluster entered + an unusable state, likely due to the user terminating or restarting it outside the jobs + service. TerminationTypeType: type: string x-enum: @@ -16116,22 +21957,16 @@ components: - INTERNAL_ERROR - SUCCESS description: >- - * `SUCCESS`: The run terminated without any issues * `INTERNAL_ERROR`: - An error occurred in the + - ``SUCCESS``: The run terminated without any issues - Databricks platform. Please look at the [status page] or contact support - if the issue persists. - - * `CLIENT_ERROR`: The run was terminated because of an error caused by + - ``INTERNAL_ERROR``: An error occurred in the Databricks platform. + Please look at the `status + page `__ or contact support if the issue persists. + - ``CLIENT_ERROR``: The run was terminated because of an error caused by user input or the job - - configuration. * `CLOUD_FAILURE`: The run was terminated because of an - issue with your cloud - - provider. - - - [status page]: https://status.databricks.com/ + configuration. + - ``CLOUD_FAILURE``: The run was terminated because of an issue with + your cloud provider. TriggerType: type: string x-enum: @@ -16147,48 +21982,46 @@ components: The type of trigger that fired this run. - * `PERIODIC`: Schedules that periodically trigger runs, such as a cron - scheduler. * `ONE_TIME`: - - One time triggers that fire a single run. This occurs you triggered a - single run on demand - - through the UI or the API. * `RETRY`: Indicates a run that is triggered - as a retry of a - - previously failed run. This occurs when you request to re-run the job in - case of failures. * + - ``PERIODIC``: Schedules that periodically trigger runs, such as a cron + scheduler. - `RUN_JOB_TASK`: Indicates a run that is triggered using a Run Job task. - * `FILE_ARRIVAL`: + - ``ONE_TIME``: One time triggers that fire a single run. This occurs + you triggered a single run + on demand through the UI or the API. + - ``RETRY``: Indicates a run that is triggered as a retry of a + previously failed run. This + occurs when you request to re-run the job in case of failures. + - ``RUN_JOB_TASK``: Indicates a run that is triggered using a Run Job + task. - Indicates a run that is triggered by a file arrival. * `CONTINUOUS`: - Indicates a run that is + - ``FILE_ARRIVAL``: Indicates a run that is triggered by a file arrival. - triggered by a continuous job. * `TABLE`: Indicates a run that is - triggered by a table update. * + - ``CONTINUOUS``: Indicates a run that is triggered by a continuous job. - `CONTINUOUS_RESTART`: Indicates a run created by user to manually - restart a continuous job run. + - ``TABLE``: Indicates a run that is triggered by a table update. - * `MODEL`: Indicates a run that is triggered by a model update. + - ``CONTINUOUS_RESTART``: Indicates a run created by user to manually + restart a continuous job + run. + - ``MODEL``: Indicates a run that is triggered by a model update. ViewType: type: string x-enum: - DASHBOARD - NOTEBOOK - description: '* `NOTEBOOK`: Notebook view item. * `DASHBOARD`: Dashboard view item.' + description: |- + - ``NOTEBOOK``: Notebook view item. + - ``DASHBOARD``: Dashboard view item. ViewsToExport: type: string x-enum: - ALL - CODE - DASHBOARDS - description: >- - * `CODE`: Code view of the notebook. * `DASHBOARDS`: All dashboard views - of the notebook. * - - `ALL`: All views of the notebook. + description: |- + - ``CODE``: Code view of the notebook. + - ``DASHBOARDS``: All dashboard views of the notebook. + - ``ALL``: All views of the notebook. x-stackQL-resources: job_runs: id: databricks_workspace.jobs.job_runs diff --git a/providers/src/databricks_workspace/v00.00.00000/services/marketplace.yaml b/providers/src/databricks_workspace/v00.00.00000/services/marketplace.yaml index 017d22e1..4829b2be 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/marketplace.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/marketplace.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks marketplace service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.marketplace servers: - url: https://{deployment_name}.cloud.databricks.com @@ -69,13 +69,13 @@ paths: get: operationId: consumer_fulfillments_list summary: >- - Get all listings fulfillments associated with a listing. A _fulfillment_ + Get all listings fulfillments associated with a listing. A *fulfillment* is a potential installation. tags: - marketplace - consumer_fulfillments description: >- - Get all listings fulfillments associated with a listing. A _fulfillment_ + Get all listings fulfillments associated with a listing. A *fulfillment* is a potential installation. Standard installations contain metadata about the attached share or git @@ -137,17 +137,26 @@ paths: tags: - marketplace - consumer_installations - description: |- + description: >- Install payload associated with a Databricks Marketplace listing. + :param listing_id: str + :param accepted_consumer_terms: :class:`ConsumerTerms` (optional) + :param catalog_name: str (optional) + + :param mcp_connection_detail: :class:`InstallListingMcpConnectionDetail` + (optional) + :param recipient_type: :class:`DeltaSharingRecipientType` (optional) + :param repo_detail: :class:`RepoInstallation` (optional) for git repo installations :param share_name: str (optional) + :returns: :class:`Installation` parameters: - name: listing_id @@ -170,6 +179,65 @@ paths: - version catalog_name: type: string + mcp_connection_detail: + type: object + properties: + connection_name: + type: string + description: >- + Name of the Unity Catalog connection to create for the + MCP server. + base_path: + type: string + description: >- + Optional override of the MCP server base path. Defaults + to the base path on the listing. + bearer_token: + type: string + description: >- + Bearer token supplied by the consumer, for bearer-token + authentication. + catalog: + type: string + description: >- + Catalog the MCP service (and its schema-scoped UC + connection) is created under. The consumer must hold the + privileges to create a service and connection in this + catalog/schema. + host: + type: string + description: >- + Optional override of the MCP server host. Defaults to + the host configured on the listing. + oauth_credentials: + type: object + properties: + client_id: + type: string + description: >- + OAuth client id. Identifies the credential; not the + secret itself. + client_secret: + type: string + description: OAuth client secret supplied by the consumer. + required: + - client_id + - client_secret + description: >- + OAuth client credentials supplied by the consumer, for + OAuth (U2M/M2M) authentication. + schema: + type: string + description: >- + Schema (within ``catalog``) the MCP service and its UC + connection are created under. + required: + - connection_name + description: >- + Consumer-supplied details for creating the Unity Catalog + HTTP connection when installing an MCP + listing. Values the provider configures on the listing (port, credential type, OAuth + token/authorization endpoints, scope, exchange method) are not repeated here. recipient_type: type: string x-enum: @@ -369,12 +437,13 @@ paths: the installation table as well as interact with external services according to the fields not included - in the installation table 1. + in the installation table + - the token will be rotate if the rotateToken flag is true 2. the token - will be forcibly rotate if the + 1. the token will be rotate if the rotateToken flag is true - rotateToken flag is true and the tokenInfo field is empty + 2. the token will be forcibly rotate if the rotateToken flag is true and + the tokenInfo field is empty :param listing_id: str @@ -770,8 +839,8 @@ paths: :param provider_ids: List[str] (optional) Matches any of the following provider ids - :param tags: List[:class:`ListingTag`] (optional) - Matches any of the following tags + :param tags: :class:`ListingTag` (optional) + Matches listings with this tag :returns: Iterator over :class:`Listing` parameters: @@ -831,10 +900,8 @@ paths: in: query required: false schema: - type: array - items: - $ref: '#/components/schemas/ListingTag' - description: Matches any of the following tags + $ref: '#/components/schemas/ListingTag' + description: Matches listings with this tag responses: '200': description: Success @@ -2356,13 +2423,15 @@ paths: attributes -- see the documentation for details. parent_id: type: string - description: TODO make the following fields required marketplace_file_type: type: string x-enum: - APP + - COMMIT_DRAWDOWN_REQUEST_ATTACHMENT + - EMBEDDED_MARKDOWN - EMBEDDED_NOTEBOOK - PROVIDER_ICON + - SCREENSHOT description: >- Create a collection of name/value pairs. @@ -2663,6 +2732,7 @@ paths: - LOOKUP_TABLES - MANUFACTURING - MEDIA + - OPEN_SOURCE - OTHER - PUBLIC_SECTOR - RETAIL @@ -3144,15 +3214,17 @@ paths: details. parent_id: type: string - description: TODO make the following fields required id: type: string marketplace_file_type: type: string x-enum: - APP + - COMMIT_DRAWDOWN_REQUEST_ATTACHMENT + - EMBEDDED_MARKDOWN - EMBEDDED_NOTEBOOK - PROVIDER_ICON + - SCREENSHOT description: >- Create a collection of name/value pairs. @@ -3212,6 +3284,7 @@ paths: status: type: string x-enum: + - FILE_STATUS_COMMIT_DRAWDOWN - FILE_STATUS_PUBLISHED - FILE_STATUS_SANITIZATION_FAILED - FILE_STATUS_SANITIZING @@ -3295,8 +3368,7 @@ paths: description: >- What the pricing model is (e.g. paid, subscription, paid upfront); should only be present if cost is - paid TODO: Not used yet, should deprecate if we will - never use it + paid privacy_policy_link: type: string size: @@ -3665,6 +3737,7 @@ paths: - LOOKUP_TABLES - MANUFACTURING - MEDIA + - OPEN_SOURCE - OTHER - PUBLIC_SECTOR - RETAIL @@ -4146,15 +4219,17 @@ paths: details. parent_id: type: string - description: TODO make the following fields required id: type: string marketplace_file_type: type: string x-enum: - APP + - COMMIT_DRAWDOWN_REQUEST_ATTACHMENT + - EMBEDDED_MARKDOWN - EMBEDDED_NOTEBOOK - PROVIDER_ICON + - SCREENSHOT description: >- Create a collection of name/value pairs. @@ -4214,6 +4289,7 @@ paths: status: type: string x-enum: + - FILE_STATUS_COMMIT_DRAWDOWN - FILE_STATUS_PUBLISHED - FILE_STATUS_SANITIZATION_FAILED - FILE_STATUS_SANITIZING @@ -4297,8 +4373,7 @@ paths: description: >- What the pricing model is (e.g. paid, subscription, paid upfront); should only be present if cost is - paid TODO: Not used yet, should deprecate if we will - never use it + paid privacy_policy_link: type: string size: @@ -4779,14 +4854,14 @@ paths: post: operationId: provider_provider_analytics_dashboards_create summary: >- - Create provider analytics dashboard. Returns Marketplace specific `id`. - Not to be confused with the + Create provider analytics dashboard. Returns Marketplace specific + ``id``. Not to be confused with the tags: - marketplace - provider_provider_analytics_dashboards description: >- - Create provider analytics dashboard. Returns Marketplace specific `id`. - Not to be confused with the + Create provider analytics dashboard. Returns Marketplace specific + ``id``. Not to be confused with the Lakeview dashboard id. @@ -5416,7 +5491,6 @@ components: $ref: '#/components/schemas/FileParentType' parent_id: type: string - description: TODO make the following fields required GetExchangeResponse: type: object properties: @@ -5467,6 +5541,64 @@ components: properties: provider: $ref: '#/components/schemas/ProviderInfo' + InstallListingMcpConnectionDetail: + type: object + properties: + connection_name: + type: string + description: Name of the Unity Catalog connection to create for the MCP server. + base_path: + type: string + description: >- + Optional override of the MCP server base path. Defaults to the base + path on the listing. + bearer_token: + type: string + description: >- + Bearer token supplied by the consumer, for bearer-token + authentication. + catalog: + type: string + description: >- + Catalog the MCP service (and its schema-scoped UC connection) is + created under. The consumer must hold the privileges to create a + service and connection in this catalog/schema. + host: + type: string + description: >- + Optional override of the MCP server host. Defaults to the host + configured on the listing. + oauth_credentials: + $ref: >- + #/components/schemas/InstallListingMcpConnectionDetailOauthCredentials + description: >- + OAuth client credentials supplied by the consumer, for OAuth + (U2M/M2M) authentication. + schema: + type: string + description: >- + Schema (within ``catalog``) the MCP service and its UC connection + are created under. + required: + - connection_name + description: >- + Consumer-supplied details for creating the Unity Catalog HTTP connection + when installing an MCP + listing. Values the provider configures on the listing (port, credential type, OAuth + token/authorization endpoints, scope, exchange method) are not repeated here. + InstallListingMcpConnectionDetailOauthCredentials: + type: object + properties: + client_id: + type: string + description: OAuth client id. Identifies the credential; not the secret itself. + client_secret: + type: string + description: OAuth client secret supplied by the consumer. + required: + - client_id + - client_secret + description: OAuth client credentials supplied by the consumer for an MCP connection. Installation: type: object properties: @@ -5672,8 +5804,7 @@ components: type: string description: >- What the pricing model is (e.g. paid, subscription, paid upfront); - should only be present if cost is paid TODO: Not used yet, should - deprecate if we will never use it + should only be present if cost is paid privacy_policy_link: type: string size: @@ -6075,6 +6206,7 @@ components: - LOOKUP_TABLES - MANUFACTURING - MEDIA + - OPEN_SOURCE - OTHER - PUBLIC_SECTOR - RETAIL @@ -6340,6 +6472,7 @@ components: FileStatus: type: string x-enum: + - FILE_STATUS_COMMIT_DRAWDOWN - FILE_STATUS_PUBLISHED - FILE_STATUS_SANITIZATION_FAILED - FILE_STATUS_SANITIZING @@ -6603,8 +6736,11 @@ components: type: string x-enum: - APP + - COMMIT_DRAWDOWN_REQUEST_ATTACHMENT + - EMBEDDED_MARKDOWN - EMBEDDED_NOTEBOOK - PROVIDER_ICON + - SCREENSHOT description: |- Create a collection of name/value pairs. diff --git a/providers/src/databricks_workspace/v00.00.00000/services/ml.yaml b/providers/src/databricks_workspace/v00.00.00000/services/ml.yaml index 916a3edf..1d57a795 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/ml.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/ml.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks ml service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.ml servers: - url: https://{deployment_name}.cloud.databricks.com @@ -34,8 +34,13 @@ paths: same name already exists. - Throws `RESOURCE_ALREADY_EXISTS` if an experiment with the given name - exists. + Throws ``RESOURCE_ALREADY_EXISTS`` if an experiment with the given name + exists. Note: In some + + contexts, this error may be remapped to ``ALREADY_EXISTS``. To be safe, + clients should check for both + + error codes. :param name: str @@ -48,6 +53,10 @@ paths: depends on the storage backend. All storage backends are guaranteed to support tag keys up to 250 bytes in size and tag values up to 5000 bytes in size. All storage backends are also guaranteed to support up to 20 tags per request. + :param trace_location: :class:`ExperimentTraceLocation` (optional) + The location where the experiment's traces are stored. When set, the underlying storage is + provisioned and the experiment's traces are routed to it. When unset, traces are stored in the + default MLflow backend. This field cannot be updated after the experiment is created. :returns: :class:`CreateExperimentResponse` requestBody: @@ -84,6 +93,49 @@ paths: keys up to 250 bytes in size and tag values up to 5000 bytes in size. All storage backends are also guaranteed to support up to 20 tags per request. + trace_location: + type: object + properties: + uc_trace_location: + type: object + properties: + catalog: + type: string + description: The name of the Unity Catalog catalog. + schema: + type: string + description: >- + The name of the Unity Catalog schema within + ``catalog``. + effective_table_prefix: + type: string + description: >- + The trace-table prefix actually in effect: + ``table_prefix`` if it was set on creation, + otherwise the server-generated default. + table_prefix: + type: string + description: >- + The prefix for the trace tables, which are named + ``{catalog}.{schema}.{table_prefix}_otel_*``. May + only contain letters, digits, and underscores, and + may be at most 238 characters. When unset, a + server-generated prefix derived from the experiment + ID is used and this field stays empty on read; the + resolved value is always available in + ``effective_table_prefix``. + required: + - catalog + - schema + description: >- + A Unity Catalog schema where the experiment's traces are + stored as Delta tables. + description: >- + The location where the experiment's traces are stored. When + set, the underlying storage is provisioned and the + experiment's traces are routed to it. When unset, traces are + stored in the default MLflow backend. This field cannot be + updated after the experiment is created. required: - name responses: @@ -211,10 +263,10 @@ paths: Creates a new run within an experiment. A run is usually a single execution of a machine learning or - data ETL pipeline. MLflow uses runs to track the `mlflowParam`, - `mlflowMetric`, and `mlflowRunTag` + data ETL pipeline. MLflow uses runs to track the ``mlflowParam``, + ``mlflowMetric``, and - associated with a single execution. + ``mlflowRunTag`` associated with a single execution. :param experiment_id: str (optional) @@ -687,7 +739,7 @@ paths: will return one of them. - Throws `RESOURCE_DOES_NOT_EXIST` if no experiment with the specified + Throws ``RESOURCE_DOES_NOT_EXIST`` if no experiment with the specified name exists. @@ -785,8 +837,8 @@ paths: :param run_id: str (optional) ID of the run from which to fetch metric values. Must be provided. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run from which to fetch metric values. This field will - be removed in a future MLflow version. + [Deprecated, use ``run_id`` instead] ID of the run from which to fetch metric values. This field + will be removed in a future MLflow version. :returns: Iterator over :class:`Metric` parameters: @@ -823,8 +875,8 @@ paths: schema: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run from which to fetch - metric values. This field will be removed in a future MLflow + [Deprecated, use ``run_id`` instead] ID of the run from which to + fetch metric values. This field will be removed in a future MLflow version. responses: '200': @@ -845,6 +897,48 @@ paths: message: type: string x-stackql-sdk-source: ExperimentsAPI + /api/2.0/mlflow/logged-models:batchGet: + get: + operationId: experiments_get_logged_models + summary: Batch endpoint for getting logged models from a list of model IDs + tags: + - ml + - experiments + description: |- + Batch endpoint for getting logged models from a list of model IDs + + :param model_ids: List[str] (optional) + The IDs of the logged models to retrieve. Max threshold is 100. + + :returns: :class:`GetLoggedModelsRequestResponse` + parameters: + - name: model_ids + in: query + required: false + schema: + type: array + items: + type: string + description: The IDs of the logged models to retrieve. Max threshold is 100. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GetLoggedModelsRequestResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ExperimentsAPI /api/2.0/permissions/experiments/{experiment_id}/permissionLevels: get: operationId: experiments_get_permission_levels @@ -1100,7 +1194,7 @@ paths: :param run_id: str ID of the run to fetch. Must be provided. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run to fetch. This field will be removed in a future + [Deprecated, use ``run_id`` instead] ID of the run to fetch. This field will be removed in a future MLflow version. :returns: :class:`GetRunResponse` @@ -1117,7 +1211,7 @@ paths: schema: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run to fetch. This + [Deprecated, use ``run_id`` instead] ID of the run to fetch. This field will be removed in a future MLflow version. responses: '200': @@ -1142,30 +1236,30 @@ paths: get: operationId: experiments_list_artifacts summary: >- - List artifacts for a run. Takes an optional `artifact_path` prefix which - if specified, the response + List artifacts for a run. Takes an optional ``artifact_path`` prefix + which if specified, the response tags: - ml - experiments description: >- - List artifacts for a run. Takes an optional `artifact_path` prefix which - if specified, the response + List artifacts for a run. Takes an optional ``artifact_path`` prefix + which if specified, the response contains only artifacts with the specified prefix. A maximum of 1000 artifacts will be retrieved for - UC Volumes. Please call `/api/2.0/fs/directories{directory_path}` for - listing artifacts in UC Volumes, + UC Volumes. Please call ``/api/2.0/fs/directories{directory_path}`` for + listing artifacts in UC - which supports pagination. See [List directory contents | Files + Volumes, which supports pagination. See [List directory contents | Files API](/api/workspace/files/listdirectorycontents). :param page_token: str (optional) - The token indicating the page of artifact results to fetch. `page_token` is not supported when + The token indicating the page of artifact results to fetch. ``page_token`` is not supported when listing artifacts in UC Volumes. A maximum of 1000 artifacts will be retrieved for UC Volumes. - Please call `/api/2.0/fs/directories{directory_path}` for listing artifacts in UC Volumes, which + Please call ``/api/2.0/fs/directories{directory_path}`` for listing artifacts in UC Volumes, which supports pagination. See [List directory contents | Files API](/api/workspace/files/listdirectorycontents). :param path: str (optional) @@ -1173,8 +1267,8 @@ paths: :param run_id: str (optional) ID of the run whose artifacts to list. Must be provided. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run whose artifacts to list. This field will be removed - in a future MLflow version. + [Deprecated, use ``run_id`` instead] ID of the run whose artifacts to list. This field will be + removed in a future MLflow version. :returns: Iterator over :class:`FileInfo` parameters: @@ -1185,11 +1279,12 @@ paths: type: string description: >- The token indicating the page of artifact results to fetch. - `page_token` is not supported when listing artifacts in UC Volumes. - A maximum of 1000 artifacts will be retrieved for UC Volumes. Please - call `/api/2.0/fs/directories{directory_path}` for listing artifacts - in UC Volumes, which supports pagination. See [List directory - contents | Files API](/api/workspace/files/listdirectorycontents). + ``page_token`` is not supported when listing artifacts in UC + Volumes. A maximum of 1000 artifacts will be retrieved for UC + Volumes. Please call ``/api/2.0/fs/directories{directory_path}`` for + listing artifacts in UC Volumes, which supports pagination. See + [List directory contents | Files + API](/api/workspace/files/listdirectorycontents). - name: path in: query required: false @@ -1210,8 +1305,8 @@ paths: schema: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run whose artifacts to - list. This field will be removed in a future MLflow version. + [Deprecated, use ``run_id`` instead] ID of the run whose artifacts + to list. This field will be removed in a future MLflow version. responses: '200': description: Success @@ -1242,8 +1337,8 @@ paths: Gets a list of all experiments. :param max_results: int (optional) - Maximum number of experiments desired. If `max_results` is unspecified, return all experiments. If - `max_results` is too large, it'll be automatically capped at 1000. Callers of this endpoint are + Maximum number of experiments desired. If ``max_results`` is unspecified, return all experiments. If + ``max_results`` is too large, it'll be automatically capped at 1000. Callers of this endpoint are encouraged to pass max_results explicitly and leverage page_token to iterate through experiments. :param page_token: str (optional) Token indicating the page of experiments to fetch @@ -1258,11 +1353,11 @@ paths: schema: type: integer description: >- - Maximum number of experiments desired. If `max_results` is - unspecified, return all experiments. If `max_results` is too large, - it'll be automatically capped at 1000. Callers of this endpoint are - encouraged to pass max_results explicitly and leverage page_token to - iterate through experiments. + Maximum number of experiments desired. If ``max_results`` is + unspecified, return all experiments. If ``max_results`` is too + large, it'll be automatically capped at 1000. Callers of this + endpoint are encouraged to pass max_results explicitly and leverage + page_token to iterate through experiments. - name: page_token in: query required: false @@ -1325,47 +1420,32 @@ paths: The overwrite behavior for metrics, params, and tags is as follows: - * Metrics: metric values are never overwritten. Logging a metric (key, + - Metrics: metric values are never overwritten. Logging a metric (key, value, timestamp) appends to - - the set of values for the metric with the provided key. - - - * Tags: tag values can be overwritten by successive writes to the same + the set of values for the metric with the provided key. + - Tags: tag values can be overwritten by successive writes to the same tag key. That is, if multiple - - tag values with the same key are provided in the same API request, the - last-provided tag value is - - written. Logging the same tag (key, value) is permitted. Specifically, - logging a tag is idempotent. - - - * Parameters: once written, param values cannot be changed (attempting + tag values with the same key are provided in the same API request, the last-provided tag value is + written. Logging the same tag (key, value) is permitted. Specifically, logging a tag is idempotent. + - Parameters: once written, param values cannot be changed (attempting to overwrite a param value will + result in an error). However, logging the same param (key, value) is permitted. Specifically, + logging a param is idempotent. - result in an error). However, logging the same param (key, value) is - permitted. Specifically, logging - - a param is idempotent. - - - Request Limits ------------------------------- A single JSON-serialized - API request may be up to 1 MB + Request Limits - in size and contain: + A single JSON-serialized API request may be up to 1 MB in size and + contain: - * No more than 1000 metrics, params, and tags in total + - No more than 1000 metrics, params, and tags in total - * Up to 1000 metrics + - Up to 1000 metrics + - Up to 100 params - * Up to 100 params - - - * Up to 100 tags + - Up to 100 tags For example, a valid request might contain 900 metrics, 50 params, and @@ -1378,11 +1458,10 @@ paths: values: - * Metric keys, param keys, and tag keys can be up to 250 characters in + - Metric keys, param keys, and tag keys can be up to 250 characters in length - - * Parameter and tag values can be up to 250 characters in length + - Parameter and tag values can be up to 250 characters in length :param metrics: List[:class:`Metric`] (optional) @@ -1721,7 +1800,7 @@ paths: :param run_id: str (optional) ID of the run under which to log the metric. Must be provided. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run under which to log the metric. This field will be + [Deprecated, use ``run_id`` instead] ID of the run under which to log the metric. This field will be removed in a future MLflow version. :param step: int (optional) Step at which to log the metric @@ -1766,9 +1845,9 @@ paths: run_uuid: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run under which - to log the metric. This field will be removed in a future - MLflow version. + [Deprecated, use ``run_id`` instead] ID of the run under + which to log the metric. This field will be removed in a + future MLflow version. step: type: integer description: Step at which to log the metric @@ -1923,7 +2002,7 @@ paths: :param run_id: str (optional) ID of the run under which to log the param. Must be provided. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run under which to log the param. This field will be + [Deprecated, use ``run_id`` instead] ID of the run under which to log the param. This field will be removed in a future MLflow version. requestBody: content: @@ -1947,9 +2026,9 @@ paths: run_uuid: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run under which - to log the param. This field will be removed in a future - MLflow version. + [Deprecated, use ``run_id`` instead] ID of the run under + which to log the param. This field will be removed in a + future MLflow version. required: - key - value @@ -1987,8 +2066,8 @@ paths: also restored. - Throws `RESOURCE_DOES_NOT_EXIST` if experiment was never created or was - permanently deleted. + Throws ``RESOURCE_DOES_NOT_EXIST`` if experiment was never created or + was permanently deleted. :param experiment_id: str @@ -2033,7 +2112,7 @@ paths: metrics, params, and tags. - Throws `RESOURCE_DOES_NOT_EXIST` if the run was never created or was + Throws ``RESOURCE_DOES_NOT_EXIST`` if the run was never created or was permanently deleted. @@ -2229,7 +2308,7 @@ paths: :param datasets: List[:class:`SearchLoggedModelsDataset`] (optional) List of datasets on which to apply the metrics filter clauses. For example, a filter with - `metrics.accuracy > 0.9` and dataset info with name "test_dataset" means we will return all logged + ``metrics.accuracy > 0.9`` and dataset info with name "test_dataset" means we will return all logged models with accuracy > 0.9 on the test_dataset. Metric values from ANY dataset matching the criteria are considered. If no datasets are specified, then metrics across all datasets are considered in the filter. @@ -2268,9 +2347,9 @@ paths: - dataset_name description: >- List of datasets on which to apply the metrics filter - clauses. For example, a filter with `metrics.accuracy > 0.9` - and dataset info with name "test_dataset" means we will - return all logged models with accuracy > 0.9 on the + clauses. For example, a filter with ``metrics.accuracy > + 0.9`` and dataset info with name "test_dataset" means we + will return all logged models with accuracy > 0.9 on the test_dataset. Metric values from ANY dataset matching the criteria are considered. If no datasets are specified, then metrics across all datasets are considered in the filter. @@ -2359,7 +2438,7 @@ paths: description: |- Searches for runs that satisfy expressions. - Search expressions can use `mlflowMetric` and `mlflowParam` keys. + Search expressions can use ``mlflowMetric`` and ``mlflowParam`` keys. :param experiment_ids: List[str] (optional) List of experiment IDs to search over. @@ -2368,20 +2447,20 @@ paths: syntax is a subset of SQL that supports ANDing together binary operations between a param, metric, or tag and a constant. - Example: `metrics.rmse < 1 and params.model_class = 'LogisticRegression'` + Example: ``metrics.rmse < 1 and params.model_class = 'LogisticRegression'`` You can select columns with special characters (hyphen, space, period, etc.) by using double quotes: - `metrics."model class" = 'LinearRegression' and tags."user-name" = 'Tomas'` + ``metrics."model class" = 'LinearRegression' and tags."user-name" = 'Tomas'`` - Supported operators are `=`, `!=`, `>`, `>=`, `<`, and `<=`. + Supported operators are ``=``, ``!=``, ``>``, ``>=``, ``<``, and ``<=``. :param max_results: int (optional) Maximum number of runs desired. Max threshold is 50000 :param order_by: List[str] (optional) List of columns to be ordered by, including attributes, params, metrics, and tags with an optional - `"DESC"` or `"ASC"` annotation, where `"ASC"` is the default. Example: `["params.input DESC", - "metrics.alpha ASC", "metrics.rmse"]`. Tiebreaks are done by start_time `DESC` followed by `run_id` - for runs with the same start time (and this is the default ordering criterion if order_by is not - provided). + ``"DESC"`` or ``"ASC"`` annotation, where ``"ASC"`` is the default. Example: ``["params.input DESC", + "metrics.alpha ASC", "metrics.rmse"]``. Tiebreaks are done by start_time ``DESC`` followed by + ``run_id`` for runs with the same start time (and this is the default ordering criterion if order_by + is not provided). :param page_token: str (optional) Token for the current page of runs. :param run_view_type: :class:`ViewType` (optional) @@ -2406,12 +2485,12 @@ paths: allows returning a subset of runs. The syntax is a subset of SQL that supports ANDing together binary operations between a param, metric, or tag and a constant. Example: - `metrics.rmse < 1 and params.model_class = - 'LogisticRegression'` You can select columns with special + ``metrics.rmse < 1 and params.model_class = + 'LogisticRegression'`` You can select columns with special characters (hyphen, space, period, etc.) by using double - quotes: `metrics."model class" = 'LinearRegression' and - tags."user-name" = 'Tomas'` Supported operators are `=`, - `!=`, `>`, `>=`, `<`, and `<=`. + quotes: ``metrics."model class" = 'LinearRegression' and + tags."user-name" = 'Tomas'`` Supported operators are ``=``, + ``!=``, ``>``, ``>=``, ``<``, and ``<=``. max_results: type: integer description: Maximum number of runs desired. Max threshold is 50000 @@ -2421,13 +2500,13 @@ paths: type: string description: >- List of columns to be ordered by, including attributes, - params, metrics, and tags with an optional `"DESC"` or - `"ASC"` annotation, where `"ASC"` is the default. Example: - `["params.input DESC", "metrics.alpha ASC", - "metrics.rmse"]`. Tiebreaks are done by start_time `DESC` - followed by `run_id` for runs with the same start time (and - this is the default ordering criterion if order_by is not - provided). + params, metrics, and tags with an optional ``"DESC"`` or + ``"ASC"`` annotation, where ``"ASC"`` is the default. + Example: ``["params.input DESC", "metrics.alpha ASC", + "metrics.rmse"]``. Tiebreaks are done by start_time ``DESC`` + followed by ``run_id`` for runs with the same start time + (and this is the default ordering criterion if order_by is + not provided). page_token: type: string description: Token for the current page of runs. @@ -2593,7 +2672,7 @@ paths: :param run_id: str (optional) ID of the run under which to log the tag. Must be provided. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run under which to log the tag. This field will be + [Deprecated, use ``run_id`` instead] ID of the run under which to log the tag. This field will be removed in a future MLflow version. requestBody: content: @@ -2615,8 +2694,8 @@ paths: run_uuid: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run under which - to log the tag. This field will be removed in a future + [Deprecated, use ``run_id`` instead] ID of the run under + which to log the tag. This field will be removed in a future MLflow version. required: - key @@ -2698,7 +2777,7 @@ paths: :param run_name: str (optional) Updated name of the run. :param run_uuid: str (optional) - [Deprecated, use `run_id` instead] ID of the run to update. This field will be removed in a future + [Deprecated, use ``run_id`` instead] ID of the run to update. This field will be removed in a future MLflow version. :param status: :class:`UpdateRunStatus` (optional) Updated status of the run. @@ -2722,8 +2801,9 @@ paths: run_uuid: type: string description: >- - [Deprecated, use `run_id` instead] ID of the run to update. - This field will be removed in a future MLflow version. + [Deprecated, use ``run_id`` instead] ID of the run to + update. This field will be removed in a future MLflow + version. status: type: string x-enum: @@ -2788,7 +2868,26 @@ paths: description: >- The quartz cron expression that defines the schedule of the materialization pipeline. The - schedule is evaluated in the UTC timezone. + schedule is evaluated in the UTC timezone. Hidden + from GraphQL: superseded by the ``trigger`` oneof + (cron_schedule_trigger), so not exposed to Catalog + Explorer. + cron_schedule_trigger: + type: object + properties: + cron_expression: + type: string + description: >- + The cron expression defining the schedule + (e.g., "0 0 * * *" for daily at midnight). + description: >- + A cron-based schedule trigger for the + materialization pipeline. + is_online: + type: boolean + description: >- + True if this is an online materialized feature. + False if it is an offline materialized feature. last_materialization_time: type: string description: >- @@ -2797,7 +2896,9 @@ paths: pipeline has not run yet, this field will be null. materialized_feature_id: type: string - description: Unique identifier for the materialized feature. + description: >- + Server-assigned unique identifier for the + materialized feature. offline_store_config: type: object properties: @@ -2817,7 +2918,9 @@ paths: - catalog_name - schema_name - table_name_prefix - description: Configuration for offline store destination. + description: >- + Destination for writing feature values to an + offline Delta table. online_store_config: type: object properties: @@ -2850,7 +2953,9 @@ paths: - schema_name - table_name_prefix - online_store_name - description: Configuration for online store destination. + description: >- + Destination for writing feature values to an + online Lakebase table. pipeline_schedule_state: type: string x-enum: @@ -2859,13 +2964,43 @@ paths: - SNAPSHOT description: >- The schedule state of the materialization - pipeline. + pipeline. Hidden from GraphQL: being deprecated, + so not exposed to Catalog Explorer. + streaming_mode: + type: object + properties: + freshness_target: + type: string + description: >- + The desired data freshness for feature + materialization, expressed as a duration + string (e.g. "1 minute"). + mode: + type: string + x-enum: + - STREAMING_MODE_TYPE_MBM + - STREAMING_MODE_TYPE_RTM + description: >- + The type of streaming mode used by the + materialization pipeline. + description: >- + The Structured Streaming trigger mode used for + materialization. Real-time mode (RTM) targets + sub-second latency for operational workloads; + micro-batch mode (MBM) favors cost efficiency for + ETL and analytics workloads. table_name: type: string description: >- The fully qualified Unity Catalog path to the table containing the materialized feature (Delta table or Lakebase table). Output only. + table_trigger: + type: object + properties: {} + description: >- + A trigger that fires when the upstream source + table changes. required: - feature_name description: >- @@ -2964,17 +3099,25 @@ paths: filter_condition. Should contains all the columns needed (eg. "SELECT *, col_a + col_b AS col_c FROM x.y.z WHERE col_a > 0" would have - `transformation_sql` "*, col_a + col_b AS + ``transformation_sql`` "*, col_a + col_b AS col_c") If transformation_sql is not provided, all columns of the delta table are present in the DataSource dataframe. required: - full_name + description: A Delta table data source. kafka_source: type: object properties: name: type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Any subsequent + functions operate against this dataframe. entity_column_identifiers: type: array items: @@ -3005,8 +3148,86 @@ paths: instead. Kept for backwards compatibility. The timeseries column identifier of the Kafka source. + transformation_sql: + type: string + description: >- + The pipeline runs these SQL statements + immediately after conversion into the schema + specified on the KafkaConfig object. required: - name + description: A Kafka stream data source. + request_source: + type: object + properties: + flat_schema: + type: object + properties: + fields: + type: array + items: + type: object + properties: + name: + type: string + description: The name of the field. + data_type: + type: string + x-enum: + - BINARY + - BOOLEAN + - DATE + - DECIMAL + - DOUBLE + - FLOAT + - INTEGER + - LONG + - SHORT + - STRING + - TIMESTAMP + description: The scalar data type of the field. + required: + - name + - data_type + description: >- + A single field definition within a + FlatSchema, specifying the field name and + its scalar data + type. Does not support nested or complex types (arrays, maps, structs). + description: The list of fields in this schema. + required: + - fields + description: A flat schema with scalar-typed fields only. + description: A request-time data source. + stream_source: + type: object + properties: + full_name: + type: string + description: >- + Three-part full name of the Stream + (catalog.schema.stream). + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Any subsequent + functions operate against this dataframe. + filter_condition: + type: string + description: >- + The filter condition applied to the source data + before aggregation. + transformation_sql: + type: string + description: >- + The pipeline runs these SQL statements + immediately after conversion into the schema + specified on the Stream object. + required: + - full_name + description: A Stream data source. description: The data source of the feature. function: type: object @@ -3061,7 +3282,13 @@ paths: type: string description: >- The input column from which the average is - computed. + computed. For Kafka sources, use + dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf + node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; + migrate to dot notation. required: - input description: Computes the average of values. @@ -3072,7 +3299,13 @@ paths: type: string description: >- The input column from which the count is - computed. + computed. For Kafka sources, use + dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf + node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; + migrate to dot notation. required: - input description: Computes the count of values. @@ -3087,6 +3320,40 @@ paths: required: - input description: Returns the first value. + first_distinct: + type: object + properties: + input: + type: string + description: >- + The input column from which the first N + distinct values are returned. + 'n': + type: integer + description: The number of distinct values to return. + required: + - input + - 'n' + description: >- + Returns the first N distinct values, ordered by + the feature's timeseries column. + first_n: + type: object + properties: + input: + type: string + description: >- + The input column from which the first N + values are returned. + 'n': + type: integer + description: The number of values to return. + required: + - input + - 'n' + description: >- + Returns the first N values, ordered by the + feature's timeseries column. last: type: object properties: @@ -3098,6 +3365,40 @@ paths: required: - input description: Returns the last value. + last_distinct: + type: object + properties: + input: + type: string + description: >- + The input column from which the last N + distinct values are returned. + 'n': + type: integer + description: The number of distinct values to return. + required: + - input + - 'n' + description: >- + Returns the last N distinct values, ordered by + the feature's timeseries column. + last_n: + type: object + properties: + input: + type: string + description: >- + The input column from which the last N + values are returned. + 'n': + type: integer + description: The number of values to return. + required: + - input + - 'n' + description: >- + Returns the last N values, ordered by the + feature's timeseries column. max: type: object properties: @@ -3127,7 +3428,13 @@ paths: type: string description: >- The input column from which the population - standard deviation is computed. + standard deviation is computed. For Kafka + sources, use dot-prefixed path notation + (e.g., "value.amount"). For nested fields, + the leaf node name is used. Colon-prefixed + notation (e.g., "value:amount") is supported + for backwards compatibility but is + deprecated; migrate to dot notation. required: - input description: Computes the population standard deviation. @@ -3149,7 +3456,13 @@ paths: type: string description: >- The input column from which the sum is - computed. + computed. For Kafka sources, use + dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf + node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; + migrate to dot notation. required: - input description: Computes the sum of values. @@ -3161,6 +3474,9 @@ paths: properties: window_duration: type: string + description: >- + The duration of the continuous window + (must be positive). offset: type: string description: >- @@ -3168,19 +3484,71 @@ paths: (must be non-positive). required: - window_duration - sliding: + description: >- + Deprecated: use RollingWindow with ``delay`` + instead. + rolling: type: object properties: + delay: + type: string + description: >- + The delay applied to the end of the + rolling window (must be non-negative). + For example, delay=1d shifts the window + end 1 day before the evaluation time. + window_duration: + type: string + description: >- + The duration of the rolling window. Must + be positive when set; absent means + lifetime (aggregate over the entity's + entire history). + description: >- + A rolling time window with an optional + delay. This is the SQL-spec-aligned + replacement for + ContinuousWindow: ``delay`` is the non-negative counterpart of the legacy non-positive + ``ContinuousWindow.offset``. + sawtooth: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the + window (must be non-negative). For + example, delay=1d shifts the window end + 1 day before the evaluation time. window_duration: type: string + description: >- + The duration of the window. Must be + positive and span more than two days + when set, so that both the batch (N-1 + day) and stale-path (N-2 day) partial + aggregates are well defined. The + duration need not be a whole number of + days (e.g. 3 days 15 minutes is + allowed). Absent means lifetime + (aggregate over the entity's entire + history). + description: >- + A sawtooth window served via the hybrid + batch + streaming path. + sliding: + type: object + properties: slide_duration: type: string + window_duration: + type: string description: >- - The slide duration (interval by which - windows advance, must be positive and - less than duration). + The duration of the sliding window. Must + be positive when set; absent means + lifetime (aggregate over the entity's + entire history). required: - - window_duration - slide_duration tumbling: type: object @@ -3215,6 +3583,55 @@ paths: - input description: Computes the sample variance. description: An aggregation function applied over a time window. + column_selection: + type: object + properties: + column: + type: string + description: >- + Column name from source to select as the feature + value. + required: + - column + description: >- + Selects the latest value of a single column in a + data source + custom_udf: + type: object + properties: + function_path: + type: string + description: >- + Fully qualified 3-part Unity Catalog path of the + function to apply. + input_bindings: + type: array + items: + type: object + properties: + parameter: + type: string + description: Name of the UC function parameter. + column: + type: string + description: >- + Source column whose value is passed for + this parameter at execution time. + required: + - parameter + - column + description: >- + Binds a single UC function parameter to a + source column. + description: >- + Binds each UC function parameter to a source + column. May be empty for zero-argument functions + (e.g. a timestamp generator). + required: + - function_path + description: >- + Applies a registered Unity Catalog function row-wise + to source columns. extra_parameters: type: array items: @@ -3260,6 +3677,16 @@ paths: AggregationFunction instead. Kept for backwards compatibility. The type of the function. description: The function by which the feature is computed. + catalog_name: + type: string + description: Name of parent catalog. + created_at: + type: string + format: date-time + description: Time at which this feature was created. + created_by: + type: string + description: Username of the feature creator. description: type: string description: The description of the feature. @@ -3315,6 +3742,14 @@ paths: or unexpected behavior. This field will be set by feature-engineering client and should be left unset by SDK and terraform users. + name: + type: string + description: >- + Name of the feature, extracted from the full three-part + name (catalog.schema.name). + schema_name: + type: string + description: Name of parent schema relative to its parent catalog. time_window: type: object properties: @@ -3323,6 +3758,9 @@ paths: properties: window_duration: type: string + description: >- + The duration of the continuous window (must be + positive). offset: type: string description: >- @@ -3330,19 +3768,65 @@ paths: non-positive). required: - window_duration - sliding: + description: >- + Deprecated: use RollingWindow with ``delay`` + instead. + rolling: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the rolling + window (must be non-negative). For example, + delay=1d shifts the window end 1 day before the + evaluation time. + window_duration: + type: string + description: >- + The duration of the rolling window. Must be + positive when set; absent means lifetime + (aggregate over the entity's entire history). + description: >- + A rolling time window with an optional delay. This + is the SQL-spec-aligned replacement for + ContinuousWindow: ``delay`` is the non-negative counterpart of the legacy non-positive + ``ContinuousWindow.offset``. + sawtooth: type: object properties: + delay: + type: string + description: >- + The delay applied to the end of the window (must + be non-negative). For example, delay=1d shifts + the window end 1 day before the evaluation time. window_duration: type: string + description: >- + The duration of the window. Must be positive and + span more than two days when set, so that both + the batch (N-1 day) and stale-path (N-2 day) + partial aggregates are well defined. The + duration need not be a whole number of days + (e.g. 3 days 15 minutes is allowed). Absent + means lifetime (aggregate over the entity's + entire history). + description: >- + A sawtooth window served via the hybrid batch + + streaming path. + sliding: + type: object + properties: slide_duration: type: string + window_duration: + type: string description: >- - The slide duration (interval by which windows - advance, must be positive and less than - duration). + The duration of the sliding window. Must be + positive when set; absent means lifetime + (aggregate over the entity's entire history). required: - - window_duration - slide_duration tumbling: type: object @@ -3401,6 +3885,10 @@ paths: description: |- List Features. + :param catalog_name: str + Name of parent catalog for features of interest. + :param schema_name: str + Name of parent schema relative to its parent catalog. :param page_size: int (optional) The maximum number of results to return. :param page_token: str (optional) @@ -3408,6 +3896,18 @@ paths: :returns: Iterator over :class:`Feature` parameters: + - name: catalog_name + in: query + required: true + schema: + type: string + description: Name of parent catalog for features of interest. + - name: schema_name + in: query + required: true + schema: + type: string + description: Name of parent schema relative to its parent catalog. - name: page_size in: query required: false @@ -3481,6 +3981,11 @@ paths: properties: assign: type: string + description: >- + A JSON string that contains the specific + topic-partitions to consume from. For example, for + '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and + 1st partitions will be consumed from. subscribe: type: string description: >- @@ -3498,29 +4003,136 @@ paths: auth_config: type: object properties: - uc_service_credential_name: - type: string - description: Authentication configuration for connection to topics. - backfill_source: - type: object - properties: - delta_table_source: + mtls_config: type: object properties: - full_name: + keystore_location: type: string - dataframe_schema: - type: string - description: >- - Schema of the resulting dataframe after - transformations, in Spark StructType JSON format - (from df.schema.json()). Required if - transformation_sql is specified. Example: - {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} - entity_columns: - type: array - items: - type: string + description: >- + Unity Catalog volume path to the JKS keystore + file containing the client certificate and + private key. e.g. + "/Volumes////client.jks". + The materialization compute must have read + permission on this volume. + keystore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the JKS keystore + password. + key_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the private key + password. Often the same value as the keystore + password (keytool's default), but provided as a + separate field because Apache Kafka requires it + as a distinct option (kafka.ssl.key.password). + truststore_location: + type: string + description: >- + Unity Catalog volume path to the JKS truststore + file containing the CA certificate(s) trusted to + verify the Kafka broker's server certificate. + e.g. + "/Volumes////truststore.jks". + truststore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the JKS truststore + password. + disable_hostname_verification: + type: boolean + description: >- + Set to true only when the broker certificate's + SAN intentionally does not match the connection + endpoint — for example when reaching the cluster + through a PrivateLink endpoint whose DNS name is + not in the broker certificate. Skipping the + hostname check removes a defense against + man-in-the-middle attacks; do not enable + casually. mTLS client authentication is + unaffected by this option. See the Apache Kafka + SSL security guide for background on this check: + https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification + required: + - keystore_location + - keystore_password_ref + - key_password_ref + - truststore_location + - truststore_password_ref + description: >- + Mutual-TLS (mTLS) authentication configuration. The + keystore (client certificate + private key) + and truststore (CAs trusted to verify the broker) live as JKS files on Unity Catalog volumes, + with their passwords stored in Databricks secret scopes. This matches the SSL setup pattern + documented at + https://docs.databricks.com/en/connect/streaming/kafka/authentication#use-ssl-to-connect-databricks-to-kafka. + + At materialization time, the generated PySpark code passes the JKS file paths and resolved + passwords through to the Kafka SSL options (kafka.ssl.keystore.location, + kafka.ssl.keystore.password, kafka.ssl.key.password, kafka.ssl.truststore.location, + kafka.ssl.truststore.password). Passwords are resolved on the Spark cluster via + dbutils.secrets.get; this message stores only references, never password values. + uc_service_credential_name: + type: string + description: >- + Name of the Unity Catalog service credential. This + value will be set under the option + databricks.serviceCredential + description: Authentication configuration for connection to topics. + backfill_source: + type: object + properties: + delta_table_name: + type: string + delta_table_source: + type: object + properties: + full_name: + type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string description: >- Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of @@ -3545,12 +4157,18 @@ paths: filter_condition. Should contains all the columns needed (eg. "SELECT *, col_a + col_b AS col_c FROM x.y.z WHERE col_a > 0" would have - `transformation_sql` "*, col_a + col_b AS + ``transformation_sql`` "*, col_a + col_b AS col_c") If transformation_sql is not provided, all columns of the delta table are present in the DataSource dataframe. required: - full_name + description: >- + Deprecated: Use delta_table_name instead. Kept for + backwards compatibility. The Delta table source + containing the historical data to backfill. Only the + delta table name is used for backfill, other fields + are ignored. description: >- A user-provided and managed source for backfilling data. Historical data is used when creating a training set @@ -3564,11 +4182,162 @@ paths: description: >- Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*) + ingestion_config: + type: object + properties: + ingestion_destination: + type: object + properties: + delta_table_name: + type: string + description: >- + The full three-part name (catalog, schema, name) + of the Delta table to be created for ingestion. + description: >- + Destination for the Databricks-managed Delta table + that holds an offline copy of the streaming data for + querying and training. This table contains both 1) + forward-filled data from the Stream and 2) + backfilled data from the BackfillSource (if + provided). This table is created and managed by + Databricks and is deleted when the Stream is + deleted. + backfill_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + historical backfill of the ingestion Delta table. + backfill_source: + type: object + properties: + delta_table_name: + type: string + delta_table_source: + type: object + properties: + full_name: + type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON + format (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string + description: >- + Deprecated: Use Feature.entity instead. Kept + for backwards compatibility. The entity + columns of the Delta table. + filter_condition: + type: string + description: >- + Single WHERE clause to filter delta table + before applying transformations. Will be + row-wise evaluated, so should only include + conditionals and projections. + timeseries_column: + type: string + description: >- + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. + The timeseries column of the Delta table. + transformation_sql: + type: string + description: >- + A single SQL SELECT expression applied after + filter_condition. Should contains all the + columns needed (eg. "SELECT *, col_a + col_b + AS col_c FROM x.y.z WHERE col_a > 0" would + have ``transformation_sql`` "*, col_a + + col_b AS col_c") If transformation_sql is + not provided, all columns of the delta table + are present in the DataSource dataframe. + required: + - full_name + description: >- + Deprecated: Use delta_table_name instead. Kept + for backwards compatibility. The Delta table + source containing the historical data to + backfill. Only the delta table name is used for + backfill, other fields are ignored. + description: >- + A user-provided source for backfilling data. + Historical data is used when creating a training set + from streaming features linked to this Stream. The + backfill data stored in this location will be copied + into the ingestion table for offline querying and + training. The schema for this source must match + exactly that of the key and payload schemas + specified for this Stream. + deduplication_columns: + type: array + items: + type: string + description: >- + Column paths used to identify duplicate rows during + ingestion; only one row per distinct combination of + these values is kept. Use dot notation for nested + fields (e.g. ``value.user_id``). Empty list means + every column is compared. + ingestion_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + forward-fill ingestion. + ingestion_pipeline_id: + type: string + description: >- + The ID of the SDP pipeline that continuously copies + new events from the streaming source into the + ingestion Delta table. + required: + - ingestion_destination + description: >- + Configuration for ingesting Kafka data into a + Databricks-managed Delta table. key_schema: type: object properties: + avro_schema: + type: string json_schema: type: string + description: >- + Schema of the JSON object in standard IETF JSON + schema format (https://json-schema.org/). + proto_schema: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and proto3 + syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message within + schema_text that describes the Kafka payload + (e.g. "Event" or "com.example.Event" if + schema_text declares a package). Identifies + which message is used to decode each Kafka + record — a .proto file may declare multiple + messages but only one represents the payload. + Must not be empty. + required: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload message + name. description: >- Schema configuration for extracting message keys from topics. At least one of key_schema and value_schema must @@ -3576,8 +4345,41 @@ paths: value_schema: type: object properties: + avro_schema: + type: string json_schema: type: string + description: >- + Schema of the JSON object in standard IETF JSON + schema format (https://json-schema.org/). + proto_schema: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and proto3 + syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message within + schema_text that describes the Kafka payload + (e.g. "Event" or "com.example.Event" if + schema_text declares a package). Identifies + which message is used to decode each Kafka + record — a .proto file may declare multiple + messages but only one represents the payload. + Must not be empty. + required: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload message + name. description: >- Schema configuration for extracting message values from topics. At least one of key_schema and value_schema must @@ -3692,7 +4494,25 @@ paths: description: >- The quartz cron expression that defines the schedule of the materialization pipeline. The schedule is evaluated - in the UTC timezone. + in the UTC timezone. Hidden from GraphQL: superseded by + the ``trigger`` oneof (cron_schedule_trigger), so not + exposed to Catalog Explorer. + cron_schedule_trigger: + type: object + properties: + cron_expression: + type: string + description: >- + The cron expression defining the schedule (e.g., "0 + 0 * * *" for daily at midnight). + description: >- + A cron-based schedule trigger for the materialization + pipeline. + is_online: + type: boolean + description: >- + True if this is an online materialized feature. False if + it is an offline materialized feature. last_materialization_time: type: string description: >- @@ -3701,7 +4521,9 @@ paths: yet, this field will be null. materialized_feature_id: type: string - description: Unique identifier for the materialized feature. + description: >- + Server-assigned unique identifier for the materialized + feature. offline_store_config: type: object properties: @@ -3721,7 +4543,9 @@ paths: - catalog_name - schema_name - table_name_prefix - description: Configuration for offline store destination. + description: >- + Destination for writing feature values to an offline + Delta table. online_store_config: type: object properties: @@ -3753,20 +4577,54 @@ paths: - schema_name - table_name_prefix - online_store_name - description: Configuration for online store destination. + description: >- + Destination for writing feature values to an online + Lakebase table. pipeline_schedule_state: type: string x-enum: - ACTIVE - PAUSED - SNAPSHOT - description: The schedule state of the materialization pipeline. + description: >- + The schedule state of the materialization pipeline. + Hidden from GraphQL: being deprecated, so not exposed to + Catalog Explorer. + streaming_mode: + type: object + properties: + freshness_target: + type: string + description: >- + The desired data freshness for feature + materialization, expressed as a duration string + (e.g. "1 minute"). + mode: + type: string + x-enum: + - STREAMING_MODE_TYPE_MBM + - STREAMING_MODE_TYPE_RTM + description: >- + The type of streaming mode used by the + materialization pipeline. + description: >- + The Structured Streaming trigger mode used for + materialization. Real-time mode (RTM) targets sub-second + latency for operational workloads; micro-batch mode + (MBM) favors cost efficiency for ETL and analytics + workloads. table_name: type: string description: >- The fully qualified Unity Catalog path to the table containing the materialized feature (Delta table or Lakebase table). Output only. + table_trigger: + type: object + properties: {} + description: >- + A trigger that fires when the upstream source table + changes. required: - feature_name description: The materialized feature to create. @@ -3852,381 +4710,1272 @@ paths: message: type: string x-stackql-sdk-source: FeatureEngineeringAPI - /api/2.0/feature-engineering/features/{full_name}: - delete: - operationId: feature_engineering_delete_feature - summary: Delete a Feature. - tags: - - ml - - feature_engineering - description: |- - Delete a Feature. - - :param full_name: str - Name of the feature to delete. - parameters: - - name: full_name - in: path - required: true - schema: - type: string - description: Name of the feature to delete. - responses: - '200': - description: Success - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: FeatureEngineeringAPI - get: - operationId: feature_engineering_get_feature - summary: Get a Feature. + /api/2.0/feature-engineering/streams: + post: + operationId: feature_engineering_create_stream + summary: >- + Create a Stream, a governed UC entity representing an external streaming + data source. tags: - ml - feature_engineering - description: |- - Get a Feature. - - :param full_name: str - Name of the feature to get. + description: >- + Create a Stream, a governed UC entity representing an external streaming + data source. - :returns: :class:`Feature` - parameters: - - name: full_name - in: path - required: true - schema: - type: string - description: Name of the feature to get. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/Feature' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: FeatureEngineeringAPI - patch: - operationId: feature_engineering_update_feature - summary: Update a Feature. - tags: - - ml - - feature_engineering - description: |- - Update a Feature. - :param full_name: str - The full three-part name (catalog, schema, name) of the feature. - :param feature: :class:`Feature` - Feature to update. - :param update_mask: str - The list of fields to update. + :param stream: :class:`Stream` + The Stream to create. - :returns: :class:`Feature` - parameters: - - name: full_name - in: path - required: true - schema: - type: string - description: The full three-part name (catalog, schema, name) of the feature. - - name: update_mask - in: query - required: true - schema: - type: string - description: The list of fields to update. + :returns: :class:`Stream` requestBody: content: application/json: schema: type: object properties: - feature: + stream: type: object properties: - full_name: + name: type: string - source: + description: >- + Full three-part (catalog.schema.stream) name of the + stream. + source_config: type: object properties: - delta_table_source: + kafka_stream_config: type: object properties: - full_name: - type: string - dataframe_schema: - type: string + subscription_mode: + type: object + properties: + assign: + type: string + description: >- + A JSON string that contains the specific + topic-partitions to consume from. For + example, for + '{"topicA":[0,1],"topicB":[2,4]}', topicA's + 0'th and 1st partitions will be consumed + from. + subscribe: + type: string + description: >- + A comma-separated list of Kafka topics to + read from. For example, + 'topicA,topicB,topicC'. + subscribe_pattern: + type: string + description: >- + A regular expression matching topics to + subscribe to. For example, 'topic.*' will + subscribe to all topics starting with + 'topic'. description: >- - Schema of the resulting dataframe after - transformations, in Spark StructType JSON format - (from df.schema.json()). Required if - transformation_sql is specified. Example: - {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} - entity_columns: - type: array - items: - type: string + Options to configure which Kafka topics to pull + data from. + extra_options: + type: object description: >- - Deprecated: Use Feature.entity instead. Kept for - backwards compatibility. The entity columns of - the Delta table. - filter_condition: - type: string - description: >- - Single WHERE clause to filter delta table before - applying transformations. Will be row-wise - evaluated, so should only include conditionals - and projections. - timeseries_column: - type: string - description: >- - Deprecated: Use Feature.timeseries_column - instead. Kept for backwards compatibility. The - timeseries column of the Delta table. - transformation_sql: - type: string - description: >- - A single SQL SELECT expression applied after - filter_condition. Should contains all the - columns needed (eg. "SELECT *, col_a + col_b AS - col_c FROM x.y.z WHERE col_a > 0" would have - `transformation_sql` "*, col_a + col_b AS - col_c") If transformation_sql is not provided, - all columns of the delta table are present in - the DataSource dataframe. + Optional Kafka source or consumer options, + validated against a server-side allowlist at + request time. Allowed keys: - + ``maxOffsetsPerTrigger`` - ``startingOffsets`` - + ``includeHeaders`` - + ``kafka.request.timeout.ms`` - + ``kafka.session.timeout.ms`` - + ``kafka.max.partition.fetch.bytes`` The + following keys are ingestion-only and are + stripped before being forwarded to the + materialization pipeline: - + ``maxOffsetsPerTrigger`` - ``startingOffsets`` + Auth and connection details belong on the parent + Stream's ``connection_config``, not here. required: - - full_name - kafka_source: + - subscription_mode + description: Configuration for Apache Kafka streams. + kinesis_stream_config: type: object properties: - name: - type: string - entity_column_identifiers: - type: array - items: - type: object - properties: - variant_expr_path: - type: string - required: - - variant_expr_path - description: >- - Deprecated: Use Feature.entity instead. Kept for - backwards compatibility. The entity column - identifiers of the Kafka source. - filter_condition: - type: string + extra_options: + type: object description: >- - The filter condition applied to the source data - before aggregation. - timeseries_column_identifier: + Optional Kinesis source options, validated + against a server-side allowlist at request time. + Auth and connection details belong on the parent + Stream's ``connection_config``, not here. + stream_arns: type: object properties: - variant_expr_path: - type: string - required: - - variant_expr_path - description: >- - Deprecated: Use Feature.timeseries_column - instead. Kept for backwards compatibility. The - timeseries column identifier of the Kafka - source. - required: - - name - description: The data source of the feature. - function: + arns: + type: array + items: + type: string + description: >- + Kinesis stream ARNs to read from. For + example, + 'arn:aws:kinesis:us-west-2:111122223333:stream/stream-a'. + description: Kinesis stream ARNs to read from. + stream_names: + type: object + properties: + names: + type: array + items: + type: string + description: Kinesis stream names to read from. + description: Kinesis stream names to read from. + description: Configuration for AWS Kinesis Data Streams. + description: >- + Source-specific configuration. Determines the streaming + platform source. + connection_config: type: object properties: - aggregation_function: + direct_mtls_config: type: object properties: - approx_count_distinct: + bootstrap_servers: + type: string + description: >- + A comma-separated list of host:port pairs for + the Kafka bootstrap servers. + mtls_config: type: object properties: - input: + keystore_location: type: string description: >- - The input column from which the approximate - count of distinct values is computed. - relative_sd: - type: number + Unity Catalog volume path to the JKS + keystore file containing the client + certificate and private key. e.g. + "/Volumes////client.jks". + The materialization compute must have read + permission on this volume. + keystore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key description: >- - The maximum relative standard deviation - allowed (default defined by Spark). - required: - - input - description: >- - Computes the approximate count of distinct - values. - approx_percentile: - type: object - properties: - input: + Secret-scope reference for the JKS keystore + password. + key_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the private key + password. Often the same value as the + keystore password (keytool's default), but + provided as a separate field because Apache + Kafka requires it as a distinct option + (kafka.ssl.key.password). + truststore_location: type: string description: >- - The input column from which the approximate - percentile is computed. - percentile: - type: number + Unity Catalog volume path to the JKS + truststore file containing the CA + certificate(s) trusted to verify the Kafka + broker's server certificate. e.g. + "/Volumes////truststore.jks". + truststore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key description: >- - The percentile value to compute (between 0 - and 1). - accuracy: - type: integer + Secret-scope reference for the JKS + truststore password. + disable_hostname_verification: + type: boolean description: >- - The accuracy parameter (higher is more - accurate but slower). + Set to true only when the broker + certificate's SAN intentionally does not + match the connection endpoint — for example + when reaching the cluster through a + PrivateLink endpoint whose DNS name is not + in the broker certificate. Skipping the + hostname check removes a defense against + man-in-the-middle attacks; do not enable + casually. mTLS client authentication is + unaffected by this option. See the Apache + Kafka SSL security guide for background on + this check: + https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification required: - - input - - percentile - description: Computes the approximate percentile of values. - avg: + - keystore_location + - keystore_password_ref + - key_password_ref + - truststore_location + - truststore_password_ref + description: Mutual-TLS authentication configuration. + required: + - bootstrap_servers + - mtls_config + description: >- + Direct mTLS configuration for stream platform + access. This is only used in the short term until UC + Kafka Connections support mTLS . Once UC Kafka + Connections support mTLS, this will be deprecated. + uc_connection_name: + type: string + description: >- + Name of an existing UC Connection for stream + platform access. Must be the correct type for the + streaming platform (e.g. a Kafka Connection for a + Kafka Stream, or a Kinesis Connection for a Kinesis + Stream). + description: >- + Specifies how to connect and authenticate to the stream + platform. + schema_config: + type: object + properties: + direct_schemas: + type: object + properties: + key_schema: type: object properties: - input: + avro_schema: + type: string + json_schema: type: string description: >- - The input column from which the average is - computed. - required: - - input - description: Computes the average of values. - count_function: + Schema of the JSON object in standard IETF + JSON schema format + (https://json-schema.org/). + proto_schema: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and + proto3 syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message + within schema_text that describes the + Kafka payload (e.g. "Event" or + "com.example.Event" if schema_text + declares a package). Identifies which + message is used to decode each Kafka + record — a .proto file may declare + multiple messages but only one + represents the payload. Must not be + empty. + required: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload + message name. + description: >- + Schema for the message key. This is only used + for Kafka streams. For Kafka, at least one of + payload_schema or key_schema must be specified. + payload_schema: type: object properties: - input: + avro_schema: + type: string + json_schema: type: string description: >- - The input column from which the count is - computed. - required: - - input - description: Computes the count of values. - first: + Schema of the JSON object in standard IETF + JSON schema format + (https://json-schema.org/). + proto_schema: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and + proto3 syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message + within schema_text that describes the + Kafka payload (e.g. "Event" or + "com.example.Event" if schema_text + declares a package). Identifies which + message is used to decode each Kafka + record — a .proto file may declare + multiple messages but only one + represents the payload. Must not be + empty. + required: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload + message name. + description: >- + Schema for the message payload. For Kafka, this + is the value schema. Unless the platform + supports another schema (e.g. keys for Kafka), + this must be specified. + description: Schema definitions provided directly on the Stream. + schema_registry_config: + type: object + properties: + api_secret_ref: type: object properties: - input: + scope: type: string - description: >- - The input column from which the first value - is returned. + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. required: - - input - description: Returns the first value. - last: + - scope + - key + description: >- + Reference to the schema registry API secret in a + Databricks secret scope. + key_schema_locator: type: object properties: - input: + format: type: string - description: >- - The input column from which the last value - is returned. + x-enum: + - FORMAT_AVRO + - FORMAT_JSON + - FORMAT_PROTOBUF + description: Serialization format for this schema. + confluent_schema: + type: object + properties: + subject: + type: string + description: >- + The Confluent schema registry subject + name. + required: + - subject + description: Confluent Schema Registry schema locator. required: - - input - description: Returns the last value. - max: + - format + description: >- + Schema locator for the message key. Only used + for Kafka streams. At least one of + payload_schema_locator or key_schema_locator + must be set. + payload_schema_locator: type: object properties: - input: + format: type: string - description: >- - The input column from which the maximum is - computed. + x-enum: + - FORMAT_AVRO + - FORMAT_JSON + - FORMAT_PROTOBUF + description: Serialization format for this schema. + confluent_schema: + type: object + properties: + subject: + type: string + description: >- + The Confluent schema registry subject + name. + required: + - subject + description: Confluent Schema Registry schema locator. required: - - input - description: Computes the maximum value. - min: + - format + description: >- + Schema locator for the message payload. For + Kafka this is the value. At least one of + payload_schema_locator or key_schema_locator + must be set. + uc_connection: + type: string + description: A Schema Registry UC Connection object. + required: + - api_secret_ref + description: Resolve schemas from an external schema registry. + description: >- + Schema definitions for the stream, provided either + directly on the Stream or resolved from an external + schema registry through a UC Connection. + ingestion_config: + type: object + properties: + ingestion_destination: + type: object + properties: + delta_table_name: + type: string + description: >- + The full three-part name (catalog, schema, name) + of the Delta table to be created for ingestion. + description: >- + Destination for the Databricks-managed Delta table + that holds an offline copy of the streaming data for + querying and training. This table contains both 1) + forward-filled data from the Stream and 2) + backfilled data from the BackfillSource (if + provided). This table is created and managed by + Databricks and is deleted when the Stream is + deleted. + backfill_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + historical backfill of the ingestion Delta table. + backfill_source: + type: object + properties: + delta_table_name: + type: string + delta_table_source: type: object properties: - input: + full_name: + type: string + dataframe_schema: type: string description: >- - The input column from which the minimum is - computed. - required: - - input - description: Computes the minimum value. - stddev_pop: - type: object - properties: - input: + Schema of the resulting dataframe after + transformations, in Spark StructType JSON + format (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string + description: >- + Deprecated: Use Feature.entity instead. Kept + for backwards compatibility. The entity + columns of the Delta table. + filter_condition: type: string description: >- - The input column from which the population - standard deviation is computed. - required: - - input - description: Computes the population standard deviation. - stddev_samp: - type: object - properties: - input: + Single WHERE clause to filter delta table + before applying transformations. Will be + row-wise evaluated, so should only include + conditionals and projections. + timeseries_column: type: string description: >- - The input column from which the sample - standard deviation is computed. - required: - - input - description: Computes the sample standard deviation. - sum: - type: object - properties: - input: + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. + The timeseries column of the Delta table. + transformation_sql: type: string description: >- - The input column from which the sum is - computed. + A single SQL SELECT expression applied after + filter_condition. Should contains all the + columns needed (eg. "SELECT *, col_a + col_b + AS col_c FROM x.y.z WHERE col_a > 0" would + have ``transformation_sql`` "*, col_a + + col_b AS col_c") If transformation_sql is + not provided, all columns of the delta table + are present in the DataSource dataframe. required: - - input - description: Computes the sum of values. - time_window: - type: object - properties: - continuous: - type: object - properties: - window_duration: - type: string - offset: + - full_name + description: >- + Deprecated: Use delta_table_name instead. Kept + for backwards compatibility. The Delta table + source containing the historical data to + backfill. Only the delta table name is used for + backfill, other fields are ignored. + description: >- + A user-provided source for backfilling data. + Historical data is used when creating a training set + from streaming features linked to this Stream. The + backfill data stored in this location will be copied + into the ingestion table for offline querying and + training. The schema for this source must match + exactly that of the key and payload schemas + specified for this Stream. + deduplication_columns: + type: array + items: + type: string + description: >- + Column paths used to identify duplicate rows during + ingestion; only one row per distinct combination of + these values is kept. Use dot notation for nested + fields (e.g. ``value.user_id``). Empty list means + every column is compared. + ingestion_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + forward-fill ingestion. + ingestion_pipeline_id: + type: string + description: >- + The ID of the SDP pipeline that continuously copies + new events from the streaming source into the + ingestion Delta table. + required: + - ingestion_destination + description: >- + Configuration for streaming data ingestion: the managed + table storing an offline copy of forward fill data and + optional historical backfill. + browse_only: + type: boolean + description: >- + Indicates whether the principal is limited to retrieving + metadata for the associated object through the BROWSE + privilege when include_browse is enabled in the request. + create_time: + type: string + format: date-time + description: Time at which this Stream was created. + created_by: + type: string + description: Username of the Stream creator. + description: + type: string + description: User-provided description. + update_time: + type: string + format: date-time + description: Time at which this Stream was last modified. + updated_by: + type: string + description: Username of user who last modified the Stream. + required: + - name + - source_config + - connection_config + - schema_config + - ingestion_config + description: The Stream to create. + required: + - stream + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Stream' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + get: + operationId: feature_engineering_list_streams + summary: List Streams under a given catalog.schema parent. + tags: + - ml + - feature_engineering + description: |- + List Streams under a given catalog.schema parent. + + :param page_size: int (optional) + The maximum number of results to return. + :param page_token: str (optional) + Pagination token to go to the next page based on a previous query. + :param parent: str (optional) + Two-part name (catalog.schema) of the parent under which to list Streams. + + :returns: Iterator over :class:`Stream` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of results to return. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token to go to the next page based on a previous query. + - name: parent + in: query + required: false + schema: + type: string + description: >- + Two-part name (catalog.schema) of the parent under which to list + Streams. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListStreamsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + /api/2.0/feature-engineering/features/{full_name}: + delete: + operationId: feature_engineering_delete_feature + summary: Delete a Feature. + tags: + - ml + - feature_engineering + description: |- + Delete a Feature. + + :param full_name: str + Name of the feature to delete. + parameters: + - name: full_name + in: path + required: true + schema: + type: string + description: Name of the feature to delete. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + get: + operationId: feature_engineering_get_feature + summary: Get a Feature. + tags: + - ml + - feature_engineering + description: |- + Get a Feature. + + :param full_name: str + Name of the feature to get. + + :returns: :class:`Feature` + parameters: + - name: full_name + in: path + required: true + schema: + type: string + description: Name of the feature to get. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Feature' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + patch: + operationId: feature_engineering_update_feature + summary: Update a Feature. + tags: + - ml + - feature_engineering + description: |- + Update a Feature. + + :param full_name: str + The full three-part name (catalog, schema, name) of the feature. This is the feature's resource + identifier; the catalog_name, schema_name, and name fields below are OUTPUT_ONLY decomposed views of + this value. + :param feature: :class:`Feature` + Feature to update. + :param update_mask: str + The list of fields to update. + + :returns: :class:`Feature` + parameters: + - name: full_name + in: path + required: true + schema: + type: string + description: >- + The full three-part name (catalog, schema, name) of the feature. + This is the feature's resource identifier; the catalog_name, + schema_name, and name fields below are OUTPUT_ONLY decomposed views + of this value. + - name: update_mask + in: query + required: true + schema: + type: string + description: The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + feature: + type: object + properties: + full_name: + type: string + source: + type: object + properties: + delta_table_source: + type: object + properties: + full_name: + type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string + description: >- + Deprecated: Use Feature.entity instead. Kept for + backwards compatibility. The entity columns of + the Delta table. + filter_condition: + type: string + description: >- + Single WHERE clause to filter delta table before + applying transformations. Will be row-wise + evaluated, so should only include conditionals + and projections. + timeseries_column: + type: string + description: >- + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. The + timeseries column of the Delta table. + transformation_sql: + type: string + description: >- + A single SQL SELECT expression applied after + filter_condition. Should contains all the + columns needed (eg. "SELECT *, col_a + col_b AS + col_c FROM x.y.z WHERE col_a > 0" would have + ``transformation_sql`` "*, col_a + col_b AS + col_c") If transformation_sql is not provided, + all columns of the delta table are present in + the DataSource dataframe. + required: + - full_name + description: A Delta table data source. + kafka_source: + type: object + properties: + name: + type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Any subsequent + functions operate against this dataframe. + entity_column_identifiers: + type: array + items: + type: object + properties: + variant_expr_path: + type: string + required: + - variant_expr_path + description: >- + Deprecated: Use Feature.entity instead. Kept for + backwards compatibility. The entity column + identifiers of the Kafka source. + filter_condition: + type: string + description: >- + The filter condition applied to the source data + before aggregation. + timeseries_column_identifier: + type: object + properties: + variant_expr_path: + type: string + required: + - variant_expr_path + description: >- + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. The + timeseries column identifier of the Kafka + source. + transformation_sql: + type: string + description: >- + The pipeline runs these SQL statements + immediately after conversion into the schema + specified on the KafkaConfig object. + required: + - name + description: A Kafka stream data source. + request_source: + type: object + properties: + flat_schema: + type: object + properties: + fields: + type: array + items: + type: object + properties: + name: + type: string + description: The name of the field. + data_type: + type: string + x-enum: + - BINARY + - BOOLEAN + - DATE + - DECIMAL + - DOUBLE + - FLOAT + - INTEGER + - LONG + - SHORT + - STRING + - TIMESTAMP + description: The scalar data type of the field. + required: + - name + - data_type + description: >- + A single field definition within a + FlatSchema, specifying the field name and + its scalar data + type. Does not support nested or complex types (arrays, maps, structs). + description: The list of fields in this schema. + required: + - fields + description: A flat schema with scalar-typed fields only. + description: A request-time data source. + stream_source: + type: object + properties: + full_name: + type: string + description: >- + Three-part full name of the Stream + (catalog.schema.stream). + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Any subsequent + functions operate against this dataframe. + filter_condition: + type: string + description: >- + The filter condition applied to the source data + before aggregation. + transformation_sql: + type: string + description: >- + The pipeline runs these SQL statements + immediately after conversion into the schema + specified on the Stream object. + required: + - full_name + description: A Stream data source. + description: The data source of the feature. + function: + type: object + properties: + aggregation_function: + type: object + properties: + approx_count_distinct: + type: object + properties: + input: + type: string + description: >- + The input column from which the approximate + count of distinct values is computed. + relative_sd: + type: number + description: >- + The maximum relative standard deviation + allowed (default defined by Spark). + required: + - input + description: >- + Computes the approximate count of distinct + values. + approx_percentile: + type: object + properties: + input: + type: string + description: >- + The input column from which the approximate + percentile is computed. + percentile: + type: number + description: >- + The percentile value to compute (between 0 + and 1). + accuracy: + type: integer + description: >- + The accuracy parameter (higher is more + accurate but slower). + required: + - input + - percentile + description: Computes the approximate percentile of values. + avg: + type: object + properties: + input: + type: string + description: >- + The input column from which the average is + computed. For Kafka sources, use + dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf + node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; + migrate to dot notation. + required: + - input + description: Computes the average of values. + count_function: + type: object + properties: + input: + type: string + description: >- + The input column from which the count is + computed. For Kafka sources, use + dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf + node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; + migrate to dot notation. + required: + - input + description: Computes the count of values. + first: + type: object + properties: + input: + type: string + description: >- + The input column from which the first value + is returned. + required: + - input + description: Returns the first value. + first_distinct: + type: object + properties: + input: + type: string + description: >- + The input column from which the first N + distinct values are returned. + 'n': + type: integer + description: The number of distinct values to return. + required: + - input + - 'n' + description: >- + Returns the first N distinct values, ordered by + the feature's timeseries column. + first_n: + type: object + properties: + input: + type: string + description: >- + The input column from which the first N + values are returned. + 'n': + type: integer + description: The number of values to return. + required: + - input + - 'n' + description: >- + Returns the first N values, ordered by the + feature's timeseries column. + last: + type: object + properties: + input: + type: string + description: >- + The input column from which the last value + is returned. + required: + - input + description: Returns the last value. + last_distinct: + type: object + properties: + input: + type: string + description: >- + The input column from which the last N + distinct values are returned. + 'n': + type: integer + description: The number of distinct values to return. + required: + - input + - 'n' + description: >- + Returns the last N distinct values, ordered by + the feature's timeseries column. + last_n: + type: object + properties: + input: + type: string + description: >- + The input column from which the last N + values are returned. + 'n': + type: integer + description: The number of values to return. + required: + - input + - 'n' + description: >- + Returns the last N values, ordered by the + feature's timeseries column. + max: + type: object + properties: + input: + type: string + description: >- + The input column from which the maximum is + computed. + required: + - input + description: Computes the maximum value. + min: + type: object + properties: + input: + type: string + description: >- + The input column from which the minimum is + computed. + required: + - input + description: Computes the minimum value. + stddev_pop: + type: object + properties: + input: + type: string + description: >- + The input column from which the population + standard deviation is computed. For Kafka + sources, use dot-prefixed path notation + (e.g., "value.amount"). For nested fields, + the leaf node name is used. Colon-prefixed + notation (e.g., "value:amount") is supported + for backwards compatibility but is + deprecated; migrate to dot notation. + required: + - input + description: Computes the population standard deviation. + stddev_samp: + type: object + properties: + input: + type: string + description: >- + The input column from which the sample + standard deviation is computed. + required: + - input + description: Computes the sample standard deviation. + sum: + type: object + properties: + input: + type: string + description: >- + The input column from which the sum is + computed. For Kafka sources, use + dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf + node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; + migrate to dot notation. + required: + - input + description: Computes the sum of values. + time_window: + type: object + properties: + continuous: + type: object + properties: + window_duration: + type: string + description: >- + The duration of the continuous window + (must be positive). + offset: type: string description: >- The offset of the continuous window (must be non-positive). required: - window_duration - sliding: + description: >- + Deprecated: use RollingWindow with ``delay`` + instead. + rolling: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the + rolling window (must be non-negative). + For example, delay=1d shifts the window + end 1 day before the evaluation time. + window_duration: + type: string + description: >- + The duration of the rolling window. Must + be positive when set; absent means + lifetime (aggregate over the entity's + entire history). + description: >- + A rolling time window with an optional + delay. This is the SQL-spec-aligned + replacement for + ContinuousWindow: ``delay`` is the non-negative counterpart of the legacy non-positive + ``ContinuousWindow.offset``. + sawtooth: type: object properties: + delay: + type: string + description: >- + The delay applied to the end of the + window (must be non-negative). For + example, delay=1d shifts the window end + 1 day before the evaluation time. window_duration: type: string + description: >- + The duration of the window. Must be + positive and span more than two days + when set, so that both the batch (N-1 + day) and stale-path (N-2 day) partial + aggregates are well defined. The + duration need not be a whole number of + days (e.g. 3 days 15 minutes is + allowed). Absent means lifetime + (aggregate over the entity's entire + history). + description: >- + A sawtooth window served via the hybrid + batch + streaming path. + sliding: + type: object + properties: slide_duration: type: string + window_duration: + type: string description: >- - The slide duration (interval by which - windows advance, must be positive and - less than duration). + The duration of the sliding window. Must + be positive when set; absent means + lifetime (aggregate over the entity's + entire history). required: - - window_duration - slide_duration tumbling: type: object @@ -4236,196 +5985,892 @@ paths: required: - window_duration description: >- - The time window over which the aggregation is - computed. - var_pop: + The time window over which the aggregation is + computed. + var_pop: + type: object + properties: + input: + type: string + description: >- + The input column from which the population + variance is computed. + required: + - input + description: Computes the population variance. + var_samp: + type: object + properties: + input: + type: string + description: >- + The input column from which the sample + variance is computed. + required: + - input + description: Computes the sample variance. + description: An aggregation function applied over a time window. + column_selection: + type: object + properties: + column: + type: string + description: >- + Column name from source to select as the feature + value. + required: + - column + description: >- + Selects the latest value of a single column in a + data source + custom_udf: + type: object + properties: + function_path: + type: string + description: >- + Fully qualified 3-part Unity Catalog path of the + function to apply. + input_bindings: + type: array + items: + type: object + properties: + parameter: + type: string + description: Name of the UC function parameter. + column: + type: string + description: >- + Source column whose value is passed for + this parameter at execution time. + required: + - parameter + - column + description: >- + Binds a single UC function parameter to a + source column. + description: >- + Binds each UC function parameter to a source + column. May be empty for zero-argument functions + (e.g. a timestamp generator). + required: + - function_path + description: >- + Applies a registered Unity Catalog function row-wise + to source columns. + extra_parameters: + type: array + items: + type: object + properties: + key: + type: string + description: The name of the parameter. + value: + type: string + description: The value of the parameter. + required: + - key + - value + description: >- + Deprecated: Use typed fields on function-specific + messages (e.g. + ApproxPercentileFunction.percentile) or AggregationFunction.ExtraParameter instead. Kept for + backwards compatibility. + description: >- + Deprecated: Use the function oneof with + AggregationFunction instead. Kept for backwards + compatibility. Extra parameters for parameterized + functions. + function_type: + type: string + x-enum: + - APPROX_COUNT_DISTINCT + - APPROX_PERCENTILE + - AVG + - COUNT + - FIRST + - LAST + - MAX + - MIN + - STDDEV_POP + - STDDEV_SAMP + - SUM + - VAR_POP + - VAR_SAMP + description: >- + Deprecated: Use the function oneof with + AggregationFunction instead. Kept for backwards + compatibility. The type of the function. + description: The function by which the feature is computed. + catalog_name: + type: string + description: Name of parent catalog. + created_at: + type: string + format: date-time + description: Time at which this feature was created. + created_by: + type: string + description: Username of the feature creator. + description: + type: string + description: The description of the feature. + entities: + type: array + items: + type: object + properties: + name: + type: string + required: + - name + description: >- + The entity columns for the feature, used as aggregation + keys and for query-time lookup. + filter_condition: + type: string + description: >- + Deprecated: Use DeltaTableSource.filter_condition or + KafkaSource.filter_condition instead. Kept for backwards + compatibility. The filter condition applied to the + source data before aggregation. + inputs: + type: array + items: + type: string + description: >- + Deprecated: Use AggregationFunction.inputs instead. Kept + for backwards compatibility. The input columns from + which the feature is computed. + lineage_context: + type: object + properties: + job_context: + type: object + properties: + job_id: + type: integer + job_run_id: + type: integer + description: The job run ID where this API was invoked. + description: Job context information including job ID and run ID. + notebook_id: + type: integer + description: The notebook ID where this API was invoked. + description: >- + Lineage context information for this feature. WARNING: + This field is primarily intended for internal use by + Databricks systems and is automatically populated when + features are created through Databricks notebooks or + jobs. Users should not manually set this field as + incorrect values may lead to inaccurate lineage tracking + or unexpected behavior. This field will be set by + feature-engineering client and should be left unset by + SDK and terraform users. + name: + type: string + description: >- + Name of the feature, extracted from the full three-part + name (catalog.schema.name). + schema_name: + type: string + description: Name of parent schema relative to its parent catalog. + time_window: + type: object + properties: + continuous: + type: object + properties: + window_duration: + type: string + description: >- + The duration of the continuous window (must be + positive). + offset: + type: string + description: >- + The offset of the continuous window (must be + non-positive). + required: + - window_duration + description: >- + Deprecated: use RollingWindow with ``delay`` + instead. + rolling: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the rolling + window (must be non-negative). For example, + delay=1d shifts the window end 1 day before the + evaluation time. + window_duration: + type: string + description: >- + The duration of the rolling window. Must be + positive when set; absent means lifetime + (aggregate over the entity's entire history). + description: >- + A rolling time window with an optional delay. This + is the SQL-spec-aligned replacement for + ContinuousWindow: ``delay`` is the non-negative counterpart of the legacy non-positive + ``ContinuousWindow.offset``. + sawtooth: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the window (must + be non-negative). For example, delay=1d shifts + the window end 1 day before the evaluation time. + window_duration: + type: string + description: >- + The duration of the window. Must be positive and + span more than two days when set, so that both + the batch (N-1 day) and stale-path (N-2 day) + partial aggregates are well defined. The + duration need not be a whole number of days + (e.g. 3 days 15 minutes is allowed). Absent + means lifetime (aggregate over the entity's + entire history). + description: >- + A sawtooth window served via the hybrid batch + + streaming path. + sliding: + type: object + properties: + slide_duration: + type: string + window_duration: + type: string + description: >- + The duration of the sliding window. Must be + positive when set; absent means lifetime + (aggregate over the entity's entire history). + required: + - slide_duration + tumbling: + type: object + properties: + window_duration: + type: string + required: + - window_duration + description: >- + Deprecated: Use + Function.aggregation_function.time_window instead. Kept + for backwards compatibility. The time window in which + the feature is computed. + timeseries_column: + type: object + properties: + name: + type: string + required: + - name + description: >- + Column recording time, used for point-in-time joins, + backfills, and aggregations. + required: + - full_name + - source + - function + description: Feature to update. + required: + - feature + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Feature' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + /api/2.0/feature-engineering/features/kafka-configs/{name}: + delete: + operationId: feature_engineering_delete_kafka_config + summary: >- + Delete a Kafka config. During PrPr, Kafka configs can be read and used + when creating features under + tags: + - ml + - feature_engineering + description: >- + Delete a Kafka config. During PrPr, Kafka configs can be read and used + when creating features under + + the entire metastore. Only the creator of the Kafka config can delete + it. + + + :param name: str + Name of the Kafka config to delete. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the Kafka config to delete. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + get: + operationId: feature_engineering_get_kafka_config + summary: >- + Get a Kafka config. During PrPr, Kafka configs can be read and used when + creating features under the + tags: + - ml + - feature_engineering + description: >- + Get a Kafka config. During PrPr, Kafka configs can be read and used when + creating features under the + + entire metastore. Only the creator of the Kafka config can delete it. + + + :param name: str + Name of the Kafka config to get. + + :returns: :class:`KafkaConfig` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Name of the Kafka config to get. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/KafkaConfig' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: FeatureEngineeringAPI + patch: + operationId: feature_engineering_update_kafka_config + summary: >- + Update a Kafka config. During PrPr, Kafka configs can be read and used + when creating features under + tags: + - ml + - feature_engineering + description: >- + Update a Kafka config. During PrPr, Kafka configs can be read and used + when creating features under + + the entire metastore. Only the creator of the Kafka config can delete + it. + + + :param name: str + Name that uniquely identifies this Kafka config within the metastore. This will be the identifier + used from the Feature object to reference these configs for a feature. Can be distinct from topic + name. + :param kafka_config: :class:`KafkaConfig` + The Kafka config to update. + :param update_mask: FieldMask + The list of fields to update. + + :returns: :class:`KafkaConfig` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Name that uniquely identifies this Kafka config within the + metastore. This will be the identifier used from the Feature object + to reference these configs for a feature. Can be distinct from topic + name. + - name: update_mask + in: query + required: true + schema: + type: object + description: The list of fields to update. + requestBody: + content: + application/json: + schema: + type: object + properties: + kafka_config: + type: object + properties: + name: + type: string + bootstrap_servers: + type: string + description: >- + A comma-separated list of host/port pairs pointing to + Kafka cluster. + subscription_mode: + type: object + properties: + assign: + type: string + description: >- + A JSON string that contains the specific + topic-partitions to consume from. For example, for + '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and + 1st partitions will be consumed from. + subscribe: + type: string + description: >- + A comma-separated list of Kafka topics to read from. + For example, 'topicA,topicB,topicC'. + subscribe_pattern: + type: string + description: >- + A regular expression matching topics to subscribe + to. For example, 'topic.*' will subscribe to all + topics starting with 'topic'. + description: >- + Options to configure which Kafka topics to pull data + from. + auth_config: + type: object + properties: + mtls_config: + type: object + properties: + keystore_location: + type: string + description: >- + Unity Catalog volume path to the JKS keystore + file containing the client certificate and + private key. e.g. + "/Volumes////client.jks". + The materialization compute must have read + permission on this volume. + keystore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the JKS keystore + password. + key_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the private key + password. Often the same value as the keystore + password (keytool's default), but provided as a + separate field because Apache Kafka requires it + as a distinct option (kafka.ssl.key.password). + truststore_location: + type: string + description: >- + Unity Catalog volume path to the JKS truststore + file containing the CA certificate(s) trusted to + verify the Kafka broker's server certificate. + e.g. + "/Volumes////truststore.jks". + truststore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the JKS truststore + password. + disable_hostname_verification: + type: boolean + description: >- + Set to true only when the broker certificate's + SAN intentionally does not match the connection + endpoint — for example when reaching the cluster + through a PrivateLink endpoint whose DNS name is + not in the broker certificate. Skipping the + hostname check removes a defense against + man-in-the-middle attacks; do not enable + casually. mTLS client authentication is + unaffected by this option. See the Apache Kafka + SSL security guide for background on this check: + https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification + required: + - keystore_location + - keystore_password_ref + - key_password_ref + - truststore_location + - truststore_password_ref + description: >- + Mutual-TLS (mTLS) authentication configuration. The + keystore (client certificate + private key) + and truststore (CAs trusted to verify the broker) live as JKS files on Unity Catalog volumes, + with their passwords stored in Databricks secret scopes. This matches the SSL setup pattern + documented at + https://docs.databricks.com/en/connect/streaming/kafka/authentication#use-ssl-to-connect-databricks-to-kafka. + + At materialization time, the generated PySpark code passes the JKS file paths and resolved + passwords through to the Kafka SSL options (kafka.ssl.keystore.location, + kafka.ssl.keystore.password, kafka.ssl.key.password, kafka.ssl.truststore.location, + kafka.ssl.truststore.password). Passwords are resolved on the Spark cluster via + dbutils.secrets.get; this message stores only references, never password values. + uc_service_credential_name: + type: string + description: >- + Name of the Unity Catalog service credential. This + value will be set under the option + databricks.serviceCredential + description: Authentication configuration for connection to topics. + backfill_source: + type: object + properties: + delta_table_name: + type: string + delta_table_source: + type: object + properties: + full_name: + type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON format + (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string + description: >- + Deprecated: Use Feature.entity instead. Kept for + backwards compatibility. The entity columns of + the Delta table. + filter_condition: + type: string + description: >- + Single WHERE clause to filter delta table before + applying transformations. Will be row-wise + evaluated, so should only include conditionals + and projections. + timeseries_column: + type: string + description: >- + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. The + timeseries column of the Delta table. + transformation_sql: + type: string + description: >- + A single SQL SELECT expression applied after + filter_condition. Should contains all the + columns needed (eg. "SELECT *, col_a + col_b AS + col_c FROM x.y.z WHERE col_a > 0" would have + ``transformation_sql`` "*, col_a + col_b AS + col_c") If transformation_sql is not provided, + all columns of the delta table are present in + the DataSource dataframe. + required: + - full_name + description: >- + Deprecated: Use delta_table_name instead. Kept for + backwards compatibility. The Delta table source + containing the historical data to backfill. Only the + delta table name is used for backfill, other fields + are ignored. + description: >- + A user-provided and managed source for backfilling data. + Historical data is used when creating a training set + from streaming features linked to this Kafka config. In + the future, a separate table will be maintained by + Databricks for forward filling data. The schema for this + source must match exactly that of the key and value + schemas specified for this Kafka config. + extra_options: + type: object + description: >- + Catch-all for miscellaneous options. Keys should be + source options or Kafka consumer options (kafka.*) + ingestion_config: + type: object + properties: + ingestion_destination: + type: object + properties: + delta_table_name: + type: string + description: >- + The full three-part name (catalog, schema, name) + of the Delta table to be created for ingestion. + description: >- + Destination for the Databricks-managed Delta table + that holds an offline copy of the streaming data for + querying and training. This table contains both 1) + forward-filled data from the Stream and 2) + backfilled data from the BackfillSource (if + provided). This table is created and managed by + Databricks and is deleted when the Stream is + deleted. + backfill_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + historical backfill of the ingestion Delta table. + backfill_source: + type: object + properties: + delta_table_name: + type: string + delta_table_source: type: object properties: - input: + full_name: + type: string + dataframe_schema: type: string description: >- - The input column from which the population - variance is computed. - required: - - input - description: Computes the population variance. - var_samp: - type: object - properties: - input: + Schema of the resulting dataframe after + transformations, in Spark StructType JSON + format (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string + description: >- + Deprecated: Use Feature.entity instead. Kept + for backwards compatibility. The entity + columns of the Delta table. + filter_condition: type: string description: >- - The input column from which the sample - variance is computed. + Single WHERE clause to filter delta table + before applying transformations. Will be + row-wise evaluated, so should only include + conditionals and projections. + timeseries_column: + type: string + description: >- + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. + The timeseries column of the Delta table. + transformation_sql: + type: string + description: >- + A single SQL SELECT expression applied after + filter_condition. Should contains all the + columns needed (eg. "SELECT *, col_a + col_b + AS col_c FROM x.y.z WHERE col_a > 0" would + have ``transformation_sql`` "*, col_a + + col_b AS col_c") If transformation_sql is + not provided, all columns of the delta table + are present in the DataSource dataframe. required: - - input - description: Computes the sample variance. - description: An aggregation function applied over a time window. - extra_parameters: + - full_name + description: >- + Deprecated: Use delta_table_name instead. Kept + for backwards compatibility. The Delta table + source containing the historical data to + backfill. Only the delta table name is used for + backfill, other fields are ignored. + description: >- + A user-provided source for backfilling data. + Historical data is used when creating a training set + from streaming features linked to this Stream. The + backfill data stored in this location will be copied + into the ingestion table for offline querying and + training. The schema for this source must match + exactly that of the key and payload schemas + specified for this Stream. + deduplication_columns: type: array items: - type: object - properties: - key: - type: string - description: The name of the parameter. - value: - type: string - description: The value of the parameter. - required: - - key - - value - description: >- - Deprecated: Use typed fields on function-specific - messages (e.g. - ApproxPercentileFunction.percentile) or AggregationFunction.ExtraParameter instead. Kept for - backwards compatibility. + type: string description: >- - Deprecated: Use the function oneof with - AggregationFunction instead. Kept for backwards - compatibility. Extra parameters for parameterized - functions. - function_type: + Column paths used to identify duplicate rows during + ingestion; only one row per distinct combination of + these values is kept. Use dot notation for nested + fields (e.g. ``value.user_id``). Empty list means + every column is compared. + ingestion_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + forward-fill ingestion. + ingestion_pipeline_id: type: string - x-enum: - - APPROX_COUNT_DISTINCT - - APPROX_PERCENTILE - - AVG - - COUNT - - FIRST - - LAST - - MAX - - MIN - - STDDEV_POP - - STDDEV_SAMP - - SUM - - VAR_POP - - VAR_SAMP description: >- - Deprecated: Use the function oneof with - AggregationFunction instead. Kept for backwards - compatibility. The type of the function. - description: The function by which the feature is computed. - description: - type: string - description: The description of the feature. - entities: - type: array - items: - type: object - properties: - name: - type: string - required: - - name - description: >- - The entity columns for the feature, used as aggregation - keys and for query-time lookup. - filter_condition: - type: string - description: >- - Deprecated: Use DeltaTableSource.filter_condition or - KafkaSource.filter_condition instead. Kept for backwards - compatibility. The filter condition applied to the - source data before aggregation. - inputs: - type: array - items: - type: string - description: >- - Deprecated: Use AggregationFunction.inputs instead. Kept - for backwards compatibility. The input columns from - which the feature is computed. - lineage_context: - type: object - properties: - job_context: - type: object - properties: - job_id: - type: integer - job_run_id: - type: integer - description: The job run ID where this API was invoked. - description: Job context information including job ID and run ID. - notebook_id: - type: integer - description: The notebook ID where this API was invoked. + The ID of the SDP pipeline that continuously copies + new events from the streaming source into the + ingestion Delta table. + required: + - ingestion_destination description: >- - Lineage context information for this feature. WARNING: - This field is primarily intended for internal use by - Databricks systems and is automatically populated when - features are created through Databricks notebooks or - jobs. Users should not manually set this field as - incorrect values may lead to inaccurate lineage tracking - or unexpected behavior. This field will be set by - feature-engineering client and should be left unset by - SDK and terraform users. - time_window: + Configuration for ingesting Kafka data into a + Databricks-managed Delta table. + key_schema: type: object properties: - continuous: + avro_schema: + type: string + json_schema: + type: string + description: >- + Schema of the JSON object in standard IETF JSON + schema format (https://json-schema.org/). + proto_schema: type: object properties: - window_duration: + schema_text: type: string - offset: + description: >- + The raw .proto file text (proto2 and proto3 + syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: type: string description: >- - The offset of the continuous window (must be - non-positive). + The fully-qualified name of the message within + schema_text that describes the Kafka payload + (e.g. "Event" or "com.example.Event" if + schema_text declares a package). Identifies + which message is used to decode each Kafka + record — a .proto file may declare multiple + messages but only one represents the payload. + Must not be empty. required: - - window_duration - sliding: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload message + name. + description: >- + Schema configuration for extracting message keys from + topics. At least one of key_schema and value_schema must + be provided. + value_schema: + type: object + properties: + avro_schema: + type: string + json_schema: + type: string + description: >- + Schema of the JSON object in standard IETF JSON + schema format (https://json-schema.org/). + proto_schema: type: object properties: - window_duration: - type: string - slide_duration: + schema_text: type: string description: >- - The slide duration (interval by which windows - advance, must be positive and less than - duration). - required: - - window_duration - - slide_duration - tumbling: - type: object - properties: - window_duration: + The raw .proto file text (proto2 and proto3 + syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: type: string + description: >- + The fully-qualified name of the message within + schema_text that describes the Kafka payload + (e.g. "Event" or "com.example.Event" if + schema_text declares a package). Identifies + which message is used to decode each Kafka + record — a .proto file may declare multiple + messages but only one represents the payload. + Must not be empty. required: - - window_duration - description: >- - Deprecated: Use - Function.aggregation_function.time_window instead. Kept - for backwards compatibility. The time window in which - the feature is computed. - timeseries_column: - type: object - properties: - name: - type: string - required: - - name + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload message + name. description: >- - Column recording time, used for point-in-time joins, - backfills, and aggregations. + Schema configuration for extracting message values from + topics. At least one of key_schema and value_schema must + be provided. required: - - full_name - - source - - function - description: Feature to update. + - name + - bootstrap_servers + - subscription_mode + - auth_config + description: The Kafka config to update. required: - - feature + - kafka_config responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/Feature' + $ref: '#/components/schemas/KafkaConfig' default: description: Error response content: @@ -4438,32 +6883,25 @@ paths: message: type: string x-stackql-sdk-source: FeatureEngineeringAPI - /api/2.0/feature-engineering/features/kafka-configs/{name}: + /api/2.0/feature-engineering/materialized-features/{materialized_feature_id}: delete: - operationId: feature_engineering_delete_kafka_config - summary: >- - Delete a Kafka config. During PrPr, Kafka configs can be read and used - when creating features under + operationId: feature_engineering_delete_materialized_feature + summary: Delete a materialized feature. tags: - ml - feature_engineering - description: >- - Delete a Kafka config. During PrPr, Kafka configs can be read and used - when creating features under - - the entire metastore. Only the creator of the Kafka config can delete - it. - + description: |- + Delete a materialized feature. - :param name: str - Name of the Kafka config to delete. + :param materialized_feature_id: str + The ID of the materialized feature to delete. parameters: - - name: name + - name: materialized_feature_id in: path required: true schema: type: string - description: Name of the Kafka config to delete. + description: The ID of the materialized feature to delete. responses: '200': description: Success @@ -4478,40 +6916,34 @@ paths: type: string message: type: string - x-stackql-sdk-source: FeatureEngineeringAPI - get: - operationId: feature_engineering_get_kafka_config - summary: >- - Get a Kafka config. During PrPr, Kafka configs can be read and used when - creating features under the + x-stackql-sdk-source: FeatureEngineeringAPI + get: + operationId: feature_engineering_get_materialized_feature + summary: Get a materialized feature. tags: - ml - feature_engineering - description: >- - Get a Kafka config. During PrPr, Kafka configs can be read and used when - creating features under the - - entire metastore. Only the creator of the Kafka config can delete it. - + description: |- + Get a materialized feature. - :param name: str - Name of the Kafka config to get. + :param materialized_feature_id: str + The ID of the materialized feature. - :returns: :class:`KafkaConfig` + :returns: :class:`MaterializedFeature` parameters: - - name: name + - name: materialized_feature_id in: path required: true schema: type: string - description: Name of the Kafka config to get. + description: The ID of the materialized feature. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/KafkaConfig' + $ref: '#/components/schemas/MaterializedFeature' default: description: Error response content: @@ -4525,185 +6957,198 @@ paths: type: string x-stackql-sdk-source: FeatureEngineeringAPI patch: - operationId: feature_engineering_update_kafka_config - summary: >- - Update a Kafka config. During PrPr, Kafka configs can be read and used - when creating features under + operationId: feature_engineering_update_materialized_feature + summary: Update a materialized feature (pause/resume). tags: - ml - feature_engineering - description: >- - Update a Kafka config. During PrPr, Kafka configs can be read and used - when creating features under - - the entire metastore. Only the creator of the Kafka config can delete - it. - + description: |- + Update a materialized feature (pause/resume). - :param name: str - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier - used from the Feature object to reference these configs for a feature. Can be distinct from topic - name. - :param kafka_config: :class:`KafkaConfig` - The Kafka config to update. - :param update_mask: FieldMask - The list of fields to update. + :param materialized_feature_id: str + Server-assigned unique identifier for the materialized feature. + :param materialized_feature: :class:`MaterializedFeature` + The materialized feature to update. + :param update_mask: str + Provide the materialization feature fields which should be updated. Currently, only the + pipeline_state field can be updated. - :returns: :class:`KafkaConfig` + :returns: :class:`MaterializedFeature` parameters: - - name: name + - name: materialized_feature_id in: path required: true schema: type: string - description: >- - Name that uniquely identifies this Kafka config within the - metastore. This will be the identifier used from the Feature object - to reference these configs for a feature. Can be distinct from topic - name. + description: Server-assigned unique identifier for the materialized feature. - name: update_mask in: query required: true schema: - type: object - description: The list of fields to update. + type: string + description: >- + Provide the materialization feature fields which should be updated. + Currently, only the pipeline_state field can be updated. requestBody: content: application/json: schema: type: object properties: - kafka_config: + materialized_feature: type: object properties: - name: + feature_name: type: string - bootstrap_servers: + description: The full name of the feature in Unity Catalog. + cron_schedule: type: string description: >- - A comma-separated list of host/port pairs pointing to - Kafka cluster. - subscription_mode: + The quartz cron expression that defines the schedule of + the materialization pipeline. The schedule is evaluated + in the UTC timezone. Hidden from GraphQL: superseded by + the ``trigger`` oneof (cron_schedule_trigger), so not + exposed to Catalog Explorer. + cron_schedule_trigger: type: object properties: - assign: - type: string - subscribe: - type: string - description: >- - A comma-separated list of Kafka topics to read from. - For example, 'topicA,topicB,topicC'. - subscribe_pattern: + cron_expression: type: string description: >- - A regular expression matching topics to subscribe - to. For example, 'topic.*' will subscribe to all - topics starting with 'topic'. + The cron expression defining the schedule (e.g., "0 + 0 * * *" for daily at midnight). description: >- - Options to configure which Kafka topics to pull data - from. - auth_config: + A cron-based schedule trigger for the materialization + pipeline. + is_online: + type: boolean + description: >- + True if this is an online materialized feature. False if + it is an offline materialized feature. + last_materialization_time: + type: string + description: >- + The timestamp when the pipeline last ran and updated the + materialized feature values. If the pipeline has not run + yet, this field will be null. + materialized_feature_id: + type: string + description: >- + Server-assigned unique identifier for the materialized + feature. + offline_store_config: type: object properties: - uc_service_credential_name: + catalog_name: type: string - description: Authentication configuration for connection to topics. - backfill_source: + description: The Unity Catalog catalog name. + schema_name: + type: string + description: The Unity Catalog schema name. + table_name_prefix: + type: string + description: >- + Prefix for Unity Catalog table name. The + materialized feature will be stored in a table with + this prefix and a generated postfix. + required: + - catalog_name + - schema_name + - table_name_prefix + description: >- + Destination for writing feature values to an offline + Delta table. + online_store_config: type: object properties: - delta_table_source: - type: object - properties: - full_name: - type: string - dataframe_schema: - type: string - description: >- - Schema of the resulting dataframe after - transformations, in Spark StructType JSON format - (from df.schema.json()). Required if - transformation_sql is specified. Example: - {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} - entity_columns: - type: array - items: - type: string - description: >- - Deprecated: Use Feature.entity instead. Kept for - backwards compatibility. The entity columns of - the Delta table. - filter_condition: - type: string - description: >- - Single WHERE clause to filter delta table before - applying transformations. Will be row-wise - evaluated, so should only include conditionals - and projections. - timeseries_column: - type: string - description: >- - Deprecated: Use Feature.timeseries_column - instead. Kept for backwards compatibility. The - timeseries column of the Delta table. - transformation_sql: - type: string - description: >- - A single SQL SELECT expression applied after - filter_condition. Should contains all the - columns needed (eg. "SELECT *, col_a + col_b AS - col_c FROM x.y.z WHERE col_a > 0" would have - `transformation_sql` "*, col_a + col_b AS - col_c") If transformation_sql is not provided, - all columns of the delta table are present in - the DataSource dataframe. - required: - - full_name + catalog_name: + type: string + description: >- + The Unity Catalog catalog name. This name is also + used as the Lakebase logical database name. Quoting + is handled by the backend where needed, do not + pre-quote it. + schema_name: + type: string + description: >- + The Unity Catalog schema name. This name is also + used as the Lakebase schema name under the database. + Quoting is handled by the backend where needed, do + not pre-quote it. + table_name_prefix: + type: string + description: >- + Prefix for Unity Catalog table name. The + materialized feature will be stored in a Lakebase + table with this prefix and a generated postfix. + online_store_name: + type: string + description: The name of the target online store. + required: + - catalog_name + - schema_name + - table_name_prefix + - online_store_name description: >- - A user-provided and managed source for backfilling data. - Historical data is used when creating a training set - from streaming features linked to this Kafka config. In - the future, a separate table will be maintained by - Databricks for forward filling data. The schema for this - source must match exactly that of the key and value - schemas specified for this Kafka config. - extra_options: - type: object + Destination for writing feature values to an online + Lakebase table. + pipeline_schedule_state: + type: string + x-enum: + - ACTIVE + - PAUSED + - SNAPSHOT description: >- - Catch-all for miscellaneous options. Keys should be - source options or Kafka consumer options (kafka.*) - key_schema: + The schedule state of the materialization pipeline. + Hidden from GraphQL: being deprecated, so not exposed to + Catalog Explorer. + streaming_mode: type: object properties: - json_schema: + freshness_target: + type: string + description: >- + The desired data freshness for feature + materialization, expressed as a duration string + (e.g. "1 minute"). + mode: type: string + x-enum: + - STREAMING_MODE_TYPE_MBM + - STREAMING_MODE_TYPE_RTM + description: >- + The type of streaming mode used by the + materialization pipeline. description: >- - Schema configuration for extracting message keys from - topics. At least one of key_schema and value_schema must - be provided. - value_schema: + The Structured Streaming trigger mode used for + materialization. Real-time mode (RTM) targets sub-second + latency for operational workloads; micro-batch mode + (MBM) favors cost efficiency for ETL and analytics + workloads. + table_name: + type: string + description: >- + The fully qualified Unity Catalog path to the table + containing the materialized feature (Delta table or + Lakebase table). Output only. + table_trigger: type: object - properties: - json_schema: - type: string + properties: {} description: >- - Schema configuration for extracting message values from - topics. At least one of key_schema and value_schema must - be provided. + A trigger that fires when the upstream source table + changes. required: - - name - - bootstrap_servers - - subscription_mode - - auth_config - description: The Kafka config to update. + - feature_name + description: The materialized feature to update. required: - - kafka_config + - materialized_feature responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/KafkaConfig' + $ref: '#/components/schemas/MaterializedFeature' default: description: Error response content: @@ -4716,25 +7161,27 @@ paths: message: type: string x-stackql-sdk-source: FeatureEngineeringAPI - /api/2.0/feature-engineering/materialized-features/{materialized_feature_id}: + /api/2.0/feature-engineering/streams/{name}: delete: - operationId: feature_engineering_delete_materialized_feature - summary: Delete a materialized feature. + operationId: feature_engineering_delete_stream + summary: Delete a Stream by its full three-part name (catalog.schema.stream). tags: - ml - feature_engineering description: |- - Delete a materialized feature. + Delete a Stream by its full three-part name (catalog.schema.stream). - :param materialized_feature_id: str - The ID of the materialized feature to delete. + :param name: str + Full three-part name (catalog.schema.stream) of the Stream to delete. parameters: - - name: materialized_feature_id + - name: name in: path required: true schema: type: string - description: The ID of the materialized feature to delete. + description: >- + Full three-part name (catalog.schema.stream) of the Stream to + delete. responses: '200': description: Success @@ -4751,32 +7198,32 @@ paths: type: string x-stackql-sdk-source: FeatureEngineeringAPI get: - operationId: feature_engineering_get_materialized_feature - summary: Get a materialized feature. + operationId: feature_engineering_get_stream + summary: Get a Stream by its full three-part name (catalog.schema.stream). tags: - ml - feature_engineering description: |- - Get a materialized feature. + Get a Stream by its full three-part name (catalog.schema.stream). - :param materialized_feature_id: str - The ID of the materialized feature. + :param name: str + Full three-part name (catalog.schema.stream) of the Stream to get. - :returns: :class:`MaterializedFeature` + :returns: :class:`Stream` parameters: - - name: materialized_feature_id + - name: name in: path required: true schema: type: string - description: The ID of the materialized feature. + description: Full three-part name (catalog.schema.stream) of the Stream to get. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/MaterializedFeature' + $ref: '#/components/schemas/Stream' default: description: Error response content: @@ -4790,142 +7237,600 @@ paths: type: string x-stackql-sdk-source: FeatureEngineeringAPI patch: - operationId: feature_engineering_update_materialized_feature - summary: Update a materialized feature (pause/resume). + operationId: feature_engineering_update_stream + summary: Update a Stream. Only fields listed in ``update_mask`` are mutated. tags: - ml - feature_engineering description: |- - Update a materialized feature (pause/resume). + Update a Stream. Only fields listed in ``update_mask`` are mutated. - :param materialized_feature_id: str - Unique identifier for the materialized feature. - :param materialized_feature: :class:`MaterializedFeature` - The materialized feature to update. - :param update_mask: str - Provide the materialization feature fields which should be updated. Currently, only the - pipeline_state field can be updated. + :param name: str + Full three-part (catalog.schema.stream) name of the stream. + :param stream: :class:`Stream` + The Stream to update. + :param update_mask: FieldMask + The list of fields to update. - :returns: :class:`MaterializedFeature` + :returns: :class:`Stream` parameters: - - name: materialized_feature_id + - name: name in: path required: true schema: type: string - description: Unique identifier for the materialized feature. + description: Full three-part (catalog.schema.stream) name of the stream. - name: update_mask in: query required: true schema: - type: string - description: >- - Provide the materialization feature fields which should be updated. - Currently, only the pipeline_state field can be updated. + type: object + description: The list of fields to update. requestBody: content: application/json: schema: type: object properties: - materialized_feature: + stream: type: object properties: - feature_name: - type: string - description: The full name of the feature in Unity Catalog. - cron_schedule: + name: type: string description: >- - The quartz cron expression that defines the schedule of - the materialization pipeline. The schedule is evaluated - in the UTC timezone. - last_materialization_time: - type: string + Full three-part (catalog.schema.stream) name of the + stream. + source_config: + type: object + properties: + kafka_stream_config: + type: object + properties: + subscription_mode: + type: object + properties: + assign: + type: string + description: >- + A JSON string that contains the specific + topic-partitions to consume from. For + example, for + '{"topicA":[0,1],"topicB":[2,4]}', topicA's + 0'th and 1st partitions will be consumed + from. + subscribe: + type: string + description: >- + A comma-separated list of Kafka topics to + read from. For example, + 'topicA,topicB,topicC'. + subscribe_pattern: + type: string + description: >- + A regular expression matching topics to + subscribe to. For example, 'topic.*' will + subscribe to all topics starting with + 'topic'. + description: >- + Options to configure which Kafka topics to pull + data from. + extra_options: + type: object + description: >- + Optional Kafka source or consumer options, + validated against a server-side allowlist at + request time. Allowed keys: - + ``maxOffsetsPerTrigger`` - ``startingOffsets`` - + ``includeHeaders`` - + ``kafka.request.timeout.ms`` - + ``kafka.session.timeout.ms`` - + ``kafka.max.partition.fetch.bytes`` The + following keys are ingestion-only and are + stripped before being forwarded to the + materialization pipeline: - + ``maxOffsetsPerTrigger`` - ``startingOffsets`` + Auth and connection details belong on the parent + Stream's ``connection_config``, not here. + required: + - subscription_mode + description: Configuration for Apache Kafka streams. + kinesis_stream_config: + type: object + properties: + extra_options: + type: object + description: >- + Optional Kinesis source options, validated + against a server-side allowlist at request time. + Auth and connection details belong on the parent + Stream's ``connection_config``, not here. + stream_arns: + type: object + properties: + arns: + type: array + items: + type: string + description: >- + Kinesis stream ARNs to read from. For + example, + 'arn:aws:kinesis:us-west-2:111122223333:stream/stream-a'. + description: Kinesis stream ARNs to read from. + stream_names: + type: object + properties: + names: + type: array + items: + type: string + description: Kinesis stream names to read from. + description: Kinesis stream names to read from. + description: Configuration for AWS Kinesis Data Streams. description: >- - The timestamp when the pipeline last ran and updated the - materialized feature values. If the pipeline has not run - yet, this field will be null. - materialized_feature_id: - type: string - description: Unique identifier for the materialized feature. - offline_store_config: + Source-specific configuration. Determines the streaming + platform source. + connection_config: type: object properties: - catalog_name: - type: string - description: The Unity Catalog catalog name. - schema_name: - type: string - description: The Unity Catalog schema name. - table_name_prefix: + direct_mtls_config: + type: object + properties: + bootstrap_servers: + type: string + description: >- + A comma-separated list of host:port pairs for + the Kafka bootstrap servers. + mtls_config: + type: object + properties: + keystore_location: + type: string + description: >- + Unity Catalog volume path to the JKS + keystore file containing the client + certificate and private key. e.g. + "/Volumes////client.jks". + The materialization compute must have read + permission on this volume. + keystore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the JKS keystore + password. + key_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the private key + password. Often the same value as the + keystore password (keytool's default), but + provided as a separate field because Apache + Kafka requires it as a distinct option + (kafka.ssl.key.password). + truststore_location: + type: string + description: >- + Unity Catalog volume path to the JKS + truststore file containing the CA + certificate(s) trusted to verify the Kafka + broker's server certificate. e.g. + "/Volumes////truststore.jks". + truststore_password_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Secret-scope reference for the JKS + truststore password. + disable_hostname_verification: + type: boolean + description: >- + Set to true only when the broker + certificate's SAN intentionally does not + match the connection endpoint — for example + when reaching the cluster through a + PrivateLink endpoint whose DNS name is not + in the broker certificate. Skipping the + hostname check removes a defense against + man-in-the-middle attacks; do not enable + casually. mTLS client authentication is + unaffected by this option. See the Apache + Kafka SSL security guide for background on + this check: + https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification + required: + - keystore_location + - keystore_password_ref + - key_password_ref + - truststore_location + - truststore_password_ref + description: Mutual-TLS authentication configuration. + required: + - bootstrap_servers + - mtls_config + description: >- + Direct mTLS configuration for stream platform + access. This is only used in the short term until UC + Kafka Connections support mTLS . Once UC Kafka + Connections support mTLS, this will be deprecated. + uc_connection_name: type: string description: >- - Prefix for Unity Catalog table name. The - materialized feature will be stored in a table with - this prefix and a generated postfix. - required: - - catalog_name - - schema_name - - table_name_prefix - description: Configuration for offline store destination. - online_store_config: + Name of an existing UC Connection for stream + platform access. Must be the correct type for the + streaming platform (e.g. a Kafka Connection for a + Kafka Stream, or a Kinesis Connection for a Kinesis + Stream). + description: >- + Specifies how to connect and authenticate to the stream + platform. + schema_config: type: object properties: - catalog_name: - type: string + direct_schemas: + type: object + properties: + key_schema: + type: object + properties: + avro_schema: + type: string + json_schema: + type: string + description: >- + Schema of the JSON object in standard IETF + JSON schema format + (https://json-schema.org/). + proto_schema: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and + proto3 syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message + within schema_text that describes the + Kafka payload (e.g. "Event" or + "com.example.Event" if schema_text + declares a package). Identifies which + message is used to decode each Kafka + record — a .proto file may declare + multiple messages but only one + represents the payload. Must not be + empty. + required: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload + message name. + description: >- + Schema for the message key. This is only used + for Kafka streams. For Kafka, at least one of + payload_schema or key_schema must be specified. + payload_schema: + type: object + properties: + avro_schema: + type: string + json_schema: + type: string + description: >- + Schema of the JSON object in standard IETF + JSON schema format + (https://json-schema.org/). + proto_schema: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and + proto3 syntax supported, see + https://protobuf.dev/programming-guides/proto3/ + and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message + within schema_text that describes the + Kafka payload (e.g. "Event" or + "com.example.Event" if schema_text + declares a package). Identifies which + message is used to decode each Kafka + record — a .proto file may declare + multiple messages but only one + represents the payload. Must not be + empty. + required: + - schema_text + - message_name + description: >- + Protocol Buffer schema with its payload + message name. + description: >- + Schema for the message payload. For Kafka, this + is the value schema. Unless the platform + supports another schema (e.g. keys for Kafka), + this must be specified. + description: Schema definitions provided directly on the Stream. + schema_registry_config: + type: object + properties: + api_secret_ref: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Reference to the schema registry API secret in a + Databricks secret scope. + key_schema_locator: + type: object + properties: + format: + type: string + x-enum: + - FORMAT_AVRO + - FORMAT_JSON + - FORMAT_PROTOBUF + description: Serialization format for this schema. + confluent_schema: + type: object + properties: + subject: + type: string + description: >- + The Confluent schema registry subject + name. + required: + - subject + description: Confluent Schema Registry schema locator. + required: + - format + description: >- + Schema locator for the message key. Only used + for Kafka streams. At least one of + payload_schema_locator or key_schema_locator + must be set. + payload_schema_locator: + type: object + properties: + format: + type: string + x-enum: + - FORMAT_AVRO + - FORMAT_JSON + - FORMAT_PROTOBUF + description: Serialization format for this schema. + confluent_schema: + type: object + properties: + subject: + type: string + description: >- + The Confluent schema registry subject + name. + required: + - subject + description: Confluent Schema Registry schema locator. + required: + - format + description: >- + Schema locator for the message payload. For + Kafka this is the value. At least one of + payload_schema_locator or key_schema_locator + must be set. + uc_connection: + type: string + description: A Schema Registry UC Connection object. + required: + - api_secret_ref + description: Resolve schemas from an external schema registry. + description: >- + Schema definitions for the stream, provided either + directly on the Stream or resolved from an external + schema registry through a UC Connection. + ingestion_config: + type: object + properties: + ingestion_destination: + type: object + properties: + delta_table_name: + type: string + description: >- + The full three-part name (catalog, schema, name) + of the Delta table to be created for ingestion. description: >- - The Unity Catalog catalog name. This name is also - used as the Lakebase logical database name. Quoting - is handled by the backend where needed, do not - pre-quote it. - schema_name: - type: string + Destination for the Databricks-managed Delta table + that holds an offline copy of the streaming data for + querying and training. This table contains both 1) + forward-filled data from the Stream and 2) + backfilled data from the BackfillSource (if + provided). This table is created and managed by + Databricks and is deleted when the Stream is + deleted. + backfill_job_id: + type: integer description: >- - The Unity Catalog schema name. This name is also - used as the Lakebase schema name under the database. - Quoting is handled by the backend where needed, do - not pre-quote it. - table_name_prefix: - type: string + The ID of the Databricks Job that performs the + historical backfill of the ingestion Delta table. + backfill_source: + type: object + properties: + delta_table_name: + type: string + delta_table_source: + type: object + properties: + full_name: + type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after + transformations, in Spark StructType JSON + format (from df.schema.json()). Required if + transformation_sql is specified. Example: + {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]} + entity_columns: + type: array + items: + type: string + description: >- + Deprecated: Use Feature.entity instead. Kept + for backwards compatibility. The entity + columns of the Delta table. + filter_condition: + type: string + description: >- + Single WHERE clause to filter delta table + before applying transformations. Will be + row-wise evaluated, so should only include + conditionals and projections. + timeseries_column: + type: string + description: >- + Deprecated: Use Feature.timeseries_column + instead. Kept for backwards compatibility. + The timeseries column of the Delta table. + transformation_sql: + type: string + description: >- + A single SQL SELECT expression applied after + filter_condition. Should contains all the + columns needed (eg. "SELECT *, col_a + col_b + AS col_c FROM x.y.z WHERE col_a > 0" would + have ``transformation_sql`` "*, col_a + + col_b AS col_c") If transformation_sql is + not provided, all columns of the delta table + are present in the DataSource dataframe. + required: + - full_name + description: >- + Deprecated: Use delta_table_name instead. Kept + for backwards compatibility. The Delta table + source containing the historical data to + backfill. Only the delta table name is used for + backfill, other fields are ignored. description: >- - Prefix for Unity Catalog table name. The - materialized feature will be stored in a Lakebase - table with this prefix and a generated postfix. - online_store_name: + A user-provided source for backfilling data. + Historical data is used when creating a training set + from streaming features linked to this Stream. The + backfill data stored in this location will be copied + into the ingestion table for offline querying and + training. The schema for this source must match + exactly that of the key and payload schemas + specified for this Stream. + deduplication_columns: + type: array + items: + type: string + description: >- + Column paths used to identify duplicate rows during + ingestion; only one row per distinct combination of + these values is kept. Use dot notation for nested + fields (e.g. ``value.user_id``). Empty list means + every column is compared. + ingestion_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the + forward-fill ingestion. + ingestion_pipeline_id: type: string - description: The name of the target online store. + description: >- + The ID of the SDP pipeline that continuously copies + new events from the streaming source into the + ingestion Delta table. required: - - catalog_name - - schema_name - - table_name_prefix - - online_store_name - description: Configuration for online store destination. - pipeline_schedule_state: + - ingestion_destination + description: >- + Configuration for streaming data ingestion: the managed + table storing an offline copy of forward fill data and + optional historical backfill. + browse_only: + type: boolean + description: >- + Indicates whether the principal is limited to retrieving + metadata for the associated object through the BROWSE + privilege when include_browse is enabled in the request. + create_time: type: string - x-enum: - - ACTIVE - - PAUSED - - SNAPSHOT - description: The schedule state of the materialization pipeline. - table_name: + format: date-time + description: Time at which this Stream was created. + created_by: + type: string + description: Username of the Stream creator. + description: + type: string + description: User-provided description. + update_time: type: string - description: >- - The fully qualified Unity Catalog path to the table - containing the materialized feature (Delta table or - Lakebase table). Output only. + format: date-time + description: Time at which this Stream was last modified. + updated_by: + type: string + description: Username of user who last modified the Stream. required: - - feature_name - description: The materialized feature to update. + - name + - source_config + - connection_config + - schema_config + - ingestion_config + description: The Stream to update. required: - - materialized_feature + - stream responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/MaterializedFeature' + $ref: '#/components/schemas/Stream' default: description: Error response content: @@ -5332,6 +8237,15 @@ paths: description: >- The publish mode of the pipeline that syncs the online table with the source table. + full_feature_name: + type: string + description: >- + Full Unity Catalog name of one of the features + materialized in the source table, used to derive the + synced online table's entity and timeseries columns. + Required for view sources without a UC + PrimaryKeyConstraint; ignored when the source already + has one. required: - online_store - online_table_name @@ -5397,8 +8311,10 @@ paths: :param include_features: List[str] (optional) Specifies the list of feature columns to include in model training. These columns must exist in the training data and be of type string, numerical, or boolean. If not specified, no additional features - will be included. Note: Certain columns are automatically handled: - Automatically excluded: - split_column, target_column, custom_weights_column. - Automatically included: time_column. + will be included. Note: Certain columns are automatically handled: + + - Automatically excluded: split_column, target_column, custom_weights_column. + - Automatically included: time_column. :param max_runtime: int (optional) The maximum duration for the experiment in minutes. The experiment stops automatically if it exceeds this limit. @@ -5962,13 +8878,10 @@ paths: :param stage: str Target stage of the transition. Valid values are: - * `None`: The initial stage of a model version. - - * `Staging`: Staging or pre-production stage. - - * `Production`: Production stage. - - * `Archived`: Archived stage. + - ``None``: The initial stage of a model version. + - ``Staging``: Staging or pre-production stage. + - ``Production``: Production stage. + - ``Archived``: Archived stage. :param archive_existing_versions: bool Specifies whether to archive all current model versions in the target stage. :param comment: str (optional) @@ -5990,10 +8903,11 @@ paths: stage: type: string description: >- - Target stage of the transition. Valid values are: * `None`: - The initial stage of a model version. * `Staging`: Staging - or pre-production stage. * `Production`: Production stage. * - `Archived`: Archived stage. + Target stage of the transition. Valid values are: - + ``None``: The initial stage of a model version. - + ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived + stage. archive_existing_versions: type: boolean description: >- @@ -6102,7 +9016,7 @@ paths: Creates a new registered model with the name specified in the request body. Throws - `RESOURCE_ALREADY_EXISTS` if a registered model with the given name + ``RESOURCE_ALREADY_EXISTS`` if a registered model with the given name exists. @@ -6177,7 +9091,7 @@ paths: :param description: str (optional) Optional description for model version. :param run_id: str (optional) - MLflow run ID for correlation, if `source` was generated by an experiment run in MLflow tracking + MLflow run ID for correlation, if ``source`` was generated by an experiment run in MLflow tracking server :param run_link: str (optional) MLflow run link - this is the exact link of the run that generated this model version, potentially @@ -6204,8 +9118,8 @@ paths: run_id: type: string description: >- - MLflow run ID for correlation, if `source` was generated by - an experiment run in MLflow tracking server + MLflow run ID for correlation, if ``source`` was generated + by an experiment run in MLflow tracking server run_link: type: string description: >- @@ -6262,13 +9176,10 @@ paths: :param stage: str Target stage of the transition. Valid values are: - * `None`: The initial stage of a model version. - - * `Staging`: Staging or pre-production stage. - - * `Production`: Production stage. - - * `Archived`: Archived stage. + - ``None``: The initial stage of a model version. + - ``Staging``: Staging or pre-production stage. + - ``Production``: Production stage. + - ``Archived``: Archived stage. :param comment: str (optional) User-provided comment on the action. @@ -6288,10 +9199,11 @@ paths: stage: type: string description: >- - Target stage of the transition. Valid values are: * `None`: - The initial stage of a model version. * `Staging`: Staging - or pre-production stage. * `Production`: Production stage. * - `Archived`: Archived stage. + Target stage of the transition. Valid values are: - + ``None``: The initial stage of a model version. - + ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived + stage. comment: type: string description: User-provided comment on the action. @@ -6333,34 +9245,24 @@ paths: :param events: List[:class:`RegistryWebhookEvent`] - Events that can trigger a registry webhook: * `MODEL_VERSION_CREATED`: A new model version was - created for the associated model. - - * `MODEL_VERSION_TRANSITIONED_STAGE`: A model version’s stage was changed. - - * `TRANSITION_REQUEST_CREATED`: A user requested a model version’s stage be transitioned. - - * `COMMENT_CREATED`: A user wrote a comment on a registered model. - - * `REGISTERED_MODEL_CREATED`: A new registered model was created. This event type can only be - specified for a registry-wide webhook, which can be created by not specifying a model name in the - create request. - - * `MODEL_VERSION_TAG_SET`: A user set a tag on the model version. - - * `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model version was transitioned to staging. - - * `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version was transitioned to production. - - * `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version was archived. - - * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A user requested a model version be transitioned to - staging. - - * `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested a model version be transitioned to - production. - - * `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a model version be archived. + Events that can trigger a registry webhook: + + - ``MODEL_VERSION_CREATED``: A new model version was created for the associated model. + - ``MODEL_VERSION_TRANSITIONED_STAGE``: A model version’s stage was changed. + - ``TRANSITION_REQUEST_CREATED``: A user requested a model version’s stage be transitioned. + - ``COMMENT_CREATED``: A user wrote a comment on a registered model. + - ``REGISTERED_MODEL_CREATED``: A new registered model was created. This event type can only be + specified for a registry-wide webhook, which can be created by not specifying a model name in the + create request. + - ``MODEL_VERSION_TAG_SET``: A user set a tag on the model version. + - ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A model version was transitioned to staging. + - ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model version was transitioned to production. + - ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version was archived. + - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A user requested a model version be transitioned to + staging. + - ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user requested a model version be transitioned to + production. + - ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested a model version be archived. :param description: str (optional) User-specified description for the webhook. :param http_url_spec: :class:`HttpUrlSpec` (optional) @@ -6372,12 +9274,12 @@ paths: events across all versions of all registered models. :param status: :class:`RegistryWebhookStatus` (optional) Enable or disable triggering the webhook, or put the webhook into test mode. The default is - `ACTIVE`: * `ACTIVE`: Webhook is triggered when an associated event happens. + ``ACTIVE``: - * `DISABLED`: Webhook is not triggered. - - * `TEST_MODE`: Webhook can be triggered through the test endpoint, but is not triggered on a real - event. + - ``ACTIVE``: Webhook is triggered when an associated event happens. + - ``DISABLED``: Webhook is not triggered. + - ``TEST_MODE``: Webhook can be triggered through the test endpoint, but is not triggered on a real + event. :returns: :class:`CreateWebhookResponse` requestBody: @@ -6456,29 +9358,29 @@ paths: attributes -- see the documentation for details. description: >- - Events that can trigger a registry webhook: * - `MODEL_VERSION_CREATED`: A new model version was created for - the associated model. * `MODEL_VERSION_TRANSITIONED_STAGE`: - A model version’s stage was changed. * - `TRANSITION_REQUEST_CREATED`: A user requested a model - version’s stage be transitioned. * `COMMENT_CREATED`: A user - wrote a comment on a registered model. * - `REGISTERED_MODEL_CREATED`: A new registered model was - created. This event type can only be specified for a - registry-wide webhook, which can be created by not - specifying a model name in the create request. * - `MODEL_VERSION_TAG_SET`: A user set a tag on the model - version. * `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model - version was transitioned to staging. * - `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version - was transitioned to production. * - `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version - was archived. * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A - user requested a model version be transitioned to staging. * - `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested - a model version be transitioned to production. * - `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a - model version be archived. + Events that can trigger a registry webhook: - + ``MODEL_VERSION_CREATED``: A new model version was created + for the associated model. - + ``MODEL_VERSION_TRANSITIONED_STAGE``: A model version’s + stage was changed. - ``TRANSITION_REQUEST_CREATED``: A user + requested a model version’s stage be transitioned. - + ``COMMENT_CREATED``: A user wrote a comment on a registered + model. - ``REGISTERED_MODEL_CREATED``: A new registered + model was created. This event type can only be specified for + a registry-wide webhook, which can be created by not + specifying a model name in the create request. - + ``MODEL_VERSION_TAG_SET``: A user set a tag on the model + version. - ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A + model version was transitioned to staging. - + ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model + version was transitioned to production. - + ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version + was archived. - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A + user requested a model version be transitioned to staging. - + ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user + requested a model version be transitioned to production. - + ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested + a model version be archived. description: type: string description: User-specified description for the webhook. @@ -6492,7 +9394,7 @@ paths: description: >- Value of the authorization header that should be sent in the request sent by the wehbook. It should be of the - form `" "`. If set to an empty + form ``" "``. If set to an empty string, no authorization header will be included in the request. enable_ssl_verification: @@ -6553,11 +9455,11 @@ paths: - TEST_MODE description: >- Enable or disable triggering the webhook, or put the webhook - into test mode. The default is `ACTIVE`: * `ACTIVE`: Webhook - is triggered when an associated event happens. * `DISABLED`: - Webhook is not triggered. * `TEST_MODE`: Webhook can be - triggered through the test endpoint, but is not triggered on - a real event. + into test mode. The default is ``ACTIVE``: - ``ACTIVE``: + Webhook is triggered when an associated event happens. - + ``DISABLED``: Webhook is not triggered. - ``TEST_MODE``: + Webhook can be triggered through the test endpoint, but is + not triggered on a real event. required: - events responses: @@ -6804,13 +9706,10 @@ paths: :param stage: str Target stage of the transition request. Valid values are: - * `None`: The initial stage of a model version. - - * `Staging`: Staging or pre-production stage. - - * `Production`: Production stage. - - * `Archived`: Archived stage. + - ``None``: The initial stage of a model version. + - ``Staging``: Staging or pre-production stage. + - ``Production``: Production stage. + - ``Archived``: Archived stage. :param creator: str Username of the user who created this request. Of the transition requests matching the specified details, only the one transition created by this user will be deleted. @@ -6837,10 +9736,10 @@ paths: schema: type: string description: >- - Target stage of the transition request. Valid values are: * `None`: - The initial stage of a model version. * `Staging`: Staging or - pre-production stage. * `Production`: Production stage. * - `Archived`: Archived stage. + Target stage of the transition request. Valid values are: - + ``None``: The initial stage of a model version. - ``Staging``: + Staging or pre-production stage. - ``Production``: Production stage. + - ``Archived``: Archived stage. - name: creator in: query required: true @@ -6969,22 +9868,19 @@ paths: operationId: model_registry_get_model summary: >- Get the details of a model. This is a Databricks workspace version of - the [MLflow endpoint] that also + the `MLflow endpoint tags: - ml - model_registry description: >- Get the details of a model. This is a Databricks workspace version of - the [MLflow endpoint] that also - - returns the model's Databricks workspace ID and the permission level of - the requesting user on the - - model. + the `MLflow endpoint + `__ + that also returns the - [MLflow endpoint]: - https://www.mlflow.org/docs/latest/rest-api.html#get-registeredmodel + model's Databricks workspace ID and the permission level of the + requesting user on the model. :param name: str @@ -7358,13 +10254,13 @@ paths: operationId: model_registry_list_models summary: >- Lists all available registered models, up to the limit specified in - __max_results__. + **max_results**. tags: - ml - model_registry description: >- Lists all available registered models, up to the limit specified in - __max_results__. + **max_results**. :param max_results: int (optional) @@ -7468,37 +10364,28 @@ paths: :param events: List[:class:`RegistryWebhookEvent`] (optional) - Events that trigger the webhook. * `MODEL_VERSION_CREATED`: A new model version was created for the - associated model. - - * `MODEL_VERSION_TRANSITIONED_STAGE`: A model version’s stage was changed. - - * `TRANSITION_REQUEST_CREATED`: A user requested a model version’s stage be transitioned. - - * `COMMENT_CREATED`: A user wrote a comment on a registered model. - - * `REGISTERED_MODEL_CREATED`: A new registered model was created. This event type can only be - specified for a registry-wide webhook, which can be created by not specifying a model name in the - create request. - - * `MODEL_VERSION_TAG_SET`: A user set a tag on the model version. - - * `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model version was transitioned to staging. - - * `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version was transitioned to production. - - * `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version was archived. - - * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A user requested a model version be transitioned to - staging. - - * `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested a model version be transitioned to - production. - - * `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a model version be archived. - - If `events` is specified, any webhook with one or more of the specified trigger events is included - in the output. If `events` is not specified, webhooks of all event types are included in the output. + Events that trigger the webhook. + + - ``MODEL_VERSION_CREATED``: A new model version was created for the associated model. + - ``MODEL_VERSION_TRANSITIONED_STAGE``: A model version’s stage was changed. + - ``TRANSITION_REQUEST_CREATED``: A user requested a model version’s stage be transitioned. + - ``COMMENT_CREATED``: A user wrote a comment on a registered model. + - ``REGISTERED_MODEL_CREATED``: A new registered model was created. This event type can only be + specified for a registry-wide webhook, which can be created by not specifying a model name in the + create request. + - ``MODEL_VERSION_TAG_SET``: A user set a tag on the model version. + - ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A model version was transitioned to staging. + - ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model version was transitioned to production. + - ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version was archived. + - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A user requested a model version be transitioned to + staging. + - ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user requested a model version be transitioned to + production. + - ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested a model version be archived. + + If ``events`` is specified, any webhook with one or more of the specified trigger events is included + in the output. If ``events`` is not specified, webhooks of all event types are included in the + output. :param max_results: int (optional) :param model_name: str (optional) @@ -7517,30 +10404,30 @@ paths: items: $ref: '#/components/schemas/RegistryWebhookEvent' description: >- - Events that trigger the webhook. * `MODEL_VERSION_CREATED`: A new - model version was created for the associated model. * - `MODEL_VERSION_TRANSITIONED_STAGE`: A model version’s stage was - changed. * `TRANSITION_REQUEST_CREATED`: A user requested a model - version’s stage be transitioned. * `COMMENT_CREATED`: A user wrote a - comment on a registered model. * `REGISTERED_MODEL_CREATED`: A new - registered model was created. This event type can only be specified - for a registry-wide webhook, which can be created by not specifying - a model name in the create request. * `MODEL_VERSION_TAG_SET`: A - user set a tag on the model version. * - `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model version was - transitioned to staging. * - `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version was - transitioned to production. * - `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version was - archived. * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A user - requested a model version be transitioned to staging. * - `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested a model - version be transitioned to production. * - `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a model - version be archived. If `events` is specified, any webhook with one - or more of the specified trigger events is included in the output. - If `events` is not specified, webhooks of all event types are - included in the output. + Events that trigger the webhook. - ``MODEL_VERSION_CREATED``: A new + model version was created for the associated model. - + ``MODEL_VERSION_TRANSITIONED_STAGE``: A model version’s stage was + changed. - ``TRANSITION_REQUEST_CREATED``: A user requested a model + version’s stage be transitioned. - ``COMMENT_CREATED``: A user wrote + a comment on a registered model. - ``REGISTERED_MODEL_CREATED``: A + new registered model was created. This event type can only be + specified for a registry-wide webhook, which can be created by not + specifying a model name in the create request. - + ``MODEL_VERSION_TAG_SET``: A user set a tag on the model version. - + ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A model version was + transitioned to staging. - + ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model version was + transitioned to production. - + ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version was + archived. - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A user + requested a model version be transitioned to staging. - + ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user requested a + model version be transitioned to production. - + ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested a model + version be archived. If ``events`` is specified, any webhook with + one or more of the specified trigger events is included in the + output. If ``events`` is not specified, webhooks of all event types + are included in the output. - name: max_results in: query required: false @@ -7597,13 +10484,10 @@ paths: :param stage: str Target stage of the transition. Valid values are: - * `None`: The initial stage of a model version. - - * `Staging`: Staging or pre-production stage. - - * `Production`: Production stage. - - * `Archived`: Archived stage. + - ``None``: The initial stage of a model version. + - ``Staging``: Staging or pre-production stage. + - ``Production``: Production stage. + - ``Archived``: Archived stage. :param comment: str (optional) User-provided comment on the action. @@ -7623,10 +10507,11 @@ paths: stage: type: string description: >- - Target stage of the transition. Valid values are: * `None`: - The initial stage of a model version. * `Staging`: Staging - or pre-production stage. * `Production`: Production stage. * - `Archived`: Archived stage. + Target stage of the transition. Valid values are: - + ``None``: The initial stage of a model version. - + ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived + stage. comment: type: string description: User-provided comment on the action. @@ -7666,7 +10551,7 @@ paths: :param name: str Registered model unique name identifier. :param new_name: str (optional) - If provided, updates the name for this `registered_model`. + If provided, updates the name for this ``registered_model``. :returns: :class:`RenameModelResponse` requestBody: @@ -7680,7 +10565,7 @@ paths: description: Registered model unique name identifier. new_name: type: string - description: If provided, updates the name for this `registered_model`. + description: If provided, updates the name for this ``registered_model``. required: - name responses: @@ -7705,12 +10590,12 @@ paths: /api/2.0/mlflow/model-versions/search: get: operationId: model_registry_search_model_versions - summary: Searches for specific model versions based on the supplied __filter__. + summary: Searches for specific model versions based on the supplied **filter**. tags: - ml - model_registry description: |- - Searches for specific model versions based on the supplied __filter__. + Searches for specific model versions based on the supplied **filter**. :param filter: str (optional) String filter condition, like "name='my-model-name'". Must be a single boolean condition, with @@ -7781,12 +10666,12 @@ paths: /api/2.0/mlflow/registered-models/search: get: operationId: model_registry_search_models - summary: Search for registered models based on the specified __filter__. + summary: Search for registered models based on the specified **filter**. tags: - ml - model_registry description: |- - Search for registered models based on the specified __filter__. + Search for registered models based on the specified **filter**. :param filter: str (optional) String filter condition, like "name LIKE 'my-model-name'". Interpreted in the backend automatically @@ -7874,8 +10759,8 @@ paths: Unique name of the model. :param key: str Name of the tag. Maximum size depends on storage backend. If a tag with this name already exists, - its preexisting value will be replaced by the specified `value`. All storage backends are guaranteed - to support key values up to 250 bytes in size. + its preexisting value will be replaced by the specified ``value``. All storage backends are + guaranteed to support key values up to 250 bytes in size. :param value: str String value of the tag being logged. Maximum size depends on storage backend. All storage backends are guaranteed to support key values up to 5000 bytes in size. @@ -7893,7 +10778,7 @@ paths: description: >- Name of the tag. Maximum size depends on storage backend. If a tag with this name already exists, its preexisting value - will be replaced by the specified `value`. All storage + will be replaced by the specified ``value``. All storage backends are guaranteed to support key values up to 250 bytes in size. value: @@ -7937,8 +10822,8 @@ paths: Model version number. :param key: str Name of the tag. Maximum size depends on storage backend. If a tag with this name already exists, - its preexisting value will be replaced by the specified `value`. All storage backends are guaranteed - to support key values up to 250 bytes in size. + its preexisting value will be replaced by the specified ``value``. All storage backends are + guaranteed to support key values up to 250 bytes in size. :param value: str String value of the tag being logged. Maximum size depends on storage backend. All storage backends are guaranteed to support key values up to 5000 bytes in size. @@ -7959,7 +10844,7 @@ paths: description: >- Name of the tag. Maximum size depends on storage backend. If a tag with this name already exists, its preexisting value - will be replaced by the specified `value`. All storage + will be replaced by the specified ``value``. All storage backends are guaranteed to support key values up to 250 bytes in size. value: @@ -8001,8 +10886,8 @@ paths: :param id: str Webhook ID :param event: :class:`RegistryWebhookEvent` (optional) - If `event` is specified, the test trigger uses the specified event. If `event` is not specified, the - test trigger uses a randomly chosen event associated with the webhook. + If ``event`` is specified, the test trigger uses the specified event. If ``event`` is not specified, + the test trigger uses a randomly chosen event associated with the webhook. :returns: :class:`TestRegistryWebhookResponse` requestBody: @@ -8030,9 +10915,10 @@ paths: - TRANSITION_REQUEST_TO_PRODUCTION_CREATED - TRANSITION_REQUEST_TO_STAGING_CREATED description: >- - If `event` is specified, the test trigger uses the specified - event. If `event` is not specified, the test trigger uses a - randomly chosen event associated with the webhook. + If ``event`` is specified, the test trigger uses the + specified event. If ``event`` is not specified, the test + trigger uses a randomly chosen event associated with the + webhook. required: - id responses: @@ -8059,20 +10945,18 @@ paths: operationId: model_registry_transition_stage summary: >- Transition a model version's stage. This is a Databricks workspace - version of the [MLflow endpoint] + version of the `MLflow endpoint tags: - ml - model_registry description: >- Transition a model version's stage. This is a Databricks workspace - version of the [MLflow endpoint] - - that also accepts a comment associated with the transition to be - recorded. + version of the `MLflow endpoint + `__ + that also accepts - [MLflow endpoint]: - https://www.mlflow.org/docs/latest/rest-api.html#transition-modelversion-stage + a comment associated with the transition to be recorded. :param name: str @@ -8082,13 +10966,10 @@ paths: :param stage: str Target stage of the transition. Valid values are: - * `None`: The initial stage of a model version. - - * `Staging`: Staging or pre-production stage. - - * `Production`: Production stage. - - * `Archived`: Archived stage. + - ``None``: The initial stage of a model version. + - ``Staging``: Staging or pre-production stage. + - ``Production``: Production stage. + - ``Archived``: Archived stage. :param archive_existing_versions: bool Specifies whether to archive all current model versions in the target stage. :param comment: str (optional) @@ -8110,10 +10991,11 @@ paths: stage: type: string description: >- - Target stage of the transition. Valid values are: * `None`: - The initial stage of a model version. * `Staging`: Staging - or pre-production stage. * `Production`: Production stage. * - `Archived`: Archived stage. + Target stage of the transition. Valid values are: - + ``None``: The initial stage of a model version. - + ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived + stage. archive_existing_versions: type: boolean description: >- @@ -8209,7 +11091,7 @@ paths: :param name: str Registered model unique name identifier. :param description: str (optional) - If provided, updates the description for this `registered_model`. + If provided, updates the description for this ``registered_model``. :returns: :class:`UpdateModelResponse` requestBody: @@ -8225,7 +11107,7 @@ paths: type: string description: >- If provided, updates the description for this - `registered_model`. + ``registered_model``. required: - name responses: @@ -8262,7 +11144,7 @@ paths: :param version: str Model version number :param description: str (optional) - If provided, updates the description for this `registered_model`. + If provided, updates the description for this ``registered_model``. :returns: :class:`UpdateModelVersionResponse` requestBody: @@ -8281,7 +11163,7 @@ paths: type: string description: >- If provided, updates the description for this - `registered_model`. + ``registered_model``. required: - name - version @@ -8323,34 +11205,24 @@ paths: :param description: str (optional) User-specified description for the webhook. :param events: List[:class:`RegistryWebhookEvent`] (optional) - Events that can trigger a registry webhook: * `MODEL_VERSION_CREATED`: A new model version was - created for the associated model. - - * `MODEL_VERSION_TRANSITIONED_STAGE`: A model version’s stage was changed. - - * `TRANSITION_REQUEST_CREATED`: A user requested a model version’s stage be transitioned. - - * `COMMENT_CREATED`: A user wrote a comment on a registered model. - - * `REGISTERED_MODEL_CREATED`: A new registered model was created. This event type can only be - specified for a registry-wide webhook, which can be created by not specifying a model name in the - create request. - - * `MODEL_VERSION_TAG_SET`: A user set a tag on the model version. - - * `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model version was transitioned to staging. - - * `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version was transitioned to production. - - * `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version was archived. - - * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A user requested a model version be transitioned to - staging. - - * `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested a model version be transitioned to - production. - - * `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a model version be archived. + Events that can trigger a registry webhook: + + - ``MODEL_VERSION_CREATED``: A new model version was created for the associated model. + - ``MODEL_VERSION_TRANSITIONED_STAGE``: A model version’s stage was changed. + - ``TRANSITION_REQUEST_CREATED``: A user requested a model version’s stage be transitioned. + - ``COMMENT_CREATED``: A user wrote a comment on a registered model. + - ``REGISTERED_MODEL_CREATED``: A new registered model was created. This event type can only be + specified for a registry-wide webhook, which can be created by not specifying a model name in the + create request. + - ``MODEL_VERSION_TAG_SET``: A user set a tag on the model version. + - ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A model version was transitioned to staging. + - ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model version was transitioned to production. + - ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version was archived. + - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A user requested a model version be transitioned to + staging. + - ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user requested a model version be transitioned to + production. + - ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested a model version be archived. :param http_url_spec: :class:`HttpUrlSpec` (optional) :param job_spec: :class:`JobSpec` (optional) @@ -8441,29 +11313,29 @@ paths: attributes -- see the documentation for details. description: >- - Events that can trigger a registry webhook: * - `MODEL_VERSION_CREATED`: A new model version was created for - the associated model. * `MODEL_VERSION_TRANSITIONED_STAGE`: - A model version’s stage was changed. * - `TRANSITION_REQUEST_CREATED`: A user requested a model - version’s stage be transitioned. * `COMMENT_CREATED`: A user - wrote a comment on a registered model. * - `REGISTERED_MODEL_CREATED`: A new registered model was - created. This event type can only be specified for a - registry-wide webhook, which can be created by not - specifying a model name in the create request. * - `MODEL_VERSION_TAG_SET`: A user set a tag on the model - version. * `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model - version was transitioned to staging. * - `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version - was transitioned to production. * - `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version - was archived. * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A - user requested a model version be transitioned to staging. * - `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested - a model version be transitioned to production. * - `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a - model version be archived. + Events that can trigger a registry webhook: - + ``MODEL_VERSION_CREATED``: A new model version was created + for the associated model. - + ``MODEL_VERSION_TRANSITIONED_STAGE``: A model version’s + stage was changed. - ``TRANSITION_REQUEST_CREATED``: A user + requested a model version’s stage be transitioned. - + ``COMMENT_CREATED``: A user wrote a comment on a registered + model. - ``REGISTERED_MODEL_CREATED``: A new registered + model was created. This event type can only be specified for + a registry-wide webhook, which can be created by not + specifying a model name in the create request. - + ``MODEL_VERSION_TAG_SET``: A user set a tag on the model + version. - ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A + model version was transitioned to staging. - + ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model + version was transitioned to production. - + ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version + was archived. - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A + user requested a model version be transitioned to staging. - + ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user + requested a model version be transitioned to production. - + ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested + a model version be archived. http_url_spec: type: object properties: @@ -8474,7 +11346,7 @@ paths: description: >- Value of the authorization header that should be sent in the request sent by the wehbook. It should be of the - form `" "`. If set to an empty + form ``" "``. If set to an empty string, no authorization header will be included in the request. enable_ssl_verification: @@ -8527,17 +11399,17 @@ paths: Enable or disable triggering the webhook, or put the webhook into test mode. The default is - `ACTIVE`: * `ACTIVE`: Webhook is triggered when an - associated event happens. + ``ACTIVE``: - * `DISABLED`: Webhook is not triggered. + - ``ACTIVE``: Webhook is triggered when an associated event + happens. + - ``DISABLED``: Webhook is not triggered. - * `TEST_MODE`: Webhook can be triggered through the test + - ``TEST_MODE``: Webhook can be triggered through the test endpoint, but is not triggered on a - - real event. + real event. required: - id responses: @@ -8578,9 +11450,9 @@ components: type: string description: >- Source stage of the transition (if the activity is stage transition - related). Valid values are: * `None`: The initial stage of a model - version. * `Staging`: Staging or pre-production stage. * - `Production`: Production stage. * `Archived`: Archived stage. + related). Valid values are: - ``None``: The initial stage of a model + version. - ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived stage. id: type: string description: Unique identifier for the object. @@ -8593,16 +11465,16 @@ components: type: string description: >- Comment made by system, for example explaining an activity of type - `SYSTEM_TRANSITION`. It usually describes a side effect, such as a + ``SYSTEM_TRANSITION``. It usually describes a side effect, such as a version being archived as part of another version's stage transition, and may not be returned for some activity types. to_stage: type: string description: >- Target stage of the transition (if the activity is stage transition - related). Valid values are: * `None`: The initial stage of a model - version. * `Staging`: Staging or pre-production stage. * - `Production`: Production stage. * `Archived`: Archived stage. + related). Valid values are: - ``None``: The initial stage of a model + version. - ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived stage. user_id: type: string description: The username of the user that created the object. @@ -8623,8 +11495,16 @@ components: $ref: '#/components/schemas/CountFunction' first: $ref: '#/components/schemas/FirstFunction' + first_distinct: + $ref: '#/components/schemas/FirstDistinctFunction' + first_n: + $ref: '#/components/schemas/FirstNFunction' last: $ref: '#/components/schemas/LastFunction' + last_distinct: + $ref: '#/components/schemas/LastDistinctFunction' + last_n: + $ref: '#/components/schemas/LastNFunction' max: $ref: '#/components/schemas/MaxFunction' min: @@ -8683,22 +11563,39 @@ components: AuthConfig: type: object properties: + mtls_config: + $ref: '#/components/schemas/MtlsConfig' uc_service_credential_name: type: string + description: >- + Name of the Unity Catalog service credential. This value will be set + under the option databricks.serviceCredential AvgFunction: type: object properties: input: type: string - description: The input column from which the average is computed. + description: >- + The input column from which the average is computed. For Kafka + sources, use dot-prefixed path notation (e.g., "value.amount"). For + nested fields, the leaf node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for backwards compatibility but + is deprecated; migrate to dot notation. required: - input description: Computes the average of values. BackfillSource: type: object properties: + delta_table_name: + type: string delta_table_source: $ref: '#/components/schemas/DeltaTableSource' + description: >- + Deprecated: Use delta_table_name instead. Kept for backwards + compatibility. The Delta table source containing the historical data + to backfill. Only the delta table name is used for backfill, other + fields are ignored. BatchCreateMaterializedFeaturesResponse: type: object properties: @@ -8713,6 +11610,18 @@ components: type: string required: - variant_expr_path + ColumnSelection: + type: object + properties: + column: + type: string + description: Column name from source to select as the feature value. + required: + - column + description: >- + A ColumnSelection function, equivalent to the LAST() record of an entity + over a lifetime + ContinuousWindow CommentObject: type: object properties: @@ -8749,17 +11658,24 @@ components: properties: window_duration: type: string + description: The duration of the continuous window (must be positive). offset: type: string description: The offset of the continuous window (must be non-positive). required: - window_duration + description: 'Deprecated: use RollingWindow with ``delay`` instead.' CountFunction: type: object properties: input: type: string - description: The input column from which the count is computed. + description: >- + The input column from which the count is computed. For Kafka + sources, use dot-prefixed path notation (e.g., "value.amount"). For + nested fields, the leaf node name is used. Colon-prefixed notation + (e.g., "value:amount") is supported for backwards compatibility but + is deprecated; migrate to dot notation. required: - input description: Computes the count of values. @@ -8815,13 +11731,52 @@ components: properties: webhook: $ref: '#/components/schemas/RegistryWebhook' + CronSchedule: + type: object + properties: + cron_expression: + type: string + description: >- + The cron expression defining the schedule (e.g., "0 0 * * *" for + daily at midnight). + description: A cron-based schedule trigger for the materialization pipeline. + CustomUdf: + type: object + properties: + function_path: + type: string + description: Fully qualified 3-part Unity Catalog path of the function to apply. + input_bindings: + type: array + items: + $ref: '#/components/schemas/InputBinding' + description: >- + Binds each UC function parameter to a source column. May be empty + for zero-argument functions (e.g. a timestamp generator). + required: + - function_path + description: >- + A CustomUdf function applies a registered Unity Catalog function + row-wise to source columns, + producing a single output column per row. DataSource: type: object properties: delta_table_source: $ref: '#/components/schemas/DeltaTableSource' + description: A Delta table data source. kafka_source: $ref: '#/components/schemas/KafkaSource' + description: A Kafka stream data source. + request_source: + $ref: '#/components/schemas/RequestSource' + description: A request-time data source. + stream_source: + $ref: '#/components/schemas/StreamSource' + description: A Stream data source. + description: >- + Specifies the data source backing a feature. Exactly one source type + must be set. Dataset: type: object properties: @@ -8961,12 +11916,49 @@ components: description: >- A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT *, col_a + col_b - AS col_c FROM x.y.z WHERE col_a > 0" would have `transformation_sql` - "*, col_a + col_b AS col_c") If transformation_sql is not provided, - all columns of the delta table are present in the DataSource - dataframe. + AS col_c FROM x.y.z WHERE col_a > 0" would have + ``transformation_sql`` "*, col_a + col_b AS col_c") If + transformation_sql is not provided, all columns of the delta table + are present in the DataSource dataframe. required: - full_name + DirectMtlsConfig: + type: object + properties: + bootstrap_servers: + type: string + description: >- + A comma-separated list of host:port pairs for the Kafka bootstrap + servers. + mtls_config: + $ref: '#/components/schemas/MtlsConfig' + description: Mutual-TLS authentication configuration. + required: + - bootstrap_servers + - mtls_config + description: >- + Direct connection configs for mTLS, as Kafka Connections do not support + mTLS yet . Temporarily + used until UC Kafka Connections gain mTLS support. + DirectSchemas: + type: object + properties: + key_schema: + $ref: '#/components/schemas/SchemaConfig' + description: >- + Schema for the message key. This is only used for Kafka streams. For + Kafka, at least one of payload_schema or key_schema must be + specified. + payload_schema: + $ref: '#/components/schemas/SchemaConfig' + description: >- + Schema for the message payload. For Kafka, this is the value schema. + Unless the platform supports another schema (e.g. keys for Kafka), + this must be specified. + description: >- + Schema definitions provided directly on the Stream, as opposed to + referencing a schema registry. + To resolve schemas from a registry instead, use SchemaRegistryConfig. EntityColumn: type: object properties: @@ -9002,6 +11994,12 @@ components: items: $ref: '#/components/schemas/ExperimentTag' description: 'Tags: Additional metadata key-value pairs.' + trace_location: + $ref: '#/components/schemas/ExperimentTraceLocation' + description: >- + The location where the experiment's traces are stored. Unset when + traces are stored in the default MLflow backend. This field cannot + be updated after the experiment is created. description: An experiment and its metadata. ExperimentAccessControlRequest: type: object @@ -9074,6 +12072,15 @@ components: type: string description: The tag value. description: A tag for an experiment. + ExperimentTraceLocation: + type: object + properties: + uc_trace_location: + $ref: '#/components/schemas/UcTraceLocation' + description: >- + A Unity Catalog schema where the experiment's traces are stored as + Delta tables. + description: The storage location for an experiment's traces. Feature: type: object properties: @@ -9085,6 +12092,16 @@ components: function: $ref: '#/components/schemas/Function' description: The function by which the feature is computed. + catalog_name: + type: string + description: Name of parent catalog. + created_at: + type: string + format: date-time + description: Time at which this feature was created. + created_by: + type: string + description: Username of the feature creator. description: type: string description: The description of the feature. @@ -9120,6 +12137,14 @@ components: incorrect values may lead to inaccurate lineage tracking or unexpected behavior. This field will be set by feature-engineering client and should be left unset by SDK and terraform users. + name: + type: string + description: >- + Name of the feature, extracted from the full three-part name + (catalog.schema.name). + schema_name: + type: string + description: Name of parent schema relative to its parent catalog. time_window: $ref: '#/components/schemas/TimeWindow' description: >- @@ -9191,6 +12216,22 @@ components: required: - key description: Represents a tag on a feature in a feature table. + FieldDefinition: + type: object + properties: + name: + type: string + description: The name of the field. + data_type: + $ref: '#/components/schemas/ScalarDataType' + description: The scalar data type of the field. + required: + - name + - data_type + description: >- + A single field definition within a FlatSchema, specifying the field name + and its scalar data + type. Does not support nested or complex types (arrays, maps, structs). FileInfo: type: object properties: @@ -9209,6 +12250,23 @@ components: properties: model: $ref: '#/components/schemas/LoggedModel' + FirstDistinctFunction: + type: object + properties: + input: + type: string + description: >- + The input column from which the first N distinct values are + returned. + 'n': + type: integer + description: The number of distinct values to return. + required: + - input + - 'n' + description: >- + Returns the first N distinct values, ordered by the feature's timeseries + column. FirstFunction: type: object properties: @@ -9218,6 +12276,33 @@ components: required: - input description: Returns the first value. + FirstNFunction: + type: object + properties: + input: + type: string + description: The input column from which the first N values are returned. + 'n': + type: integer + description: The number of values to return. + required: + - input + - 'n' + description: Returns the first N values, ordered by the feature's timeseries column. + FlatSchema: + type: object + properties: + fields: + type: array + items: + $ref: '#/components/schemas/FieldDefinition' + description: The list of fields in this schema. + required: + - fields + description: >- + A flat (non-nested) schema for request-time fields, defined as an + ordered list of field + definitions. This schema only supports scalar types. ForecastingExperiment: type: object properties: @@ -9238,6 +12323,14 @@ components: properties: aggregation_function: $ref: '#/components/schemas/AggregationFunction' + column_selection: + $ref: '#/components/schemas/ColumnSelection' + description: Selects the latest value of a single column in a data source + custom_udf: + $ref: '#/components/schemas/CustomUdf' + description: >- + Applies a registered Unity Catalog function row-wise to source + columns. extra_parameters: type: array items: @@ -9296,6 +12389,13 @@ components: properties: model: $ref: '#/components/schemas/LoggedModel' + GetLoggedModelsRequestResponse: + type: object + properties: + models: + type: array + items: + $ref: '#/components/schemas/LoggedModel' GetMetricHistoryResponse: type: object properties: @@ -9345,8 +12445,8 @@ components: type: string description: >- Value of the authorization header that should be sent in the request - sent by the wehbook. It should be of the form `" - "`. If set to an empty string, no authorization header + sent by the wehbook. It should be of the form ``" + "``. If set to an empty string, no authorization header will be included in the request. enable_ssl_verification: type: boolean @@ -9377,6 +12477,84 @@ components: description: >- External HTTPS URL called on event trigger (by using a POST request). + IngestionConfig: + type: object + properties: + ingestion_destination: + $ref: '#/components/schemas/IngestionDestination' + description: >- + Destination for the Databricks-managed Delta table that holds an + offline copy of the streaming data for querying and training. This + table contains both 1) forward-filled data from the Stream and 2) + backfilled data from the BackfillSource (if provided). This table is + created and managed by Databricks and is deleted when the Stream is + deleted. + backfill_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the historical backfill + of the ingestion Delta table. + backfill_source: + $ref: '#/components/schemas/BackfillSource' + description: >- + A user-provided source for backfilling data. Historical data is used + when creating a training set from streaming features linked to this + Stream. The backfill data stored in this location will be copied + into the ingestion table for offline querying and training. The + schema for this source must match exactly that of the key and + payload schemas specified for this Stream. + deduplication_columns: + type: array + items: + type: string + description: >- + Column paths used to identify duplicate rows during ingestion; only + one row per distinct combination of these values is kept. Use dot + notation for nested fields (e.g. ``value.user_id``). Empty list + means every column is compared. + ingestion_job_id: + type: integer + description: >- + The ID of the Databricks Job that performs the forward-fill + ingestion. + ingestion_pipeline_id: + type: string + description: >- + The ID of the SDP pipeline that continuously copies new events from + the streaming source into the ingestion Delta table. + required: + - ingestion_destination + description: >- + Configuration for the Databricks-managed ingestion pipeline. Groups the + ingestion destination + (required) and optional backfill source. + IngestionDestination: + type: object + properties: + delta_table_name: + type: string + description: >- + The full three-part name (catalog, schema, name) of the Delta table + to be created for ingestion. + description: >- + Destination for the Databricks-managed Delta table that holds an offline + copy of the streaming + data for querying and training. + InputBinding: + type: object + properties: + parameter: + type: string + description: Name of the UC function parameter. + column: + type: string + description: >- + Source column whose value is passed for this parameter at execution + time. + required: + - parameter + - column + description: Binds a single UC function parameter to a source column. InputTag: type: object properties: @@ -9454,6 +12632,11 @@ components: description: >- Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*) + ingestion_config: + $ref: '#/components/schemas/IngestionConfig' + description: >- + Configuration for ingesting Kafka data into a Databricks-managed + Delta table. key_schema: $ref: '#/components/schemas/SchemaConfig' description: >- @@ -9474,6 +12657,12 @@ components: properties: name: type: string + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after transformations, in Spark + StructType JSON format (from df.schema.json()). Any subsequent + functions operate against this dataframe. entity_column_identifiers: type: array items: @@ -9490,8 +12679,92 @@ components: Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column identifier of the Kafka source. + transformation_sql: + type: string + description: >- + The pipeline runs these SQL statements immediately after conversion + into the schema specified on the KafkaConfig object. required: - name + KafkaStreamConfig: + type: object + properties: + subscription_mode: + $ref: '#/components/schemas/KafkaSubscriptionMode' + description: Options to configure which Kafka topics to pull data from. + extra_options: + type: object + description: >- + Optional Kafka source or consumer options, validated against a + server-side allowlist at request time. Allowed keys: - + ``maxOffsetsPerTrigger`` - ``startingOffsets`` - ``includeHeaders`` + - ``kafka.request.timeout.ms`` - ``kafka.session.timeout.ms`` - + ``kafka.max.partition.fetch.bytes`` The following keys are + ingestion-only and are stripped before being forwarded to the + materialization pipeline: - ``maxOffsetsPerTrigger`` - + ``startingOffsets`` Auth and connection details belong on the parent + Stream's ``connection_config``, not here. + required: + - subscription_mode + description: Kafka-specific configuration for a Stream. + KafkaSubscriptionMode: + type: object + properties: + assign: + type: string + description: >- + A JSON string that contains the specific topic-partitions to consume + from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's + 0'th and 1st partitions will be consumed from. + subscribe: + type: string + description: >- + A comma-separated list of Kafka topics to read from. For example, + 'topicA,topicB,topicC'. + subscribe_pattern: + type: string + description: >- + A regular expression matching topics to subscribe to. For example, + 'topic.*' will subscribe to all topics starting with 'topic'. + description: >- + Subscription mode for Kafka topic selection, matching standard Spark + Structured Streaming + options. + KinesisStreamConfig: + type: object + properties: + extra_options: + type: object + description: >- + Optional Kinesis source options, validated against a server-side + allowlist at request time. Auth and connection details belong on the + parent Stream's ``connection_config``, not here. + stream_arns: + $ref: '#/components/schemas/StreamArnList' + description: Kinesis stream ARNs to read from. + stream_names: + $ref: '#/components/schemas/StreamNameList' + description: Kinesis stream names to read from. + description: >- + Kinesis-specific configuration for a Stream. For the underlying + connector and its source + options, see the Databricks documentation on connecting to Amazon Kinesis + (https://docs.databricks.com/aws/en/connect/streaming/kinesis). + LastDistinctFunction: + type: object + properties: + input: + type: string + description: The input column from which the last N distinct values are returned. + 'n': + type: integer + description: The number of distinct values to return. + required: + - input + - 'n' + description: >- + Returns the last N distinct values, ordered by the feature's timeseries + column. LastFunction: type: object properties: @@ -9501,6 +12774,19 @@ components: required: - input description: Returns the last value. + LastNFunction: + type: object + properties: + input: + type: string + description: The input column from which the last N values are returned. + 'n': + type: integer + description: The number of values to return. + required: + - input + - 'n' + description: Returns the last N values, ordered by the feature's timeseries column. LineageContext: type: object properties: @@ -9527,9 +12813,7 @@ components: feature_table_name: type: string description: Feature table name - description: >- - Feature for model version. ([ML-57150] Renamed from Feature to - LinkedFeature) + description: Feature for model version. ListArtifactsResponse: type: object properties: @@ -9629,6 +12913,27 @@ components: items: $ref: '#/components/schemas/RegistryWebhook' description: Array of registry webhooks. + ListStreamsResponse: + type: object + properties: + next_page_token: + type: string + description: Pagination token to request the next page of results for this query. + streams: + type: array + items: + $ref: '#/components/schemas/Stream' + description: List of Streams. + description: |- + Response to a ListStreamsRequest. + + NOTE: Results are post-filtered by access permission on each stream's ingestion table. This + means: + + - Returned results may be fewer than page_size (including zero) + - Page token points to next unfiltered batch, not next filtered batch, and may point to an item + that will be filtered out Callers should paginate until next_page_token is empty to retrieve + all accessible streams. ListTransitionRequestsResponse: type: object properties: @@ -9765,7 +13070,16 @@ components: description: >- The quartz cron expression that defines the schedule of the materialization pipeline. The schedule is evaluated in the UTC - timezone. + timezone. Hidden from GraphQL: superseded by the ``trigger`` oneof + (cron_schedule_trigger), so not exposed to Catalog Explorer. + cron_schedule_trigger: + $ref: '#/components/schemas/CronSchedule' + description: A cron-based schedule trigger for the materialization pipeline. + is_online: + type: boolean + description: >- + True if this is an online materialized feature. False if it is an + offline materialized feature. last_materialization_time: type: string description: >- @@ -9774,19 +13088,33 @@ components: field will be null. materialized_feature_id: type: string - description: Unique identifier for the materialized feature. + description: Server-assigned unique identifier for the materialized feature. offline_store_config: $ref: '#/components/schemas/OfflineStoreConfig' + description: Destination for writing feature values to an offline Delta table. online_store_config: $ref: '#/components/schemas/OnlineStoreConfig' + description: Destination for writing feature values to an online Lakebase table. pipeline_schedule_state: $ref: '#/components/schemas/MaterializedFeaturePipelineScheduleState' - description: The schedule state of the materialization pipeline. + description: >- + The schedule state of the materialization pipeline. Hidden from + GraphQL: being deprecated, so not exposed to Catalog Explorer. + streaming_mode: + $ref: '#/components/schemas/StreamingMode' + description: >- + The Structured Streaming trigger mode used for materialization. + Real-time mode (RTM) targets sub-second latency for operational + workloads; micro-batch mode (MBM) favors cost efficiency for ETL and + analytics workloads. table_name: type: string description: >- The fully qualified Unity Catalog path to the table containing the materialized feature (Delta table or Lakebase table). Output only. + table_trigger: + $ref: '#/components/schemas/TableTrigger' + description: A trigger that fires when the upstream source table changes. required: - feature_name description: >- @@ -9852,11 +13180,11 @@ components: type: integer description: type: string - description: Description of this `registered_model`. + description: Description of this ``registered_model``. last_updated_timestamp: type: integer description: >- - Timestamp recorded when metadata for this `registered_model` was + Timestamp recorded when metadata for this ``registered_model`` was last updated. latest_versions: type: array @@ -9864,7 +13192,7 @@ components: $ref: '#/components/schemas/ModelVersion' description: >- Collection of latest model versions for each stage. Only contains - models with current `READY` status. + models with current ``READY`` status. name: type: string description: Unique name for the model. @@ -9874,10 +13202,10 @@ components: $ref: '#/components/schemas/ModelTag' description: >- Tags: Additional metadata key-value pairs for this - `registered_model`. + ``registered_model``. user_id: type: string - description: User that created this `registered_model` + description: User that created this ``registered_model`` ModelDatabricks: type: object properties: @@ -9952,14 +13280,14 @@ components: type: integer current_stage: type: string - description: Current stage for this `model_version`. + description: Current stage for this ``model_version``. description: type: string - description: Description of this `model_version`. + description: Description of this ``model_version``. last_updated_timestamp: type: integer description: >- - Timestamp recorded when metadata for this `model_version` was last + Timestamp recorded when metadata for this ``model_version`` was last updated. name: type: string @@ -9967,8 +13295,8 @@ components: run_id: type: string description: >- - MLflow run ID used when creating `model_version`, if `source` was - generated by an experiment run stored in MLflow tracking server. + MLflow run ID used when creating ``model_version``, if ``source`` + was generated by an experiment run stored in MLflow tracking server. run_link: type: string description: 'Run Link: Direct link to the run that generated this version' @@ -9976,21 +13304,23 @@ components: type: string description: >- URI indicating the location of the source model artifacts, used when - creating `model_version` + creating ``model_version`` status: $ref: '#/components/schemas/ModelVersionStatus' - description: Current status of `model_version` + description: Current status of ``model_version`` status_message: type: string - description: Details on current `status`, if it is pending or failed. + description: Details on current ``status``, if it is pending or failed. tags: type: array items: $ref: '#/components/schemas/ModelVersionTag' - description: 'Tags: Additional metadata key-value pairs for this `model_version`.' + description: >- + Tags: Additional metadata key-value pairs for this + ``model_version``. user_id: type: string - description: User that created this `model_version`. + description: User that created this ``model_version``. version: type: string description: Model's version number. @@ -10012,7 +13342,7 @@ components: the model version. feature_list: $ref: '#/components/schemas/FeatureList' - description: Feature lineage of `model_version`. + description: Feature lineage of ``model_version``. last_updated_timestamp: type: integer description: >- @@ -10026,9 +13356,9 @@ components: items: $ref: '#/components/schemas/Activity' description: >- - Open requests for this `model_versions`. Gap in sequence number is + Open requests for this ``model_versions``. Gap in sequence number is intentional and is done in order to match field sequence numbers of - `ModelVersion` proto message + ``ModelVersion`` proto message permission_level: $ref: '#/components/schemas/PermissionLevel' run_id: @@ -10072,6 +13402,66 @@ components: value: type: string description: The tag value. + MtlsConfig: + type: object + properties: + keystore_location: + type: string + description: >- + Unity Catalog volume path to the JKS keystore file containing the + client certificate and private key. e.g. + "/Volumes////client.jks". The + materialization compute must have read permission on this volume. + keystore_password_ref: + $ref: '#/components/schemas/SecretScopeReference' + description: Secret-scope reference for the JKS keystore password. + key_password_ref: + $ref: '#/components/schemas/SecretScopeReference' + description: >- + Secret-scope reference for the private key password. Often the same + value as the keystore password (keytool's default), but provided as + a separate field because Apache Kafka requires it as a distinct + option (kafka.ssl.key.password). + truststore_location: + type: string + description: >- + Unity Catalog volume path to the JKS truststore file containing the + CA certificate(s) trusted to verify the Kafka broker's server + certificate. e.g. + "/Volumes////truststore.jks". + truststore_password_ref: + $ref: '#/components/schemas/SecretScopeReference' + description: Secret-scope reference for the JKS truststore password. + disable_hostname_verification: + type: boolean + description: >- + Set to true only when the broker certificate's SAN intentionally + does not match the connection endpoint — for example when reaching + the cluster through a PrivateLink endpoint whose DNS name is not in + the broker certificate. Skipping the hostname check removes a + defense against man-in-the-middle attacks; do not enable casually. + mTLS client authentication is unaffected by this option. See the + Apache Kafka SSL security guide for background on this check: + https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification + required: + - keystore_location + - keystore_password_ref + - key_password_ref + - truststore_location + - truststore_password_ref + description: >- + Mutual-TLS (mTLS) authentication configuration. The keystore (client + certificate + private key) + and truststore (CAs trusted to verify the broker) live as JKS files on Unity Catalog volumes, + with their passwords stored in Databricks secret scopes. This matches the SSL setup pattern + documented at + https://docs.databricks.com/en/connect/streaming/kafka/authentication#use-ssl-to-connect-databricks-to-kafka. + + At materialization time, the generated PySpark code passes the JKS file paths and resolved + passwords through to the Kafka SSL options (kafka.ssl.keystore.location, + kafka.ssl.keystore.password, kafka.ssl.key.password, kafka.ssl.truststore.location, + kafka.ssl.truststore.password). Passwords are resolved on the Spark cluster via + dbutils.secrets.get; this message stores only references, never password values. OfflineStoreConfig: type: object properties: @@ -10165,6 +13555,30 @@ components: type: string description: Value associated with this param. description: Param associated with a run. + ProtoSchemaSpec: + type: object + properties: + schema_text: + type: string + description: >- + The raw .proto file text (proto2 and proto3 syntax supported, see + https://protobuf.dev/programming-guides/proto3/ and + https://protobuf.dev/programming-guides/proto2/). + message_name: + type: string + description: >- + The fully-qualified name of the message within schema_text that + describes the Kafka payload (e.g. "Event" or "com.example.Event" if + schema_text declares a package). Identifies which message is used to + decode each Kafka record — a .proto file may declare multiple + messages but only one represents the payload. Must not be empty. + required: + - schema_text + - message_name + description: >- + A Protocol Buffer schema paired with the name of the message within it + that describes the Kafka + payload. A .proto file may declare multiple messages; message_name disambiguates. PublishSpec: type: object properties: @@ -10180,6 +13594,13 @@ components: description: >- The publish mode of the pipeline that syncs the online table with the source table. + full_feature_name: + type: string + description: >- + Full Unity Catalog name of one of the features materialized in the + source table, used to derive the synced online table's entity and + timeseries columns. Required for view sources without a UC + PrimaryKeyConstraint; ignored when the source already has one. required: - online_store - online_table_name @@ -10268,26 +13689,26 @@ components: items: $ref: '#/components/schemas/RegistryWebhookEvent' description: >- - Events that can trigger a registry webhook: * - `MODEL_VERSION_CREATED`: A new model version was created for the - associated model. * `MODEL_VERSION_TRANSITIONED_STAGE`: A model - version’s stage was changed. * `TRANSITION_REQUEST_CREATED`: A user - requested a model version’s stage be transitioned. * - `COMMENT_CREATED`: A user wrote a comment on a registered model. * - `REGISTERED_MODEL_CREATED`: A new registered model was created. This - event type can only be specified for a registry-wide webhook, which - can be created by not specifying a model name in the create request. - * `MODEL_VERSION_TAG_SET`: A user set a tag on the model version. * - `MODEL_VERSION_TRANSITIONED_TO_STAGING`: A model version was - transitioned to staging. * - `MODEL_VERSION_TRANSITIONED_TO_PRODUCTION`: A model version was - transitioned to production. * - `MODEL_VERSION_TRANSITIONED_TO_ARCHIVED`: A model version was - archived. * `TRANSITION_REQUEST_TO_STAGING_CREATED`: A user - requested a model version be transitioned to staging. * - `TRANSITION_REQUEST_TO_PRODUCTION_CREATED`: A user requested a model - version be transitioned to production. * - `TRANSITION_REQUEST_TO_ARCHIVED_CREATED`: A user requested a model + Events that can trigger a registry webhook: - + ``MODEL_VERSION_CREATED``: A new model version was created for the + associated model. - ``MODEL_VERSION_TRANSITIONED_STAGE``: A model + version’s stage was changed. - ``TRANSITION_REQUEST_CREATED``: A + user requested a model version’s stage be transitioned. - + ``COMMENT_CREATED``: A user wrote a comment on a registered model. - + ``REGISTERED_MODEL_CREATED``: A new registered model was created. + This event type can only be specified for a registry-wide webhook, + which can be created by not specifying a model name in the create + request. - ``MODEL_VERSION_TAG_SET``: A user set a tag on the model + version. - ``MODEL_VERSION_TRANSITIONED_TO_STAGING``: A model + version was transitioned to staging. - + ``MODEL_VERSION_TRANSITIONED_TO_PRODUCTION``: A model version was + transitioned to production. - + ``MODEL_VERSION_TRANSITIONED_TO_ARCHIVED``: A model version was + archived. - ``TRANSITION_REQUEST_TO_STAGING_CREATED``: A user + requested a model version be transitioned to staging. - + ``TRANSITION_REQUEST_TO_PRODUCTION_CREATED``: A user requested a + model version be transitioned to production. - + ``TRANSITION_REQUEST_TO_ARCHIVED_CREATED``: A user requested a model version be archived. http_url_spec: $ref: '#/components/schemas/HttpUrlSpecWithoutSecret' @@ -10316,6 +13737,16 @@ components: properties: registered_model: $ref: '#/components/schemas/Model' + RequestSource: + type: object + properties: + flat_schema: + $ref: '#/components/schemas/FlatSchema' + description: A flat schema with scalar-typed fields only. + description: >- + A request-time data source whose value is provided at inference time: + offline batch scoring or + online serving endpoint RestoreExperimentResponse: type: object properties: {} @@ -10327,6 +13758,25 @@ components: properties: runs_restored: type: integer + RollingWindow: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the rolling window (must be + non-negative). For example, delay=1d shifts the window end 1 day + before the evaluation time. + window_duration: + type: string + description: >- + The duration of the rolling window. Must be positive when set; + absent means lifetime (aggregate over the entity's entire history). + description: >- + A rolling time window with an optional delay. This is the + SQL-spec-aligned replacement for + ContinuousWindow: ``delay`` is the non-negative counterpart of the legacy non-positive + ``ContinuousWindow.offset``. Run: type: object properties: @@ -10428,11 +13878,105 @@ components: type: string description: The tag value. description: Tag for a run. + SawtoothWindow: + type: object + properties: + delay: + type: string + description: >- + The delay applied to the end of the window (must be non-negative). + For example, delay=1d shifts the window end 1 day before the + evaluation time. + window_duration: + type: string + description: >- + The duration of the window. Must be positive and span more than two + days when set, so that both the batch (N-1 day) and stale-path (N-2 + day) partial aggregates are well defined. The duration need not be a + whole number of days (e.g. 3 days 15 minutes is allowed). Absent + means lifetime (aggregate over the entity's entire history). + description: >- + A sawtooth window served via the hybrid batch + streaming path. The + batch pipeline maintains + daily partial aggregates for the bulk of the window while the streaming pipeline maintains the + most recent day(s), and serving merges them on read. Same field shape as RollingWindow, but a + distinct type so the control plane can explicitly identify hybrid (sawtooth) features rather + than inferring hybrid behavior from window_duration. SchemaConfig: type: object properties: + avro_schema: + type: string json_schema: type: string + description: >- + Schema of the JSON object in standard IETF JSON schema format + (https://json-schema.org/). + proto_schema: + $ref: '#/components/schemas/ProtoSchemaSpec' + description: Protocol Buffer schema with its payload message name. + SchemaLocator: + type: object + properties: + format: + $ref: '#/components/schemas/SchemaLocatorFormat' + description: Serialization format for this schema. + confluent_schema: + $ref: '#/components/schemas/SchemaLocatorConfluentSchema' + description: Confluent Schema Registry schema locator. + required: + - format + description: >- + Schema locator for one side (payload or key) of a message. Identifies + which schema to use in the + schema registry and the serialization format. + SchemaLocatorConfluentSchema: + type: object + properties: + subject: + type: string + description: The Confluent schema registry subject name. + required: + - subject + description: >- + Confluent Schema Registry schema locator. The value to provide for + ``subject`` depends on the + naming strategy configured in your registry: + + - TopicNameStrategy (default): "{topic}-key" or "{topic}-value" e.g. for topic "transactions" + use "transactions-value" for the payload and "transactions-key" for the key. + - RecordNameStrategy: the fully-qualified record name e.g. "com.example.Payment" for Avro, the + bare message name (without package) for Protobuf, or the ``title`` field value for JSON. + - TopicRecordNameStrategy: "{topic}-{fully-qualified-record-name}" e.g. + "transactions-com.example.Payment". + SchemaRegistryConfig: + type: object + properties: + api_secret_ref: + $ref: '#/components/schemas/SecretScopeReference' + description: >- + Reference to the schema registry API secret in a Databricks secret + scope. + key_schema_locator: + $ref: '#/components/schemas/SchemaLocator' + description: >- + Schema locator for the message key. Only used for Kafka streams. At + least one of payload_schema_locator or key_schema_locator must be + set. + payload_schema_locator: + $ref: '#/components/schemas/SchemaLocator' + description: >- + Schema locator for the message payload. For Kafka this is the value. + At least one of payload_schema_locator or key_schema_locator must be + set. + uc_connection: + type: string + description: A Schema Registry UC Connection object. + required: + - api_secret_ref + description: >- + Configuration for resolving a Stream's schema from an external schema + registry (e.g. Confluent). SearchExperimentsResponse: type: object properties: @@ -10525,6 +14069,22 @@ components: items: $ref: '#/components/schemas/Run' description: Runs that match the search criteria. + SecretScopeReference: + type: object + properties: + scope: + type: string + description: The Databricks secret scope name. + key: + type: string + description: The key within the scope. + required: + - scope + - key + description: >- + Reference to an entry in a Databricks secret scope. The referenced value + is fetched on the Spark + cluster at materialization time via dbutils.secrets.get(scope, key). SetExperimentTagResponse: type: object properties: {} @@ -10543,15 +14103,14 @@ components: SlidingWindow: type: object properties: - window_duration: - type: string slide_duration: type: string + window_duration: + type: string description: >- - The slide duration (interval by which windows advance, must be - positive and less than duration). + The duration of the sliding window. Must be positive when set; + absent means lifetime (aggregate over the entity's entire history). required: - - window_duration - slide_duration StddevPopFunction: type: object @@ -10560,7 +14119,10 @@ components: type: string description: >- The input column from which the population standard deviation is - computed. + computed. For Kafka sources, use dot-prefixed path notation (e.g., + "value.amount"). For nested fields, the leaf node name is used. + Colon-prefixed notation (e.g., "value:amount") is supported for + backwards compatibility but is deprecated; migrate to dot notation. required: - input description: Computes the population standard deviation. @@ -10575,11 +14137,171 @@ components: required: - input description: Computes the sample standard deviation. + Stream: + type: object + properties: + name: + type: string + description: Full three-part (catalog.schema.stream) name of the stream. + source_config: + $ref: '#/components/schemas/StreamSourceConfig' + description: >- + Source-specific configuration. Determines the streaming platform + source. + connection_config: + $ref: '#/components/schemas/StreamConnectionConfig' + description: Specifies how to connect and authenticate to the stream platform. + schema_config: + $ref: '#/components/schemas/StreamSchemaConfig' + description: >- + Schema definitions for the stream, provided either directly on the + Stream or resolved from an external schema registry through a UC + Connection. + ingestion_config: + $ref: '#/components/schemas/IngestionConfig' + description: >- + Configuration for streaming data ingestion: the managed table + storing an offline copy of forward fill data and optional historical + backfill. + browse_only: + type: boolean + description: >- + Indicates whether the principal is limited to retrieving metadata + for the associated object through the BROWSE privilege when + include_browse is enabled in the request. + create_time: + type: string + format: date-time + description: Time at which this Stream was created. + created_by: + type: string + description: Username of the Stream creator. + description: + type: string + description: User-provided description. + update_time: + type: string + format: date-time + description: Time at which this Stream was last modified. + updated_by: + type: string + description: Username of user who last modified the Stream. + required: + - name + - source_config + - connection_config + - schema_config + - ingestion_config + description: >- + A Stream is a governed UC entity representing an external streaming data + source. The + source_config oneof determines the streaming platform source (e.g. Kafka, Kinesis, etc.). + StreamArnList: + type: object + properties: + arns: + type: array + items: + type: string + description: >- + Kinesis stream ARNs to read from. For example, + 'arn:aws:kinesis:us-west-2:111122223333:stream/stream-a'. + description: A list of Kinesis stream ARNs to read from. + StreamConnectionConfig: + type: object + properties: + direct_mtls_config: + $ref: '#/components/schemas/DirectMtlsConfig' + description: >- + Direct mTLS configuration for stream platform access. This is only + used in the short term until UC Kafka Connections support mTLS . + Once UC Kafka Connections support mTLS, this will be deprecated. + uc_connection_name: + type: string + description: >- + Name of an existing UC Connection for stream platform access. Must + be the correct type for the streaming platform (e.g. a Kafka + Connection for a Kafka Stream, or a Kinesis Connection for a Kinesis + Stream). + description: Specifies how to connect and authenticate to the stream platform. + StreamNameList: + type: object + properties: + names: + type: array + items: + type: string + description: Kinesis stream names to read from. + description: A list of Kinesis stream names to read from. + StreamSchemaConfig: + type: object + properties: + direct_schemas: + $ref: '#/components/schemas/DirectSchemas' + description: Schema definitions provided directly on the Stream. + schema_registry_config: + $ref: '#/components/schemas/SchemaRegistryConfig' + description: Resolve schemas from an external schema registry. + description: >- + Schema definitions for the stream. Feature store supports both direct + schemas and schema + registries. + StreamSource: + type: object + properties: + full_name: + type: string + description: Three-part full name of the Stream (catalog.schema.stream). + dataframe_schema: + type: string + description: >- + Schema of the resulting dataframe after transformations, in Spark + StructType JSON format (from df.schema.json()). Any subsequent + functions operate against this dataframe. + filter_condition: + type: string + description: The filter condition applied to the source data before aggregation. + transformation_sql: + type: string + description: >- + The pipeline runs these SQL statements immediately after conversion + into the schema specified on the Stream object. + required: + - full_name + description: A Stream entity used as a data source for a feature. + StreamSourceConfig: + type: object + properties: + kafka_stream_config: + $ref: '#/components/schemas/KafkaStreamConfig' + description: Configuration for Apache Kafka streams. + kinesis_stream_config: + $ref: '#/components/schemas/KinesisStreamConfig' + description: Configuration for AWS Kinesis Data Streams. + description: Source-specific configuration. Determines the streaming platform source. + StreamingMode: + type: object + properties: + freshness_target: + type: string + description: >- + The desired data freshness for feature materialization, expressed as + a duration string (e.g. "1 minute"). + mode: + $ref: '#/components/schemas/StreamingModeStreamingModeType' + description: The type of streaming mode used by the materialization pipeline. + description: >- + The streaming mode configuration for a streaming materialization + pipeline. SubscriptionMode: type: object properties: assign: type: string + description: >- + A JSON string that contains the specific topic-partitions to consume + from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's + 0'th and 1st partitions will be consumed from. subscribe: type: string description: >- @@ -10590,15 +14312,25 @@ components: description: >- A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'. + description: 'Deprecated: Use KafkaSubscriptionMode instead.' SumFunction: type: object properties: input: type: string - description: The input column from which the sum is computed. + description: >- + The input column from which the sum is computed. For Kafka sources, + use dot-prefixed path notation (e.g., "value.amount"). For nested + fields, the leaf node name is used. Colon-prefixed notation (e.g., + "value:amount") is supported for backwards compatibility but is + deprecated; migrate to dot notation. required: - input description: Computes the sum of values. + TableTrigger: + type: object + properties: {} + description: A trigger that fires when the upstream source table changes. TestRegistryWebhookResponse: type: object properties: @@ -10612,6 +14344,11 @@ components: properties: continuous: $ref: '#/components/schemas/ContinuousWindow' + rolling: + $ref: '#/components/schemas/RollingWindow' + sawtooth: + $ref: '#/components/schemas/SawtoothWindow' + description: A sawtooth window served via the hybrid batch + streaming path. sliding: $ref: '#/components/schemas/SlidingWindow' tumbling: @@ -10643,9 +14380,9 @@ components: type: string description: >- Target stage of the transition (if the activity is stage transition - related). Valid values are: * `None`: The initial stage of a model - version. * `Staging`: Staging or pre-production stage. * - `Production`: Production stage. * `Archived`: Archived stage. + related). Valid values are: - ``None``: The initial stage of a model + version. - ``Staging``: Staging or pre-production stage. - + ``Production``: Production stage. - ``Archived``: Archived stage. user_id: type: string description: The username of the user that created the object. @@ -10665,6 +14402,36 @@ components: type: string required: - window_duration + UcTraceLocation: + type: object + properties: + catalog: + type: string + description: The name of the Unity Catalog catalog. + schema: + type: string + description: The name of the Unity Catalog schema within ``catalog``. + effective_table_prefix: + type: string + description: >- + The trace-table prefix actually in effect: ``table_prefix`` if it + was set on creation, otherwise the server-generated default. + table_prefix: + type: string + description: >- + The prefix for the trace tables, which are named + ``{catalog}.{schema}.{table_prefix}_otel_*``. May only contain + letters, digits, and underscores, and may be at most 238 characters. + When unset, a server-generated prefix derived from the experiment ID + is used and this field stays empty on read; the resolved value is + always available in ``effective_table_prefix``. + required: + - catalog + - schema + description: >- + A Unity Catalog trace storage location. Traces are stored as Delta + tables in the specified + catalog and schema. UpdateCommentResponse: type: object properties: @@ -10724,20 +14491,22 @@ components: activity or comment. - For activities, valid values are: * `APPROVE_TRANSITION_REQUEST`: - Approve a transition request + For activities, valid values are: + + - ``APPROVE_TRANSITION_REQUEST``: Approve a transition request - * `REJECT_TRANSITION_REQUEST`: Reject a transition request + - ``REJECT_TRANSITION_REQUEST``: Reject a transition request + - ``CANCEL_TRANSITION_REQUEST``: Cancel (delete) a transition request - * `CANCEL_TRANSITION_REQUEST`: Cancel (delete) a transition request + For comments, valid values are: - For comments, valid values are: * `EDIT_COMMENT`: Edit the comment + - ``EDIT_COMMENT``: Edit the comment - * `DELETE_COMMENT`: Delete the comment + - ``DELETE_COMMENT``: Delete the comment ActivityType: type: string x-enum: @@ -10749,29 +14518,26 @@ components: - REQUESTED_TRANSITION - SYSTEM_TRANSITION description: >- - Type of activity. Valid values are: * `APPLIED_TRANSITION`: User applied - the corresponding stage - - transition. + Type of activity. Valid values are: - * `REQUESTED_TRANSITION`: User requested the corresponding stage + - ``APPLIED_TRANSITION``: User applied the corresponding stage transition. + - ``REQUESTED_TRANSITION``: User requested the corresponding stage + transition. - * `CANCELLED_REQUEST`: User cancelled an existing transition request. - - - * `APPROVED_REQUEST`: User approved the corresponding stage transition. - + - ``CANCELLED_REQUEST``: User cancelled an existing transition request. - * `REJECTED_REQUEST`: User rejected the coressponding stage transition. + - ``APPROVED_REQUEST``: User approved the corresponding stage + transition. + - ``REJECTED_REQUEST``: User rejected the coressponding stage + transition. - * `SYSTEM_TRANSITION`: For events performed as a side effect, such as + - ``SYSTEM_TRANSITION``: For events performed as a side effect, such as archiving existing model - - versions in a stage. + versions in a stage. CommentActivityAction: type: string x-enum: @@ -10785,20 +14551,22 @@ components: activity or comment. - For activities, valid values are: * `APPROVE_TRANSITION_REQUEST`: - Approve a transition request + For activities, valid values are: + + - ``APPROVE_TRANSITION_REQUEST``: Approve a transition request - * `REJECT_TRANSITION_REQUEST`: Reject a transition request + - ``REJECT_TRANSITION_REQUEST``: Reject a transition request + - ``CANCEL_TRANSITION_REQUEST``: Cancel (delete) a transition request - * `CANCEL_TRANSITION_REQUEST`: Cancel (delete) a transition request + For comments, valid values are: - For comments, valid values are: * `EDIT_COMMENT`: Edit the comment + - ``EDIT_COMMENT``: Edit the comment - * `DELETE_COMMENT`: Delete the comment + - ``DELETE_COMMENT``: Delete the comment ExperimentPermissionLevel: type: string x-enum: @@ -10929,17 +14697,16 @@ components: - PENDING_REGISTRATION - READY description: >- - The status of the model version. Valid values are: * - `PENDING_REGISTRATION`: Request to register + The status of the model version. Valid values are: - a new model version is pending as server performs background tasks. - - * `FAILED_REGISTRATION`: Request to register a new model version has + - ``PENDING_REGISTRATION``: Request to register a new model version is + pending as server + performs background tasks. + - ``FAILED_REGISTRATION``: Request to register a new model version has failed. - - * `READY`: Model version is ready for use. + - ``READY``: Model version is ready for use. OnlineStoreState: type: string x-enum: @@ -11063,13 +14830,16 @@ components: .. note:: Experimental: This entity may change or be removed in a future release without - warning. Email subscription types for registry notifications: - - `ALL_EVENTS`: Subscribed to all + warning. Email subscription types for registry notifications: - events. - `DEFAULT`: Default subscription type. - `SUBSCRIBED`: - Subscribed to notifications. - - `UNSUBSCRIBED`: Not subscribed to notifications. + - ``ALL_EVENTS``: Subscribed to all events. + + - ``DEFAULT``: Default subscription type. + + - ``SUBSCRIBED``: Subscribed to notifications. + + - ``UNSUBSCRIBED``: Not subscribed to notifications. RegistryWebhookEvent: type: string x-enum: @@ -11132,17 +14902,16 @@ components: Enable or disable triggering the webhook, or put the webhook into test mode. The default is - `ACTIVE`: * `ACTIVE`: Webhook is triggered when an associated event - happens. + ``ACTIVE``: - * `DISABLED`: Webhook is not triggered. + - ``ACTIVE``: Webhook is triggered when an associated event happens. + - ``DISABLED``: Webhook is not triggered. - * `TEST_MODE`: Webhook can be triggered through the test endpoint, but + - ``TEST_MODE``: Webhook can be triggered through the test endpoint, but is not triggered on a - - real event. + real event. RunInfoStatus: type: string x-enum: @@ -11152,6 +14921,32 @@ components: - RUNNING - SCHEDULED description: Status of a run. + ScalarDataType: + type: string + x-enum: + - BINARY + - BOOLEAN + - DATE + - DECIMAL + - DOUBLE + - FLOAT + - INTEGER + - LONG + - SHORT + - STRING + - TIMESTAMP + description: >- + Scalar data types for request-time field definitions. Only flat + (non-nested) types are + + supported. + SchemaLocatorFormat: + type: string + x-enum: + - FORMAT_AVRO + - FORMAT_JSON + - FORMAT_PROTOBUF + description: Supported serialization formats for a schema registry schema. Status: type: string x-enum: @@ -11159,17 +14954,58 @@ components: - PENDING_REGISTRATION - READY description: >- - The status of the model version. Valid values are: * - `PENDING_REGISTRATION`: Request to register + The status of the model version. Valid values are: - a new model version is pending as server performs background tasks. - - * `FAILED_REGISTRATION`: Request to register a new model version has + - ``PENDING_REGISTRATION``: Request to register a new model version is + pending as server + performs background tasks. + - ``FAILED_REGISTRATION``: Request to register a new model version has failed. + - ``READY``: Model version is ready for use. + StreamingModeStreamingModeType: + type: string + x-enum: + - STREAMING_MODE_TYPE_MBM + - STREAMING_MODE_TYPE_RTM + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 - * `READY`: Model version is ready for use. + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. UpdateRunStatus: type: string x-enum: @@ -11317,6 +15153,13 @@ components: response: mediaType: application/json openAPIDocKey: '200' + get_batch: + operation: + $ref: '#/paths/~1api~12.0~1mlflow~1logged-models:batchGet/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.models log_params: config: requestBodyTranslate: @@ -11347,6 +15190,7 @@ components: sqlVerbs: select: - $ref: '#/components/x-stackQL-resources/logged_models/methods/get' + - $ref: '#/components/x-stackQL-resources/logged_models/methods/get_batch' insert: - $ref: '#/components/x-stackQL-resources/logged_models/methods/create' update: [] @@ -11800,6 +15644,64 @@ components: - $ref: >- #/components/x-stackQL-resources/feature_kafka_configs/methods/delete replace: [] + feature_engineering_streams: + id: databricks_workspace.ml.feature_engineering_streams + name: feature_engineering_streams + title: Feature Engineering Streams + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1feature-engineering~1streams/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1feature-engineering~1streams/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.streams + delete: + operation: + $ref: '#/paths/~1api~12.0~1feature-engineering~1streams~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1feature-engineering~1streams~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1feature-engineering~1streams~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/feature_engineering_streams/methods/get + - $ref: >- + #/components/x-stackQL-resources/feature_engineering_streams/methods/list + insert: + - $ref: >- + #/components/x-stackQL-resources/feature_engineering_streams/methods/create + update: + - $ref: >- + #/components/x-stackQL-resources/feature_engineering_streams/methods/update + delete: + - $ref: >- + #/components/x-stackQL-resources/feature_engineering_streams/methods/delete + replace: [] feature_store: id: databricks_workspace.ml.feature_store name: feature_store diff --git a/providers/src/databricks_workspace/v00.00.00000/services/oauth2.yaml b/providers/src/databricks_workspace/v00.00.00000/services/oauth2.yaml index cc46b45e..af68ea17 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/oauth2.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/oauth2.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks oauth2 service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.oauth2 servers: - url: https://{deployment_name}.cloud.databricks.com @@ -30,6 +30,9 @@ paths: :param lifetime: str (optional) The lifetime of the secret in seconds. If this parameter is not provided, the secret will have a default lifetime of 730 days (63072000s). + :param scopes: List[str] (optional) + OAuth API scopes to bind to the secret; a minted token cannot exceed them. Empty = unrestricted + (all-apis). E.g. all-apis, sql, iam.groups:read. :returns: :class:`CreateServicePrincipalSecretResponse` parameters: @@ -51,6 +54,14 @@ paths: The lifetime of the secret in seconds. If this parameter is not provided, the secret will have a default lifetime of 730 days (63072000s). + scopes: + type: array + items: + type: string + description: >- + OAuth API scopes to bind to the secret; a minted token + cannot exceed them. Empty = unrestricted (all-apis). E.g. + all-apis, sql, iam.groups:read. responses: '200': description: Success @@ -90,12 +101,12 @@ paths: :param page_size: int (optional) :param page_token: str (optional) - An opaque page token which was the `next_page_token` in the response of the previous request to list - the secrets for this service principal. Provide this token to retrieve the next page of secret - entries. When providing a `page_token`, all other parameters provided to the request must match the - previous request. To list all of the secrets for a service principal, it is necessary to continue - requesting pages of entries until the response contains no `next_page_token`. Note that the number - of entries returned must not be used to determine when the listing is complete. + An opaque page token which was the ``next_page_token`` in the response of the previous request to + list the secrets for this service principal. Provide this token to retrieve the next page of secret + entries. When providing a ``page_token``, all other parameters provided to the request must match + the previous request. To list all of the secrets for a service principal, it is necessary to + continue requesting pages of entries until the response contains no ``next_page_token``. Note that + the number of entries returned must not be used to determine when the listing is complete. :returns: Iterator over :class:`SecretInfo` parameters: @@ -116,15 +127,15 @@ paths: schema: type: string description: >- - An opaque page token which was the `next_page_token` in the response - of the previous request to list the secrets for this service - principal. Provide this token to retrieve the next page of secret - entries. When providing a `page_token`, all other parameters - provided to the request must match the previous request. To list all - of the secrets for a service principal, it is necessary to continue - requesting pages of entries until the response contains no - `next_page_token`. Note that the number of entries returned must not - be used to determine when the listing is complete. + An opaque page token which was the ``next_page_token`` in the + response of the previous request to list the secrets for this + service principal. Provide this token to retrieve the next page of + secret entries. When providing a ``page_token``, all other + parameters provided to the request must match the previous request. + To list all of the secrets for a service principal, it is necessary + to continue requesting pages of entries until the response contains + no ``next_page_token``. Note that the number of entries returned + must not be used to determine when the listing is complete. responses: '200': description: Success @@ -198,9 +209,20 @@ components: description: >- OAuth client-secret generated by the Databricks. If this is a confidential OAuth app client-secret will be generated. + client_secret_expire_time: + type: string + format: date-time + description: >- + The expiration time of the client secret. If not present, the secret + does not expire. integration_id: type: string description: Unique integration id for the custom OAuth app + principal_id: + type: integer + description: >- + The principal id of the service principal associated with this OAuth + app. CreatePublishedAppIntegrationOutput: type: object properties: @@ -219,6 +241,13 @@ components: id: type: string description: ID of the secret + scopes: + type: array + items: + type: string + description: >- + OAuth API scopes bound to this secret. Empty = unrestricted + (all-apis). secret: type: string description: Secret Value @@ -249,9 +278,9 @@ components: type: string description: >- Resource name for the federation policy. Example values include - `accounts//federationPolicies/my-federation-policy` for - Account Federation Policies, and - `accounts//servicePrincipals//federationPolicies/my-federation-policy` + ``accounts//federationPolicies/my-federation-policy`` + for Account Federation Policies, and + ``accounts//servicePrincipals//federationPolicies/my-federation-policy`` for Service Principal Federation Policies. Typically an output parameter, which does not need to be specified in create or update requests. If specified in a request, must match the value in the @@ -294,6 +323,11 @@ components: name: type: string description: The display name of the custom OAuth app + principal_id: + type: integer + description: >- + The principal id of the service principal associated with this OAuth + app. redirect_urls: type: array items: @@ -477,6 +511,13 @@ components: id: type: string description: ID of the secret + scopes: + type: array + items: + type: string + description: >- + OAuth API scopes bound to this secret. Empty = unrestricted + (all-apis). secret_hash: type: string description: Secret Hash diff --git a/providers/src/databricks_workspace/v00.00.00000/services/pipelines.yaml b/providers/src/databricks_workspace/v00.00.00000/services/pipelines.yaml index c2b75592..617179bb 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/pipelines.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/pipelines.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks pipelines service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.pipelines servers: - url: https://{deployment_name}.cloud.databricks.com @@ -15,6 +15,50 @@ servers: description: The Databricks Workspace Deployment Name default: dbc-abcd0123-a1bc paths: + /api/2.0/pipelines/{pipeline_id}/environment/apply: + post: + operationId: pipelines_apply_environment + summary: >- + Applies the current pipeline environment onto the pipeline compute. The + environment applied can be + tags: + - pipelines + description: >- + Applies the current pipeline environment onto the pipeline compute. The + environment applied can be + + used by subsequent dev-mode updates. + + + :param pipeline_id: str + + + :returns: :class:`ApplyEnvironmentRequestResponse` + parameters: + - name: pipeline_id + in: path + required: true + schema: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ApplyEnvironmentRequestResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PipelinesAPI /api/2.0/pipelines/{pipeline_id}/clone: post: operationId: pipelines_clone @@ -40,11 +84,12 @@ paths: :param budget_policy_id: str (optional) Budget policy of this pipeline. :param catalog: str (optional) - A catalog in Unity Catalog to publish data from this pipeline to. If `target` is specified, tables - in this pipeline are published to a `target` schema inside `catalog` (for example, - `catalog`.`target`.`table`). If `target` is not specified, no data is published to Unity Catalog. + A catalog in Unity Catalog to publish data from this pipeline to. If ``target`` is specified, tables + in this pipeline are published to a ``target`` schema inside ``catalog`` (for example, + ``catalog``.``target``.``table``). If ``target`` is not specified, no data is published to Unity + Catalog. :param channel: str (optional) - DLT Release Channel that specifies which version to use. + SDP Release Channel that specifies which version to use. :param clone_mode: :class:`CloneMode` (optional) The type of clone to perform. Currently, only deep copies are supported :param clusters: List[:class:`PipelineCluster`] (optional) @@ -52,7 +97,7 @@ paths: :param configuration: Dict[str,str] (optional) String-String configuration for this pipeline execution. :param continuous: bool (optional) - Whether the pipeline is continuous or triggered. This replaces `trigger`. + Whether the pipeline is continuous or triggered. This replaces ``trigger``. :param deployment: :class:`PipelineDeployment` (optional) Deployment type of this pipeline. :param development: bool (optional) @@ -87,6 +132,8 @@ paths: Whether Photon is enabled for this pipeline. :param restart_window: :class:`RestartWindow` (optional) Restart window of this pipeline. + :param rewind_generation_interval: :class:`PeriodicTrigger` (optional) + Interval at which rewind points are generated during pipeline execution. :param root_path: str (optional) Root path for this pipeline. This is used as the root directory when editing the pipeline in the Databricks user interface and it is added to sys.path when executing Python sources during pipeline @@ -95,17 +142,19 @@ paths: The default schema (database) where tables are read from or published to. :param serverless: bool (optional) Whether serverless compute is enabled for this pipeline. + :param serverless_compute_id: str (optional) + Serverless compute ID specified by the user for serverless pipelines. :param storage: str (optional) DBFS root directory for storing checkpoints and tables. :param tags: Dict[str,str] (optional) A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline. :param target: str (optional) - Target schema (database) to add tables in this pipeline to. Exactly one of `schema` or `target` must - be specified. To publish to Unity Catalog, also specify `catalog`. This legacy field is deprecated - for pipeline creation in favor of the `schema` field. + Target schema (database) to add tables in this pipeline to. Exactly one of ``schema`` or ``target`` + must be specified. To publish to Unity Catalog, also specify ``catalog``. This legacy field is + deprecated for pipeline creation in favor of the ``schema`` field. :param trigger: :class:`PipelineTrigger` (optional) - Which pipeline trigger to use. Deprecated: Use `continuous` instead. + Which pipeline trigger to use. Deprecated: Use ``continuous`` instead. :param usage_policy_id: str (optional) Usage policy of this pipeline. @@ -135,13 +184,14 @@ paths: type: string description: >- A catalog in Unity Catalog to publish data from this - pipeline to. If `target` is specified, tables in this - pipeline are published to a `target` schema inside `catalog` - (for example, `catalog`.`target`.`table`). If `target` is - not specified, no data is published to Unity Catalog. + pipeline to. If ``target`` is specified, tables in this + pipeline are published to a ``target`` schema inside + ``catalog`` (for example, ``catalog``.``target``.``table``). + If ``target`` is not specified, no data is published to + Unity Catalog. channel: type: string - description: DLT Release Channel that specifies which version to use. + description: SDP Release Channel that specifies which version to use. clone_mode: type: string x-enum: @@ -165,8 +215,8 @@ paths: type: integer description: >- The maximum number of workers to which the cluster - can scale up when overloaded. `max_workers` must - be strictly greater than `min_workers`. + can scale up when overloaded. ``max_workers`` must + be strictly greater than ``min_workers``. mode: type: string x-enum: @@ -178,8 +228,8 @@ paths: resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for - `updates` clusters only. The legacy autoscaling - feature is used for `maintenance` clusters. + ``updates`` clusters only. The legacy autoscaling + feature is used for ``maintenance`` clusters. required: - min_workers - max_workers @@ -206,21 +256,21 @@ paths: long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will - be delivered to the destination every `5 mins`. The + be delivered to the destination every ``5 mins``. The destination of driver logs is - `$destination/$clusterId/driver`, while the + ``$destination/$clusterId/driver``, while the destination of executor logs is - `$destination/$clusterId/executor`. + ``$destination/$clusterId/executor``. custom_tags: type: object description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to - `default_tags`. Notes: - Currently, Databricks allows - at most 45 custom tags - Clusters can only reuse cloud - resources if the resources' tags are a subset of the - cluster tags + ``default_tags``. Notes: - Currently, Databricks + allows at most 45 custom tags - Clusters can only + reuse cloud resources if the resources' tags are a + subset of the cluster tags driver_instance_pool_id: type: string description: >- @@ -233,7 +283,7 @@ paths: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will - be set as the same value as `node_type_id` defined + be set as the same value as ``node_type_id`` defined above. enable_local_disk_encryption: type: boolean @@ -252,8 +302,9 @@ paths: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If - `cluster_log_conf` is specified, init script logs are - sent to `//init_scripts`. + ``cluster_log_conf`` is specified, init script logs + are sent to + ``//init_scripts``. instance_pool_id: type: string description: >- @@ -263,10 +314,10 @@ paths: type: string description: >- A label for the cluster specification, either - `default` to configure the default cluster, or - `maintenance` to configure the maintenance cluster. + ``default`` to configure the default cluster, or + ``maintenance`` to configure the maintenance cluster. This field is optional. The default value is - `default`. + ``default``. node_type_id: type: string description: >- @@ -281,16 +332,17 @@ paths: type: integer description: >- Number of worker nodes that this cluster should have. - A cluster has one Spark Driver and `num_workers` - Executors for a total of `num_workers` + 1 Spark + A cluster has one Spark Driver and ``num_workers`` + Executors for a total of ``num_workers`` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase - from 5 to 10 as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually + increase from 5 to 10 as the new nodes are + provisioned. policy_id: type: string description: >- @@ -308,17 +360,17 @@ paths: An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as - is (i.e., `export X='Y'`) while launching the driver + is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional set of - `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them - to `$SPARK_DAEMON_JAVA_OPTS` as shown in the example - below. This ensures that all default databricks - managed environmental variables are included as well. - Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending + them to ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the + example below. This ensures that all default + databricks managed environmental variables are + included as well. Example Spark environment variables: + ``{"SPARK_WORKER_MEMORY": "28000m", + "SPARK_LOCAL_DIRS": "/local_disk0"}`` or + ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` ssh_public_keys: type: array items: @@ -326,8 +378,9 @@ paths: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private - keys can be used to login with the user name `ubuntu` - on port `2200`. Up to 10 keys can be specified. + keys can be used to login with the user name + ``ubuntu`` on port ``2200``. Up to 10 keys can be + specified. description: Cluster settings for this pipeline deployment. configuration: type: object @@ -336,7 +389,7 @@ paths: type: boolean description: >- Whether the pipeline is continuous or triggered. This - replaces `trigger`. + replaces ``trigger``. deployment: type: object properties: @@ -345,15 +398,30 @@ paths: x-enum: - BUNDLE description: >- - The deployment method that manages the pipeline: - - BUNDLE: The pipeline is managed by a + The deployment method that manages the pipeline: + - Databricks Asset Bundle. + - BUNDLE: The pipeline is managed by a Databricks Asset + Bundle. + deployment_id: + type: string + description: >- + ID of the deployment that manages this pipeline. Only + set when ``kind`` is ``BUNDLE``. Used to look up + deployment metadata from the Deployment Metadata + service. metadata_file_path: type: string description: >- The path to the file containing metadata about the deployment. + version_id: + type: string + description: >- + ID of the version of the deployment that produced this + pipeline. Only set when ``kind`` is ``BUNDLE``. + Identifies a specific snapshot of the deployment in the + Deployment Metadata service. required: - kind description: Deployment type of this pipeline. @@ -393,7 +461,7 @@ paths: versions and their included packages, see https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ The value should be a string representing the - environment version number, for example: `"4"`. + environment version number, for example: ``"4"``. description: >- Environment specification for this pipeline used to install dependencies. @@ -571,7 +639,7 @@ paths: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. - The `source_catalog` fields in objects of + The ``source_catalog`` fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from. ingestion_gateway_id: @@ -637,6 +705,38 @@ paths: required: - enabled description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. exclude_columns: type: array items: @@ -649,7 +749,7 @@ paths: all other columns including future ones will be automatically included for ingestion. This field in mutually - exclusive with `include_columns`. + exclusive with ``include_columns``. include_columns: type: array items: @@ -663,7 +763,7 @@ paths: all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with - `exclude_columns`. + ``exclude_columns``. primary_keys: type: array items: @@ -689,9 +789,9 @@ paths: is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. deletion_condition: type: string description: >- @@ -701,7 +801,7 @@ paths: as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, + ``hard_deletion_sync_interval_in_seconds``, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds @@ -761,6 +861,36 @@ paths: Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. workday_report_parameters: type: object properties: @@ -803,7 +933,7 @@ paths: (Optional) Additional custom parameters for Workday Report This field is deprecated and should not be used. Use - `parameters` instead. + ``parameters`` instead. description: >- (Optional) Additional custom parameters for Workday Report @@ -832,1660 +962,3554 @@ paths: tables are created in this destination schema. The pipeline fails If a table with the same name already exists. - source_catalog: - type: string + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string description: >- - The source catalog name. Might be optional - depending on the type of source. - table_configuration: + Deprecated: Use + connector_options.confluence_options instead. + connector_options: type: object properties: - auto_full_refresh_policy: + akamai_options: type: object properties: - enabled: - type: boolean + config_ids: + type: array + items: + type: integer description: >- - (Required, Mutable) Whether to enable - auto full refresh or not. - min_interval_hours: - type: integer + (Required) List of Akamai configuration + IDs to ingest data from. A config ID is + a unique identifier for a security + configuration used to generate WAF logs. + description: Akamai specific options for ingestion + api_source_connector_options: + type: object + properties: + options: + type: object description: >- - (Optional, Mutable) Specify the minimum - interval in hours between the timestamp - at which a table was last full refreshed - and the current timestamp for triggering - auto full If unspecified and - autoFullRefresh is enabled then by - default min_interval_hours is 24 hours. - required: - - enabled - description: Policy for auto full refresh. - exclude_columns: - type: array - items: - type: string + Arbitrary key-value configuration + options for the API Source connector. description: >- - A list of column names to be excluded for - the ingestion. When not specified, - include_columns fully controls what - columns to be ingested. When specified, - all other columns including future ones - will be automatically included for - ingestion. This field in mutually - exclusive with `include_columns`. - include_columns: - type: array - items: - type: string + Connector-specific options for API Source + connectors. + bing_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + report_type: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: >- + (Required) Microsoft Advertising report + type (e.g. CampaignPerformance, + KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the + report, given as Microsoft Advertising + column enum values for the selected + report type (e.g. AccountId, + CampaignName, TimePeriod, Impressions, + Clicks, Spend). Required key columns are + always included. Available columns are + report-type-specific. For the full list + supported by a given report type, see + its column value set (substitute the + report name, e.g. campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: >- + (Optional) Time granularity for report + aggregation. Defaults to DAILY. + required: + - report_type + - columns + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Bing Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + during incremental sync to capture + late-arriving conversion and attribution + data. If not specified, defaults to 7 + days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. If not + specified, defaults to 2 years before + today. description: >- - A list of column names to be included for - the ingestion. When not specified, all - columns except ones in exclude_columns - will be included. Future columns will be - automatically included. When specified, - all other future columns will be - automatically excluded from ingestion. - This field in mutually exclusive with - `exclude_columns`. - primary_keys: - type: array - items: - type: string + Bing Ads specific options for ingestion + pipelines. + community_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the community connector. description: >- - The primary key of the table used to apply - changes. - query_based_connector_config: + Options for community connectors with + arbitrary configuration. + confluence_options: type: object properties: - cursor_columns: + include_confluence_spaces: type: array items: type: string description: >- - The names of the monotonically - increasing columns in the source table - that are used to enable the table to be - read and ingested incrementally through - structured streaming. The columns are - allowed to have repeated values but have - to be non-decreasing. If the source data - is merged into the destination (e.g., - using SCD Type 1 or Type 2), these - columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. - deletion_condition: + (Optional) Spaces to filter Confluence + data on + description: Confluence specific options for ingestion + gdrive_options: + type: object + properties: + entity_type: type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION description: >- - Specifies a SQL WHERE condition that - specifies that the source row has been - deleted. This is sometimes referred to - as "soft-deletes". For example: - "Operation = 'DELETE'" or "is_deleted = - true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, - one for soft-deletes and the other for - hard-deletes. See also the - hard_deletion_sync_min_interval_in_seconds - field for handling of "hard deletes" - where the source rows are physically - removed from the table. - hard_deletion_sync_min_interval_in_seconds: - type: integer + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + scope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE description: >- - Specifies the minimum interval (in - seconds) between snapshots on primary - keys for detecting and synchronizing - hard deletions—i.e., rows that have been - physically removed from the source - table. This interval acts as a lower - bound. If ingestion runs less frequently - than this value, hard deletion - synchronization will align with the - actual ingestion frequency instead of - happening more often. If not set, hard - deletion synchronization via snapshots - is disabled. This field is mutable and - can be updated without triggering a full - snapshot. - description: >- - Configurations that are only applicable - for query-based ingestion connectors. - row_filter: - type: string - description: >- - (Optional, Immutable) The row filter - condition to be applied to the table. It - must not contain the WHERE keyword, only - the actual filter condition. It must be in - DBSQL format. - salesforce_include_formula_fields: - type: boolean - description: >- - If true, formula fields defined in the - table are included in the ingestion. This - setting is only valid for the Salesforce - connector - scd_type: - type: string - x-enum: - - APPEND_ONLY - - SCD_TYPE_1 - - SCD_TYPE_2 - description: The SCD type to use to ingest the table. - sequence_by: - type: array - items: - type: string - description: >- - The column names specifying the logical - order of events in the source data. Spark - Declarative Pipelines uses this sequencing - to handle change events that arrive out of - order. - workday_report_parameters: + The scope of the Google Drive ingestion. + **ONLY ONE** of scope or url must be + specified. If scope is specified, we + will ingest files from the scope. If url + is specified, we will ingest files from + the url. + url: + type: string + description: Google Drive URL. + github_options: type: object properties: - incremental: - type: boolean - parameters: - type: object - description: >- - Parameters for the Workday report. Each - key represents the parameter name (e.g., - "start_date", "end_date"), and the - corresponding value is a SQL-like - expression used to compute the parameter - value at runtime. Example: { - "start_date": "{ - coalesce(current_offset(), - date(\"2025-02-01\")) }", "end_date": "{ - current_date() - INTERVAL 1 DAY }" } - report_parameters: + repository_branch_selections: type: array items: type: object properties: - key: - type: string - value: + repository_id: type: string description: >- - Value for the report parameter. Possible - values it can take are these sql - functions: 1. coalesce(current_offset(), - date("YYYY-MM-DD")) -> if - current_offset() is null, then the - passed date, else current_offset() 2. - current_date() 3. - date_sub(current_date(), x) -> subtract - x (some non-negative integer) days from - current date + (Required) GitHub repository id + (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this + repository, e.g. ["main", "dev"]. + required: + - repository_id + description: >- + A repository and the branches to ingest + for it. description: >- - (Optional) Additional custom parameters - for Workday Report This field is - deprecated and should not be used. Use - `parameters` instead. - description: >- - (Optional) Additional custom parameters - for Workday Report - description: >- - Configuration settings to control the - ingestion of tables. These settings are - applied to all tables in this schema and - override the table_configuration defined in - the IngestionPipelineDefinition object. - required: - - source_schema - - destination_catalog - - destination_schema - description: Select all tables from a specific source schema. - table: - type: object - properties: - source_table: - type: string - destination_catalog: - type: string - description: Required. Destination catalog to store table. - destination_schema: - type: string - description: Required. Destination schema to store table. - destination_table: - type: string - description: >- - Optional. Destination table name. The pipeline - fails if a table with that name already - exists. If not set, the source table name is - used. - source_catalog: - type: string - description: >- - Source catalog name. Might be optional - depending on the type of source. - source_schema: - type: string - description: >- - Schema name in the source database. Might be - optional depending on the type of source. - table_configuration: - type: object - properties: - auto_full_refresh_policy: + (Optional) Branches to ingest per + repository. A repository not listed here + is ingested on its default branch. + description: GitHub-specific ingestion options. + google_ads_options: type: object properties: - enabled: - type: boolean + manager_account_id: + type: string description: >- - (Required, Mutable) Whether to enable - auto full refresh or not. - min_interval_hours: + (Optional at this level) Manager Account + ID (also called MCC Account ID) used to + list and access customer accounts under + this manager account. Overrides + GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query + (e.g. "ad_group_ad", "keyword_view", + "search_term_view"). Must be a resource + that has metrics. Values are validated + against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. + "metrics.clicks", + "metrics.cost_micros"). Multiple values + are joined into the GAQL SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in + fully-qualified GAQL form (e.g. + "ad_group_ad.ad.id", + "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select + (e.g. "segments.date", + "segments.device"). Must include at + least one of segments.date, + segments.week, or segments.month — that + segment is used as the incremental + cursor for the table. + required: + - resource + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Google Ads custom report: + the connector synthesizes a GAQL query + from the resource, fields, segments, and + metrics specified here. When unset, the + table must match one of the connector's + prebuilt sources. + lookback_window_days: type: integer description: >- - (Optional, Mutable) Specify the minimum - interval in hours between the timestamp - at which a table was last full refreshed - and the current timestamp for triggering - auto full If unspecified and - autoFullRefresh is enabled then by - default min_interval_hours is 24 hours. + (Optional) Number of days to look back + for report tables to capture + late-arriving data. If not specified, + defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years of + historical data. required: - - enabled - description: Policy for auto full refresh. - exclude_columns: - type: array - items: - type: string - description: >- - A list of column names to be excluded for - the ingestion. When not specified, - include_columns fully controls what - columns to be ingested. When specified, - all other columns including future ones - will be automatically included for - ingestion. This field in mutually - exclusive with `include_columns`. - include_columns: - type: array - items: - type: string - description: >- - A list of column names to be included for - the ingestion. When not specified, all - columns except ones in exclude_columns - will be included. Future columns will be - automatically included. When specified, - all other future columns will be - automatically excluded from ingestion. - This field in mutually exclusive with - `exclude_columns`. - primary_keys: - type: array - items: - type: string + - manager_account_id description: >- - The primary key of the table used to apply - changes. - query_based_connector_config: + Google Ads specific options for ingestion + (object-level). When set, these values + override the + corresponding fields in GoogleAdsConfig (source_configurations). + jira_options: type: object properties: - cursor_columns: + include_jira_spaces: type: array items: type: string description: >- - The names of the monotonically - increasing columns in the source table - that are used to enable the table to be - read and ingested incrementally through - structured streaming. The columns are - allowed to have repeated values but have - to be non-decreasing. If the source data - is merged into the destination (e.g., - using SCD Type 1 or Type 2), these - columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. - deletion_condition: + (Optional) Projects to filter Jira data + on + use_audit_logs: + type: boolean + description: >- + (Optional) Whether the pipeline reads + Jira's audit log to detect changes for + the tables it ingests. Reading the audit + log requires a paid Jira plan and + appropriate admin permissions. For + example, the issues table uses the audit + log to detect and propagate deletes: + when true, rows deleted in Jira are + removed from the destination table; when + false, the destination retains all + previously ingested rows and only + reflects inserts and updates. If not + specified, defaults to false. + description: Jira specific options for ingestion + kafka_options: + type: object + properties: + client_config: + type: object + key_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + key. If not specified, the key is left + as raw bytes. + max_offsets_per_trigger: + type: integer + description: >- + Internal option to control the maximum + number of offsets to process per + trigger. + max_partitions: type: string description: >- - Specifies a SQL WHERE condition that - specifies that the source row has been - deleted. This is sometimes referred to - as "soft-deletes". For example: - "Operation = 'DELETE'" or "is_deleted = - true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, - one for soft-deletes and the other for - hard-deletes. See also the - hard_deletion_sync_min_interval_in_seconds - field for handling of "hard deletes" - where the source rows are physically - removed from the table. - hard_deletion_sync_min_interval_in_seconds: + (Optional) Controls how Kafka offset + ranges are bin-packed into Spark input + partitions. Defaults to "auto" when + unset, which lets the engine choose the + input partition count from measured + throughput. Set to "none" to disable + bin-packing, so each offset range maps + to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when + no checkpoint exists. Valid values: + "latest" and "earliest". Defaults to + "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to + matching topics. Only one of topics or + topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of + topics or topic_pattern must be + specified. + value_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + value. If not specified, the value is + left as raw bytes. + linkedin_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + finder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + (Required) adAnalytics finder. See + LinkedInAdsFinder. + entity_granularity: + type: array + items: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the + report. Open vocabulary (not an enum): + the valid set is large (~100) and + evolves with the LinkedIn adAnalytics + API, so values are passed through + verbatim. If empty, a pivot-safe default + core set is ingested: impressions, + clicks, costInLocalCurrency, + externalWebsiteConversions (valid for + every pivot). Ignored for + attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics + struct). + time_granularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + (Optional) Time aggregation. Defaults to + DAILY when unspecified. Used by + analytics/statistics; ignored for + attributedRevenueMetrics. + required: + - finder + description: >- + (Optional) Custom report definition. + Only valid on a table object. When set, + the table is synthesized from + /rest/adAnalytics using the finder, + pivots, time granularity and metrics + here. When unset, the table must match + one of the connector's prebuilt sources. + lookback_window_days: type: integer description: >- - Specifies the minimum interval (in - seconds) between snapshots on primary - keys for detecting and synchronizing - hard deletions—i.e., rows that have been - physically removed from the source - table. This interval acts as a lower - bound. If ingestion runs less frequently - than this value, hard deletion - synchronization will align with the - actual ingestion frequency instead of - happening more often. If not set, hard - deletion synchronization via snapshots - is disabled. This field is mutable and - can be updated without triggering a full - snapshot. - description: >- - Configurations that are only applicable - for query-based ingestion connectors. - row_filter: - type: string - description: >- - (Optional, Immutable) The row filter - condition to be applied to the table. It - must not contain the WHERE keyword, only - the actual filter condition. It must be in - DBSQL format. - salesforce_include_formula_fields: - type: boolean - description: >- - If true, formula fields defined in the - table are included in the ingestion. This - setting is only valid for the Salesforce - connector - scd_type: - type: string - x-enum: - - APPEND_ONLY - - SCD_TYPE_1 - - SCD_TYPE_2 - description: The SCD type to use to ingest the table. - sequence_by: - type: array - items: - type: string + (Optional) Days to look back during + incremental sync for late-arriving data. + If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables, YYYY-MM-DD. + Earliest date from which to sync + historical data; overrides the default + when set. For finder + attributedRevenueMetrics, this must be + between 30 and 366 days before today. If + not specified, defaults to 1 year of + history. description: >- - The column names specifying the logical - order of events in the source data. Spark - Declarative Pipelines uses this sequencing - to handle change events that arrive out of - order. - workday_report_parameters: + LinkedIn Ads specific options for + ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + marketo_options: type: object properties: - incremental: - type: boolean - parameters: - type: object + sync_start_date: + type: string description: >- - Parameters for the Workday report. Each - key represents the parameter name (e.g., - "start_date", "end_date"), and the - corresponding value is a SQL-like - expression used to compute the parameter - value at runtime. Example: { - "start_date": "{ - coalesce(current_offset(), - date(\"2025-02-01\")) }", "end_date": "{ - current_date() - INTERVAL 1 DAY }" } - report_parameters: + (Optional) Start date for the initial + sync in YYYY-MM-DD format. This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + description: Marketo specific options for ingestion + meta_ads_options: + type: object + properties: + action_attribution_windows: type: array items: - type: object - properties: - key: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) + Action attribution windows for insights + reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) + Timing used to report action statistics + (impression, conversion, mixed, or + lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit + data during sync to capture updated + conversion data from the API, shared by + prebuilt and custom reports. + custom_report_options: + type: object + properties: + action_attribution_windows: + type: array + items: type: string - value: + description: >- + (Optional) Action attribution windows + for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: type: string - description: >- - Value for the report parameter. Possible - values it can take are these sql - functions: 1. coalesce(current_offset(), - date("YYYY-MM-DD")) -> if - current_offset() is null, then the - passed date, else current_offset() 2. - current_date() 3. - date_sub(current_date(), x) -> subtract - x (some non-negative integer) days from - current date + description: >- + (Optional) Action breakdowns to + configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action + statistics (impression, conversion, + mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdowns to configure for + data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull + (account, ad, adset, campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to + aggregate statistics (all_days, monthly + or number of days) description: >- - (Optional) Additional custom parameters - for Workday Report This field is - deprecated and should not be used. Use - `parameters` instead. + (Optional) Per-table custom report + definition. When set, defines the shape + of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). + Supersedes the deprecated flat + report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.level) Granularity + of data to pull (account, ad, adset, + campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd + format (e.g. 2025-01-15). Data added + after this date will be ingested, shared + by prebuilt and custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.time_increment) + Value in string by which to aggregate + statistics (can take all_days, monthly + or number of days) description: >- - (Optional) Additional custom parameters - for Workday Report - description: >- - Configuration settings to control the - ingestion of tables. These settings override - the table_configuration defined in the - IngestionPipelineDefinition object and the - SchemaSpec. - required: - - source_table - - destination_catalog - - destination_schema - description: Select a specific source table. - description: >- - Required. Settings specifying tables to replicate and - the destination for the replicated tables. - source_configurations: - type: array - items: - type: object - properties: - catalog: - type: object - properties: - postgres: - type: object - properties: - slot_config: + Meta Marketing (Meta Ads) specific options + for ingestion + outlook_options: type: object properties: - publication_name: + attachment_mode: type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY description: >- - The name of the publication to use for - the Postgres source - slot_name: + (Optional) Controls which attachments to + ingest. If not specified, defaults to + ALL. + body_format: type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN description: >- - The name of the logical replication slot - to use for the Postgres source + (Optional) Defines how the body_content + column is populated. TEXT_HTML: + Preserves full formatting, links, and + styling. TEXT_PLAIN: Converts body to + plain text. Recommended for AI/RAG + pipelines to reduce token usage and + noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to + include in the sync. If not specified, + all folders will be synced. Examples: + Inbox, Sent Items, Custom_Folder Filter + semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync + (e.g. mailbox email addresses or + identifiers). If not specified, all + accessible mailboxes are ingested. + Filter semantics: OR between different + mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender + address. Uses exact email match. + Examples: user@vendor.com, + alerts@system.io, noreply@company.com If + not specified, emails from all senders + will be synced. Filter semantics: OR + between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject + line. Values ending with "*" use prefix + match (subject starts with the part + before "*"); otherwise substring match + (subject contains the value). Examples: + "Invoice" (substring), "Re:*" (prefix), + "Support Ticket", "URGENT*" If not + specified, emails with all subjects will + be synced. Filter semantics: OR between + different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. Format: + YYYY-MM-DD (e.g., 2024-01-01) This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + subject_filter: + type: array + items: + type: string + description: >- + Deprecated. Use include_subjects + instead. + description: Outlook specific options for ingestion + rabbitmq_options: + type: object + properties: + queue: + type: string + description: >- + (Required) RabbitMQ queue name to + consume from. + required: + - queue description: >- - Optional. The Postgres slot configuration - to use for logical replication + RabbitMQ specific options for ingestion. + Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + reddit_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group + report data by. Examples: CAMPAIGN_ID, + DATE, COUNTRY, REGION, AD_ID. Must + include at least one time dimension + (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the + report (maps to the Reddit Ads API + ``fields`` parameter). Examples: + IMPRESSIONS, CLICKS, SPEND, CPC, CTR. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Reddit Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. If not + specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years + ago. + description: Reddit Ads specific options for ingestion + sharepoint_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + description: >- + (Optional) File ingestion options for + processing files. + url: + type: string + description: Required. The SharePoint URL. + smartsheet_options: + type: object + properties: + enforce_schema: + type: boolean + description: >- + (Optional) When true, maps each column + to its Smartsheet-declared type + (Text/Number/Date/ Checkbox/etc.). Cells + that do not conform to the declared type + are set to NULL. When false, all columns + land as STRING. Use false for sheets + with irregular data or columns that + frequently violate their own declared + type. If not specified, defaults to + true. + description: Smartsheet specific options for ingestion + tiktok_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + (Optional) Data level for the report. If + not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the + report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", + "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the + report (e.g. "spend", "impressions", + "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime + metrics (all-time aggregated data). When + true, the report returns all-time data. + If not specified, defaults to false. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + (Optional) Report type for the TikTok + Ads API. If not specified, defaults to + BASIC. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined TikTok Ads custom report: + the connector synthesizes a report + request from the dimensions, metrics, + report type, and data level specified + here. Supersedes the deprecated + top-level + dimensions/metrics/report_type/ + data_level/query_lifetime fields above. + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + Deprecated. Use + custom_report_options.data_level + instead. + dimensions: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.dimensions + instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. + metrics: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: >- + Deprecated. Use + custom_report_options.query_lifetime + instead. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + Deprecated. Use + custom_report_options.report_type + instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + description: TikTok Ads specific options for ingestion + zendesk_support_options: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD + format for the initial sync. This + determines the earliest date from which + to sync historical data. + description: >- + Zendesk Support specific options for + ingestion + description: (Optional) Source Specific Connector Options + fanout_options: + type: object + properties: + fanout_by: + type: string + description: >- + Column path or SQL expression whose value + determines the destination table. Supports + dotted paths (e.g. "value.event_name") and + expressions (e.g. + "value:event_name::string"). + transforms: + type: array + items: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + Specifies how to transform binary data + into structured data. + description: >- + Optional transforms applied to each + route's DataFrame before writing to the + destination table. description: >- - Postgres-specific catalog-level configuration - parameters + Fanout options for multi-table routing from + streaming sources. When set, records are + routed to destination tables based on a + per-record routing key. The key value becomes + the table name: + {destination_catalog}.{destination_schema}.{key_value}. + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use connector_options.jira_options + instead. source_catalog: type: string - description: Source catalog name - description: >- - SourceCatalogConfig contains catalog-level custom - configuration parameters for each source - description: Top-level source configurations - source_type: - type: string - x-enum: - - BIGQUERY - - DYNAMICS365 - - FOREIGN_CATALOG - - GA4_RAW_DATA - - MANAGED_POSTGRESQL - - MYSQL - - NETSUITE - - ORACLE - - POSTGRESQL - - SALESFORCE - - SERVICENOW - - SHAREPOINT - - SQLSERVER - - TERADATA - - WORKDAY_RAAS - description: >- - The type of the foreign source. The source type will be - inferred from the source connection or ingestion - gateway. This field is output only and will be ignored - if provided. - table_configuration: - type: object - properties: - auto_full_refresh_policy: - type: object - properties: - enabled: - type: boolean - description: >- - (Required, Mutable) Whether to enable auto full - refresh or not. - min_interval_hours: - type: integer - description: >- - (Optional, Mutable) Specify the minimum interval - in hours between the timestamp at which a table - was last full refreshed and the current - timestamp for triggering auto full If - unspecified and autoFullRefresh is enabled then - by default min_interval_hours is 24 hours. - required: - - enabled - description: Policy for auto full refresh. - exclude_columns: - type: array - items: - type: string - description: >- - A list of column names to be excluded for the - ingestion. When not specified, include_columns fully - controls what columns to be ingested. When - specified, all other columns including future ones - will be automatically included for ingestion. This - field in mutually exclusive with `include_columns`. - include_columns: - type: array - items: - type: string - description: >- - A list of column names to be included for the - ingestion. When not specified, all columns except - ones in exclude_columns will be included. Future - columns will be automatically included. When - specified, all other future columns will be - automatically excluded from ingestion. This field in - mutually exclusive with `exclude_columns`. - primary_keys: - type: array - items: - type: string - description: The primary key of the table used to apply changes. - query_based_connector_config: - type: object - properties: - cursor_columns: - type: array - items: - type: string - description: >- - The names of the monotonically increasing - columns in the source table that are used to - enable the table to be read and ingested - incrementally through structured streaming. The - columns are allowed to have repeated values but - have to be non-decreasing. If the source data is - merged into the destination (e.g., using SCD - Type 1 or Type 2), these columns will implicitly - define the `sequence_by` behavior. You can still - explicitly set `sequence_by` to override this - default. - deletion_condition: - type: string - description: >- - Specifies a SQL WHERE condition that specifies - that the source row has been deleted. This is - sometimes referred to as "soft-deletes". For - example: "Operation = 'DELETE'" or "is_deleted = - true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, one - for soft-deletes and the other for hard-deletes. - See also the - hard_deletion_sync_min_interval_in_seconds field - for handling of "hard deletes" where the source - rows are physically removed from the table. - hard_deletion_sync_min_interval_in_seconds: - type: integer - description: >- - Specifies the minimum interval (in seconds) - between snapshots on primary keys for detecting - and synchronizing hard deletions—i.e., rows that - have been physically removed from the source - table. This interval acts as a lower bound. If - ingestion runs less frequently than this value, - hard deletion synchronization will align with - the actual ingestion frequency instead of - happening more often. If not set, hard deletion - synchronization via snapshots is disabled. This - field is mutable and can be updated without - triggering a full snapshot. - description: >- - Configurations that are only applicable for - query-based ingestion connectors. - row_filter: - type: string - description: >- - (Optional, Immutable) The row filter condition to be - applied to the table. It must not contain the WHERE - keyword, only the actual filter condition. It must - be in DBSQL format. - salesforce_include_formula_fields: - type: boolean - description: >- - If true, formula fields defined in the table are - included in the ingestion. This setting is only - valid for the Salesforce connector - scd_type: - type: string - x-enum: - - APPEND_ONLY - - SCD_TYPE_1 - - SCD_TYPE_2 - description: The SCD type to use to ingest the table. - sequence_by: - type: array - items: - type: string - description: >- - The column names specifying the logical order of - events in the source data. Spark Declarative - Pipelines uses this sequencing to handle change - events that arrive out of order. - workday_report_parameters: - type: object - properties: - incremental: - type: boolean - parameters: - type: object - description: >- - Parameters for the Workday report. Each key - represents the parameter name (e.g., - "start_date", "end_date"), and the corresponding - value is a SQL-like expression used to compute - the parameter value at runtime. Example: { - "start_date": "{ coalesce(current_offset(), - date(\"2025-02-01\")) }", "end_date": "{ - current_date() - INTERVAL 1 DAY }" } - report_parameters: - type: array - items: + description: >- + The source catalog name. Might be optional + depending on the type of source. + table_configuration: type: object properties: - key: - type: string - value: + auto_full_refresh_policy: + type: object + properties: + enabled: + type: boolean + description: >- + (Required, Mutable) Whether to enable + auto full refresh or not. + min_interval_hours: + type: integer + description: >- + (Optional, Mutable) Specify the minimum + interval in hours between the timestamp + at which a table was last full refreshed + and the current timestamp for triggering + auto full If unspecified and + autoFullRefresh is enabled then by + default min_interval_hours is 24 hours. + required: + - enabled + description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + exclude_columns: + type: array + items: + type: string + description: >- + A list of column names to be excluded for + the ingestion. When not specified, + include_columns fully controls what + columns to be ingested. When specified, + all other columns including future ones + will be automatically included for + ingestion. This field in mutually + exclusive with ``include_columns``. + include_columns: + type: array + items: + type: string + description: >- + A list of column names to be included for + the ingestion. When not specified, all + columns except ones in exclude_columns + will be included. Future columns will be + automatically included. When specified, + all other future columns will be + automatically excluded from ingestion. + This field in mutually exclusive with + ``exclude_columns``. + primary_keys: + type: array + items: + type: string + description: >- + The primary key of the table used to apply + changes. + query_based_connector_config: + type: object + properties: + cursor_columns: + type: array + items: + type: string + description: >- + The names of the monotonically + increasing columns in the source table + that are used to enable the table to be + read and ingested incrementally through + structured streaming. The columns are + allowed to have repeated values but have + to be non-decreasing. If the source data + is merged into the destination (e.g., + using SCD Type 1 or Type 2), these + columns will implicitly define the + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. + deletion_condition: + type: string + description: >- + Specifies a SQL WHERE condition that + specifies that the source row has been + deleted. This is sometimes referred to + as "soft-deletes". For example: + "Operation = 'DELETE'" or "is_deleted = + true". This field is orthogonal to + ``hard_deletion_sync_interval_in_seconds``, + one for soft-deletes and the other for + hard-deletes. See also the + hard_deletion_sync_min_interval_in_seconds + field for handling of "hard deletes" + where the source rows are physically + removed from the table. + hard_deletion_sync_min_interval_in_seconds: + type: integer + description: >- + Specifies the minimum interval (in + seconds) between snapshots on primary + keys for detecting and synchronizing + hard deletions—i.e., rows that have been + physically removed from the source + table. This interval acts as a lower + bound. If ingestion runs less frequently + than this value, hard deletion + synchronization will align with the + actual ingestion frequency instead of + happening more often. If not set, hard + deletion synchronization via snapshots + is disabled. This field is mutable and + can be updated without triggering a full + snapshot. + description: >- + Configurations that are only applicable + for query-based ingestion connectors. + row_filter: type: string description: >- - Value for the report parameter. Possible - values it can take are these sql - functions: 1. coalesce(current_offset(), - date("YYYY-MM-DD")) -> if current_offset() - is null, then the passed date, else - current_offset() 2. current_date() 3. - date_sub(current_date(), x) -> subtract x - (some non-negative integer) days from - current date - description: >- - (Optional) Additional custom parameters for - Workday Report This field is deprecated and - should not be used. Use `parameters` instead. - description: >- - (Optional) Additional custom parameters for Workday - Report - description: >- - Configuration settings to control the ingestion of - tables. These settings are applied to all tables in the - pipeline. - description: >- - The configuration for a managed ingestion pipeline. These - settings cannot be used with the 'libraries', 'schema', - 'target', or 'catalog' settings. - libraries: - type: array - items: - type: object - properties: - file: - type: object - properties: - path: - type: string - glob: - type: object - properties: - include: - type: string - description: >- - The unified field to include source codes. Each entry - can be a notebook path, a file path, or a folder path - that ends `/**`. This field cannot be used together - with `notebook` or `file`. - jar: - type: string - description: >- - URI of the jar to be installed. Currently only DBFS is - supported. - maven: - type: string - description: Specification of a maven library to be installed. - notebook: - type: object - properties: - path: - type: string - description: >- - The path to a notebook that defines a pipeline and is - stored in the Databricks workspace. - whl: - type: string - description: URI of the whl to be installed. - description: Libraries or code needed by this deployment. - name: - type: string - description: Friendly identifier for this pipeline. - notifications: - type: array - items: - type: object - properties: - alerts: - type: array - items: - type: string - email_recipients: - type: array - items: - type: string - description: >- - A list of email addresses notified when a configured - alert is triggered. - description: List of notification settings for this pipeline. - photon: - type: boolean - description: Whether Photon is enabled for this pipeline. - restart_window: - type: object - properties: - start_hour: - type: integer - days_of_week: - type: array - items: - type: string - x-enum: - - FRIDAY - - MONDAY - - SATURDAY - - SUNDAY - - THURSDAY - - TUESDAY - - WEDNESDAY - description: >- - Days of week in which the window is allowed to happen. - If not specified all days of the week + (Optional, Immutable) The row filter + condition to be applied to the table. It + must not contain the WHERE keyword, only + the actual filter condition. It must be in + DBSQL format. + salesforce_include_formula_fields: + type: boolean + description: >- + If true, formula fields defined in the + table are included in the ingestion. This + setting is only valid for the Salesforce + connector + scd_type: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: The SCD type to use to ingest the table. + sequence_by: + type: array + items: + type: string + description: >- + The column names specifying the logical + order of events in the source data. Spark + Declarative Pipelines uses this sequencing + to handle change events that arrive out of + order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. + workday_report_parameters: + type: object + properties: + incremental: + type: boolean + parameters: + type: object + description: >- + Parameters for the Workday report. Each + key represents the parameter name (e.g., + "start_date", "end_date"), and the + corresponding value is a SQL-like + expression used to compute the parameter + value at runtime. Example: { + "start_date": "{ + coalesce(current_offset(), + date(\"2025-02-01\")) }", "end_date": "{ + current_date() - INTERVAL 1 DAY }" } + report_parameters: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: >- + Value for the report parameter. Possible + values it can take are these sql + functions: 1. coalesce(current_offset(), + date("YYYY-MM-DD")) -> if + current_offset() is null, then the + passed date, else current_offset() 2. + current_date() 3. + date_sub(current_date(), x) -> subtract + x (some non-negative integer) days from + current date + description: >- + (Optional) Additional custom parameters + for Workday Report This field is + deprecated and should not be used. Use + ``parameters`` instead. + description: >- + (Optional) Additional custom parameters + for Workday Report + description: >- + Configuration settings to control the + ingestion of tables. These settings are + applied to all tables in this schema and + override the table_configuration defined in + the IngestionPipelineDefinition object. + required: + - source_schema + - destination_catalog + - destination_schema + description: Select all tables from a specific source schema. + table: + type: object + properties: + source_table: + type: string + destination_catalog: + type: string + description: Required. Destination catalog to store table. + destination_schema: + type: string + description: Required. Destination schema to store table. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use + connector_options.confluence_options instead. + connector_options: + type: object + properties: + akamai_options: + type: object + properties: + config_ids: + type: array + items: + type: integer + description: >- + (Required) List of Akamai configuration + IDs to ingest data from. A config ID is + a unique identifier for a security + configuration used to generate WAF logs. + description: Akamai specific options for ingestion + api_source_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the API Source connector. + description: >- + Connector-specific options for API Source + connectors. + bing_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + report_type: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: >- + (Required) Microsoft Advertising report + type (e.g. CampaignPerformance, + KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the + report, given as Microsoft Advertising + column enum values for the selected + report type (e.g. AccountId, + CampaignName, TimePeriod, Impressions, + Clicks, Spend). Required key columns are + always included. Available columns are + report-type-specific. For the full list + supported by a given report type, see + its column value set (substitute the + report name, e.g. campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: >- + (Optional) Time granularity for report + aggregation. Defaults to DAILY. + required: + - report_type + - columns + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Bing Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + during incremental sync to capture + late-arriving conversion and attribution + data. If not specified, defaults to 7 + days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. If not + specified, defaults to 2 years before + today. + description: >- + Bing Ads specific options for ingestion + pipelines. + community_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the community connector. + description: >- + Options for community connectors with + arbitrary configuration. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + (Optional) Spaces to filter Confluence + data on + description: Confluence specific options for ingestion + gdrive_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION + description: >- + Create a collection of name/value pairs. - will be used. - description: >- - Days of week in which the restart is allowed to happen - (within a five-hour window starting at start_hour). If - not specified all days of the week will be used. - time_zone_id: - type: string - description: >- - Time zone id of restart window. See - https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html - for details. If not specified, UTC will be used. - required: - - start_hour - description: Restart window of this pipeline. - root_path: - type: string - description: >- - Root path for this pipeline. This is used as the root - directory when editing the pipeline in the Databricks user - interface and it is added to sys.path when executing Python - sources during pipeline execution. - schema: - type: string - description: >- - The default schema (database) where tables are read from or - published to. - serverless: - type: boolean - description: Whether serverless compute is enabled for this pipeline. - storage: - type: string - description: DBFS root directory for storing checkpoints and tables. - tags: - type: object - description: >- - A map of tags associated with the pipeline. These are - forwarded to the cluster as cluster tags, and are therefore - subject to the same limitations. A maximum of 25 tags can be - added to the pipeline. - target: - type: string - description: >- - Target schema (database) to add tables in this pipeline to. - Exactly one of `schema` or `target` must be specified. To - publish to Unity Catalog, also specify `catalog`. This - legacy field is deprecated for pipeline creation in favor of - the `schema` field. - trigger: - type: object - properties: - cron: - type: object - properties: - quartz_cron_schedule: - type: string - timezone_id: - type: string - manual: - type: object - properties: {} - description: >- - Which pipeline trigger to use. Deprecated: Use `continuous` - instead. - usage_policy_id: - type: string - description: Usage policy of this pipeline. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ClonePipelineResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PipelinesAPI - /api/2.0/pipelines: - post: - operationId: pipelines_create - summary: >- - Creates a new data processing pipeline based on the requested - configuration. If successful, this - tags: - - pipelines - description: >- - Creates a new data processing pipeline based on the requested - configuration. If successful, this - method returns the ID of the new pipeline. + Example enumeration: - :param allow_duplicate_names: bool (optional) - If false, deployment will fail if name conflicts with that of another pipeline. - :param budget_policy_id: str (optional) - Budget policy of this pipeline. - :param catalog: str (optional) - A catalog in Unity Catalog to publish data from this pipeline to. If `target` is specified, tables - in this pipeline are published to a `target` schema inside `catalog` (for example, - `catalog`.`target`.`table`). If `target` is not specified, no data is published to Unity Catalog. - :param channel: str (optional) - DLT Release Channel that specifies which version to use. - :param clusters: List[:class:`PipelineCluster`] (optional) - Cluster settings for this pipeline deployment. - :param configuration: Dict[str,str] (optional) - String-String configuration for this pipeline execution. - :param continuous: bool (optional) - Whether the pipeline is continuous or triggered. This replaces `trigger`. - :param deployment: :class:`PipelineDeployment` (optional) - Deployment type of this pipeline. - :param development: bool (optional) - Whether the pipeline is in Development mode. Defaults to false. - :param dry_run: bool (optional) + >>> class Color(Enum): - :param edition: str (optional) - Pipeline product edition. - :param environment: :class:`PipelinesEnvironment` (optional) - Environment specification for this pipeline used to install dependencies. - :param event_log: :class:`EventLogSpec` (optional) - Event log configuration for this pipeline - :param filters: :class:`Filters` (optional) - Filters on which Pipeline packages to include in the deployed graph. - :param gateway_definition: :class:`IngestionGatewayPipelineDefinition` - (optional) - The definition of a gateway pipeline to support change data capture. - :param id: str (optional) - Unique identifier for this pipeline. - :param ingestion_definition: :class:`IngestionPipelineDefinition` - (optional) - The configuration for a managed ingestion pipeline. These settings cannot be used with the - 'libraries', 'schema', 'target', or 'catalog' settings. - :param libraries: List[:class:`PipelineLibrary`] (optional) - Libraries or code needed by this deployment. - :param name: str (optional) - Friendly identifier for this pipeline. - :param notifications: List[:class:`Notifications`] (optional) - List of notification settings for this pipeline. - :param photon: bool (optional) - Whether Photon is enabled for this pipeline. - :param restart_window: :class:`RestartWindow` (optional) - Restart window of this pipeline. - :param root_path: str (optional) - Root path for this pipeline. This is used as the root directory when editing the pipeline in the - Databricks user interface and it is added to sys.path when executing Python sources during pipeline - execution. - :param run_as: :class:`RunAs` (optional) + ... RED = 1 - :param schema: str (optional) - The default schema (database) where tables are read from or published to. - :param serverless: bool (optional) - Whether serverless compute is enabled for this pipeline. - :param storage: str (optional) - DBFS root directory for storing checkpoints and tables. - :param tags: Dict[str,str] (optional) - A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and - are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline. - :param target: str (optional) - Target schema (database) to add tables in this pipeline to. Exactly one of `schema` or `target` must - be specified. To publish to Unity Catalog, also specify `catalog`. This legacy field is deprecated - for pipeline creation in favor of the `schema` field. - :param trigger: :class:`PipelineTrigger` (optional) - Which pipeline trigger to use. Deprecated: Use `continuous` instead. - :param usage_policy_id: str (optional) - Usage policy of this pipeline. + ... BLUE = 2 - :returns: :class:`CreatePipelineResponse` - requestBody: - content: - application/json: - schema: - type: object - properties: - allow_duplicate_names: - type: boolean - description: >- - If false, deployment will fail if name conflicts with that - of another pipeline. - budget_policy_id: - type: string - description: Budget policy of this pipeline. - catalog: - type: string - description: >- - A catalog in Unity Catalog to publish data from this - pipeline to. If `target` is specified, tables in this - pipeline are published to a `target` schema inside `catalog` - (for example, `catalog`.`target`.`table`). If `target` is - not specified, no data is published to Unity Catalog. - channel: - type: string - description: DLT Release Channel that specifies which version to use. - clusters: - type: array - items: - type: object - properties: - apply_policy_default_values: - type: boolean - autoscale: - type: object - properties: - min_workers: - type: integer - max_workers: - type: integer - description: >- - The maximum number of workers to which the cluster - can scale up when overloaded. `max_workers` must - be strictly greater than `min_workers`. - mode: - type: string - x-enum: - - ENHANCED - - LEGACY - description: >- - Databricks Enhanced Autoscaling optimizes cluster - utilization by automatically allocating cluster - resources based on workload volume, with minimal - impact to the data processing latency of your - pipelines. Enhanced Autoscaling is available for - `updates` clusters only. The legacy autoscaling - feature is used for `maintenance` clusters. - required: - - min_workers - - max_workers - description: >- - Parameters needed in order to automatically scale - clusters up and down based on load. Note: autoscaling - works best with DB runtime versions 3.0 or later. - aws_attributes: - type: string - description: >- - Attributes related to clusters running on Amazon Web - Services. If not specified at cluster creation, a set - of default values will be used. - azure_attributes: - type: string - description: >- - Attributes related to clusters running on Microsoft - Azure. If not specified at cluster creation, a set of - default values will be used. - cluster_log_conf: - type: string - description: >- - The configuration for delivering spark logs to a - long-term storage destination. Only dbfs destinations - are supported. Only one destination can be specified - for one cluster. If the conf is given, the logs will - be delivered to the destination every `5 mins`. The - destination of driver logs is - `$destination/$clusterId/driver`, while the - destination of executor logs is - `$destination/$clusterId/executor`. - custom_tags: - type: object - description: >- - Additional tags for cluster resources. Databricks will - tag all cluster resources (e.g., AWS instances and EBS - volumes) with these tags in addition to - `default_tags`. Notes: - Currently, Databricks allows - at most 45 custom tags - Clusters can only reuse cloud - resources if the resources' tags are a subset of the - cluster tags - driver_instance_pool_id: - type: string - description: >- - The optional ID of the instance pool for the driver of - the cluster belongs. The pool cluster uses the - instance pool with id (instance_pool_id) if the driver - pool is not assigned. - driver_node_type_id: - type: string - description: >- - The node type of the Spark driver. Note that this - field is optional; if unset, the driver node type will - be set as the same value as `node_type_id` defined - above. - enable_local_disk_encryption: - type: boolean - description: >- - Whether to enable local disk encryption for the - cluster. - gcp_attributes: - type: string - description: >- - Attributes related to clusters running on Google Cloud - Platform. If not specified at cluster creation, a set - of default values will be used. - init_scripts: - type: string - description: >- - The configuration for storing init scripts. Any number - of destinations can be specified. The scripts are - executed sequentially in the order provided. If - `cluster_log_conf` is specified, init script logs are - sent to `//init_scripts`. - instance_pool_id: - type: string - description: >- - The optional ID of the instance pool to which the - cluster belongs. - label: - type: string - description: >- - A label for the cluster specification, either - `default` to configure the default cluster, or - `maintenance` to configure the maintenance cluster. - This field is optional. The default value is - `default`. - node_type_id: - type: string - description: >- - This field encodes, through a single value, the - resources available to each of the Spark nodes in this - cluster. For example, the Spark nodes can be - provisioned and optimized for memory or compute - intensive workloads. A list of available node types - can be retrieved by using the - :method:clusters/listNodeTypes API call. - num_workers: - type: integer - description: >- - Number of worker nodes that this cluster should have. - A cluster has one Spark Driver and `num_workers` - Executors for a total of `num_workers` + 1 Spark - nodes. Note: When reading the properties of a cluster, - this field reflects the desired number of workers - rather than the actual current number of workers. For - instance, if a cluster is resized from 5 to 10 - workers, this field will immediately be updated to - reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase - from 5 to 10 as the new nodes are provisioned. - policy_id: - type: string - description: >- - The ID of the cluster policy used to create the - cluster if applicable. - spark_conf: - type: object - description: >- - An object containing a set of optional, user-specified - Spark configuration key-value pairs. See - :method:clusters/create for more details. - spark_env_vars: - type: object - description: >- - An object containing a set of optional, user-specified - environment variable key-value pairs. Please note that - key-value pair of the form (X,Y) will be exported as - is (i.e., `export X='Y'`) while launching the driver - and workers. In order to specify an additional set of - `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them - to `$SPARK_DAEMON_JAVA_OPTS` as shown in the example - below. This ensures that all default databricks - managed environmental variables are included as well. - Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` - ssh_public_keys: - type: array - items: - type: string - description: >- - SSH public key contents that will be added to each - Spark node in this cluster. The corresponding private - keys can be used to login with the user name `ubuntu` - on port `2200`. Up to 10 keys can be specified. - description: Cluster settings for this pipeline deployment. - configuration: - type: object - description: String-String configuration for this pipeline execution. - continuous: - type: boolean - description: >- - Whether the pipeline is continuous or triggered. This - replaces `trigger`. - deployment: - type: object - properties: - kind: - type: string - x-enum: - - BUNDLE - description: >- - The deployment method that manages the pipeline: - - BUNDLE: The pipeline is managed by a + ... GREEN = 3 - Databricks Asset Bundle. - metadata_file_path: - type: string - description: >- - The path to the file containing metadata about the - deployment. - required: - - kind - description: Deployment type of this pipeline. - development: - type: boolean - description: >- - Whether the pipeline is in Development mode. Defaults to - false. - dry_run: - type: boolean - edition: - type: string - description: Pipeline product edition. - environment: - type: object - properties: - dependencies: - type: array - items: - type: string - description: >- - List of pip dependencies, as supported by the version of - pip in this environment. Each dependency is a pip - requirement file line - https://pip.pypa.io/en/stable/reference/requirements-file-format/ - Allowed dependency could be , - , (WSFS or Volumes - in Databricks), - environment_version: - type: string - description: >- - The environment version of the serverless Python - environment used to execute customer Python code. Each - environment version includes a specific Python version - and a curated set of pre-installed libraries with - defined versions, providing a stable and reproducible - execution environment. Databricks supports a three-year - lifecycle for each environment version. For available - versions and their included packages, see - https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ - The value should be a string representing the - environment version number, for example: `"4"`. - description: >- - Environment specification for this pipeline used to install - dependencies. - event_log: - type: object - properties: - catalog: - type: string - description: The UC catalog the event log is published under. - name: - type: string - description: The name the event log is published to in UC. - schema: - type: string - description: The UC schema the event log is published under. - description: Event log configuration for this pipeline - filters: - type: object - properties: - exclude: - type: array - items: - type: string - include: - type: array - items: - type: string - description: Paths to include. - description: >- - Filters on which Pipeline packages to include in the - deployed graph. - gateway_definition: - type: object - properties: - connection_name: - type: string - gateway_storage_catalog: - type: string - description: >- - Required, Immutable. The name of the catalog for the - gateway pipeline's storage location. - gateway_storage_schema: - type: string - description: >- - Required, Immutable. The name of the schema for the - gateway pipelines's storage location. - connection_id: - type: string - description: >- - [Deprecated, use connection_name instead] Immutable. The - Unity Catalog connection that this gateway pipeline uses - to communicate with the source. - connection_parameters: - type: object - properties: - source_catalog: - type: string - description: >- - Optional, Internal. Parameters required to establish an - initial connection with the source. - gateway_storage_name: - type: string - description: >- - Optional. The Unity Catalog-compatible name for the - gateway storage location. This is the destination to use - for the data that is extracted by the gateway. Spark - Declarative Pipelines system will automatically create - the storage location under the catalog and schema. - required: - - connection_name - - gateway_storage_catalog - - gateway_storage_schema - description: >- - The definition of a gateway pipeline to support change data - capture. - id: - type: string - description: Unique identifier for this pipeline. - ingestion_definition: - type: object - properties: - connection_name: - type: string - connector_type: - type: string - x-enum: - - CDC - - QUERY_BASED - description: >- - (Optional) Connector Type for sources. Ex: CDC, Query - Based. - data_staging_options: - type: object - properties: - catalog_name: - type: string - description: >- - (Required, Immutable) The name of the catalog for - the connector's staging storage location. - schema_name: - type: string - description: >- - (Required, Immutable) The name of the schema for the - connector's staging storage location. - volume_name: - type: string - description: >- - (Optional) The Unity Catalog-compatible name for the - storage location. This is the volume to use for the - data that is extracted by the connector. Spark - Declarative Pipelines system will automatically - create the volume under the catalog and schema. For - Combined Cdc Managed Ingestion pipelines default - name for the volume would be : - __databricks_ingestion_gateway_staging_data-$pipelineId - required: - - catalog_name - - schema_name - description: >- - (Optional) Location of staged data storage. This is - required for migration from Cdc Managed Ingestion - Pipeline with Gateway pipeline to Combined Cdc Managed - Ingestion Pipeline. If not specified, the volume for - staged data will be created in catalog and schema/target - specified in the top level pipeline definition. - full_refresh_window: - type: object - properties: - start_hour: - type: integer - description: >- - An integer between 0 and 23 denoting the start hour - for the window in the 24-hour day. - days_of_week: - type: array - items: - type: string - x-enum: - - FRIDAY - - MONDAY - - SATURDAY - - SUNDAY - - THURSDAY - - TUESDAY - - WEDNESDAY - description: >- - Days of week in which the window is allowed to - happen. If not specified all days of the week - will be used. - description: >- - Days of week in which the window is allowed to - happen If not specified all days of the week will be - used. - time_zone_id: - type: string - description: >- - Time zone id of window. See - https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html - for details. If not specified, UTC will be used. - required: - - start_hour - description: >- - (Optional) A window that specifies a set of time ranges - for snapshot queries in CDC. - ingest_from_uc_foreign_catalog: - type: boolean - description: >- - Immutable. If set to true, the pipeline will ingest - tables from the UC foreign catalogs directly without the - need to specify a UC connection or ingestion gateway. - The `source_catalog` fields in objects of - IngestionConfig are interpreted as the UC foreign - catalogs to ingest from. - ingestion_gateway_id: - type: string - description: >- - Identifier for the gateway that is used by this - ingestion pipeline to communicate with the source - database. This is used with CDC connectors to databases - like SQL Server using a gateway pipeline (connector_type - = CDC). Under certain conditions, this can be replaced - with connection_name to change the connector to Combined - Cdc Managed Ingestion Pipeline. - netsuite_jar_path: - type: string - description: >- - Netsuite only configuration. When the field is set for a - netsuite connector, the jar stored in the field will be - validated and added to the classpath of pipeline's - cluster. - objects: - type: array - items: - type: object - properties: - report: - type: object - properties: - source_url: - type: string - destination_catalog: - type: string - description: Required. Destination catalog to store table. - destination_schema: - type: string - description: Required. Destination schema to store table. - destination_table: - type: string - description: >- - Required. Destination table name. The pipeline - fails if a table with that name already - exists. - table_configuration: - type: object - properties: - auto_full_refresh_policy: + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + scope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE + description: >- + The scope of the Google Drive ingestion. + **ONLY ONE** of scope or url must be + specified. If scope is specified, we + will ingest files from the scope. If url + is specified, we will ingest files from + the url. + url: + type: string + description: Google Drive URL. + github_options: type: object properties: - enabled: - type: boolean + repository_branch_selections: + type: array + items: + type: object + properties: + repository_id: + type: string + description: >- + (Required) GitHub repository id + (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this + repository, e.g. ["main", "dev"]. + required: + - repository_id + description: >- + A repository and the branches to ingest + for it. description: >- - (Required, Mutable) Whether to enable - auto full refresh or not. - min_interval_hours: + (Optional) Branches to ingest per + repository. A repository not listed here + is ingested on its default branch. + description: GitHub-specific ingestion options. + google_ads_options: + type: object + properties: + manager_account_id: + type: string + description: >- + (Optional at this level) Manager Account + ID (also called MCC Account ID) used to + list and access customer accounts under + this manager account. Overrides + GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query + (e.g. "ad_group_ad", "keyword_view", + "search_term_view"). Must be a resource + that has metrics. Values are validated + against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. + "metrics.clicks", + "metrics.cost_micros"). Multiple values + are joined into the GAQL SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in + fully-qualified GAQL form (e.g. + "ad_group_ad.ad.id", + "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select + (e.g. "segments.date", + "segments.device"). Must include at + least one of segments.date, + segments.week, or segments.month — that + segment is used as the incremental + cursor for the table. + required: + - resource + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Google Ads custom report: + the connector synthesizes a GAQL query + from the resource, fields, segments, and + metrics specified here. When unset, the + table must match one of the connector's + prebuilt sources. + lookback_window_days: type: integer description: >- - (Optional, Mutable) Specify the minimum - interval in hours between the timestamp - at which a table was last full refreshed - and the current timestamp for triggering - auto full If unspecified and - autoFullRefresh is enabled then by - default min_interval_hours is 24 hours. + (Optional) Number of days to look back + for report tables to capture + late-arriving data. If not specified, + defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years of + historical data. required: - - enabled - description: Policy for auto full refresh. - exclude_columns: - type: array - items: - type: string - description: >- - A list of column names to be excluded for - the ingestion. When not specified, - include_columns fully controls what - columns to be ingested. When specified, - all other columns including future ones - will be automatically included for - ingestion. This field in mutually - exclusive with `include_columns`. - include_columns: - type: array - items: - type: string - description: >- - A list of column names to be included for - the ingestion. When not specified, all - columns except ones in exclude_columns - will be included. Future columns will be - automatically included. When specified, - all other future columns will be - automatically excluded from ingestion. - This field in mutually exclusive with - `exclude_columns`. - primary_keys: - type: array - items: - type: string + - manager_account_id description: >- - The primary key of the table used to apply - changes. - query_based_connector_config: + Google Ads specific options for ingestion + (object-level). When set, these values + override the + corresponding fields in GoogleAdsConfig (source_configurations). + jira_options: type: object properties: - cursor_columns: + include_jira_spaces: type: array items: type: string description: >- - The names of the monotonically - increasing columns in the source table - that are used to enable the table to be - read and ingested incrementally through - structured streaming. The columns are - allowed to have repeated values but have - to be non-decreasing. If the source data - is merged into the destination (e.g., - using SCD Type 1 or Type 2), these - columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. - deletion_condition: - type: string + (Optional) Projects to filter Jira data + on + use_audit_logs: + type: boolean description: >- - Specifies a SQL WHERE condition that - specifies that the source row has been - deleted. This is sometimes referred to - as "soft-deletes". For example: - "Operation = 'DELETE'" or "is_deleted = - true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, - one for soft-deletes and the other for - hard-deletes. See also the - hard_deletion_sync_min_interval_in_seconds - field for handling of "hard deletes" - where the source rows are physically - removed from the table. - hard_deletion_sync_min_interval_in_seconds: + (Optional) Whether the pipeline reads + Jira's audit log to detect changes for + the tables it ingests. Reading the audit + log requires a paid Jira plan and + appropriate admin permissions. For + example, the issues table uses the audit + log to detect and propagate deletes: + when true, rows deleted in Jira are + removed from the destination table; when + false, the destination retains all + previously ingested rows and only + reflects inserts and updates. If not + specified, defaults to false. + description: Jira specific options for ingestion + kafka_options: + type: object + properties: + client_config: + type: object + key_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + key. If not specified, the key is left + as raw bytes. + max_offsets_per_trigger: type: integer description: >- - Specifies the minimum interval (in - seconds) between snapshots on primary - keys for detecting and synchronizing - hard deletions—i.e., rows that have been - physically removed from the source - table. This interval acts as a lower - bound. If ingestion runs less frequently - than this value, hard deletion - synchronization will align with the - actual ingestion frequency instead of - happening more often. If not set, hard - deletion synchronization via snapshots - is disabled. This field is mutable and - can be updated without triggering a full - snapshot. - description: >- - Configurations that are only applicable - for query-based ingestion connectors. - row_filter: - type: string - description: >- - (Optional, Immutable) The row filter - condition to be applied to the table. It - must not contain the WHERE keyword, only - the actual filter condition. It must be in - DBSQL format. - salesforce_include_formula_fields: - type: boolean - description: >- - If true, formula fields defined in the - table are included in the ingestion. This - setting is only valid for the Salesforce - connector - scd_type: - type: string - x-enum: - - APPEND_ONLY - - SCD_TYPE_1 - - SCD_TYPE_2 - description: The SCD type to use to ingest the table. - sequence_by: - type: array - items: - type: string - description: >- - The column names specifying the logical - order of events in the source data. Spark - Declarative Pipelines uses this sequencing - to handle change events that arrive out of - order. - workday_report_parameters: + Internal option to control the maximum + number of offsets to process per + trigger. + max_partitions: + type: string + description: >- + (Optional) Controls how Kafka offset + ranges are bin-packed into Spark input + partitions. Defaults to "auto" when + unset, which lets the engine choose the + input partition count from measured + throughput. Set to "none" to disable + bin-packing, so each offset range maps + to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when + no checkpoint exists. Valid values: + "latest" and "earliest". Defaults to + "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to + matching topics. Only one of topics or + topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of + topics or topic_pattern must be + specified. + value_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + value. If not specified, the value is + left as raw bytes. + linkedin_ads_options: type: object properties: - incremental: - type: boolean - parameters: + custom_report_options: type: object - description: >- - Parameters for the Workday report. Each - key represents the parameter name (e.g., - "start_date", "end_date"), and the - corresponding value is a SQL-like - expression used to compute the parameter - value at runtime. Example: { - "start_date": "{ - coalesce(current_offset(), - date(\"2025-02-01\")) }", "end_date": "{ - current_date() - INTERVAL 1 DAY }" } - report_parameters: + properties: + finder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + (Required) adAnalytics finder. See + LinkedInAdsFinder. + entity_granularity: + type: array + items: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the + report. Open vocabulary (not an enum): + the valid set is large (~100) and + evolves with the LinkedIn adAnalytics + API, so values are passed through + verbatim. If empty, a pivot-safe default + core set is ingested: impressions, + clicks, costInLocalCurrency, + externalWebsiteConversions (valid for + every pivot). Ignored for + attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics + struct). + time_granularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + (Optional) Time aggregation. Defaults to + DAILY when unspecified. Used by + analytics/statistics; ignored for + attributedRevenueMetrics. + required: + - finder + description: >- + (Optional) Custom report definition. + Only valid on a table object. When set, + the table is synthesized from + /rest/adAnalytics using the finder, + pivots, time granularity and metrics + here. When unset, the table must match + one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Days to look back during + incremental sync for late-arriving data. + If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables, YYYY-MM-DD. + Earliest date from which to sync + historical data; overrides the default + when set. For finder + attributedRevenueMetrics, this must be + between 30 and 366 days before today. If + not specified, defaults to 1 year of + history. + description: >- + LinkedIn Ads specific options for + ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + marketo_options: + type: object + properties: + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + description: Marketo specific options for ingestion + meta_ads_options: + type: object + properties: + action_attribution_windows: type: array items: - type: object - properties: - key: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) + Action attribution windows for insights + reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) + Timing used to report action statistics + (impression, conversion, mixed, or + lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit + data during sync to capture updated + conversion data from the API, shared by + prebuilt and custom reports. + custom_report_options: + type: object + properties: + action_attribution_windows: + type: array + items: type: string - value: + description: >- + (Optional) Action attribution windows + for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: type: string - description: >- - Value for the report parameter. Possible - values it can take are these sql - functions: 1. coalesce(current_offset(), - date("YYYY-MM-DD")) -> if - current_offset() is null, then the - passed date, else current_offset() 2. - current_date() 3. - date_sub(current_date(), x) -> subtract - x (some non-negative integer) days from - current date + description: >- + (Optional) Action breakdowns to + configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action + statistics (impression, conversion, + mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdowns to configure for + data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull + (account, ad, adset, campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to + aggregate statistics (all_days, monthly + or number of days) description: >- - (Optional) Additional custom parameters - for Workday Report This field is - deprecated and should not be used. Use - `parameters` instead. + (Optional) Per-table custom report + definition. When set, defines the shape + of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). + Supersedes the deprecated flat + report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.level) Granularity + of data to pull (account, ad, adset, + campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd + format (e.g. 2025-01-15). Data added + after this date will be ingested, shared + by prebuilt and custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.time_increment) + Value in string by which to aggregate + statistics (can take all_days, monthly + or number of days) description: >- - (Optional) Additional custom parameters - for Workday Report - description: >- - Configuration settings to control the - ingestion of tables. These settings override - the table_configuration defined in the - IngestionPipelineDefinition object. - required: - - source_url - - destination_catalog - - destination_schema - schema: - type: object - properties: - source_schema: - type: string - destination_catalog: - type: string - description: Required. Destination catalog to store tables. - destination_schema: + Meta Marketing (Meta Ads) specific options + for ingestion + outlook_options: + type: object + properties: + attachment_mode: + type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY + description: >- + (Optional) Controls which attachments to + ingest. If not specified, defaults to + ALL. + body_format: + type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN + description: >- + (Optional) Defines how the body_content + column is populated. TEXT_HTML: + Preserves full formatting, links, and + styling. TEXT_PLAIN: Converts body to + plain text. Recommended for AI/RAG + pipelines to reduce token usage and + noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to + include in the sync. If not specified, + all folders will be synced. Examples: + Inbox, Sent Items, Custom_Folder Filter + semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync + (e.g. mailbox email addresses or + identifiers). If not specified, all + accessible mailboxes are ingested. + Filter semantics: OR between different + mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender + address. Uses exact email match. + Examples: user@vendor.com, + alerts@system.io, noreply@company.com If + not specified, emails from all senders + will be synced. Filter semantics: OR + between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject + line. Values ending with "*" use prefix + match (subject starts with the part + before "*"); otherwise substring match + (subject contains the value). Examples: + "Invoice" (substring), "Re:*" (prefix), + "Support Ticket", "URGENT*" If not + specified, emails with all subjects will + be synced. Filter semantics: OR between + different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. Format: + YYYY-MM-DD (e.g., 2024-01-01) This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + subject_filter: + type: array + items: + type: string + description: >- + Deprecated. Use include_subjects + instead. + description: Outlook specific options for ingestion + rabbitmq_options: + type: object + properties: + queue: + type: string + description: >- + (Required) RabbitMQ queue name to + consume from. + required: + - queue + description: >- + RabbitMQ specific options for ingestion. + Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + reddit_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group + report data by. Examples: CAMPAIGN_ID, + DATE, COUNTRY, REGION, AD_ID. Must + include at least one time dimension + (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the + report (maps to the Reddit Ads API + ``fields`` parameter). Examples: + IMPRESSIONS, CLICKS, SPEND, CPC, CTR. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Reddit Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. If not + specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years + ago. + description: Reddit Ads specific options for ingestion + sharepoint_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + description: >- + (Optional) File ingestion options for + processing files. + url: + type: string + description: Required. The SharePoint URL. + smartsheet_options: + type: object + properties: + enforce_schema: + type: boolean + description: >- + (Optional) When true, maps each column + to its Smartsheet-declared type + (Text/Number/Date/ Checkbox/etc.). Cells + that do not conform to the declared type + are set to NULL. When false, all columns + land as STRING. Use false for sheets + with irregular data or columns that + frequently violate their own declared + type. If not specified, defaults to + true. + description: Smartsheet specific options for ingestion + tiktok_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + (Optional) Data level for the report. If + not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the + report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", + "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the + report (e.g. "spend", "impressions", + "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime + metrics (all-time aggregated data). When + true, the report returns all-time data. + If not specified, defaults to false. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + (Optional) Report type for the TikTok + Ads API. If not specified, defaults to + BASIC. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined TikTok Ads custom report: + the connector synthesizes a report + request from the dimensions, metrics, + report type, and data level specified + here. Supersedes the deprecated + top-level + dimensions/metrics/report_type/ + data_level/query_lifetime fields above. + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + Deprecated. Use + custom_report_options.data_level + instead. + dimensions: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.dimensions + instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. + metrics: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: >- + Deprecated. Use + custom_report_options.query_lifetime + instead. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + Deprecated. Use + custom_report_options.report_type + instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + description: TikTok Ads specific options for ingestion + zendesk_support_options: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD + format for the initial sync. This + determines the earliest date from which + to sync historical data. + description: >- + Zendesk Support specific options for + ingestion + description: (Optional) Source Specific Connector Options + destination_table: type: string description: >- - Required. Destination schema to store tables - in. Tables with the same name as the source - tables are created in this destination schema. - The pipeline fails If a table with the same - name already exists. + Optional. Destination table name. The pipeline + fails if a table with that name already + exists. If not set, the source table name is + used. + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use connector_options.jira_options + instead. source_catalog: type: string description: >- - The source catalog name. Might be optional + Source catalog name. Might be optional depending on the type of source. + source_schema: + type: string + description: >- + Schema name in the source database. Might be + optional depending on the type of source. table_configuration: type: object properties: @@ -2510,6 +4534,38 @@ paths: required: - enabled description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. exclude_columns: type: array items: @@ -2522,7 +4578,7 @@ paths: all other columns including future ones will be automatically included for ingestion. This field in mutually - exclusive with `include_columns`. + exclusive with ``include_columns``. include_columns: type: array items: @@ -2536,7 +4592,7 @@ paths: all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with - `exclude_columns`. + ``exclude_columns``. primary_keys: type: array items: @@ -2562,9 +4618,9 @@ paths: is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. deletion_condition: type: string description: >- @@ -2574,7 +4630,7 @@ paths: as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, + ``hard_deletion_sync_interval_in_seconds``, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds @@ -2634,6 +4690,36 @@ paths: Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. workday_report_parameters: type: object properties: @@ -2676,251 +4762,18 @@ paths: (Optional) Additional custom parameters for Workday Report This field is deprecated and should not be used. Use - `parameters` instead. + ``parameters`` instead. description: >- (Optional) Additional custom parameters for Workday Report description: >- Configuration settings to control the - ingestion of tables. These settings are - applied to all tables in this schema and - override the table_configuration defined in - the IngestionPipelineDefinition object. + ingestion of tables. These settings override + the table_configuration defined in the + IngestionPipelineDefinition object and the + SchemaSpec. required: - - source_schema - - destination_catalog - - destination_schema - description: Select all tables from a specific source schema. - table: - type: object - properties: - source_table: - type: string - destination_catalog: - type: string - description: Required. Destination catalog to store table. - destination_schema: - type: string - description: Required. Destination schema to store table. - destination_table: - type: string - description: >- - Optional. Destination table name. The pipeline - fails if a table with that name already - exists. If not set, the source table name is - used. - source_catalog: - type: string - description: >- - Source catalog name. Might be optional - depending on the type of source. - source_schema: - type: string - description: >- - Schema name in the source database. Might be - optional depending on the type of source. - table_configuration: - type: object - properties: - auto_full_refresh_policy: - type: object - properties: - enabled: - type: boolean - description: >- - (Required, Mutable) Whether to enable - auto full refresh or not. - min_interval_hours: - type: integer - description: >- - (Optional, Mutable) Specify the minimum - interval in hours between the timestamp - at which a table was last full refreshed - and the current timestamp for triggering - auto full If unspecified and - autoFullRefresh is enabled then by - default min_interval_hours is 24 hours. - required: - - enabled - description: Policy for auto full refresh. - exclude_columns: - type: array - items: - type: string - description: >- - A list of column names to be excluded for - the ingestion. When not specified, - include_columns fully controls what - columns to be ingested. When specified, - all other columns including future ones - will be automatically included for - ingestion. This field in mutually - exclusive with `include_columns`. - include_columns: - type: array - items: - type: string - description: >- - A list of column names to be included for - the ingestion. When not specified, all - columns except ones in exclude_columns - will be included. Future columns will be - automatically included. When specified, - all other future columns will be - automatically excluded from ingestion. - This field in mutually exclusive with - `exclude_columns`. - primary_keys: - type: array - items: - type: string - description: >- - The primary key of the table used to apply - changes. - query_based_connector_config: - type: object - properties: - cursor_columns: - type: array - items: - type: string - description: >- - The names of the monotonically - increasing columns in the source table - that are used to enable the table to be - read and ingested incrementally through - structured streaming. The columns are - allowed to have repeated values but have - to be non-decreasing. If the source data - is merged into the destination (e.g., - using SCD Type 1 or Type 2), these - columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. - deletion_condition: - type: string - description: >- - Specifies a SQL WHERE condition that - specifies that the source row has been - deleted. This is sometimes referred to - as "soft-deletes". For example: - "Operation = 'DELETE'" or "is_deleted = - true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, - one for soft-deletes and the other for - hard-deletes. See also the - hard_deletion_sync_min_interval_in_seconds - field for handling of "hard deletes" - where the source rows are physically - removed from the table. - hard_deletion_sync_min_interval_in_seconds: - type: integer - description: >- - Specifies the minimum interval (in - seconds) between snapshots on primary - keys for detecting and synchronizing - hard deletions—i.e., rows that have been - physically removed from the source - table. This interval acts as a lower - bound. If ingestion runs less frequently - than this value, hard deletion - synchronization will align with the - actual ingestion frequency instead of - happening more often. If not set, hard - deletion synchronization via snapshots - is disabled. This field is mutable and - can be updated without triggering a full - snapshot. - description: >- - Configurations that are only applicable - for query-based ingestion connectors. - row_filter: - type: string - description: >- - (Optional, Immutable) The row filter - condition to be applied to the table. It - must not contain the WHERE keyword, only - the actual filter condition. It must be in - DBSQL format. - salesforce_include_formula_fields: - type: boolean - description: >- - If true, formula fields defined in the - table are included in the ingestion. This - setting is only valid for the Salesforce - connector - scd_type: - type: string - x-enum: - - APPEND_ONLY - - SCD_TYPE_1 - - SCD_TYPE_2 - description: The SCD type to use to ingest the table. - sequence_by: - type: array - items: - type: string - description: >- - The column names specifying the logical - order of events in the source data. Spark - Declarative Pipelines uses this sequencing - to handle change events that arrive out of - order. - workday_report_parameters: - type: object - properties: - incremental: - type: boolean - parameters: - type: object - description: >- - Parameters for the Workday report. Each - key represents the parameter name (e.g., - "start_date", "end_date"), and the - corresponding value is a SQL-like - expression used to compute the parameter - value at runtime. Example: { - "start_date": "{ - coalesce(current_offset(), - date(\"2025-02-01\")) }", "end_date": "{ - current_date() - INTERVAL 1 DAY }" } - report_parameters: - type: array - items: - type: object - properties: - key: - type: string - value: - type: string - description: >- - Value for the report parameter. Possible - values it can take are these sql - functions: 1. coalesce(current_offset(), - date("YYYY-MM-DD")) -> if - current_offset() is null, then the - passed date, else current_offset() 2. - current_date() 3. - date_sub(current_date(), x) -> subtract - x (some non-negative integer) days from - current date - description: >- - (Optional) Additional custom parameters - for Workday Report This field is - deprecated and should not be used. Use - `parameters` instead. - description: >- - (Optional) Additional custom parameters - for Workday Report - description: >- - Configuration settings to control the - ingestion of tables. These settings override - the table_configuration defined in the - IngestionPipelineDefinition object and the - SchemaSpec. - required: - - source_table + - source_table - destination_catalog - destination_schema description: Select a specific source table. @@ -2932,6 +4785,17 @@ paths: items: type: object properties: + api_source_connector_config: + type: object + properties: + configs: + type: object + description: >- + Arbitrary key-value configuration values for + the API Source connector. + description: >- + Top-level configuration for API Source connectors + with arbitrary configuration. catalog: type: object properties: @@ -2960,28 +4824,123 @@ paths: source_catalog: type: string description: Source catalog name - description: >- - SourceCatalogConfig contains catalog-level custom - configuration parameters for each source + description: Catalog-level source configuration parameters + google_ads_config: + type: object + properties: + manager_account_id: + type: string description: Top-level source configurations source_type: type: string x-enum: + - ADOBE_CAMPAIGNS + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AKAMAI_WAF + - AMPLITUDE + - API_SOURCE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS - BIGQUERY + - BING_ADS + - CERIDIAN_DAYFORCE + - COMMUNITY + - CONFLUENCE + - CROWDSTRIKE_EVENT_STREAM + - DELIGHTED - DYNAMICS365 + - EPIC_CLARITY - FOREIGN_CATALOG + - FRESHSERVICE + - FRONT - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB + - GMAIL + - GONG + - GOOGLE_ADS + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_DRIVE + - GOOGLE_SEARCH_CONSOLE + - GOOGLE_WORKSPACE + - GUIDEWIRE + - GURU + - HIBOB + - HUBSPOT + - IRONCLAD + - JIRA + - KAFKA + - LINEAR + - LINKEDIN_ADS + - M365_AUDIT_LOGS - MANAGED_POSTGRESQL + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MICROSOFT_TEAMS + - MONDAY_COM - MYSQL + - NETSKOPE_LOGS - NETSUITE + - NOTION + - OKTA_SYSTEM_LOGS + - ONE_PASSWORD_EVENT_LOGS - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PARTNERSTACK + - PENDO + - PINTEREST_ADS - POSTGRESQL + - PROOFPOINT_SIEM + - QUICKBOOKS + - RABBITMQ + - REDDIT_ADS + - REDSHIFT - SALESFORCE + - SALESFORCE_MARKETING_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID - SERVICENOW - SHAREPOINT + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SLACK_AUDIT_LOGS + - SMARTSHEET + - SNAPCHAT_ADS + - SPLUNK + - SQLDW - SQLSERVER + - SQUARE - TERADATA + - TIKTOK_ADS + - VEEVA + - VEEVA_VAULT + - VERKADA + - WIZ_AUDIT_LOGS + - WORKDAY_ACTIVITY_LOGGING + - WORKDAY_HCM - WORKDAY_RAAS + - X_ADS + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS description: >- The type of the foreign source. The source type will be inferred from the source connection or ingestion @@ -3010,6 +4969,33 @@ paths: required: - enabled description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering the + destination table. When specified, the destination + Delta table will be clustered by these columns. This + can improve query performance when filtering on + these columns. Note: clustering_columns in table + specific configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or clustering_columns, added + as separate fields as we cannot have repeated field + in oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the destination + table. When enabled, Delta will automatically + optimize the data layout based on the clustering + columns for improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline definition. + Note: we can only provide enable_auto_clustering or + clustering_columns, added as separate fields as we + cannot have repeated field in oneof. exclude_columns: type: array items: @@ -3020,7 +5006,8 @@ paths: controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This - field in mutually exclusive with `include_columns`. + field in mutually exclusive with + ``include_columns``. include_columns: type: array items: @@ -3032,7 +5019,7 @@ paths: columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in - mutually exclusive with `exclude_columns`. + mutually exclusive with ``exclude_columns``. primary_keys: type: array items: @@ -3054,9 +5041,9 @@ paths: have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly - define the `sequence_by` behavior. You can still - explicitly set `sequence_by` to override this - default. + define the ``sequence_by`` behavior. You can + still explicitly set ``sequence_by`` to override + this default. deletion_condition: type: string description: >- @@ -3065,7 +5052,7 @@ paths: sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, one + ``hard_deletion_sync_interval_in_seconds``, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field @@ -3118,6 +5105,33 @@ paths: events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added to each + ingested record to hold per row source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is written and + maintained in the destination table. Mutually + exclusive with scd_type — a 400 error is returned if + both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination table. + These are key-value pairs that configure various + Delta table behaviors or any user defined + properties. Example: {"delta.feature.variantType": + "supported", "delta.enableTypeWidening": "true"} + Note: table_properties in table specific + configuration will override the table_properties of + the pipeline definition. workday_report_parameters: type: object properties: @@ -3156,7 +5170,7 @@ paths: description: >- (Optional) Additional custom parameters for Workday Report This field is deprecated and - should not be used. Use `parameters` instead. + should not be used. Use ``parameters`` instead. description: >- (Optional) Additional custom parameters for Workday Report @@ -3186,8 +5200,8 @@ paths: description: >- The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path - that ends `/**`. This field cannot be used together - with `notebook` or `file`. + that ends ``/**``. This field cannot be used together + with ``notebook`` or ``file``. jar: type: string description: >- @@ -3266,6 +5280,24 @@ paths: required: - start_hour description: Restart window of this pipeline. + rewind_generation_interval: + type: object + properties: + interval: + type: integer + description: >- + An integer value representing the interval quantity for + the schedule. + unit: + type: string + x-enum: + - DAYS + - HOURS + - WEEKS + description: A time unit for the schedule. + description: >- + Interval at which rewind points are generated during + pipeline execution. root_path: type: string description: >- @@ -3273,27 +5305,6 @@ paths: directory when editing the pipeline in the Databricks user interface and it is added to sys.path when executing Python sources during pipeline execution. - run_as: - type: object - properties: - service_principal_name: - type: string - description: >- - Application ID of an active service principal. Setting - this field requires the `servicePrincipal/user` role. - user_name: - type: string - description: >- - The email of an active workspace user. Users can only - set this field to their own email. - description: >- - Write-only setting, available only in Create/Update calls. - Specifies the user or service - principal that the pipeline runs as. If not specified, the pipeline runs as the user who created - the pipeline. - - Only `user_name` or `service_principal_name` can be specified. If both are specified, an error - is thrown. schema: type: string description: >- @@ -3302,6 +5313,11 @@ paths: serverless: type: boolean description: Whether serverless compute is enabled for this pipeline. + serverless_compute_id: + type: string + description: >- + Serverless compute ID specified by the user for serverless + pipelines. storage: type: string description: DBFS root directory for storing checkpoints and tables. @@ -3316,10 +5332,10 @@ paths: type: string description: >- Target schema (database) to add tables in this pipeline to. - Exactly one of `schema` or `target` must be specified. To - publish to Unity Catalog, also specify `catalog`. This + Exactly one of ``schema`` or ``target`` must be specified. + To publish to Unity Catalog, also specify ``catalog``. This legacy field is deprecated for pipeline creation in favor of - the `schema` field. + the ``schema`` field. trigger: type: object properties: @@ -3334,8 +5350,8 @@ paths: type: object properties: {} description: >- - Which pipeline trigger to use. Deprecated: Use `continuous` - instead. + Which pipeline trigger to use. Deprecated: Use + ``continuous`` instead. usage_policy_id: type: string description: Usage policy of this pipeline. @@ -3345,95 +5361,7 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/CreatePipelineResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PipelinesAPI - get: - operationId: pipelines_list_pipelines - summary: Lists pipelines defined in the Spark Declarative Pipelines system. - tags: - - pipelines - description: |- - Lists pipelines defined in the Spark Declarative Pipelines system. - - :param filter: str (optional) - Select a subset of results based on the specified criteria. The supported filters are: - - * `notebook=''` to select pipelines that reference the provided notebook path. * `name LIKE - '[pattern]'` to select pipelines with a name that matches pattern. Wildcards are supported, for - example: `name LIKE '%shopping%'` - - Composite filters are not supported. This field is optional. - :param max_results: int (optional) - The maximum number of entries to return in a single page. The system may return fewer than - max_results events in a response, even if there are more events available. This field is optional. - The default value is 25. The maximum value is 100. An error is returned if the value of max_results - is greater than 100. - :param order_by: List[str] (optional) - A list of strings specifying the order of results. Supported order_by fields are id and name. The - default is id asc. This field is optional. - :param page_token: str (optional) - Page token returned by previous call - - :returns: Iterator over :class:`PipelineStateInfo` - parameters: - - name: filter - in: query - required: false - schema: - type: string - description: >- - Select a subset of results based on the specified criteria. The - supported filters are: * `notebook=''` to select pipelines - that reference the provided notebook path. * `name LIKE '[pattern]'` - to select pipelines with a name that matches pattern. Wildcards are - supported, for example: `name LIKE '%shopping%'` Composite filters - are not supported. This field is optional. - - name: max_results - in: query - required: false - schema: - type: integer - description: >- - The maximum number of entries to return in a single page. The system - may return fewer than max_results events in a response, even if - there are more events available. This field is optional. The default - value is 25. The maximum value is 100. An error is returned if the - value of max_results is greater than 100. - - name: order_by - in: query - required: false - schema: - type: array - items: - type: string - description: >- - A list of strings specifying the order of results. Supported - order_by fields are id and name. The default is id asc. This field - is optional. - - name: page_token - in: query - required: false - schema: - type: string - description: Page token returned by previous call - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListPipelinesResponse' + $ref: '#/components/schemas/ClonePipelineResponse' default: description: Error response content: @@ -3446,133 +5374,50 @@ paths: message: type: string x-stackql-sdk-source: PipelinesAPI - /api/2.0/pipelines/{pipeline_id}: - delete: - operationId: pipelines_delete + /api/2.0/pipelines: + post: + operationId: pipelines_create summary: >- - Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline - deletion will cascade to all + Creates a new data processing pipeline based on the requested + configuration. If successful, this tags: - pipelines description: >- - Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline - deletion will cascade to all - - pipeline tables. Please reach out to Databricks support for assistance - to undo this action. - - - :param pipeline_id: str - - :param force: bool (optional) - If true, deletion will proceed even if resource cleanup fails. By default, deletion will fail if - resources cleanup is required but fails. - parameters: - - name: pipeline_id - in: path - required: true - schema: - type: string - - name: force - in: query - required: false - schema: - type: boolean - description: >- - If true, deletion will proceed even if resource cleanup fails. By - default, deletion will fail if resources cleanup is required but - fails. - responses: - '200': - description: Success - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PipelinesAPI - get: - operationId: pipelines_get - summary: Get a pipeline. - tags: - - pipelines - description: |- - Get a pipeline. - - :param pipeline_id: str + Creates a new data processing pipeline based on the requested + configuration. If successful, this - :returns: :class:`GetPipelineResponse` - parameters: - - name: pipeline_id - in: path - required: true - schema: - type: string - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/GetPipelineResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PipelinesAPI - put: - operationId: pipelines_update - summary: Updates a pipeline with the supplied configuration. - tags: - - pipelines - description: >- - Updates a pipeline with the supplied configuration. + method returns the ID of the new pipeline. - :param pipeline_id: str - Unique identifier for this pipeline. :param allow_duplicate_names: bool (optional) - If false, deployment will fail if name has changed and conflicts the name of another pipeline. + If false, deployment will fail if name conflicts with that of another pipeline. :param budget_policy_id: str (optional) Budget policy of this pipeline. :param catalog: str (optional) - A catalog in Unity Catalog to publish data from this pipeline to. If `target` is specified, tables - in this pipeline are published to a `target` schema inside `catalog` (for example, - `catalog`.`target`.`table`). If `target` is not specified, no data is published to Unity Catalog. + A catalog in Unity Catalog to publish data from this pipeline to. If ``target`` is specified, tables + in this pipeline are published to a ``target`` schema inside ``catalog`` (for example, + ``catalog``.``target``.``table``). If ``target`` is not specified, no data is published to Unity + Catalog. :param channel: str (optional) - DLT Release Channel that specifies which version to use. + SDP Release Channel that specifies which version to use. :param clusters: List[:class:`PipelineCluster`] (optional) Cluster settings for this pipeline deployment. :param configuration: Dict[str,str] (optional) String-String configuration for this pipeline execution. :param continuous: bool (optional) - Whether the pipeline is continuous or triggered. This replaces `trigger`. + Whether the pipeline is continuous or triggered. This replaces ``trigger``. :param deployment: :class:`PipelineDeployment` (optional) Deployment type of this pipeline. :param development: bool (optional) Whether the pipeline is in Development mode. Defaults to false. + :param dry_run: bool (optional) + :param edition: str (optional) Pipeline product edition. :param environment: :class:`PipelinesEnvironment` (optional) Environment specification for this pipeline used to install dependencies. :param event_log: :class:`EventLogSpec` (optional) Event log configuration for this pipeline - :param expected_last_modified: int (optional) - If present, the last-modified time of the pipeline settings before the edit. If the settings were - modified after that time, then the request will fail with a conflict. :param filters: :class:`Filters` (optional) Filters on which Pipeline packages to include in the deployed graph. :param gateway_definition: :class:`IngestionGatewayPipelineDefinition` @@ -3590,10 +5435,15 @@ paths: Friendly identifier for this pipeline. :param notifications: List[:class:`Notifications`] (optional) List of notification settings for this pipeline. + :param parameters: Dict[str,str] (optional) + Key/value map of default parameters to use for pipeline execution. Maximum total size: 10k + characters (JSON format) :param photon: bool (optional) Whether Photon is enabled for this pipeline. :param restart_window: :class:`RestartWindow` (optional) Restart window of this pipeline. + :param rewind_generation_interval: :class:`PeriodicTrigger` (optional) + Interval at which rewind points are generated during pipeline execution. :param root_path: str (optional) Root path for this pipeline. This is used as the root directory when editing the pipeline in the Databricks user interface and it is added to sys.path when executing Python sources during pipeline @@ -3604,26 +5454,23 @@ paths: The default schema (database) where tables are read from or published to. :param serverless: bool (optional) Whether serverless compute is enabled for this pipeline. + :param serverless_compute_id: str (optional) + Serverless compute ID specified by the user for serverless pipelines. :param storage: str (optional) DBFS root directory for storing checkpoints and tables. :param tags: Dict[str,str] (optional) A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline. :param target: str (optional) - Target schema (database) to add tables in this pipeline to. Exactly one of `schema` or `target` must - be specified. To publish to Unity Catalog, also specify `catalog`. This legacy field is deprecated - for pipeline creation in favor of the `schema` field. + Target schema (database) to add tables in this pipeline to. Exactly one of ``schema`` or ``target`` + must be specified. To publish to Unity Catalog, also specify ``catalog``. This legacy field is + deprecated for pipeline creation in favor of the ``schema`` field. :param trigger: :class:`PipelineTrigger` (optional) - Which pipeline trigger to use. Deprecated: Use `continuous` instead. + Which pipeline trigger to use. Deprecated: Use ``continuous`` instead. :param usage_policy_id: str (optional) Usage policy of this pipeline. - parameters: - - name: pipeline_id - in: path - required: true - schema: - type: string - description: Unique identifier for this pipeline. + + :returns: :class:`CreatePipelineResponse` requestBody: content: application/json: @@ -3633,8 +5480,8 @@ paths: allow_duplicate_names: type: boolean description: >- - If false, deployment will fail if name has changed and - conflicts the name of another pipeline. + If false, deployment will fail if name conflicts with that + of another pipeline. budget_policy_id: type: string description: Budget policy of this pipeline. @@ -3642,13 +5489,14 @@ paths: type: string description: >- A catalog in Unity Catalog to publish data from this - pipeline to. If `target` is specified, tables in this - pipeline are published to a `target` schema inside `catalog` - (for example, `catalog`.`target`.`table`). If `target` is - not specified, no data is published to Unity Catalog. + pipeline to. If ``target`` is specified, tables in this + pipeline are published to a ``target`` schema inside + ``catalog`` (for example, ``catalog``.``target``.``table``). + If ``target`` is not specified, no data is published to + Unity Catalog. channel: type: string - description: DLT Release Channel that specifies which version to use. + description: SDP Release Channel that specifies which version to use. clusters: type: array items: @@ -3665,8 +5513,8 @@ paths: type: integer description: >- The maximum number of workers to which the cluster - can scale up when overloaded. `max_workers` must - be strictly greater than `min_workers`. + can scale up when overloaded. ``max_workers`` must + be strictly greater than ``min_workers``. mode: type: string x-enum: @@ -3678,8 +5526,8 @@ paths: resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for - `updates` clusters only. The legacy autoscaling - feature is used for `maintenance` clusters. + ``updates`` clusters only. The legacy autoscaling + feature is used for ``maintenance`` clusters. required: - min_workers - max_workers @@ -3706,21 +5554,21 @@ paths: long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will - be delivered to the destination every `5 mins`. The + be delivered to the destination every ``5 mins``. The destination of driver logs is - `$destination/$clusterId/driver`, while the + ``$destination/$clusterId/driver``, while the destination of executor logs is - `$destination/$clusterId/executor`. + ``$destination/$clusterId/executor``. custom_tags: type: object description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to - `default_tags`. Notes: - Currently, Databricks allows - at most 45 custom tags - Clusters can only reuse cloud - resources if the resources' tags are a subset of the - cluster tags + ``default_tags``. Notes: - Currently, Databricks + allows at most 45 custom tags - Clusters can only + reuse cloud resources if the resources' tags are a + subset of the cluster tags driver_instance_pool_id: type: string description: >- @@ -3733,7 +5581,7 @@ paths: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will - be set as the same value as `node_type_id` defined + be set as the same value as ``node_type_id`` defined above. enable_local_disk_encryption: type: boolean @@ -3752,8 +5600,9 @@ paths: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If - `cluster_log_conf` is specified, init script logs are - sent to `//init_scripts`. + ``cluster_log_conf`` is specified, init script logs + are sent to + ``//init_scripts``. instance_pool_id: type: string description: >- @@ -3763,10 +5612,10 @@ paths: type: string description: >- A label for the cluster specification, either - `default` to configure the default cluster, or - `maintenance` to configure the maintenance cluster. + ``default`` to configure the default cluster, or + ``maintenance`` to configure the maintenance cluster. This field is optional. The default value is - `default`. + ``default``. node_type_id: type: string description: >- @@ -3781,16 +5630,17 @@ paths: type: integer description: >- Number of worker nodes that this cluster should have. - A cluster has one Spark Driver and `num_workers` - Executors for a total of `num_workers` + 1 Spark + A cluster has one Spark Driver and ``num_workers`` + Executors for a total of ``num_workers`` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase - from 5 to 10 as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually + increase from 5 to 10 as the new nodes are + provisioned. policy_id: type: string description: >- @@ -3808,17 +5658,17 @@ paths: An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as - is (i.e., `export X='Y'`) while launching the driver + is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional set of - `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them - to `$SPARK_DAEMON_JAVA_OPTS` as shown in the example - below. This ensures that all default databricks - managed environmental variables are included as well. - Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS - -Dspark.shuffle.service.enabled=true"}` + ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending + them to ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the + example below. This ensures that all default + databricks managed environmental variables are + included as well. Example Spark environment variables: + ``{"SPARK_WORKER_MEMORY": "28000m", + "SPARK_LOCAL_DIRS": "/local_disk0"}`` or + ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` ssh_public_keys: type: array items: @@ -3826,8 +5676,9 @@ paths: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private - keys can be used to login with the user name `ubuntu` - on port `2200`. Up to 10 keys can be specified. + keys can be used to login with the user name + ``ubuntu`` on port ``2200``. Up to 10 keys can be + specified. description: Cluster settings for this pipeline deployment. configuration: type: object @@ -3836,7 +5687,7 @@ paths: type: boolean description: >- Whether the pipeline is continuous or triggered. This - replaces `trigger`. + replaces ``trigger``. deployment: type: object properties: @@ -3845,15 +5696,30 @@ paths: x-enum: - BUNDLE description: >- - The deployment method that manages the pipeline: - - BUNDLE: The pipeline is managed by a + The deployment method that manages the pipeline: + - Databricks Asset Bundle. + - BUNDLE: The pipeline is managed by a Databricks Asset + Bundle. + deployment_id: + type: string + description: >- + ID of the deployment that manages this pipeline. Only + set when ``kind`` is ``BUNDLE``. Used to look up + deployment metadata from the Deployment Metadata + service. metadata_file_path: type: string description: >- The path to the file containing metadata about the deployment. + version_id: + type: string + description: >- + ID of the version of the deployment that produced this + pipeline. Only set when ``kind`` is ``BUNDLE``. + Identifies a specific snapshot of the deployment in the + Deployment Metadata service. required: - kind description: Deployment type of this pipeline. @@ -3862,6 +5728,8 @@ paths: description: >- Whether the pipeline is in Development mode. Defaults to false. + dry_run: + type: boolean edition: type: string description: Pipeline product edition. @@ -3893,7 +5761,7 @@ paths: versions and their included packages, see https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ The value should be a string representing the - environment version number, for example: `"4"`. + environment version number, for example: ``"4"``. description: >- Environment specification for this pipeline used to install dependencies. @@ -3910,12 +5778,6 @@ paths: type: string description: The UC schema the event log is published under. description: Event log configuration for this pipeline - expected_last_modified: - type: integer - description: >- - If present, the last-modified time of the pipeline settings - before the edit. If the settings were modified after that - time, then the request will fail with a conflict. filters: type: object properties: @@ -4049,70 +5911,7592 @@ paths: Days of week in which the window is allowed to happen. If not specified all days of the week - will be used. - description: >- - Days of week in which the window is allowed to - happen If not specified all days of the week will be - used. - time_zone_id: - type: string - description: >- - Time zone id of window. See - https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html - for details. If not specified, UTC will be used. - required: - - start_hour - description: >- - (Optional) A window that specifies a set of time ranges - for snapshot queries in CDC. - ingest_from_uc_foreign_catalog: - type: boolean - description: >- - Immutable. If set to true, the pipeline will ingest - tables from the UC foreign catalogs directly without the - need to specify a UC connection or ingestion gateway. - The `source_catalog` fields in objects of - IngestionConfig are interpreted as the UC foreign - catalogs to ingest from. - ingestion_gateway_id: - type: string - description: >- - Identifier for the gateway that is used by this - ingestion pipeline to communicate with the source - database. This is used with CDC connectors to databases - like SQL Server using a gateway pipeline (connector_type - = CDC). Under certain conditions, this can be replaced - with connection_name to change the connector to Combined - Cdc Managed Ingestion Pipeline. - netsuite_jar_path: - type: string - description: >- - Netsuite only configuration. When the field is set for a - netsuite connector, the jar stored in the field will be - validated and added to the classpath of pipeline's - cluster. - objects: - type: array - items: - type: object - properties: - report: - type: object - properties: - source_url: - type: string - destination_catalog: - type: string - description: Required. Destination catalog to store table. - destination_schema: - type: string - description: Required. Destination schema to store table. - destination_table: + will be used. + description: >- + Days of week in which the window is allowed to + happen If not specified all days of the week will be + used. + time_zone_id: + type: string + description: >- + Time zone id of window. See + https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html + for details. If not specified, UTC will be used. + required: + - start_hour + description: >- + (Optional) A window that specifies a set of time ranges + for snapshot queries in CDC. + ingest_from_uc_foreign_catalog: + type: boolean + description: >- + Immutable. If set to true, the pipeline will ingest + tables from the UC foreign catalogs directly without the + need to specify a UC connection or ingestion gateway. + The ``source_catalog`` fields in objects of + IngestionConfig are interpreted as the UC foreign + catalogs to ingest from. + ingestion_gateway_id: + type: string + description: >- + Identifier for the gateway that is used by this + ingestion pipeline to communicate with the source + database. This is used with CDC connectors to databases + like SQL Server using a gateway pipeline (connector_type + = CDC). Under certain conditions, this can be replaced + with connection_name to change the connector to Combined + Cdc Managed Ingestion Pipeline. + netsuite_jar_path: + type: string + description: >- + Netsuite only configuration. When the field is set for a + netsuite connector, the jar stored in the field will be + validated and added to the classpath of pipeline's + cluster. + objects: + type: array + items: + type: object + properties: + report: + type: object + properties: + source_url: + type: string + destination_catalog: + type: string + description: Required. Destination catalog to store table. + destination_schema: + type: string + description: Required. Destination schema to store table. + destination_table: + type: string + description: >- + Required. Destination table name. The pipeline + fails if a table with that name already + exists. + table_configuration: + type: object + properties: + auto_full_refresh_policy: + type: object + properties: + enabled: + type: boolean + description: >- + (Required, Mutable) Whether to enable + auto full refresh or not. + min_interval_hours: + type: integer + description: >- + (Optional, Mutable) Specify the minimum + interval in hours between the timestamp + at which a table was last full refreshed + and the current timestamp for triggering + auto full If unspecified and + autoFullRefresh is enabled then by + default min_interval_hours is 24 hours. + required: + - enabled + description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + exclude_columns: + type: array + items: + type: string + description: >- + A list of column names to be excluded for + the ingestion. When not specified, + include_columns fully controls what + columns to be ingested. When specified, + all other columns including future ones + will be automatically included for + ingestion. This field in mutually + exclusive with ``include_columns``. + include_columns: + type: array + items: + type: string + description: >- + A list of column names to be included for + the ingestion. When not specified, all + columns except ones in exclude_columns + will be included. Future columns will be + automatically included. When specified, + all other future columns will be + automatically excluded from ingestion. + This field in mutually exclusive with + ``exclude_columns``. + primary_keys: + type: array + items: + type: string + description: >- + The primary key of the table used to apply + changes. + query_based_connector_config: + type: object + properties: + cursor_columns: + type: array + items: + type: string + description: >- + The names of the monotonically + increasing columns in the source table + that are used to enable the table to be + read and ingested incrementally through + structured streaming. The columns are + allowed to have repeated values but have + to be non-decreasing. If the source data + is merged into the destination (e.g., + using SCD Type 1 or Type 2), these + columns will implicitly define the + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. + deletion_condition: + type: string + description: >- + Specifies a SQL WHERE condition that + specifies that the source row has been + deleted. This is sometimes referred to + as "soft-deletes". For example: + "Operation = 'DELETE'" or "is_deleted = + true". This field is orthogonal to + ``hard_deletion_sync_interval_in_seconds``, + one for soft-deletes and the other for + hard-deletes. See also the + hard_deletion_sync_min_interval_in_seconds + field for handling of "hard deletes" + where the source rows are physically + removed from the table. + hard_deletion_sync_min_interval_in_seconds: + type: integer + description: >- + Specifies the minimum interval (in + seconds) between snapshots on primary + keys for detecting and synchronizing + hard deletions—i.e., rows that have been + physically removed from the source + table. This interval acts as a lower + bound. If ingestion runs less frequently + than this value, hard deletion + synchronization will align with the + actual ingestion frequency instead of + happening more often. If not set, hard + deletion synchronization via snapshots + is disabled. This field is mutable and + can be updated without triggering a full + snapshot. + description: >- + Configurations that are only applicable + for query-based ingestion connectors. + row_filter: + type: string + description: >- + (Optional, Immutable) The row filter + condition to be applied to the table. It + must not contain the WHERE keyword, only + the actual filter condition. It must be in + DBSQL format. + salesforce_include_formula_fields: + type: boolean + description: >- + If true, formula fields defined in the + table are included in the ingestion. This + setting is only valid for the Salesforce + connector + scd_type: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: The SCD type to use to ingest the table. + sequence_by: + type: array + items: + type: string + description: >- + The column names specifying the logical + order of events in the source data. Spark + Declarative Pipelines uses this sequencing + to handle change events that arrive out of + order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. + workday_report_parameters: + type: object + properties: + incremental: + type: boolean + parameters: + type: object + description: >- + Parameters for the Workday report. Each + key represents the parameter name (e.g., + "start_date", "end_date"), and the + corresponding value is a SQL-like + expression used to compute the parameter + value at runtime. Example: { + "start_date": "{ + coalesce(current_offset(), + date(\"2025-02-01\")) }", "end_date": "{ + current_date() - INTERVAL 1 DAY }" } + report_parameters: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: >- + Value for the report parameter. Possible + values it can take are these sql + functions: 1. coalesce(current_offset(), + date("YYYY-MM-DD")) -> if + current_offset() is null, then the + passed date, else current_offset() 2. + current_date() 3. + date_sub(current_date(), x) -> subtract + x (some non-negative integer) days from + current date + description: >- + (Optional) Additional custom parameters + for Workday Report This field is + deprecated and should not be used. Use + ``parameters`` instead. + description: >- + (Optional) Additional custom parameters + for Workday Report + description: >- + Configuration settings to control the + ingestion of tables. These settings override + the table_configuration defined in the + IngestionPipelineDefinition object. + required: + - source_url + - destination_catalog + - destination_schema + schema: + type: object + properties: + source_schema: + type: string + destination_catalog: + type: string + description: Required. Destination catalog to store tables. + destination_schema: + type: string + description: >- + Required. Destination schema to store tables + in. Tables with the same name as the source + tables are created in this destination schema. + The pipeline fails If a table with the same + name already exists. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use + connector_options.confluence_options instead. + connector_options: + type: object + properties: + akamai_options: + type: object + properties: + config_ids: + type: array + items: + type: integer + description: >- + (Required) List of Akamai configuration + IDs to ingest data from. A config ID is + a unique identifier for a security + configuration used to generate WAF logs. + description: Akamai specific options for ingestion + api_source_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the API Source connector. + description: >- + Connector-specific options for API Source + connectors. + bing_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + report_type: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: >- + (Required) Microsoft Advertising report + type (e.g. CampaignPerformance, + KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the + report, given as Microsoft Advertising + column enum values for the selected + report type (e.g. AccountId, + CampaignName, TimePeriod, Impressions, + Clicks, Spend). Required key columns are + always included. Available columns are + report-type-specific. For the full list + supported by a given report type, see + its column value set (substitute the + report name, e.g. campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: >- + (Optional) Time granularity for report + aggregation. Defaults to DAILY. + required: + - report_type + - columns + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Bing Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + during incremental sync to capture + late-arriving conversion and attribution + data. If not specified, defaults to 7 + days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. If not + specified, defaults to 2 years before + today. + description: >- + Bing Ads specific options for ingestion + pipelines. + community_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the community connector. + description: >- + Options for community connectors with + arbitrary configuration. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + (Optional) Spaces to filter Confluence + data on + description: Confluence specific options for ingestion + gdrive_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + scope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE + description: >- + The scope of the Google Drive ingestion. + **ONLY ONE** of scope or url must be + specified. If scope is specified, we + will ingest files from the scope. If url + is specified, we will ingest files from + the url. + url: + type: string + description: Google Drive URL. + github_options: + type: object + properties: + repository_branch_selections: + type: array + items: + type: object + properties: + repository_id: + type: string + description: >- + (Required) GitHub repository id + (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this + repository, e.g. ["main", "dev"]. + required: + - repository_id + description: >- + A repository and the branches to ingest + for it. + description: >- + (Optional) Branches to ingest per + repository. A repository not listed here + is ingested on its default branch. + description: GitHub-specific ingestion options. + google_ads_options: + type: object + properties: + manager_account_id: + type: string + description: >- + (Optional at this level) Manager Account + ID (also called MCC Account ID) used to + list and access customer accounts under + this manager account. Overrides + GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query + (e.g. "ad_group_ad", "keyword_view", + "search_term_view"). Must be a resource + that has metrics. Values are validated + against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. + "metrics.clicks", + "metrics.cost_micros"). Multiple values + are joined into the GAQL SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in + fully-qualified GAQL form (e.g. + "ad_group_ad.ad.id", + "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select + (e.g. "segments.date", + "segments.device"). Must include at + least one of segments.date, + segments.week, or segments.month — that + segment is used as the incremental + cursor for the table. + required: + - resource + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Google Ads custom report: + the connector synthesizes a GAQL query + from the resource, fields, segments, and + metrics specified here. When unset, the + table must match one of the connector's + prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables to capture + late-arriving data. If not specified, + defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years of + historical data. + required: + - manager_account_id + description: >- + Google Ads specific options for ingestion + (object-level). When set, these values + override the + corresponding fields in GoogleAdsConfig (source_configurations). + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + (Optional) Projects to filter Jira data + on + use_audit_logs: + type: boolean + description: >- + (Optional) Whether the pipeline reads + Jira's audit log to detect changes for + the tables it ingests. Reading the audit + log requires a paid Jira plan and + appropriate admin permissions. For + example, the issues table uses the audit + log to detect and propagate deletes: + when true, rows deleted in Jira are + removed from the destination table; when + false, the destination retains all + previously ingested rows and only + reflects inserts and updates. If not + specified, defaults to false. + description: Jira specific options for ingestion + kafka_options: + type: object + properties: + client_config: + type: object + key_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + key. If not specified, the key is left + as raw bytes. + max_offsets_per_trigger: + type: integer + description: >- + Internal option to control the maximum + number of offsets to process per + trigger. + max_partitions: + type: string + description: >- + (Optional) Controls how Kafka offset + ranges are bin-packed into Spark input + partitions. Defaults to "auto" when + unset, which lets the engine choose the + input partition count from measured + throughput. Set to "none" to disable + bin-packing, so each offset range maps + to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when + no checkpoint exists. Valid values: + "latest" and "earliest". Defaults to + "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to + matching topics. Only one of topics or + topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of + topics or topic_pattern must be + specified. + value_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + value. If not specified, the value is + left as raw bytes. + linkedin_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + finder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + (Required) adAnalytics finder. See + LinkedInAdsFinder. + entity_granularity: + type: array + items: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the + report. Open vocabulary (not an enum): + the valid set is large (~100) and + evolves with the LinkedIn adAnalytics + API, so values are passed through + verbatim. If empty, a pivot-safe default + core set is ingested: impressions, + clicks, costInLocalCurrency, + externalWebsiteConversions (valid for + every pivot). Ignored for + attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics + struct). + time_granularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + (Optional) Time aggregation. Defaults to + DAILY when unspecified. Used by + analytics/statistics; ignored for + attributedRevenueMetrics. + required: + - finder + description: >- + (Optional) Custom report definition. + Only valid on a table object. When set, + the table is synthesized from + /rest/adAnalytics using the finder, + pivots, time granularity and metrics + here. When unset, the table must match + one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Days to look back during + incremental sync for late-arriving data. + If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables, YYYY-MM-DD. + Earliest date from which to sync + historical data; overrides the default + when set. For finder + attributedRevenueMetrics, this must be + between 30 and 366 days before today. If + not specified, defaults to 1 year of + history. + description: >- + LinkedIn Ads specific options for + ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + marketo_options: + type: object + properties: + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + description: Marketo specific options for ingestion + meta_ads_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) + Action attribution windows for insights + reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) + Timing used to report action statistics + (impression, conversion, mixed, or + lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit + data during sync to capture updated + conversion data from the API, shared by + prebuilt and custom reports. + custom_report_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional) Action attribution windows + for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional) Action breakdowns to + configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action + statistics (impression, conversion, + mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdowns to configure for + data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull + (account, ad, adset, campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to + aggregate statistics (all_days, monthly + or number of days) + description: >- + (Optional) Per-table custom report + definition. When set, defines the shape + of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). + Supersedes the deprecated flat + report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.level) Granularity + of data to pull (account, ad, adset, + campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd + format (e.g. 2025-01-15). Data added + after this date will be ingested, shared + by prebuilt and custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.time_increment) + Value in string by which to aggregate + statistics (can take all_days, monthly + or number of days) + description: >- + Meta Marketing (Meta Ads) specific options + for ingestion + outlook_options: + type: object + properties: + attachment_mode: + type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY + description: >- + (Optional) Controls which attachments to + ingest. If not specified, defaults to + ALL. + body_format: + type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN + description: >- + (Optional) Defines how the body_content + column is populated. TEXT_HTML: + Preserves full formatting, links, and + styling. TEXT_PLAIN: Converts body to + plain text. Recommended for AI/RAG + pipelines to reduce token usage and + noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to + include in the sync. If not specified, + all folders will be synced. Examples: + Inbox, Sent Items, Custom_Folder Filter + semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync + (e.g. mailbox email addresses or + identifiers). If not specified, all + accessible mailboxes are ingested. + Filter semantics: OR between different + mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender + address. Uses exact email match. + Examples: user@vendor.com, + alerts@system.io, noreply@company.com If + not specified, emails from all senders + will be synced. Filter semantics: OR + between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject + line. Values ending with "*" use prefix + match (subject starts with the part + before "*"); otherwise substring match + (subject contains the value). Examples: + "Invoice" (substring), "Re:*" (prefix), + "Support Ticket", "URGENT*" If not + specified, emails with all subjects will + be synced. Filter semantics: OR between + different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. Format: + YYYY-MM-DD (e.g., 2024-01-01) This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + subject_filter: + type: array + items: + type: string + description: >- + Deprecated. Use include_subjects + instead. + description: Outlook specific options for ingestion + rabbitmq_options: + type: object + properties: + queue: + type: string + description: >- + (Required) RabbitMQ queue name to + consume from. + required: + - queue + description: >- + RabbitMQ specific options for ingestion. + Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + reddit_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group + report data by. Examples: CAMPAIGN_ID, + DATE, COUNTRY, REGION, AD_ID. Must + include at least one time dimension + (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the + report (maps to the Reddit Ads API + ``fields`` parameter). Examples: + IMPRESSIONS, CLICKS, SPEND, CPC, CTR. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Reddit Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. If not + specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years + ago. + description: Reddit Ads specific options for ingestion + sharepoint_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + description: >- + (Optional) File ingestion options for + processing files. + url: + type: string + description: Required. The SharePoint URL. + smartsheet_options: + type: object + properties: + enforce_schema: + type: boolean + description: >- + (Optional) When true, maps each column + to its Smartsheet-declared type + (Text/Number/Date/ Checkbox/etc.). Cells + that do not conform to the declared type + are set to NULL. When false, all columns + land as STRING. Use false for sheets + with irregular data or columns that + frequently violate their own declared + type. If not specified, defaults to + true. + description: Smartsheet specific options for ingestion + tiktok_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + (Optional) Data level for the report. If + not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the + report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", + "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the + report (e.g. "spend", "impressions", + "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime + metrics (all-time aggregated data). When + true, the report returns all-time data. + If not specified, defaults to false. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + (Optional) Report type for the TikTok + Ads API. If not specified, defaults to + BASIC. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined TikTok Ads custom report: + the connector synthesizes a report + request from the dimensions, metrics, + report type, and data level specified + here. Supersedes the deprecated + top-level + dimensions/metrics/report_type/ + data_level/query_lifetime fields above. + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + Deprecated. Use + custom_report_options.data_level + instead. + dimensions: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.dimensions + instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. + metrics: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: >- + Deprecated. Use + custom_report_options.query_lifetime + instead. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + Deprecated. Use + custom_report_options.report_type + instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + description: TikTok Ads specific options for ingestion + zendesk_support_options: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD + format for the initial sync. This + determines the earliest date from which + to sync historical data. + description: >- + Zendesk Support specific options for + ingestion + description: (Optional) Source Specific Connector Options + fanout_options: + type: object + properties: + fanout_by: + type: string + description: >- + Column path or SQL expression whose value + determines the destination table. Supports + dotted paths (e.g. "value.event_name") and + expressions (e.g. + "value:event_name::string"). + transforms: + type: array + items: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + Specifies how to transform binary data + into structured data. + description: >- + Optional transforms applied to each + route's DataFrame before writing to the + destination table. + description: >- + Fanout options for multi-table routing from + streaming sources. When set, records are + routed to destination tables based on a + per-record routing key. The key value becomes + the table name: + {destination_catalog}.{destination_schema}.{key_value}. + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use connector_options.jira_options + instead. + source_catalog: + type: string + description: >- + The source catalog name. Might be optional + depending on the type of source. + table_configuration: + type: object + properties: + auto_full_refresh_policy: + type: object + properties: + enabled: + type: boolean + description: >- + (Required, Mutable) Whether to enable + auto full refresh or not. + min_interval_hours: + type: integer + description: >- + (Optional, Mutable) Specify the minimum + interval in hours between the timestamp + at which a table was last full refreshed + and the current timestamp for triggering + auto full If unspecified and + autoFullRefresh is enabled then by + default min_interval_hours is 24 hours. + required: + - enabled + description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + exclude_columns: + type: array + items: + type: string + description: >- + A list of column names to be excluded for + the ingestion. When not specified, + include_columns fully controls what + columns to be ingested. When specified, + all other columns including future ones + will be automatically included for + ingestion. This field in mutually + exclusive with ``include_columns``. + include_columns: + type: array + items: + type: string + description: >- + A list of column names to be included for + the ingestion. When not specified, all + columns except ones in exclude_columns + will be included. Future columns will be + automatically included. When specified, + all other future columns will be + automatically excluded from ingestion. + This field in mutually exclusive with + ``exclude_columns``. + primary_keys: + type: array + items: + type: string + description: >- + The primary key of the table used to apply + changes. + query_based_connector_config: + type: object + properties: + cursor_columns: + type: array + items: + type: string + description: >- + The names of the monotonically + increasing columns in the source table + that are used to enable the table to be + read and ingested incrementally through + structured streaming. The columns are + allowed to have repeated values but have + to be non-decreasing. If the source data + is merged into the destination (e.g., + using SCD Type 1 or Type 2), these + columns will implicitly define the + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. + deletion_condition: + type: string + description: >- + Specifies a SQL WHERE condition that + specifies that the source row has been + deleted. This is sometimes referred to + as "soft-deletes". For example: + "Operation = 'DELETE'" or "is_deleted = + true". This field is orthogonal to + ``hard_deletion_sync_interval_in_seconds``, + one for soft-deletes and the other for + hard-deletes. See also the + hard_deletion_sync_min_interval_in_seconds + field for handling of "hard deletes" + where the source rows are physically + removed from the table. + hard_deletion_sync_min_interval_in_seconds: + type: integer + description: >- + Specifies the minimum interval (in + seconds) between snapshots on primary + keys for detecting and synchronizing + hard deletions—i.e., rows that have been + physically removed from the source + table. This interval acts as a lower + bound. If ingestion runs less frequently + than this value, hard deletion + synchronization will align with the + actual ingestion frequency instead of + happening more often. If not set, hard + deletion synchronization via snapshots + is disabled. This field is mutable and + can be updated without triggering a full + snapshot. + description: >- + Configurations that are only applicable + for query-based ingestion connectors. + row_filter: + type: string + description: >- + (Optional, Immutable) The row filter + condition to be applied to the table. It + must not contain the WHERE keyword, only + the actual filter condition. It must be in + DBSQL format. + salesforce_include_formula_fields: + type: boolean + description: >- + If true, formula fields defined in the + table are included in the ingestion. This + setting is only valid for the Salesforce + connector + scd_type: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: The SCD type to use to ingest the table. + sequence_by: + type: array + items: + type: string + description: >- + The column names specifying the logical + order of events in the source data. Spark + Declarative Pipelines uses this sequencing + to handle change events that arrive out of + order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. + workday_report_parameters: + type: object + properties: + incremental: + type: boolean + parameters: + type: object + description: >- + Parameters for the Workday report. Each + key represents the parameter name (e.g., + "start_date", "end_date"), and the + corresponding value is a SQL-like + expression used to compute the parameter + value at runtime. Example: { + "start_date": "{ + coalesce(current_offset(), + date(\"2025-02-01\")) }", "end_date": "{ + current_date() - INTERVAL 1 DAY }" } + report_parameters: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: >- + Value for the report parameter. Possible + values it can take are these sql + functions: 1. coalesce(current_offset(), + date("YYYY-MM-DD")) -> if + current_offset() is null, then the + passed date, else current_offset() 2. + current_date() 3. + date_sub(current_date(), x) -> subtract + x (some non-negative integer) days from + current date + description: >- + (Optional) Additional custom parameters + for Workday Report This field is + deprecated and should not be used. Use + ``parameters`` instead. + description: >- + (Optional) Additional custom parameters + for Workday Report + description: >- + Configuration settings to control the + ingestion of tables. These settings are + applied to all tables in this schema and + override the table_configuration defined in + the IngestionPipelineDefinition object. + required: + - source_schema + - destination_catalog + - destination_schema + description: Select all tables from a specific source schema. + table: + type: object + properties: + source_table: + type: string + destination_catalog: + type: string + description: Required. Destination catalog to store table. + destination_schema: + type: string + description: Required. Destination schema to store table. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use + connector_options.confluence_options instead. + connector_options: + type: object + properties: + akamai_options: + type: object + properties: + config_ids: + type: array + items: + type: integer + description: >- + (Required) List of Akamai configuration + IDs to ingest data from. A config ID is + a unique identifier for a security + configuration used to generate WAF logs. + description: Akamai specific options for ingestion + api_source_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the API Source connector. + description: >- + Connector-specific options for API Source + connectors. + bing_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + report_type: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: >- + (Required) Microsoft Advertising report + type (e.g. CampaignPerformance, + KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the + report, given as Microsoft Advertising + column enum values for the selected + report type (e.g. AccountId, + CampaignName, TimePeriod, Impressions, + Clicks, Spend). Required key columns are + always included. Available columns are + report-type-specific. For the full list + supported by a given report type, see + its column value set (substitute the + report name, e.g. campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: >- + (Optional) Time granularity for report + aggregation. Defaults to DAILY. + required: + - report_type + - columns + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Bing Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + during incremental sync to capture + late-arriving conversion and attribution + data. If not specified, defaults to 7 + days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. If not + specified, defaults to 2 years before + today. + description: >- + Bing Ads specific options for ingestion + pipelines. + community_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the community connector. + description: >- + Options for community connectors with + arbitrary configuration. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + (Optional) Spaces to filter Confluence + data on + description: Confluence specific options for ingestion + gdrive_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + scope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE + description: >- + The scope of the Google Drive ingestion. + **ONLY ONE** of scope or url must be + specified. If scope is specified, we + will ingest files from the scope. If url + is specified, we will ingest files from + the url. + url: + type: string + description: Google Drive URL. + github_options: + type: object + properties: + repository_branch_selections: + type: array + items: + type: object + properties: + repository_id: + type: string + description: >- + (Required) GitHub repository id + (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this + repository, e.g. ["main", "dev"]. + required: + - repository_id + description: >- + A repository and the branches to ingest + for it. + description: >- + (Optional) Branches to ingest per + repository. A repository not listed here + is ingested on its default branch. + description: GitHub-specific ingestion options. + google_ads_options: + type: object + properties: + manager_account_id: + type: string + description: >- + (Optional at this level) Manager Account + ID (also called MCC Account ID) used to + list and access customer accounts under + this manager account. Overrides + GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query + (e.g. "ad_group_ad", "keyword_view", + "search_term_view"). Must be a resource + that has metrics. Values are validated + against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. + "metrics.clicks", + "metrics.cost_micros"). Multiple values + are joined into the GAQL SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in + fully-qualified GAQL form (e.g. + "ad_group_ad.ad.id", + "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select + (e.g. "segments.date", + "segments.device"). Must include at + least one of segments.date, + segments.week, or segments.month — that + segment is used as the incremental + cursor for the table. + required: + - resource + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Google Ads custom report: + the connector synthesizes a GAQL query + from the resource, fields, segments, and + metrics specified here. When unset, the + table must match one of the connector's + prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables to capture + late-arriving data. If not specified, + defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years of + historical data. + required: + - manager_account_id + description: >- + Google Ads specific options for ingestion + (object-level). When set, these values + override the + corresponding fields in GoogleAdsConfig (source_configurations). + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + (Optional) Projects to filter Jira data + on + use_audit_logs: + type: boolean + description: >- + (Optional) Whether the pipeline reads + Jira's audit log to detect changes for + the tables it ingests. Reading the audit + log requires a paid Jira plan and + appropriate admin permissions. For + example, the issues table uses the audit + log to detect and propagate deletes: + when true, rows deleted in Jira are + removed from the destination table; when + false, the destination retains all + previously ingested rows and only + reflects inserts and updates. If not + specified, defaults to false. + description: Jira specific options for ingestion + kafka_options: + type: object + properties: + client_config: + type: object + key_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + key. If not specified, the key is left + as raw bytes. + max_offsets_per_trigger: + type: integer + description: >- + Internal option to control the maximum + number of offsets to process per + trigger. + max_partitions: + type: string + description: >- + (Optional) Controls how Kafka offset + ranges are bin-packed into Spark input + partitions. Defaults to "auto" when + unset, which lets the engine choose the + input partition count from measured + throughput. Set to "none" to disable + bin-packing, so each offset range maps + to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when + no checkpoint exists. Valid values: + "latest" and "earliest". Defaults to + "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to + matching topics. Only one of topics or + topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of + topics or topic_pattern must be + specified. + value_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + value. If not specified, the value is + left as raw bytes. + linkedin_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + finder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + (Required) adAnalytics finder. See + LinkedInAdsFinder. + entity_granularity: + type: array + items: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the + report. Open vocabulary (not an enum): + the valid set is large (~100) and + evolves with the LinkedIn adAnalytics + API, so values are passed through + verbatim. If empty, a pivot-safe default + core set is ingested: impressions, + clicks, costInLocalCurrency, + externalWebsiteConversions (valid for + every pivot). Ignored for + attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics + struct). + time_granularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + (Optional) Time aggregation. Defaults to + DAILY when unspecified. Used by + analytics/statistics; ignored for + attributedRevenueMetrics. + required: + - finder + description: >- + (Optional) Custom report definition. + Only valid on a table object. When set, + the table is synthesized from + /rest/adAnalytics using the finder, + pivots, time granularity and metrics + here. When unset, the table must match + one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Days to look back during + incremental sync for late-arriving data. + If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables, YYYY-MM-DD. + Earliest date from which to sync + historical data; overrides the default + when set. For finder + attributedRevenueMetrics, this must be + between 30 and 366 days before today. If + not specified, defaults to 1 year of + history. + description: >- + LinkedIn Ads specific options for + ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + marketo_options: + type: object + properties: + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + description: Marketo specific options for ingestion + meta_ads_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) + Action attribution windows for insights + reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) + Timing used to report action statistics + (impression, conversion, mixed, or + lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit + data during sync to capture updated + conversion data from the API, shared by + prebuilt and custom reports. + custom_report_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional) Action attribution windows + for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional) Action breakdowns to + configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action + statistics (impression, conversion, + mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdowns to configure for + data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull + (account, ad, adset, campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to + aggregate statistics (all_days, monthly + or number of days) + description: >- + (Optional) Per-table custom report + definition. When set, defines the shape + of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). + Supersedes the deprecated flat + report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.level) Granularity + of data to pull (account, ad, adset, + campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd + format (e.g. 2025-01-15). Data added + after this date will be ingested, shared + by prebuilt and custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.time_increment) + Value in string by which to aggregate + statistics (can take all_days, monthly + or number of days) + description: >- + Meta Marketing (Meta Ads) specific options + for ingestion + outlook_options: + type: object + properties: + attachment_mode: + type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY + description: >- + (Optional) Controls which attachments to + ingest. If not specified, defaults to + ALL. + body_format: + type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN + description: >- + (Optional) Defines how the body_content + column is populated. TEXT_HTML: + Preserves full formatting, links, and + styling. TEXT_PLAIN: Converts body to + plain text. Recommended for AI/RAG + pipelines to reduce token usage and + noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to + include in the sync. If not specified, + all folders will be synced. Examples: + Inbox, Sent Items, Custom_Folder Filter + semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync + (e.g. mailbox email addresses or + identifiers). If not specified, all + accessible mailboxes are ingested. + Filter semantics: OR between different + mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender + address. Uses exact email match. + Examples: user@vendor.com, + alerts@system.io, noreply@company.com If + not specified, emails from all senders + will be synced. Filter semantics: OR + between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject + line. Values ending with "*" use prefix + match (subject starts with the part + before "*"); otherwise substring match + (subject contains the value). Examples: + "Invoice" (substring), "Re:*" (prefix), + "Support Ticket", "URGENT*" If not + specified, emails with all subjects will + be synced. Filter semantics: OR between + different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. Format: + YYYY-MM-DD (e.g., 2024-01-01) This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + subject_filter: + type: array + items: + type: string + description: >- + Deprecated. Use include_subjects + instead. + description: Outlook specific options for ingestion + rabbitmq_options: + type: object + properties: + queue: + type: string + description: >- + (Required) RabbitMQ queue name to + consume from. + required: + - queue + description: >- + RabbitMQ specific options for ingestion. + Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + reddit_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group + report data by. Examples: CAMPAIGN_ID, + DATE, COUNTRY, REGION, AD_ID. Must + include at least one time dimension + (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the + report (maps to the Reddit Ads API + ``fields`` parameter). Examples: + IMPRESSIONS, CLICKS, SPEND, CPC, CTR. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Reddit Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. If not + specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years + ago. + description: Reddit Ads specific options for ingestion + sharepoint_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + description: >- + (Optional) File ingestion options for + processing files. + url: + type: string + description: Required. The SharePoint URL. + smartsheet_options: + type: object + properties: + enforce_schema: + type: boolean + description: >- + (Optional) When true, maps each column + to its Smartsheet-declared type + (Text/Number/Date/ Checkbox/etc.). Cells + that do not conform to the declared type + are set to NULL. When false, all columns + land as STRING. Use false for sheets + with irregular data or columns that + frequently violate their own declared + type. If not specified, defaults to + true. + description: Smartsheet specific options for ingestion + tiktok_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + (Optional) Data level for the report. If + not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the + report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", + "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the + report (e.g. "spend", "impressions", + "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime + metrics (all-time aggregated data). When + true, the report returns all-time data. + If not specified, defaults to false. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + (Optional) Report type for the TikTok + Ads API. If not specified, defaults to + BASIC. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined TikTok Ads custom report: + the connector synthesizes a report + request from the dimensions, metrics, + report type, and data level specified + here. Supersedes the deprecated + top-level + dimensions/metrics/report_type/ + data_level/query_lifetime fields above. + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + Deprecated. Use + custom_report_options.data_level + instead. + dimensions: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.dimensions + instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. + metrics: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: >- + Deprecated. Use + custom_report_options.query_lifetime + instead. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + Deprecated. Use + custom_report_options.report_type + instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + description: TikTok Ads specific options for ingestion + zendesk_support_options: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD + format for the initial sync. This + determines the earliest date from which + to sync historical data. + description: >- + Zendesk Support specific options for + ingestion + description: (Optional) Source Specific Connector Options + destination_table: + type: string + description: >- + Optional. Destination table name. The pipeline + fails if a table with that name already + exists. If not set, the source table name is + used. + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use connector_options.jira_options + instead. + source_catalog: + type: string + description: >- + Source catalog name. Might be optional + depending on the type of source. + source_schema: + type: string + description: >- + Schema name in the source database. Might be + optional depending on the type of source. + table_configuration: + type: object + properties: + auto_full_refresh_policy: + type: object + properties: + enabled: + type: boolean + description: >- + (Required, Mutable) Whether to enable + auto full refresh or not. + min_interval_hours: + type: integer + description: >- + (Optional, Mutable) Specify the minimum + interval in hours between the timestamp + at which a table was last full refreshed + and the current timestamp for triggering + auto full If unspecified and + autoFullRefresh is enabled then by + default min_interval_hours is 24 hours. + required: + - enabled + description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + exclude_columns: + type: array + items: + type: string + description: >- + A list of column names to be excluded for + the ingestion. When not specified, + include_columns fully controls what + columns to be ingested. When specified, + all other columns including future ones + will be automatically included for + ingestion. This field in mutually + exclusive with ``include_columns``. + include_columns: + type: array + items: + type: string + description: >- + A list of column names to be included for + the ingestion. When not specified, all + columns except ones in exclude_columns + will be included. Future columns will be + automatically included. When specified, + all other future columns will be + automatically excluded from ingestion. + This field in mutually exclusive with + ``exclude_columns``. + primary_keys: + type: array + items: + type: string + description: >- + The primary key of the table used to apply + changes. + query_based_connector_config: + type: object + properties: + cursor_columns: + type: array + items: + type: string + description: >- + The names of the monotonically + increasing columns in the source table + that are used to enable the table to be + read and ingested incrementally through + structured streaming. The columns are + allowed to have repeated values but have + to be non-decreasing. If the source data + is merged into the destination (e.g., + using SCD Type 1 or Type 2), these + columns will implicitly define the + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. + deletion_condition: + type: string + description: >- + Specifies a SQL WHERE condition that + specifies that the source row has been + deleted. This is sometimes referred to + as "soft-deletes". For example: + "Operation = 'DELETE'" or "is_deleted = + true". This field is orthogonal to + ``hard_deletion_sync_interval_in_seconds``, + one for soft-deletes and the other for + hard-deletes. See also the + hard_deletion_sync_min_interval_in_seconds + field for handling of "hard deletes" + where the source rows are physically + removed from the table. + hard_deletion_sync_min_interval_in_seconds: + type: integer + description: >- + Specifies the minimum interval (in + seconds) between snapshots on primary + keys for detecting and synchronizing + hard deletions—i.e., rows that have been + physically removed from the source + table. This interval acts as a lower + bound. If ingestion runs less frequently + than this value, hard deletion + synchronization will align with the + actual ingestion frequency instead of + happening more often. If not set, hard + deletion synchronization via snapshots + is disabled. This field is mutable and + can be updated without triggering a full + snapshot. + description: >- + Configurations that are only applicable + for query-based ingestion connectors. + row_filter: + type: string + description: >- + (Optional, Immutable) The row filter + condition to be applied to the table. It + must not contain the WHERE keyword, only + the actual filter condition. It must be in + DBSQL format. + salesforce_include_formula_fields: + type: boolean + description: >- + If true, formula fields defined in the + table are included in the ingestion. This + setting is only valid for the Salesforce + connector + scd_type: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: The SCD type to use to ingest the table. + sequence_by: + type: array + items: + type: string + description: >- + The column names specifying the logical + order of events in the source data. Spark + Declarative Pipelines uses this sequencing + to handle change events that arrive out of + order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. + workday_report_parameters: + type: object + properties: + incremental: + type: boolean + parameters: + type: object + description: >- + Parameters for the Workday report. Each + key represents the parameter name (e.g., + "start_date", "end_date"), and the + corresponding value is a SQL-like + expression used to compute the parameter + value at runtime. Example: { + "start_date": "{ + coalesce(current_offset(), + date(\"2025-02-01\")) }", "end_date": "{ + current_date() - INTERVAL 1 DAY }" } + report_parameters: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: >- + Value for the report parameter. Possible + values it can take are these sql + functions: 1. coalesce(current_offset(), + date("YYYY-MM-DD")) -> if + current_offset() is null, then the + passed date, else current_offset() 2. + current_date() 3. + date_sub(current_date(), x) -> subtract + x (some non-negative integer) days from + current date + description: >- + (Optional) Additional custom parameters + for Workday Report This field is + deprecated and should not be used. Use + ``parameters`` instead. + description: >- + (Optional) Additional custom parameters + for Workday Report + description: >- + Configuration settings to control the + ingestion of tables. These settings override + the table_configuration defined in the + IngestionPipelineDefinition object and the + SchemaSpec. + required: + - source_table + - destination_catalog + - destination_schema + description: Select a specific source table. + description: >- + Required. Settings specifying tables to replicate and + the destination for the replicated tables. + source_configurations: + type: array + items: + type: object + properties: + api_source_connector_config: + type: object + properties: + configs: + type: object + description: >- + Arbitrary key-value configuration values for + the API Source connector. + description: >- + Top-level configuration for API Source connectors + with arbitrary configuration. + catalog: + type: object + properties: + postgres: + type: object + properties: + slot_config: + type: object + properties: + publication_name: + type: string + description: >- + The name of the publication to use for + the Postgres source + slot_name: + type: string + description: >- + The name of the logical replication slot + to use for the Postgres source + description: >- + Optional. The Postgres slot configuration + to use for logical replication + description: >- + Postgres-specific catalog-level configuration + parameters + source_catalog: + type: string + description: Source catalog name + description: Catalog-level source configuration parameters + google_ads_config: + type: object + properties: + manager_account_id: + type: string + description: Top-level source configurations + source_type: + type: string + x-enum: + - ADOBE_CAMPAIGNS + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AKAMAI_WAF + - AMPLITUDE + - API_SOURCE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS + - BIGQUERY + - BING_ADS + - CERIDIAN_DAYFORCE + - COMMUNITY + - CONFLUENCE + - CROWDSTRIKE_EVENT_STREAM + - DELIGHTED + - DYNAMICS365 + - EPIC_CLARITY + - FOREIGN_CATALOG + - FRESHSERVICE + - FRONT + - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB + - GMAIL + - GONG + - GOOGLE_ADS + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_DRIVE + - GOOGLE_SEARCH_CONSOLE + - GOOGLE_WORKSPACE + - GUIDEWIRE + - GURU + - HIBOB + - HUBSPOT + - IRONCLAD + - JIRA + - KAFKA + - LINEAR + - LINKEDIN_ADS + - M365_AUDIT_LOGS + - MANAGED_POSTGRESQL + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MICROSOFT_TEAMS + - MONDAY_COM + - MYSQL + - NETSKOPE_LOGS + - NETSUITE + - NOTION + - OKTA_SYSTEM_LOGS + - ONE_PASSWORD_EVENT_LOGS + - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PARTNERSTACK + - PENDO + - PINTEREST_ADS + - POSTGRESQL + - PROOFPOINT_SIEM + - QUICKBOOKS + - RABBITMQ + - REDDIT_ADS + - REDSHIFT + - SALESFORCE + - SALESFORCE_MARKETING_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID + - SERVICENOW + - SHAREPOINT + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SLACK_AUDIT_LOGS + - SMARTSHEET + - SNAPCHAT_ADS + - SPLUNK + - SQLDW + - SQLSERVER + - SQUARE + - TERADATA + - TIKTOK_ADS + - VEEVA + - VEEVA_VAULT + - VERKADA + - WIZ_AUDIT_LOGS + - WORKDAY_ACTIVITY_LOGGING + - WORKDAY_HCM + - WORKDAY_RAAS + - X_ADS + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS + description: >- + The type of the foreign source. The source type will be + inferred from the source connection or ingestion + gateway. This field is output only and will be ignored + if provided. + table_configuration: + type: object + properties: + auto_full_refresh_policy: + type: object + properties: + enabled: + type: boolean + description: >- + (Required, Mutable) Whether to enable auto full + refresh or not. + min_interval_hours: + type: integer + description: >- + (Optional, Mutable) Specify the minimum interval + in hours between the timestamp at which a table + was last full refreshed and the current + timestamp for triggering auto full If + unspecified and autoFullRefresh is enabled then + by default min_interval_hours is 24 hours. + required: + - enabled + description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering the + destination table. When specified, the destination + Delta table will be clustered by these columns. This + can improve query performance when filtering on + these columns. Note: clustering_columns in table + specific configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or clustering_columns, added + as separate fields as we cannot have repeated field + in oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the destination + table. When enabled, Delta will automatically + optimize the data layout based on the clustering + columns for improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline definition. + Note: we can only provide enable_auto_clustering or + clustering_columns, added as separate fields as we + cannot have repeated field in oneof. + exclude_columns: + type: array + items: + type: string + description: >- + A list of column names to be excluded for the + ingestion. When not specified, include_columns fully + controls what columns to be ingested. When + specified, all other columns including future ones + will be automatically included for ingestion. This + field in mutually exclusive with + ``include_columns``. + include_columns: + type: array + items: + type: string + description: >- + A list of column names to be included for the + ingestion. When not specified, all columns except + ones in exclude_columns will be included. Future + columns will be automatically included. When + specified, all other future columns will be + automatically excluded from ingestion. This field in + mutually exclusive with ``exclude_columns``. + primary_keys: + type: array + items: + type: string + description: The primary key of the table used to apply changes. + query_based_connector_config: + type: object + properties: + cursor_columns: + type: array + items: + type: string + description: >- + The names of the monotonically increasing + columns in the source table that are used to + enable the table to be read and ingested + incrementally through structured streaming. The + columns are allowed to have repeated values but + have to be non-decreasing. If the source data is + merged into the destination (e.g., using SCD + Type 1 or Type 2), these columns will implicitly + define the ``sequence_by`` behavior. You can + still explicitly set ``sequence_by`` to override + this default. + deletion_condition: + type: string + description: >- + Specifies a SQL WHERE condition that specifies + that the source row has been deleted. This is + sometimes referred to as "soft-deletes". For + example: "Operation = 'DELETE'" or "is_deleted = + true". This field is orthogonal to + ``hard_deletion_sync_interval_in_seconds``, one + for soft-deletes and the other for hard-deletes. + See also the + hard_deletion_sync_min_interval_in_seconds field + for handling of "hard deletes" where the source + rows are physically removed from the table. + hard_deletion_sync_min_interval_in_seconds: + type: integer + description: >- + Specifies the minimum interval (in seconds) + between snapshots on primary keys for detecting + and synchronizing hard deletions—i.e., rows that + have been physically removed from the source + table. This interval acts as a lower bound. If + ingestion runs less frequently than this value, + hard deletion synchronization will align with + the actual ingestion frequency instead of + happening more often. If not set, hard deletion + synchronization via snapshots is disabled. This + field is mutable and can be updated without + triggering a full snapshot. + description: >- + Configurations that are only applicable for + query-based ingestion connectors. + row_filter: + type: string + description: >- + (Optional, Immutable) The row filter condition to be + applied to the table. It must not contain the WHERE + keyword, only the actual filter condition. It must + be in DBSQL format. + salesforce_include_formula_fields: + type: boolean + description: >- + If true, formula fields defined in the table are + included in the ingestion. This setting is only + valid for the Salesforce connector + scd_type: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: The SCD type to use to ingest the table. + sequence_by: + type: array + items: + type: string + description: >- + The column names specifying the logical order of + events in the source data. Spark Declarative + Pipelines uses this sequencing to handle change + events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added to each + ingested record to hold per row source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is written and + maintained in the destination table. Mutually + exclusive with scd_type — a 400 error is returned if + both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination table. + These are key-value pairs that configure various + Delta table behaviors or any user defined + properties. Example: {"delta.feature.variantType": + "supported", "delta.enableTypeWidening": "true"} + Note: table_properties in table specific + configuration will override the table_properties of + the pipeline definition. + workday_report_parameters: + type: object + properties: + incremental: + type: boolean + parameters: + type: object + description: >- + Parameters for the Workday report. Each key + represents the parameter name (e.g., + "start_date", "end_date"), and the corresponding + value is a SQL-like expression used to compute + the parameter value at runtime. Example: { + "start_date": "{ coalesce(current_offset(), + date(\"2025-02-01\")) }", "end_date": "{ + current_date() - INTERVAL 1 DAY }" } + report_parameters: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: >- + Value for the report parameter. Possible + values it can take are these sql + functions: 1. coalesce(current_offset(), + date("YYYY-MM-DD")) -> if current_offset() + is null, then the passed date, else + current_offset() 2. current_date() 3. + date_sub(current_date(), x) -> subtract x + (some non-negative integer) days from + current date + description: >- + (Optional) Additional custom parameters for + Workday Report This field is deprecated and + should not be used. Use ``parameters`` instead. + description: >- + (Optional) Additional custom parameters for Workday + Report + description: >- + Configuration settings to control the ingestion of + tables. These settings are applied to all tables in the + pipeline. + description: >- + The configuration for a managed ingestion pipeline. These + settings cannot be used with the 'libraries', 'schema', + 'target', or 'catalog' settings. + libraries: + type: array + items: + type: object + properties: + file: + type: object + properties: + path: + type: string + glob: + type: object + properties: + include: + type: string + description: >- + The unified field to include source codes. Each entry + can be a notebook path, a file path, or a folder path + that ends ``/**``. This field cannot be used together + with ``notebook`` or ``file``. + jar: + type: string + description: >- + URI of the jar to be installed. Currently only DBFS is + supported. + maven: + type: string + description: Specification of a maven library to be installed. + notebook: + type: object + properties: + path: + type: string + description: >- + The path to a notebook that defines a pipeline and is + stored in the Databricks workspace. + whl: + type: string + description: URI of the whl to be installed. + description: Libraries or code needed by this deployment. + name: + type: string + description: Friendly identifier for this pipeline. + notifications: + type: array + items: + type: object + properties: + alerts: + type: array + items: + type: string + email_recipients: + type: array + items: + type: string + description: >- + A list of email addresses notified when a configured + alert is triggered. + description: List of notification settings for this pipeline. + parameters: + type: object + description: >- + Key/value map of default parameters to use for pipeline + execution. Maximum total size: 10k characters (JSON format) + photon: + type: boolean + description: Whether Photon is enabled for this pipeline. + restart_window: + type: object + properties: + start_hour: + type: integer + days_of_week: + type: array + items: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: >- + Days of week in which the window is allowed to happen. + If not specified all days of the week + + will be used. + description: >- + Days of week in which the restart is allowed to happen + (within a five-hour window starting at start_hour). If + not specified all days of the week will be used. + time_zone_id: + type: string + description: >- + Time zone id of restart window. See + https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html + for details. If not specified, UTC will be used. + required: + - start_hour + description: Restart window of this pipeline. + rewind_generation_interval: + type: object + properties: + interval: + type: integer + description: >- + An integer value representing the interval quantity for + the schedule. + unit: + type: string + x-enum: + - DAYS + - HOURS + - WEEKS + description: A time unit for the schedule. + description: >- + Interval at which rewind points are generated during + pipeline execution. + root_path: + type: string + description: >- + Root path for this pipeline. This is used as the root + directory when editing the pipeline in the Databricks user + interface and it is added to sys.path when executing Python + sources during pipeline execution. + run_as: + type: object + properties: + service_principal_name: + type: string + description: >- + Application ID of an active service principal. Setting + this field requires the ``servicePrincipal/user`` role. + user_name: + type: string + description: >- + The email of an active workspace user. Users can only + set this field to their own email. + description: >- + Write-only setting, available only in Create/Update calls. + Specifies the user or service + principal that the pipeline runs as. If not specified, the pipeline runs as the user who created + the pipeline. + + Only ``user_name`` or ``service_principal_name`` can be specified. If both are specified, an + error is thrown. + schema: + type: string + description: >- + The default schema (database) where tables are read from or + published to. + serverless: + type: boolean + description: Whether serverless compute is enabled for this pipeline. + serverless_compute_id: + type: string + description: >- + Serverless compute ID specified by the user for serverless + pipelines. + storage: + type: string + description: DBFS root directory for storing checkpoints and tables. + tags: + type: object + description: >- + A map of tags associated with the pipeline. These are + forwarded to the cluster as cluster tags, and are therefore + subject to the same limitations. A maximum of 25 tags can be + added to the pipeline. + target: + type: string + description: >- + Target schema (database) to add tables in this pipeline to. + Exactly one of ``schema`` or ``target`` must be specified. + To publish to Unity Catalog, also specify ``catalog``. This + legacy field is deprecated for pipeline creation in favor of + the ``schema`` field. + trigger: + type: object + properties: + cron: + type: object + properties: + quartz_cron_schedule: + type: string + timezone_id: + type: string + manual: + type: object + properties: {} + description: >- + Which pipeline trigger to use. Deprecated: Use + ``continuous`` instead. + usage_policy_id: + type: string + description: Usage policy of this pipeline. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/CreatePipelineResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PipelinesAPI + get: + operationId: pipelines_list_pipelines + summary: Lists pipelines defined in the Spark Declarative Pipelines system. + tags: + - pipelines + description: |- + Lists pipelines defined in the Spark Declarative Pipelines system. + + :param filter: str (optional) + Select a subset of results based on the specified criteria. The supported filters are: + + - ``notebook=''`` to select pipelines that reference the provided notebook path. + - ``name LIKE '[pattern]'`` to select pipelines with a name that matches pattern. Wildcards are + supported, for example: ``name LIKE '%shopping%'`` + + Composite filters are not supported. This field is optional. + :param max_results: int (optional) + The maximum number of entries to return in a single page. The system may return fewer than + max_results events in a response, even if there are more events available. This field is optional. + The default value is 25. The maximum value is 100. An error is returned if the value of max_results + is greater than 100. + :param order_by: List[str] (optional) + A list of strings specifying the order of results. Supported order_by fields are id and name. The + default is id asc. This field is optional. + :param page_token: str (optional) + Page token returned by previous call + + :returns: Iterator over :class:`PipelineStateInfo` + parameters: + - name: filter + in: query + required: false + schema: + type: string + description: >- + Select a subset of results based on the specified criteria. The + supported filters are: - ``notebook=''`` to select pipelines + that reference the provided notebook path. - ``name LIKE + '[pattern]'`` to select pipelines with a name that matches pattern. + Wildcards are supported, for example: ``name LIKE '%shopping%'`` + Composite filters are not supported. This field is optional. + - name: max_results + in: query + required: false + schema: + type: integer + description: >- + The maximum number of entries to return in a single page. The system + may return fewer than max_results events in a response, even if + there are more events available. This field is optional. The default + value is 25. The maximum value is 100. An error is returned if the + value of max_results is greater than 100. + - name: order_by + in: query + required: false + schema: + type: array + items: + type: string + description: >- + A list of strings specifying the order of results. Supported + order_by fields are id and name. The default is id asc. This field + is optional. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token returned by previous call + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListPipelinesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PipelinesAPI + /api/2.0/pipelines/{pipeline_id}: + delete: + operationId: pipelines_delete + summary: >- + Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline + deletion will cascade to all + tags: + - pipelines + description: >- + Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline + deletion will cascade to all + + pipeline tables. Please reach out to Databricks support for assistance + to undo this action. + + + :param pipeline_id: str + + :param cascade: bool (optional) + If false, pipeline deletion will not cascade to its datasets (MVs, STs, Views). By default, this + parameter will be true and all tables will be deleted with the pipeline. + :param delete_datasets: bool (optional) + Deprecated: Use ``cascade`` instead. + :param force: bool (optional) + If true, deletion will proceed even if resource cleanup fails. By default, deletion will fail if + resources cleanup is required but fails. + parameters: + - name: pipeline_id + in: path + required: true + schema: + type: string + - name: cascade + in: query + required: false + schema: + type: boolean + description: >- + If false, pipeline deletion will not cascade to its datasets (MVs, + STs, Views). By default, this parameter will be true and all tables + will be deleted with the pipeline. + - name: delete_datasets + in: query + required: false + schema: + type: boolean + description: 'Deprecated: Use ``cascade`` instead.' + - name: force + in: query + required: false + schema: + type: boolean + description: >- + If true, deletion will proceed even if resource cleanup fails. By + default, deletion will fail if resources cleanup is required but + fails. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PipelinesAPI + get: + operationId: pipelines_get + summary: Get a pipeline. + tags: + - pipelines + description: |- + Get a pipeline. + + :param pipeline_id: str + + :returns: :class:`GetPipelineResponse` + parameters: + - name: pipeline_id + in: path + required: true + schema: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GetPipelineResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PipelinesAPI + put: + operationId: pipelines_update + summary: Updates a pipeline with the supplied configuration. + tags: + - pipelines + description: >- + Updates a pipeline with the supplied configuration. + + + :param pipeline_id: str + Unique identifier for this pipeline. + :param allow_duplicate_names: bool (optional) + If false, deployment will fail if name has changed and conflicts the name of another pipeline. + :param budget_policy_id: str (optional) + Budget policy of this pipeline. + :param catalog: str (optional) + A catalog in Unity Catalog to publish data from this pipeline to. If ``target`` is specified, tables + in this pipeline are published to a ``target`` schema inside ``catalog`` (for example, + ``catalog``.``target``.``table``). If ``target`` is not specified, no data is published to Unity + Catalog. + :param channel: str (optional) + SDP Release Channel that specifies which version to use. + :param clusters: List[:class:`PipelineCluster`] (optional) + Cluster settings for this pipeline deployment. + :param configuration: Dict[str,str] (optional) + String-String configuration for this pipeline execution. + :param continuous: bool (optional) + Whether the pipeline is continuous or triggered. This replaces ``trigger``. + :param deployment: :class:`PipelineDeployment` (optional) + Deployment type of this pipeline. + :param development: bool (optional) + Whether the pipeline is in Development mode. Defaults to false. + :param edition: str (optional) + Pipeline product edition. + :param environment: :class:`PipelinesEnvironment` (optional) + Environment specification for this pipeline used to install dependencies. + :param event_log: :class:`EventLogSpec` (optional) + Event log configuration for this pipeline + :param expected_last_modified: int (optional) + If present, the last-modified time of the pipeline settings before the edit. If the settings were + modified after that time, then the request will fail with a conflict. + :param filters: :class:`Filters` (optional) + Filters on which Pipeline packages to include in the deployed graph. + :param gateway_definition: :class:`IngestionGatewayPipelineDefinition` + (optional) + The definition of a gateway pipeline to support change data capture. + :param id: str (optional) + Unique identifier for this pipeline. + :param ingestion_definition: :class:`IngestionPipelineDefinition` + (optional) + The configuration for a managed ingestion pipeline. These settings cannot be used with the + 'libraries', 'schema', 'target', or 'catalog' settings. + :param libraries: List[:class:`PipelineLibrary`] (optional) + Libraries or code needed by this deployment. + :param name: str (optional) + Friendly identifier for this pipeline. + :param notifications: List[:class:`Notifications`] (optional) + List of notification settings for this pipeline. + :param parameters: Dict[str,str] (optional) + Key/value map of default parameters to use for pipeline execution. Maximum total size: 10k + characters (JSON format) + :param photon: bool (optional) + Whether Photon is enabled for this pipeline. + :param restart_window: :class:`RestartWindow` (optional) + Restart window of this pipeline. + :param rewind_generation_interval: :class:`PeriodicTrigger` (optional) + Interval at which rewind points are generated during pipeline execution. + :param root_path: str (optional) + Root path for this pipeline. This is used as the root directory when editing the pipeline in the + Databricks user interface and it is added to sys.path when executing Python sources during pipeline + execution. + :param run_as: :class:`RunAs` (optional) + + :param schema: str (optional) + The default schema (database) where tables are read from or published to. + :param serverless: bool (optional) + Whether serverless compute is enabled for this pipeline. + :param serverless_compute_id: str (optional) + Serverless compute ID specified by the user for serverless pipelines. + :param storage: str (optional) + DBFS root directory for storing checkpoints and tables. + :param tags: Dict[str,str] (optional) + A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and + are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline. + :param target: str (optional) + Target schema (database) to add tables in this pipeline to. Exactly one of ``schema`` or ``target`` + must be specified. To publish to Unity Catalog, also specify ``catalog``. This legacy field is + deprecated for pipeline creation in favor of the ``schema`` field. + :param trigger: :class:`PipelineTrigger` (optional) + Which pipeline trigger to use. Deprecated: Use ``continuous`` instead. + :param usage_policy_id: str (optional) + Usage policy of this pipeline. + parameters: + - name: pipeline_id + in: path + required: true + schema: + type: string + description: Unique identifier for this pipeline. + requestBody: + content: + application/json: + schema: + type: object + properties: + allow_duplicate_names: + type: boolean + description: >- + If false, deployment will fail if name has changed and + conflicts the name of another pipeline. + budget_policy_id: + type: string + description: Budget policy of this pipeline. + catalog: + type: string + description: >- + A catalog in Unity Catalog to publish data from this + pipeline to. If ``target`` is specified, tables in this + pipeline are published to a ``target`` schema inside + ``catalog`` (for example, ``catalog``.``target``.``table``). + If ``target`` is not specified, no data is published to + Unity Catalog. + channel: + type: string + description: SDP Release Channel that specifies which version to use. + clusters: + type: array + items: + type: object + properties: + apply_policy_default_values: + type: boolean + autoscale: + type: object + properties: + min_workers: + type: integer + max_workers: + type: integer + description: >- + The maximum number of workers to which the cluster + can scale up when overloaded. ``max_workers`` must + be strictly greater than ``min_workers``. + mode: + type: string + x-enum: + - ENHANCED + - LEGACY + description: >- + Databricks Enhanced Autoscaling optimizes cluster + utilization by automatically allocating cluster + resources based on workload volume, with minimal + impact to the data processing latency of your + pipelines. Enhanced Autoscaling is available for + ``updates`` clusters only. The legacy autoscaling + feature is used for ``maintenance`` clusters. + required: + - min_workers + - max_workers + description: >- + Parameters needed in order to automatically scale + clusters up and down based on load. Note: autoscaling + works best with DB runtime versions 3.0 or later. + aws_attributes: + type: string + description: >- + Attributes related to clusters running on Amazon Web + Services. If not specified at cluster creation, a set + of default values will be used. + azure_attributes: + type: string + description: >- + Attributes related to clusters running on Microsoft + Azure. If not specified at cluster creation, a set of + default values will be used. + cluster_log_conf: + type: string + description: >- + The configuration for delivering spark logs to a + long-term storage destination. Only dbfs destinations + are supported. Only one destination can be specified + for one cluster. If the conf is given, the logs will + be delivered to the destination every ``5 mins``. The + destination of driver logs is + ``$destination/$clusterId/driver``, while the + destination of executor logs is + ``$destination/$clusterId/executor``. + custom_tags: + type: object + description: >- + Additional tags for cluster resources. Databricks will + tag all cluster resources (e.g., AWS instances and EBS + volumes) with these tags in addition to + ``default_tags``. Notes: - Currently, Databricks + allows at most 45 custom tags - Clusters can only + reuse cloud resources if the resources' tags are a + subset of the cluster tags + driver_instance_pool_id: + type: string + description: >- + The optional ID of the instance pool for the driver of + the cluster belongs. The pool cluster uses the + instance pool with id (instance_pool_id) if the driver + pool is not assigned. + driver_node_type_id: + type: string + description: >- + The node type of the Spark driver. Note that this + field is optional; if unset, the driver node type will + be set as the same value as ``node_type_id`` defined + above. + enable_local_disk_encryption: + type: boolean + description: >- + Whether to enable local disk encryption for the + cluster. + gcp_attributes: + type: string + description: >- + Attributes related to clusters running on Google Cloud + Platform. If not specified at cluster creation, a set + of default values will be used. + init_scripts: + type: string + description: >- + The configuration for storing init scripts. Any number + of destinations can be specified. The scripts are + executed sequentially in the order provided. If + ``cluster_log_conf`` is specified, init script logs + are sent to + ``//init_scripts``. + instance_pool_id: + type: string + description: >- + The optional ID of the instance pool to which the + cluster belongs. + label: + type: string + description: >- + A label for the cluster specification, either + ``default`` to configure the default cluster, or + ``maintenance`` to configure the maintenance cluster. + This field is optional. The default value is + ``default``. + node_type_id: + type: string + description: >- + This field encodes, through a single value, the + resources available to each of the Spark nodes in this + cluster. For example, the Spark nodes can be + provisioned and optimized for memory or compute + intensive workloads. A list of available node types + can be retrieved by using the + :method:clusters/listNodeTypes API call. + num_workers: + type: integer + description: >- + Number of worker nodes that this cluster should have. + A cluster has one Spark Driver and ``num_workers`` + Executors for a total of ``num_workers`` + 1 Spark + nodes. Note: When reading the properties of a cluster, + this field reflects the desired number of workers + rather than the actual current number of workers. For + instance, if a cluster is resized from 5 to 10 + workers, this field will immediately be updated to + reflect the target size of 10 workers, whereas the + workers listed in ``spark_info`` will gradually + increase from 5 to 10 as the new nodes are + provisioned. + policy_id: + type: string + description: >- + The ID of the cluster policy used to create the + cluster if applicable. + spark_conf: + type: object + description: >- + An object containing a set of optional, user-specified + Spark configuration key-value pairs. See + :method:clusters/create for more details. + spark_env_vars: + type: object + description: >- + An object containing a set of optional, user-specified + environment variable key-value pairs. Please note that + key-value pair of the form (X,Y) will be exported as + is (i.e., ``export X='Y'``) while launching the driver + and workers. In order to specify an additional set of + ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending + them to ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the + example below. This ensures that all default + databricks managed environmental variables are + included as well. Example Spark environment variables: + ``{"SPARK_WORKER_MEMORY": "28000m", + "SPARK_LOCAL_DIRS": "/local_disk0"}`` or + ``{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS + -Dspark.shuffle.service.enabled=true"}`` + ssh_public_keys: + type: array + items: + type: string + description: >- + SSH public key contents that will be added to each + Spark node in this cluster. The corresponding private + keys can be used to login with the user name + ``ubuntu`` on port ``2200``. Up to 10 keys can be + specified. + description: Cluster settings for this pipeline deployment. + configuration: + type: object + description: String-String configuration for this pipeline execution. + continuous: + type: boolean + description: >- + Whether the pipeline is continuous or triggered. This + replaces ``trigger``. + deployment: + type: object + properties: + kind: + type: string + x-enum: + - BUNDLE + description: >- + The deployment method that manages the pipeline: + + + - BUNDLE: The pipeline is managed by a Databricks Asset + Bundle. + deployment_id: + type: string + description: >- + ID of the deployment that manages this pipeline. Only + set when ``kind`` is ``BUNDLE``. Used to look up + deployment metadata from the Deployment Metadata + service. + metadata_file_path: + type: string + description: >- + The path to the file containing metadata about the + deployment. + version_id: + type: string + description: >- + ID of the version of the deployment that produced this + pipeline. Only set when ``kind`` is ``BUNDLE``. + Identifies a specific snapshot of the deployment in the + Deployment Metadata service. + required: + - kind + description: Deployment type of this pipeline. + development: + type: boolean + description: >- + Whether the pipeline is in Development mode. Defaults to + false. + edition: + type: string + description: Pipeline product edition. + environment: + type: object + properties: + dependencies: + type: array + items: + type: string + description: >- + List of pip dependencies, as supported by the version of + pip in this environment. Each dependency is a pip + requirement file line + https://pip.pypa.io/en/stable/reference/requirements-file-format/ + Allowed dependency could be , + , (WSFS or Volumes + in Databricks), + environment_version: + type: string + description: >- + The environment version of the serverless Python + environment used to execute customer Python code. Each + environment version includes a specific Python version + and a curated set of pre-installed libraries with + defined versions, providing a stable and reproducible + execution environment. Databricks supports a three-year + lifecycle for each environment version. For available + versions and their included packages, see + https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ + The value should be a string representing the + environment version number, for example: ``"4"``. + description: >- + Environment specification for this pipeline used to install + dependencies. + event_log: + type: object + properties: + catalog: + type: string + description: The UC catalog the event log is published under. + name: + type: string + description: The name the event log is published to in UC. + schema: + type: string + description: The UC schema the event log is published under. + description: Event log configuration for this pipeline + expected_last_modified: + type: integer + description: >- + If present, the last-modified time of the pipeline settings + before the edit. If the settings were modified after that + time, then the request will fail with a conflict. + filters: + type: object + properties: + exclude: + type: array + items: + type: string + include: + type: array + items: + type: string + description: Paths to include. + description: >- + Filters on which Pipeline packages to include in the + deployed graph. + gateway_definition: + type: object + properties: + connection_name: + type: string + gateway_storage_catalog: + type: string + description: >- + Required, Immutable. The name of the catalog for the + gateway pipeline's storage location. + gateway_storage_schema: + type: string + description: >- + Required, Immutable. The name of the schema for the + gateway pipelines's storage location. + connection_id: + type: string + description: >- + [Deprecated, use connection_name instead] Immutable. The + Unity Catalog connection that this gateway pipeline uses + to communicate with the source. + connection_parameters: + type: object + properties: + source_catalog: + type: string + description: >- + Optional, Internal. Parameters required to establish an + initial connection with the source. + gateway_storage_name: + type: string + description: >- + Optional. The Unity Catalog-compatible name for the + gateway storage location. This is the destination to use + for the data that is extracted by the gateway. Spark + Declarative Pipelines system will automatically create + the storage location under the catalog and schema. + required: + - connection_name + - gateway_storage_catalog + - gateway_storage_schema + description: >- + The definition of a gateway pipeline to support change data + capture. + id: + type: string + description: Unique identifier for this pipeline. + ingestion_definition: + type: object + properties: + connection_name: + type: string + connector_type: + type: string + x-enum: + - CDC + - QUERY_BASED + description: >- + (Optional) Connector Type for sources. Ex: CDC, Query + Based. + data_staging_options: + type: object + properties: + catalog_name: + type: string + description: >- + (Required, Immutable) The name of the catalog for + the connector's staging storage location. + schema_name: + type: string + description: >- + (Required, Immutable) The name of the schema for the + connector's staging storage location. + volume_name: + type: string + description: >- + (Optional) The Unity Catalog-compatible name for the + storage location. This is the volume to use for the + data that is extracted by the connector. Spark + Declarative Pipelines system will automatically + create the volume under the catalog and schema. For + Combined Cdc Managed Ingestion pipelines default + name for the volume would be : + __databricks_ingestion_gateway_staging_data-$pipelineId + required: + - catalog_name + - schema_name + description: >- + (Optional) Location of staged data storage. This is + required for migration from Cdc Managed Ingestion + Pipeline with Gateway pipeline to Combined Cdc Managed + Ingestion Pipeline. If not specified, the volume for + staged data will be created in catalog and schema/target + specified in the top level pipeline definition. + full_refresh_window: + type: object + properties: + start_hour: + type: integer + description: >- + An integer between 0 and 23 denoting the start hour + for the window in the 24-hour day. + days_of_week: + type: array + items: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: >- + Days of week in which the window is allowed to + happen. If not specified all days of the week + + will be used. + description: >- + Days of week in which the window is allowed to + happen If not specified all days of the week will be + used. + time_zone_id: + type: string + description: >- + Time zone id of window. See + https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html + for details. If not specified, UTC will be used. + required: + - start_hour + description: >- + (Optional) A window that specifies a set of time ranges + for snapshot queries in CDC. + ingest_from_uc_foreign_catalog: + type: boolean + description: >- + Immutable. If set to true, the pipeline will ingest + tables from the UC foreign catalogs directly without the + need to specify a UC connection or ingestion gateway. + The ``source_catalog`` fields in objects of + IngestionConfig are interpreted as the UC foreign + catalogs to ingest from. + ingestion_gateway_id: + type: string + description: >- + Identifier for the gateway that is used by this + ingestion pipeline to communicate with the source + database. This is used with CDC connectors to databases + like SQL Server using a gateway pipeline (connector_type + = CDC). Under certain conditions, this can be replaced + with connection_name to change the connector to Combined + Cdc Managed Ingestion Pipeline. + netsuite_jar_path: + type: string + description: >- + Netsuite only configuration. When the field is set for a + netsuite connector, the jar stored in the field will be + validated and added to the classpath of pipeline's + cluster. + objects: + type: array + items: + type: object + properties: + report: + type: object + properties: + source_url: + type: string + destination_catalog: + type: string + description: Required. Destination catalog to store table. + destination_schema: + type: string + description: Required. Destination schema to store table. + destination_table: + type: string + description: >- + Required. Destination table name. The pipeline + fails if a table with that name already + exists. + table_configuration: + type: object + properties: + auto_full_refresh_policy: + type: object + properties: + enabled: + type: boolean + description: >- + (Required, Mutable) Whether to enable + auto full refresh or not. + min_interval_hours: + type: integer + description: >- + (Optional, Mutable) Specify the minimum + interval in hours between the timestamp + at which a table was last full refreshed + and the current timestamp for triggering + auto full If unspecified and + autoFullRefresh is enabled then by + default min_interval_hours is 24 hours. + required: + - enabled + description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + exclude_columns: + type: array + items: + type: string + description: >- + A list of column names to be excluded for + the ingestion. When not specified, + include_columns fully controls what + columns to be ingested. When specified, + all other columns including future ones + will be automatically included for + ingestion. This field in mutually + exclusive with ``include_columns``. + include_columns: + type: array + items: + type: string + description: >- + A list of column names to be included for + the ingestion. When not specified, all + columns except ones in exclude_columns + will be included. Future columns will be + automatically included. When specified, + all other future columns will be + automatically excluded from ingestion. + This field in mutually exclusive with + ``exclude_columns``. + primary_keys: + type: array + items: + type: string + description: >- + The primary key of the table used to apply + changes. + query_based_connector_config: + type: object + properties: + cursor_columns: + type: array + items: + type: string + description: >- + The names of the monotonically + increasing columns in the source table + that are used to enable the table to be + read and ingested incrementally through + structured streaming. The columns are + allowed to have repeated values but have + to be non-decreasing. If the source data + is merged into the destination (e.g., + using SCD Type 1 or Type 2), these + columns will implicitly define the + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. + deletion_condition: + type: string + description: >- + Specifies a SQL WHERE condition that + specifies that the source row has been + deleted. This is sometimes referred to + as "soft-deletes". For example: + "Operation = 'DELETE'" or "is_deleted = + true". This field is orthogonal to + ``hard_deletion_sync_interval_in_seconds``, + one for soft-deletes and the other for + hard-deletes. See also the + hard_deletion_sync_min_interval_in_seconds + field for handling of "hard deletes" + where the source rows are physically + removed from the table. + hard_deletion_sync_min_interval_in_seconds: + type: integer + description: >- + Specifies the minimum interval (in + seconds) between snapshots on primary + keys for detecting and synchronizing + hard deletions—i.e., rows that have been + physically removed from the source + table. This interval acts as a lower + bound. If ingestion runs less frequently + than this value, hard deletion + synchronization will align with the + actual ingestion frequency instead of + happening more often. If not set, hard + deletion synchronization via snapshots + is disabled. This field is mutable and + can be updated without triggering a full + snapshot. + description: >- + Configurations that are only applicable + for query-based ingestion connectors. + row_filter: + type: string + description: >- + (Optional, Immutable) The row filter + condition to be applied to the table. It + must not contain the WHERE keyword, only + the actual filter condition. It must be in + DBSQL format. + salesforce_include_formula_fields: + type: boolean + description: >- + If true, formula fields defined in the + table are included in the ingestion. This + setting is only valid for the Salesforce + connector + scd_type: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: The SCD type to use to ingest the table. + sequence_by: + type: array + items: + type: string + description: >- + The column names specifying the logical + order of events in the source data. Spark + Declarative Pipelines uses this sequencing + to handle change events that arrive out of + order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. + workday_report_parameters: + type: object + properties: + incremental: + type: boolean + parameters: + type: object + description: >- + Parameters for the Workday report. Each + key represents the parameter name (e.g., + "start_date", "end_date"), and the + corresponding value is a SQL-like + expression used to compute the parameter + value at runtime. Example: { + "start_date": "{ + coalesce(current_offset(), + date(\"2025-02-01\")) }", "end_date": "{ + current_date() - INTERVAL 1 DAY }" } + report_parameters: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: >- + Value for the report parameter. Possible + values it can take are these sql + functions: 1. coalesce(current_offset(), + date("YYYY-MM-DD")) -> if + current_offset() is null, then the + passed date, else current_offset() 2. + current_date() 3. + date_sub(current_date(), x) -> subtract + x (some non-negative integer) days from + current date + description: >- + (Optional) Additional custom parameters + for Workday Report This field is + deprecated and should not be used. Use + ``parameters`` instead. + description: >- + (Optional) Additional custom parameters + for Workday Report + description: >- + Configuration settings to control the + ingestion of tables. These settings override + the table_configuration defined in the + IngestionPipelineDefinition object. + required: + - source_url + - destination_catalog + - destination_schema + schema: + type: object + properties: + source_schema: + type: string + destination_catalog: + type: string + description: Required. Destination catalog to store tables. + destination_schema: + type: string + description: >- + Required. Destination schema to store tables + in. Tables with the same name as the source + tables are created in this destination schema. + The pipeline fails If a table with the same + name already exists. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use + connector_options.confluence_options instead. + connector_options: + type: object + properties: + akamai_options: + type: object + properties: + config_ids: + type: array + items: + type: integer + description: >- + (Required) List of Akamai configuration + IDs to ingest data from. A config ID is + a unique identifier for a security + configuration used to generate WAF logs. + description: Akamai specific options for ingestion + api_source_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the API Source connector. + description: >- + Connector-specific options for API Source + connectors. + bing_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + report_type: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: >- + (Required) Microsoft Advertising report + type (e.g. CampaignPerformance, + KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the + report, given as Microsoft Advertising + column enum values for the selected + report type (e.g. AccountId, + CampaignName, TimePeriod, Impressions, + Clicks, Spend). Required key columns are + always included. Available columns are + report-type-specific. For the full list + supported by a given report type, see + its column value set (substitute the + report name, e.g. campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: >- + (Optional) Time granularity for report + aggregation. Defaults to DAILY. + required: + - report_type + - columns + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Bing Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + during incremental sync to capture + late-arriving conversion and attribution + data. If not specified, defaults to 7 + days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. If not + specified, defaults to 2 years before + today. + description: >- + Bing Ads specific options for ingestion + pipelines. + community_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the community connector. + description: >- + Options for community connectors with + arbitrary configuration. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + (Optional) Spaces to filter Confluence + data on + description: Confluence specific options for ingestion + gdrive_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + scope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE + description: >- + The scope of the Google Drive ingestion. + **ONLY ONE** of scope or url must be + specified. If scope is specified, we + will ingest files from the scope. If url + is specified, we will ingest files from + the url. + url: + type: string + description: Google Drive URL. + github_options: + type: object + properties: + repository_branch_selections: + type: array + items: + type: object + properties: + repository_id: + type: string + description: >- + (Required) GitHub repository id + (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this + repository, e.g. ["main", "dev"]. + required: + - repository_id + description: >- + A repository and the branches to ingest + for it. + description: >- + (Optional) Branches to ingest per + repository. A repository not listed here + is ingested on its default branch. + description: GitHub-specific ingestion options. + google_ads_options: + type: object + properties: + manager_account_id: + type: string + description: >- + (Optional at this level) Manager Account + ID (also called MCC Account ID) used to + list and access customer accounts under + this manager account. Overrides + GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query + (e.g. "ad_group_ad", "keyword_view", + "search_term_view"). Must be a resource + that has metrics. Values are validated + against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. + "metrics.clicks", + "metrics.cost_micros"). Multiple values + are joined into the GAQL SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in + fully-qualified GAQL form (e.g. + "ad_group_ad.ad.id", + "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select + (e.g. "segments.date", + "segments.device"). Must include at + least one of segments.date, + segments.week, or segments.month — that + segment is used as the incremental + cursor for the table. + required: + - resource + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Google Ads custom report: + the connector synthesizes a GAQL query + from the resource, fields, segments, and + metrics specified here. When unset, the + table must match one of the connector's + prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables to capture + late-arriving data. If not specified, + defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years of + historical data. + required: + - manager_account_id + description: >- + Google Ads specific options for ingestion + (object-level). When set, these values + override the + corresponding fields in GoogleAdsConfig (source_configurations). + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + (Optional) Projects to filter Jira data + on + use_audit_logs: + type: boolean + description: >- + (Optional) Whether the pipeline reads + Jira's audit log to detect changes for + the tables it ingests. Reading the audit + log requires a paid Jira plan and + appropriate admin permissions. For + example, the issues table uses the audit + log to detect and propagate deletes: + when true, rows deleted in Jira are + removed from the destination table; when + false, the destination retains all + previously ingested rows and only + reflects inserts and updates. If not + specified, defaults to false. + description: Jira specific options for ingestion + kafka_options: + type: object + properties: + client_config: + type: object + key_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + key. If not specified, the key is left + as raw bytes. + max_offsets_per_trigger: + type: integer + description: >- + Internal option to control the maximum + number of offsets to process per + trigger. + max_partitions: + type: string + description: >- + (Optional) Controls how Kafka offset + ranges are bin-packed into Spark input + partitions. Defaults to "auto" when + unset, which lets the engine choose the + input partition count from measured + throughput. Set to "none" to disable + bin-packing, so each offset range maps + to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when + no checkpoint exists. Valid values: + "latest" and "earliest". Defaults to + "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to + matching topics. Only one of topics or + topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of + topics or topic_pattern must be + specified. + value_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + value. If not specified, the value is + left as raw bytes. + linkedin_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + finder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + (Required) adAnalytics finder. See + LinkedInAdsFinder. + entity_granularity: + type: array + items: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the + report. Open vocabulary (not an enum): + the valid set is large (~100) and + evolves with the LinkedIn adAnalytics + API, so values are passed through + verbatim. If empty, a pivot-safe default + core set is ingested: impressions, + clicks, costInLocalCurrency, + externalWebsiteConversions (valid for + every pivot). Ignored for + attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics + struct). + time_granularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + (Optional) Time aggregation. Defaults to + DAILY when unspecified. Used by + analytics/statistics; ignored for + attributedRevenueMetrics. + required: + - finder + description: >- + (Optional) Custom report definition. + Only valid on a table object. When set, + the table is synthesized from + /rest/adAnalytics using the finder, + pivots, time granularity and metrics + here. When unset, the table must match + one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Days to look back during + incremental sync for late-arriving data. + If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables, YYYY-MM-DD. + Earliest date from which to sync + historical data; overrides the default + when set. For finder + attributedRevenueMetrics, this must be + between 30 and 366 days before today. If + not specified, defaults to 1 year of + history. + description: >- + LinkedIn Ads specific options for + ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + marketo_options: + type: object + properties: + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + description: Marketo specific options for ingestion + meta_ads_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) + Action attribution windows for insights + reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) + Timing used to report action statistics + (impression, conversion, mixed, or + lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit + data during sync to capture updated + conversion data from the API, shared by + prebuilt and custom reports. + custom_report_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional) Action attribution windows + for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional) Action breakdowns to + configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action + statistics (impression, conversion, + mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdowns to configure for + data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull + (account, ad, adset, campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to + aggregate statistics (all_days, monthly + or number of days) + description: >- + (Optional) Per-table custom report + definition. When set, defines the shape + of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). + Supersedes the deprecated flat + report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.level) Granularity + of data to pull (account, ad, adset, + campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd + format (e.g. 2025-01-15). Data added + after this date will be ingested, shared + by prebuilt and custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.time_increment) + Value in string by which to aggregate + statistics (can take all_days, monthly + or number of days) + description: >- + Meta Marketing (Meta Ads) specific options + for ingestion + outlook_options: + type: object + properties: + attachment_mode: + type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY + description: >- + (Optional) Controls which attachments to + ingest. If not specified, defaults to + ALL. + body_format: + type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN + description: >- + (Optional) Defines how the body_content + column is populated. TEXT_HTML: + Preserves full formatting, links, and + styling. TEXT_PLAIN: Converts body to + plain text. Recommended for AI/RAG + pipelines to reduce token usage and + noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to + include in the sync. If not specified, + all folders will be synced. Examples: + Inbox, Sent Items, Custom_Folder Filter + semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync + (e.g. mailbox email addresses or + identifiers). If not specified, all + accessible mailboxes are ingested. + Filter semantics: OR between different + mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender + address. Uses exact email match. + Examples: user@vendor.com, + alerts@system.io, noreply@company.com If + not specified, emails from all senders + will be synced. Filter semantics: OR + between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject + line. Values ending with "*" use prefix + match (subject starts with the part + before "*"); otherwise substring match + (subject contains the value). Examples: + "Invoice" (substring), "Re:*" (prefix), + "Support Ticket", "URGENT*" If not + specified, emails with all subjects will + be synced. Filter semantics: OR between + different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. Format: + YYYY-MM-DD (e.g., 2024-01-01) This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + subject_filter: + type: array + items: + type: string + description: >- + Deprecated. Use include_subjects + instead. + description: Outlook specific options for ingestion + rabbitmq_options: + type: object + properties: + queue: + type: string + description: >- + (Required) RabbitMQ queue name to + consume from. + required: + - queue + description: >- + RabbitMQ specific options for ingestion. + Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + reddit_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group + report data by. Examples: CAMPAIGN_ID, + DATE, COUNTRY, REGION, AD_ID. Must + include at least one time dimension + (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the + report (maps to the Reddit Ads API + ``fields`` parameter). Examples: + IMPRESSIONS, CLICKS, SPEND, CPC, CTR. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Reddit Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. If not + specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years + ago. + description: Reddit Ads specific options for ingestion + sharepoint_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + description: >- + (Optional) File ingestion options for + processing files. + url: + type: string + description: Required. The SharePoint URL. + smartsheet_options: + type: object + properties: + enforce_schema: + type: boolean + description: >- + (Optional) When true, maps each column + to its Smartsheet-declared type + (Text/Number/Date/ Checkbox/etc.). Cells + that do not conform to the declared type + are set to NULL. When false, all columns + land as STRING. Use false for sheets + with irregular data or columns that + frequently violate their own declared + type. If not specified, defaults to + true. + description: Smartsheet specific options for ingestion + tiktok_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + (Optional) Data level for the report. If + not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the + report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", + "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the + report (e.g. "spend", "impressions", + "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime + metrics (all-time aggregated data). When + true, the report returns all-time data. + If not specified, defaults to false. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + (Optional) Report type for the TikTok + Ads API. If not specified, defaults to + BASIC. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined TikTok Ads custom report: + the connector synthesizes a report + request from the dimensions, metrics, + report type, and data level specified + here. Supersedes the deprecated + top-level + dimensions/metrics/report_type/ + data_level/query_lifetime fields above. + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + Deprecated. Use + custom_report_options.data_level + instead. + dimensions: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.dimensions + instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. + metrics: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: >- + Deprecated. Use + custom_report_options.query_lifetime + instead. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + Deprecated. Use + custom_report_options.report_type + instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + description: TikTok Ads specific options for ingestion + zendesk_support_options: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD + format for the initial sync. This + determines the earliest date from which + to sync historical data. + description: >- + Zendesk Support specific options for + ingestion + description: (Optional) Source Specific Connector Options + fanout_options: + type: object + properties: + fanout_by: + type: string + description: >- + Column path or SQL expression whose value + determines the destination table. Supports + dotted paths (e.g. "value.event_name") and + expressions (e.g. + "value:event_name::string"). + transforms: + type: array + items: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + Specifies how to transform binary data + into structured data. + description: >- + Optional transforms applied to each + route's DataFrame before writing to the + destination table. + description: >- + Fanout options for multi-table routing from + streaming sources. When set, records are + routed to destination tables based on a + per-record routing key. The key value becomes + the table name: + {destination_catalog}.{destination_schema}.{key_value}. + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use connector_options.jira_options + instead. + source_catalog: type: string description: >- - Required. Destination table name. The pipeline - fails if a table with that name already - exists. + The source catalog name. Might be optional + depending on the type of source. table_configuration: type: object properties: @@ -4137,6 +13521,38 @@ paths: required: - enabled description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. exclude_columns: type: array items: @@ -4149,7 +13565,7 @@ paths: all other columns including future ones will be automatically included for ingestion. This field in mutually - exclusive with `include_columns`. + exclusive with ``include_columns``. include_columns: type: array items: @@ -4163,7 +13579,7 @@ paths: all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with - `exclude_columns`. + ``exclude_columns``. primary_keys: type: array items: @@ -4189,9 +13605,9 @@ paths: is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. deletion_condition: type: string description: >- @@ -4201,7 +13617,7 @@ paths: as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, + ``hard_deletion_sync_interval_in_seconds``, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds @@ -4261,6 +13677,36 @@ paths: Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. workday_report_parameters: type: object properties: @@ -4303,256 +13749,1555 @@ paths: (Optional) Additional custom parameters for Workday Report This field is deprecated and should not be used. Use - `parameters` instead. + ``parameters`` instead. description: >- (Optional) Additional custom parameters for Workday Report - description: >- - Configuration settings to control the - ingestion of tables. These settings override - the table_configuration defined in the - IngestionPipelineDefinition object. + description: >- + Configuration settings to control the + ingestion of tables. These settings are + applied to all tables in this schema and + override the table_configuration defined in + the IngestionPipelineDefinition object. required: - - source_url + - source_schema - destination_catalog - destination_schema - schema: + description: Select all tables from a specific source schema. + table: type: object properties: - source_schema: + source_table: type: string destination_catalog: type: string - description: Required. Destination catalog to store tables. + description: Required. Destination catalog to store table. destination_schema: type: string + description: Required. Destination schema to store table. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string description: >- - Required. Destination schema to store tables - in. Tables with the same name as the source - tables are created in this destination schema. - The pipeline fails If a table with the same - name already exists. - source_catalog: - type: string - description: >- - The source catalog name. Might be optional - depending on the type of source. - table_configuration: + Deprecated: Use + connector_options.confluence_options instead. + connector_options: type: object properties: - auto_full_refresh_policy: + akamai_options: type: object properties: - enabled: - type: boolean + config_ids: + type: array + items: + type: integer description: >- - (Required, Mutable) Whether to enable - auto full refresh or not. - min_interval_hours: + (Required) List of Akamai configuration + IDs to ingest data from. A config ID is + a unique identifier for a security + configuration used to generate WAF logs. + description: Akamai specific options for ingestion + api_source_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the API Source connector. + description: >- + Connector-specific options for API Source + connectors. + bing_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + report_type: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: >- + (Required) Microsoft Advertising report + type (e.g. CampaignPerformance, + KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the + report, given as Microsoft Advertising + column enum values for the selected + report type (e.g. AccountId, + CampaignName, TimePeriod, Impressions, + Clicks, Spend). Required key columns are + always included. Available columns are + report-type-specific. For the full list + supported by a given report type, see + its column value set (substitute the + report name, e.g. campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: >- + (Optional) Time granularity for report + aggregation. Defaults to DAILY. + required: + - report_type + - columns + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Bing Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: type: integer description: >- - (Optional, Mutable) Specify the minimum - interval in hours between the timestamp - at which a table was last full refreshed - and the current timestamp for triggering - auto full If unspecified and - autoFullRefresh is enabled then by - default min_interval_hours is 24 hours. + (Optional) Number of days to look back + during incremental sync to capture + late-arriving conversion and attribution + data. If not specified, defaults to 7 + days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. If not + specified, defaults to 2 years before + today. + description: >- + Bing Ads specific options for ingestion + pipelines. + community_connector_options: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration + options for the community connector. + description: >- + Options for community connectors with + arbitrary configuration. + confluence_options: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: >- + (Optional) Spaces to filter Confluence + data on + description: Confluence specific options for ingestion + gdrive_options: + type: object + properties: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + scope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE + description: >- + The scope of the Google Drive ingestion. + **ONLY ONE** of scope or url must be + specified. If scope is specified, we + will ingest files from the scope. If url + is specified, we will ingest files from + the url. + url: + type: string + description: Google Drive URL. + github_options: + type: object + properties: + repository_branch_selections: + type: array + items: + type: object + properties: + repository_id: + type: string + description: >- + (Required) GitHub repository id + (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this + repository, e.g. ["main", "dev"]. + required: + - repository_id + description: >- + A repository and the branches to ingest + for it. + description: >- + (Optional) Branches to ingest per + repository. A repository not listed here + is ingested on its default branch. + description: GitHub-specific ingestion options. + google_ads_options: + type: object + properties: + manager_account_id: + type: string + description: >- + (Optional at this level) Manager Account + ID (also called MCC Account ID) used to + list and access customer accounts under + this manager account. Overrides + GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query + (e.g. "ad_group_ad", "keyword_view", + "search_term_view"). Must be a resource + that has metrics. Values are validated + against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. + "metrics.clicks", + "metrics.cost_micros"). Multiple values + are joined into the GAQL SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in + fully-qualified GAQL form (e.g. + "ad_group_ad.ad.id", + "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select + (e.g. "segments.date", + "segments.device"). Must include at + least one of segments.date, + segments.week, or segments.month — that + segment is used as the incremental + cursor for the table. + required: + - resource + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Google Ads custom report: + the connector synthesizes a GAQL query + from the resource, fields, segments, and + metrics specified here. When unset, the + table must match one of the connector's + prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables to capture + late-arriving data. If not specified, + defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years of + historical data. required: - - enabled - description: Policy for auto full refresh. - exclude_columns: - type: array - items: - type: string + - manager_account_id description: >- - A list of column names to be excluded for - the ingestion. When not specified, - include_columns fully controls what - columns to be ingested. When specified, - all other columns including future ones - will be automatically included for - ingestion. This field in mutually - exclusive with `include_columns`. - include_columns: - type: array - items: - type: string + Google Ads specific options for ingestion + (object-level). When set, these values + override the + corresponding fields in GoogleAdsConfig (source_configurations). + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + (Optional) Projects to filter Jira data + on + use_audit_logs: + type: boolean + description: >- + (Optional) Whether the pipeline reads + Jira's audit log to detect changes for + the tables it ingests. Reading the audit + log requires a paid Jira plan and + appropriate admin permissions. For + example, the issues table uses the audit + log to detect and propagate deletes: + when true, rows deleted in Jira are + removed from the destination table; when + false, the destination retains all + previously ingested rows and only + reflects inserts and updates. If not + specified, defaults to false. + description: Jira specific options for ingestion + kafka_options: + type: object + properties: + client_config: + type: object + key_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + key. If not specified, the key is left + as raw bytes. + max_offsets_per_trigger: + type: integer + description: >- + Internal option to control the maximum + number of offsets to process per + trigger. + max_partitions: + type: string + description: >- + (Optional) Controls how Kafka offset + ranges are bin-packed into Spark input + partitions. Defaults to "auto" when + unset, which lets the engine choose the + input partition count from measured + throughput. Set to "none" to disable + bin-packing, so each offset range maps + to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when + no checkpoint exists. Valid values: + "latest" and "earliest". Defaults to + "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to + matching topics. Only one of topics or + topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of + topics or topic_pattern must be + specified. + value_transformer: + type: object + properties: + avro_options: + type: object + properties: + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while + deserializing records are handled. + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Avro schema at runtime instead of + providing it inline or via a file path. + format: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When + set, the transformer reads from this + column instead of the default source + column. + json_options: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: >- + Inline schema string for JSON parsing + (Spark DDL format). + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + (Optional) Schema evolution mode for + schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a + comma-separated string of "column_name + type" pairs. + output_column: + type: string + description: >- + Optional output column name. When set, + the transformed result is written to + this column instead of replacing the + input column. + protobuf_options: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: >- + Required: fully-qualified message type + name. + parse_mode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + (Optional) Parse mode for Protobuf data. + Valid values: FAILFAST, PERMISSIVE. + Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for + recursive protobuf fields. Spark SQL + does not natively support recursive + types, so recursive fields are expanded + up to this depth and truncated beyond + it. Valid values: -1 (disallow recursive + fields), 0 (drop), 1-10. + schema_registry: + type: object + properties: + confluent_options: + type: object + properties: + subject: + type: string + connection_name: + type: string + description: >- + (Optional) UC connection for registry + authentication. Specify if different + from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a + specific message from a schema that + defines multiple Protobuf messages. + Simple ("Location") or fully-qualified + ("com.example.protos.Location"). + Defaults to the first message. + description: >- + (Optional) Schema registry to resolve + the Protobuf schema at runtime instead + of providing it via desc_file_path. + description: >- + (Optional) Transformer for the message + value. If not specified, the value is + left as raw bytes. + linkedin_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + finder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + (Required) adAnalytics finder. See + LinkedInAdsFinder. + entity_granularity: + type: array + items: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the + report. Open vocabulary (not an enum): + the valid set is large (~100) and + evolves with the LinkedIn adAnalytics + API, so values are passed through + verbatim. If empty, a pivot-safe default + core set is ingested: impressions, + clicks, costInLocalCurrency, + externalWebsiteConversions (valid for + every pivot). Ignored for + attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics + struct). + time_granularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + (Optional) Time aggregation. Defaults to + DAILY when unspecified. Used by + analytics/statistics; ignored for + attributedRevenueMetrics. + required: + - finder + description: >- + (Optional) Custom report definition. + Only valid on a table object. When set, + the table is synthesized from + /rest/adAnalytics using the finder, + pivots, time granularity and metrics + here. When unset, the table must match + one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Days to look back during + incremental sync for late-arriving data. + If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables, YYYY-MM-DD. + Earliest date from which to sync + historical data; overrides the default + when set. For finder + attributedRevenueMetrics, this must be + between 30 and 366 days before today. If + not specified, defaults to 1 year of + history. description: >- - A list of column names to be included for - the ingestion. When not specified, all - columns except ones in exclude_columns - will be included. Future columns will be - automatically included. When specified, - all other future columns will be - automatically excluded from ingestion. - This field in mutually exclusive with - `exclude_columns`. - primary_keys: - type: array - items: - type: string + LinkedIn Ads specific options for + ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + marketo_options: + type: object + properties: + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + description: Marketo specific options for ingestion + meta_ads_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) + Action attribution windows for insights + reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) + Timing used to report action statistics + (impression, conversion, mixed, or + lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit + data during sync to capture updated + conversion data from the API, shared by + prebuilt and custom reports. + custom_report_options: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional) Action attribution windows + for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional) Action breakdowns to + configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action + statistics (impression, conversion, + mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdowns to configure for + data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull + (account, ad, adset, campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to + aggregate statistics (all_days, monthly + or number of days) + description: >- + (Optional) Per-table custom report + definition. When set, defines the shape + of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). + Supersedes the deprecated flat + report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.level) Granularity + of data to pull (account, ad, adset, + campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd + format (e.g. 2025-01-15). Data added + after this date will be ingested, shared + by prebuilt and custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.time_increment) + Value in string by which to aggregate + statistics (can take all_days, monthly + or number of days) description: >- - The primary key of the table used to apply - changes. - query_based_connector_config: + Meta Marketing (Meta Ads) specific options + for ingestion + outlook_options: type: object properties: - cursor_columns: + attachment_mode: + type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY + description: >- + (Optional) Controls which attachments to + ingest. If not specified, defaults to + ALL. + body_format: + type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN + description: >- + (Optional) Defines how the body_content + column is populated. TEXT_HTML: + Preserves full formatting, links, and + styling. TEXT_PLAIN: Converts body to + plain text. Recommended for AI/RAG + pipelines to reduce token usage and + noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to + include in the sync. If not specified, + all folders will be synced. Examples: + Inbox, Sent Items, Custom_Folder Filter + semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync + (e.g. mailbox email addresses or + identifiers). If not specified, all + accessible mailboxes are ingested. + Filter semantics: OR between different + mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender + address. Uses exact email match. + Examples: user@vendor.com, + alerts@system.io, noreply@company.com If + not specified, emails from all senders + will be synced. Filter semantics: OR + between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject + line. Values ending with "*" use prefix + match (subject starts with the part + before "*"); otherwise substring match + (subject contains the value). Examples: + "Invoice" (substring), "Re:*" (prefix), + "Support Ticket", "URGENT*" If not + specified, emails with all subjects will + be synced. Filter semantics: OR between + different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial + sync in YYYY-MM-DD format. Format: + YYYY-MM-DD (e.g., 2024-01-01) This + determines the earliest date from which + to sync historical data. If not + specified, complete history is ingested. + subject_filter: type: array items: type: string description: >- - The names of the monotonically - increasing columns in the source table - that are used to enable the table to be - read and ingested incrementally through - structured streaming. The columns are - allowed to have repeated values but have - to be non-decreasing. If the source data - is merged into the destination (e.g., - using SCD Type 1 or Type 2), these - columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. - deletion_condition: + Deprecated. Use include_subjects + instead. + description: Outlook specific options for ingestion + rabbitmq_options: + type: object + properties: + queue: type: string description: >- - Specifies a SQL WHERE condition that - specifies that the source row has been - deleted. This is sometimes referred to - as "soft-deletes". For example: - "Operation = 'DELETE'" or "is_deleted = - true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, - one for soft-deletes and the other for - hard-deletes. See also the - hard_deletion_sync_min_interval_in_seconds - field for handling of "hard deletes" - where the source rows are physically - removed from the table. - hard_deletion_sync_min_interval_in_seconds: + (Required) RabbitMQ queue name to + consume from. + required: + - queue + description: >- + RabbitMQ specific options for ingestion. + Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + reddit_ads_options: + type: object + properties: + custom_report_options: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group + report data by. Examples: CAMPAIGN_ID, + DATE, COUNTRY, REGION, AD_ID. Must + include at least one time dimension + (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the + report (maps to the Reddit Ads API + ``fields`` parameter). Examples: + IMPRESSIONS, CLICKS, SPEND, CPC, CTR. + description: >- + (Optional) Custom report definition. + When set, the table is treated as a + user-defined Reddit Ads custom report. + When unset, the table must match one of + the connector's prebuilt sources. + lookback_window_days: type: integer description: >- - Specifies the minimum interval (in - seconds) between snapshots on primary - keys for detecting and synchronizing - hard deletions—i.e., rows that have been - physically removed from the source - table. This interval acts as a lower - bound. If ingestion runs less frequently - than this value, hard deletion - synchronization will align with the - actual ingestion frequency instead of - happening more often. If not set, hard - deletion synchronization via snapshots - is disabled. This field is mutable and - can be updated without triggering a full - snapshot. - description: >- - Configurations that are only applicable - for query-based ingestion connectors. - row_filter: - type: string - description: >- - (Optional, Immutable) The row filter - condition to be applied to the table. It - must not contain the WHERE keyword, only - the actual filter condition. It must be in - DBSQL format. - salesforce_include_formula_fields: - type: boolean - description: >- - If true, formula fields defined in the - table are included in the ingestion. This - setting is only valid for the Salesforce - connector - scd_type: - type: string - x-enum: - - APPEND_ONLY - - SCD_TYPE_1 - - SCD_TYPE_2 - description: The SCD type to use to ingest the table. - sequence_by: - type: array - items: - type: string - description: >- - The column names specifying the logical - order of events in the source data. Spark - Declarative Pipelines uses this sequencing - to handle change events that arrive out of - order. - workday_report_parameters: + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. If not + specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + If not specified, defaults to 2 years + ago. + description: Reddit Ads specific options for ingestion + sharepoint_options: type: object properties: - incremental: + entity_type: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + file_ingestion_options: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times + occurring before the specified time. + Timestamp format: YYYY-MM-DDTHH:mm:ss + (e.g. 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching + the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + description: Generic options + format: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + type: string + x-enum: + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE + description: >- + Based on + + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work + schema_hints: + type: string + description: >- + Override inferred schema of specific + columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string + description: >- + (Optional) File ingestion options for + processing files. + url: + type: string + description: Required. The SharePoint URL. + smartsheet_options: + type: object + properties: + enforce_schema: type: boolean - parameters: + description: >- + (Optional) When true, maps each column + to its Smartsheet-declared type + (Text/Number/Date/ Checkbox/etc.). Cells + that do not conform to the declared type + are set to NULL. When false, all columns + land as STRING. Use false for sheets + with irregular data or columns that + frequently violate their own declared + type. If not specified, defaults to + true. + description: Smartsheet specific options for ingestion + tiktok_ads_options: + type: object + properties: + custom_report_options: type: object + properties: + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + (Optional) Data level for the report. If + not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the + report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", + "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the + report (e.g. "spend", "impressions", + "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime + metrics (all-time aggregated data). When + true, the report returns all-time data. + If not specified, defaults to false. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + (Optional) Report type for the TikTok + Ads API. If not specified, defaults to + BASIC. description: >- - Parameters for the Workday report. Each - key represents the parameter name (e.g., - "start_date", "end_date"), and the - corresponding value is a SQL-like - expression used to compute the parameter - value at runtime. Example: { - "start_date": "{ - coalesce(current_offset(), - date(\"2025-02-01\")) }", "end_date": "{ - current_date() - INTERVAL 1 DAY }" } - report_parameters: + (Optional) Custom report definition. + When set, the table is treated as a + user-defined TikTok Ads custom report: + the connector synthesizes a report + request from the dimensions, metrics, + report type, and data level specified + here. Supersedes the deprecated + top-level + dimensions/metrics/report_type/ + data_level/query_lifetime fields above. + data_level: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: >- + Deprecated. Use + custom_report_options.data_level + instead. + dimensions: type: array items: - type: object - properties: - key: - type: string - value: - type: string - description: >- - Value for the report parameter. Possible - values it can take are these sql - functions: 1. coalesce(current_offset(), - date("YYYY-MM-DD")) -> if - current_offset() is null, then the - passed date, else current_offset() 2. - current_date() 3. - date_sub(current_date(), x) -> subtract - x (some non-negative integer) days from - current date + type: string description: >- - (Optional) Additional custom parameters - for Workday Report This field is - deprecated and should not be used. Use - `parameters` instead. + Deprecated. Use + custom_report_options.dimensions + instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back + for report tables during incremental + sync to capture late-arriving + conversions and attribution data. + metrics: + type: array + items: + type: string + description: >- + Deprecated. Use + custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: >- + Deprecated. Use + custom_report_options.query_lifetime + instead. + report_type: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: >- + Deprecated. Use + custom_report_options.report_type + instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial + sync of report tables in YYYY-MM-DD + format. This determines the earliest + date from which to sync historical data. + description: TikTok Ads specific options for ingestion + zendesk_support_options: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD + format for the initial sync. This + determines the earliest date from which + to sync historical data. description: >- - (Optional) Additional custom parameters - for Workday Report - description: >- - Configuration settings to control the - ingestion of tables. These settings are - applied to all tables in this schema and - override the table_configuration defined in - the IngestionPipelineDefinition object. - required: - - source_schema - - destination_catalog - - destination_schema - description: Select all tables from a specific source schema. - table: - type: object - properties: - source_table: - type: string - destination_catalog: - type: string - description: Required. Destination catalog to store table. - destination_schema: - type: string - description: Required. Destination schema to store table. + Zendesk Support specific options for + ingestion + description: (Optional) Source Specific Connector Options destination_table: type: string description: >- @@ -4560,6 +15305,16 @@ paths: fails if a table with that name already exists. If not set, the source table name is used. + jira_options: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: >- + Deprecated: Use connector_options.jira_options + instead. source_catalog: type: string description: >- @@ -4594,6 +15349,38 @@ paths: required: - enabled description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering + the destination table. When specified, the + destination Delta table will be clustered + by these columns. This can improve query + performance when filtering on these + columns. Note: clustering_columns in table + specific configuration will override the + pipeline definition. Note: we can only + provide enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the + destination table. When enabled, Delta + will automatically optimize the data + layout based on the clustering columns for + improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or + clustering_columns, added as separate + fields as we cannot have repeated field in + oneof. exclude_columns: type: array items: @@ -4606,7 +15393,7 @@ paths: all other columns including future ones will be automatically included for ingestion. This field in mutually - exclusive with `include_columns`. + exclusive with ``include_columns``. include_columns: type: array items: @@ -4620,7 +15407,7 @@ paths: all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with - `exclude_columns`. + ``exclude_columns``. primary_keys: type: array items: @@ -4646,9 +15433,9 @@ paths: is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the - `sequence_by` behavior. You can still - explicitly set `sequence_by` to override - this default. + ``sequence_by`` behavior. You can still + explicitly set ``sequence_by`` to + override this default. deletion_condition: type: string description: >- @@ -4658,7 +15445,7 @@ paths: as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, + ``hard_deletion_sync_interval_in_seconds``, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds @@ -4718,6 +15505,36 @@ paths: Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added + to each ingested record to hold per row + source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is + written and maintained in the destination + table. Mutually exclusive with scd_type — + a 400 error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination + table. These are key-value pairs that + configure various Delta table behaviors or + any user defined properties. Example: + {"delta.feature.variantType": "supported", + "delta.enableTypeWidening": "true"} Note: + table_properties in table specific + configuration will override the + table_properties of the pipeline + definition. workday_report_parameters: type: object properties: @@ -4760,7 +15577,7 @@ paths: (Optional) Additional custom parameters for Workday Report This field is deprecated and should not be used. Use - `parameters` instead. + ``parameters`` instead. description: >- (Optional) Additional custom parameters for Workday Report @@ -4783,6 +15600,17 @@ paths: items: type: object properties: + api_source_connector_config: + type: object + properties: + configs: + type: object + description: >- + Arbitrary key-value configuration values for + the API Source connector. + description: >- + Top-level configuration for API Source connectors + with arbitrary configuration. catalog: type: object properties: @@ -4811,28 +15639,123 @@ paths: source_catalog: type: string description: Source catalog name - description: >- - SourceCatalogConfig contains catalog-level custom - configuration parameters for each source + description: Catalog-level source configuration parameters + google_ads_config: + type: object + properties: + manager_account_id: + type: string description: Top-level source configurations source_type: type: string x-enum: + - ADOBE_CAMPAIGNS + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AKAMAI_WAF + - AMPLITUDE + - API_SOURCE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS - BIGQUERY + - BING_ADS + - CERIDIAN_DAYFORCE + - COMMUNITY + - CONFLUENCE + - CROWDSTRIKE_EVENT_STREAM + - DELIGHTED - DYNAMICS365 + - EPIC_CLARITY - FOREIGN_CATALOG + - FRESHSERVICE + - FRONT - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB + - GMAIL + - GONG + - GOOGLE_ADS + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_DRIVE + - GOOGLE_SEARCH_CONSOLE + - GOOGLE_WORKSPACE + - GUIDEWIRE + - GURU + - HIBOB + - HUBSPOT + - IRONCLAD + - JIRA + - KAFKA + - LINEAR + - LINKEDIN_ADS + - M365_AUDIT_LOGS - MANAGED_POSTGRESQL + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MICROSOFT_TEAMS + - MONDAY_COM - MYSQL + - NETSKOPE_LOGS - NETSUITE + - NOTION + - OKTA_SYSTEM_LOGS + - ONE_PASSWORD_EVENT_LOGS - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PARTNERSTACK + - PENDO + - PINTEREST_ADS - POSTGRESQL + - PROOFPOINT_SIEM + - QUICKBOOKS + - RABBITMQ + - REDDIT_ADS + - REDSHIFT - SALESFORCE + - SALESFORCE_MARKETING_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID - SERVICENOW - SHAREPOINT + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SLACK_AUDIT_LOGS + - SMARTSHEET + - SNAPCHAT_ADS + - SPLUNK + - SQLDW - SQLSERVER + - SQUARE - TERADATA + - TIKTOK_ADS + - VEEVA + - VEEVA_VAULT + - VERKADA + - WIZ_AUDIT_LOGS + - WORKDAY_ACTIVITY_LOGGING + - WORKDAY_HCM - WORKDAY_RAAS + - X_ADS + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS description: >- The type of the foreign source. The source type will be inferred from the source connection or ingestion @@ -4861,6 +15784,33 @@ paths: required: - enabled description: Policy for auto full refresh. + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering the + destination table. When specified, the destination + Delta table will be clustered by these columns. This + can improve query performance when filtering on + these columns. Note: clustering_columns in table + specific configuration will override the pipeline + definition. Note: we can only provide + enable_auto_clustering or clustering_columns, added + as separate fields as we cannot have repeated field + in oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the destination + table. When enabled, Delta will automatically + optimize the data layout based on the clustering + columns for improved query performance. Note: + enable_auto_clustering in table specific + configuration will override the pipeline definition. + Note: we can only provide enable_auto_clustering or + clustering_columns, added as separate fields as we + cannot have repeated field in oneof. exclude_columns: type: array items: @@ -4871,7 +15821,8 @@ paths: controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This - field in mutually exclusive with `include_columns`. + field in mutually exclusive with + ``include_columns``. include_columns: type: array items: @@ -4883,7 +15834,7 @@ paths: columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in - mutually exclusive with `exclude_columns`. + mutually exclusive with ``exclude_columns``. primary_keys: type: array items: @@ -4905,9 +15856,9 @@ paths: have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly - define the `sequence_by` behavior. You can still - explicitly set `sequence_by` to override this - default. + define the ``sequence_by`` behavior. You can + still explicitly set ``sequence_by`` to override + this default. deletion_condition: type: string description: >- @@ -4916,7 +15867,7 @@ paths: sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to - `hard_deletion_sync_interval_in_seconds`, one + ``hard_deletion_sync_interval_in_seconds``, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field @@ -4969,6 +15920,33 @@ paths: events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added to each + ingested record to hold per row source metadata. + storage_mode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + (Optional) Defines how ingested data is written and + maintained in the destination table. Mutually + exclusive with scd_type — a 400 error is returned if + both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination table. + These are key-value pairs that configure various + Delta table behaviors or any user defined + properties. Example: {"delta.feature.variantType": + "supported", "delta.enableTypeWidening": "true"} + Note: table_properties in table specific + configuration will override the table_properties of + the pipeline definition. workday_report_parameters: type: object properties: @@ -5007,7 +15985,7 @@ paths: description: >- (Optional) Additional custom parameters for Workday Report This field is deprecated and - should not be used. Use `parameters` instead. + should not be used. Use ``parameters`` instead. description: >- (Optional) Additional custom parameters for Workday Report @@ -5037,8 +16015,8 @@ paths: description: >- The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path - that ends `/**`. This field cannot be used together - with `notebook` or `file`. + that ends ``/**``. This field cannot be used together + with ``notebook`` or ``file``. jar: type: string description: >- @@ -5079,6 +16057,11 @@ paths: A list of email addresses notified when a configured alert is triggered. description: List of notification settings for this pipeline. + parameters: + type: object + description: >- + Key/value map of default parameters to use for pipeline + execution. Maximum total size: 10k characters (JSON format) photon: type: boolean description: Whether Photon is enabled for this pipeline. @@ -5117,6 +16100,24 @@ paths: required: - start_hour description: Restart window of this pipeline. + rewind_generation_interval: + type: object + properties: + interval: + type: integer + description: >- + An integer value representing the interval quantity for + the schedule. + unit: + type: string + x-enum: + - DAYS + - HOURS + - WEEKS + description: A time unit for the schedule. + description: >- + Interval at which rewind points are generated during + pipeline execution. root_path: type: string description: >- @@ -5131,7 +16132,7 @@ paths: type: string description: >- Application ID of an active service principal. Setting - this field requires the `servicePrincipal/user` role. + this field requires the ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -5143,8 +16144,8 @@ paths: principal that the pipeline runs as. If not specified, the pipeline runs as the user who created the pipeline. - Only `user_name` or `service_principal_name` can be specified. If both are specified, an error - is thrown. + Only ``user_name`` or ``service_principal_name`` can be specified. If both are specified, an + error is thrown. schema: type: string description: >- @@ -5153,6 +16154,11 @@ paths: serverless: type: boolean description: Whether serverless compute is enabled for this pipeline. + serverless_compute_id: + type: string + description: >- + Serverless compute ID specified by the user for serverless + pipelines. storage: type: string description: DBFS root directory for storing checkpoints and tables. @@ -5167,10 +16173,10 @@ paths: type: string description: >- Target schema (database) to add tables in this pipeline to. - Exactly one of `schema` or `target` must be specified. To - publish to Unity Catalog, also specify `catalog`. This + Exactly one of ``schema`` or ``target`` must be specified. + To publish to Unity Catalog, also specify ``catalog``. This legacy field is deprecated for pipeline creation in favor of - the `schema` field. + the ``schema`` field. trigger: type: object properties: @@ -5185,8 +16191,8 @@ paths: type: object properties: {} description: >- - Which pipeline trigger to use. Deprecated: Use `continuous` - instead. + Which pipeline trigger to use. Deprecated: Use + ``continuous`` instead. usage_policy_id: type: string description: Usage policy of this pipeline. @@ -5494,8 +16500,12 @@ paths: The pipeline to return events for. :param filter: str (optional) Criteria to select a subset of results, expressed using a SQL-like syntax. The supported filters - are: 1. level='INFO' (or WARN or ERROR) 2. level in ('INFO', 'WARN') 3. id='[event-id]' 4. timestamp - > 'TIMESTAMP' (or >=,<,<=,=) + are: + + 1. level='INFO' (or WARN or ERROR) + 2. level in ('INFO', 'WARN') + 3. id='[event-id]' + 4. timestamp > 'TIMESTAMP' (or >=,<,<=,=) Composite expressions are supported, for example: level in ('ERROR', 'WARN') AND timestamp> '2021-07-22T06:37:33.083Z' @@ -5670,6 +16680,9 @@ paths: will be reset before the refresh. :param parameters: Dict[str,str] (optional) Key/value map of parameters to pass to the pipeline execution + :param refresh_flow_selection: List[str] (optional) + Flow names to selectively refresh. These are unioned with other selective refresh options + (refresh_selection, full_refresh_selection, etc.) to determine the final set of flows to refresh. :param refresh_selection: List[str] (optional) A list of tables to update without fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table @@ -5729,6 +16742,15 @@ paths: description: >- Key/value map of parameters to pass to the pipeline execution + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are unioned with + other selective refresh options (refresh_selection, + full_refresh_selection, etc.) to determine the final set of + flows to refresh. refresh_selection: type: array items: @@ -5750,8 +16772,8 @@ paths: type: string description: >- SQL predicate string to use as replace_where - condition. Example: `date = '2024-10-10' AND city = - 'xyz'` + condition. Example: ``date = '2024-10-10' AND city = + 'xyz'`` description: >- Specifies a replace_where predicate override for a replace where flow. @@ -5800,9 +16822,18 @@ paths: description: >- If true, this is a dry run and we should emit the RewindSummary but not perform the rewind. + rewind_point_id: + type: string + description: >- + The ID of a previously generated rewind point to rewind + to (same as the originating event log ID). Exactly one + of rewind_timestamp or rewind_point_id must be + specified. rewind_timestamp: type: string - description: The base timestamp to rewind to. Must be specified. + description: >- + The base timestamp to rewind to. Exactly one of + rewind_timestamp or rewind_point_id must be specified. description: >- The information about the requested rewind operation. If specified this is a rewind mode update. @@ -5831,6 +16862,51 @@ paths: message: type: string x-stackql-sdk-source: PipelinesAPI + /api/2.0/pipelines/{pipeline_id}/restore: + post: + operationId: pipelines_restore_pipeline + summary: >- + Restores a pipeline that was previously deleted, if within the + restoration window. All tables deleted + tags: + - pipelines + description: >- + Restores a pipeline that was previously deleted, if within the + restoration window. All tables deleted + + at pipeline deletion will be undropped as well. + + + :param pipeline_id: str + The ID of the pipeline to restore + + :returns: :class:`RestorePipelineRequestResponse` + parameters: + - name: pipeline_id + in: path + required: true + schema: + type: string + description: The ID of the pipeline to restore + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/RestorePipelineRequestResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PipelinesAPI /api/2.0/pipelines/{pipeline_id}/stop: post: operationId: pipelines_stop @@ -5879,6 +16955,41 @@ paths: x-stackql-sdk-source: PipelinesAPI components: schemas: + AkamaiOptions: + type: object + properties: + config_ids: + type: array + items: + type: integer + description: >- + (Required) List of Akamai configuration IDs to ingest data from. A + config ID is a unique identifier for a security configuration used + to generate WAF logs. + description: Akamai specific options for ingestion + ApiSourceConnectorConfig: + type: object + properties: + configs: + type: object + description: >- + Arbitrary key-value configuration values for the API Source + connector. + description: >- + Top-level configuration for API Source connectors with arbitrary + configuration. + ApiSourceConnectorOptions: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration options for the API Source + connector. + description: Options for API Source connectors with arbitrary configuration. + ApplyEnvironmentRequestResponse: + type: object + properties: {} AutoFullRefreshPolicy: type: object properties: @@ -5896,16 +17007,158 @@ components: required: - enabled description: Policy for auto full refresh. + AvroTransformerOptions: + type: object + properties: + parse_mode: + $ref: '#/components/schemas/ParseMode' + schema: + type: string + description: Inline Avro JSON schema string. + schema_file_path: + type: string + description: Path to a schema file (.avsc). + schema_registry: + $ref: '#/components/schemas/SchemaRegistryConfig' + description: >- + (Optional) Schema registry to resolve the Avro schema at runtime + instead of providing it inline or via a file path. + BingAdsOptions: + type: object + properties: + custom_report_options: + $ref: '#/components/schemas/BingAdsOptionsBingCustomReportOptions' + description: >- + (Optional) Custom report definition. When set, the table is treated + as a user-defined Bing Ads custom report. When unset, the table must + match one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back during incremental sync to + capture late-arriving conversion and attribution data. If not + specified, defaults to 7 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial sync in YYYY-MM-DD format. If + not specified, defaults to 2 years before today. + description: Bing Ads specific options for ingestion pipelines. + BingAdsOptionsBingCustomReportOptions: + type: object + properties: + report_type: + $ref: >- + #/components/schemas/BingAdsOptionsBingCustomReportOptionsBingAdsReportType + description: >- + (Required) Microsoft Advertising report type (e.g. + CampaignPerformance, KeywordPerformance). + columns: + type: array + items: + type: string + description: >- + (Required) Columns to include in the report, given as Microsoft + Advertising column enum values for the selected report type (e.g. + AccountId, CampaignName, TimePeriod, Impressions, Clicks, Spend). + Required key columns are always included. Available columns are + report-type-specific. For the full list supported by a given report + type, see its column value set (substitute the report name, e.g. + campaignperformance): + https://learn.microsoft.com/en-us/advertising/reporting-service/{reportname}reportcolumn?view=bingads-13 + granularity: + $ref: >- + #/components/schemas/BingAdsOptionsBingCustomReportOptionsBingAdsGranularity + description: >- + (Optional) Time granularity for report aggregation. Defaults to + DAILY. + required: + - report_type + - columns + description: >- + User-defined custom report options for the Bing Ads connector. Applies + only to the custom_report + table. ClonePipelineResponse: type: object properties: pipeline_id: type: string + CommunityConnectorOptions: + type: object + properties: + options: + type: object + description: >- + Arbitrary key-value configuration options for the community + connector. + description: Options for community connectors with arbitrary configuration. + ConfluenceConnectorOptions: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + description: (Optional) Spaces to filter Confluence data on + description: Confluence specific options for ingestion + ConfluentSchemaRegistryOptions: + type: object + properties: + subject: + type: string ConnectionParameters: type: object properties: source_catalog: type: string + ConnectorOptions: + type: object + properties: + akamai_options: + $ref: '#/components/schemas/AkamaiOptions' + api_source_connector_options: + $ref: '#/components/schemas/ApiSourceConnectorOptions' + description: Connector-specific options for API Source connectors. + bing_ads_options: + $ref: '#/components/schemas/BingAdsOptions' + community_connector_options: + $ref: '#/components/schemas/CommunityConnectorOptions' + confluence_options: + $ref: '#/components/schemas/ConfluenceConnectorOptions' + gdrive_options: + $ref: '#/components/schemas/GoogleDriveOptions' + github_options: + $ref: '#/components/schemas/GitHubConnectorOptions' + google_ads_options: + $ref: '#/components/schemas/GoogleAdsOptions' + jira_options: + $ref: '#/components/schemas/JiraConnectorOptions' + kafka_options: + $ref: '#/components/schemas/KafkaOptions' + linkedin_ads_options: + $ref: '#/components/schemas/LinkedInAdsOptions' + marketo_options: + $ref: '#/components/schemas/MarketoOptions' + meta_ads_options: + $ref: '#/components/schemas/MetaMarketingOptions' + outlook_options: + $ref: '#/components/schemas/OutlookOptions' + rabbitmq_options: + $ref: '#/components/schemas/RabbitmqOptions' + reddit_ads_options: + $ref: '#/components/schemas/RedditAdsOptions' + sharepoint_options: + $ref: '#/components/schemas/SharepointOptions' + smartsheet_options: + $ref: '#/components/schemas/SmartsheetOptions' + tiktok_ads_options: + $ref: '#/components/schemas/TikTokAdsOptions' + zendesk_support_options: + $ref: '#/components/schemas/ZendeskSupportOptions' + description: >- + Wrapper message for source-specific options to support multiple + connector types CreatePipelineResponse: type: object properties: @@ -5988,6 +17241,61 @@ components: type: string description: The UC schema the event log is published under. description: Configurable event log parameters. + FileFilter: + type: object + properties: + modified_after: + type: string + modified_before: + type: string + description: >- + Include files with modification times occurring before the specified + time. Timestamp format: YYYY-MM-DDTHH:mm:ss (e.g. + 2020-06-01T13:00:00) Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#modification-time-path-filters + path_filter: + type: string + description: >- + Include files with file names matching the pattern Based on + https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter + FileIngestionOptions: + type: object + properties: + corrupt_record_column: + type: string + file_filters: + type: array + items: + $ref: '#/components/schemas/FileFilter' + description: Generic options + format: + $ref: '#/components/schemas/FileIngestionOptionsFileFormat' + description: required for TableSpec + format_options: + type: object + description: >- + Format-specific options Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options#file-format-options + ignore_corrupt_files: + type: boolean + infer_column_types: + type: boolean + reader_case_sensitive: + type: boolean + description: >- + Column name case sensitivity + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#change-case-sensitive-behavior + rescued_data_column: + type: string + schema_evolution_mode: + $ref: '#/components/schemas/FileIngestionOptionsSchemaEvolutionMode' + schema_hints: + type: string + description: >- + Override inferred schema of specific columns Based on + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#override-schema-inference-with-schema-hints + single_variant_column: + type: string FileLibrary: type: object properties: @@ -6026,9 +17334,15 @@ components: effective_budget_policy_id: type: string description: Serverless budget policy ID of this pipeline. + effective_environment_version: + type: string + description: The effective environment version resolved for the pipeline. effective_publishing_mode: $ref: '#/components/schemas/PublishingMode' description: Publishing mode of the pipeline + effective_usage_policy_id: + type: string + description: Serverless usage policy ID of the pipeline. health: $ref: '#/components/schemas/GetPipelineResponseHealth' description: The health of a pipeline. @@ -6044,7 +17358,14 @@ components: newest update first. name: type: string - description: A human friendly identifier for the pipeline, taken from the `spec`. + description: >- + A human friendly identifier for the pipeline, taken from the + ``spec``. + parameters: + type: object + description: >- + Key/value map of default parameters to use for pipeline execution. + Maximum total size: 10k characters (JSON format) pipeline_id: type: string description: The ID of the pipeline. @@ -6053,8 +17374,8 @@ components: description: >- The user or service principal that the pipeline runs as, if specified in the request. This field indicates the explicit - configuration of `run_as` for the pipeline. To find the value in all - cases, explicit or implicit, use `run_as_user_name`. + configuration of ``run_as`` for the pipeline. To find the value in + all cases, explicit or implicit, use ``run_as_user_name``. run_as_user_name: type: string description: Username of the user that the pipeline will run on behalf of. @@ -6062,7 +17383,7 @@ components: $ref: '#/components/schemas/PipelineSpec' description: >- The pipeline specification. This field is not returned when called - by `ListPipelines`. + by ``ListPipelines``. state: $ref: '#/components/schemas/PipelineState' description: The pipeline state. @@ -6071,6 +17392,137 @@ components: properties: update: $ref: '#/components/schemas/UpdateInfo' + GitHubConnectorOptions: + type: object + properties: + repository_branch_selections: + type: array + items: + $ref: >- + #/components/schemas/GitHubConnectorOptionsRepositoryBranchSelection + description: >- + (Optional) Branches to ingest per repository. A repository not + listed here is ingested on its default branch. + description: GitHub-specific ingestion options. + GitHubConnectorOptionsRepositoryBranchSelection: + type: object + properties: + repository_id: + type: string + description: (Required) GitHub repository id (immutable; survives repo renames). + branches: + type: array + items: + type: string + description: >- + (Required) Branches to ingest for this repository, e.g. ["main", + "dev"]. + required: + - repository_id + description: A repository and the branches to ingest for it. + GoogleAdsConfig: + type: object + properties: + manager_account_id: + type: string + GoogleAdsCustomReportOptions: + type: object + properties: + resource: + type: string + description: >- + (Required) Google Ads resource to query (e.g. "ad_group_ad", + "keyword_view", "search_term_view"). Must be a resource that has + metrics. Values are validated against Google Ads' field-service + catalog at pipeline plan time. + metrics: + type: array + items: + type: string + description: >- + (Optional) Metric fields to select (e.g. "metrics.clicks", + "metrics.cost_micros"). Multiple values are joined into the GAQL + SELECT clause. + resource_fields: + type: array + items: + type: string + description: >- + (Optional) Resource fields to select, in fully-qualified GAQL form + (e.g. "ad_group_ad.ad.id", "ad_group_ad.status"). Multiple values + are joined into the GAQL SELECT clause. + segments: + type: array + items: + type: string + description: >- + (Optional) Segment fields to select (e.g. "segments.date", + "segments.device"). Must include at least one of segments.date, + segments.week, or segments.month — that segment is used as the + incremental cursor for the table. + required: + - resource + description: >- + User-defined custom report for the Google Ads connector. Mirrors the + resource + fields + + segments + metrics model that Google Ads GAQL exposes. The customer account this report runs + against is supplied by the source schema (namespace), not by this message. The whole message is + gated by the parent GoogleAdsOptions.custom_report_options stage; per-field stage annotations + are intentionally omitted. Only supported on table-type objects: a custom report requires a + destination table, so it cannot be specified at the schema/source level. + GoogleAdsOptions: + type: object + properties: + manager_account_id: + type: string + description: >- + (Optional at this level) Manager Account ID (also called MCC Account + ID) used to list and access customer accounts under this manager + account. Overrides GoogleAdsConfig.manager_account_id from + source_configurations when set. + custom_report_options: + $ref: '#/components/schemas/GoogleAdsCustomReportOptions' + description: >- + (Optional) Custom report definition. When set, the table is treated + as a user-defined Google Ads custom report: the connector + synthesizes a GAQL query from the resource, fields, segments, and + metrics specified here. When unset, the table must match one of the + connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back for report tables to capture + late-arriving data. If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial sync of report tables in + YYYY-MM-DD format. This determines the earliest date from which to + sync historical data. If not specified, defaults to 2 years of + historical data. + required: + - manager_account_id + description: >- + Google Ads specific options for ingestion (object-level). When set, + these values override the + corresponding fields in GoogleAdsConfig (source_configurations). + GoogleDriveOptions: + type: object + properties: + entity_type: + $ref: '#/components/schemas/GoogleDriveOptionsGoogleDriveEntityType' + file_ingestion_options: + $ref: '#/components/schemas/FileIngestionOptions' + scope: + $ref: '#/components/schemas/GoogleDriveOptionsGoogleDriveIngestionScope' + description: >- + The scope of the Google Drive ingestion. **ONLY ONE** of scope or + url must be specified. If scope is specified, we will ingest files + from the scope. If url is specified, we will ingest files from the + url. + url: + type: string + description: Google Drive URL. IngestionConfig: type: object properties: @@ -6146,7 +17598,7 @@ components: description: >- Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC - connection or ingestion gateway. The `source_catalog` fields in + connection or ingestion gateway. The ``source_catalog`` fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from. ingestion_gateway_id: @@ -6187,6 +17639,41 @@ components: description: >- Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline. + IngestionPipelineDefinitionConfluenceOptions: + type: object + properties: + include_confluence_spaces: + type: array + items: + type: string + IngestionPipelineDefinitionFanoutOptions: + type: object + properties: + fanout_by: + type: string + description: >- + Column path or SQL expression whose value determines the destination + table. Supports dotted paths (e.g. "value.event_name") and + expressions (e.g. "value:event_name::string"). + transforms: + type: array + items: + $ref: '#/components/schemas/Transformer' + description: >- + Optional transforms applied to each route's DataFrame before writing + to the destination table. + description: >- + Fanout configuration for multi-table routing from streaming sources. + Routes each input record to + a destination table based on a routing key derived from the record. The key value becomes the + table name suffix: {destination_catalog}.{destination_schema}.{key_value}. + IngestionPipelineDefinitionJiraOptions: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string IngestionPipelineDefinitionTableSpecificConfigQueryBasedConnectorConfig: type: object properties: @@ -6200,16 +17687,17 @@ components: incrementally through structured streaming. The columns are allowed to have repeated values but have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type - 2), these columns will implicitly define the `sequence_by` behavior. - You can still explicitly set `sequence_by` to override this default. + 2), these columns will implicitly define the ``sequence_by`` + behavior. You can still explicitly set ``sequence_by`` to override + this default. deletion_condition: type: string description: >- Specifies a SQL WHERE condition that specifies that the source row has been deleted. This is sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This - field is orthogonal to `hard_deletion_sync_interval_in_seconds`, one - for soft-deletes and the other for hard-deletes. See also the + field is orthogonal to ``hard_deletion_sync_interval_in_seconds``, + one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field for handling of "hard deletes" where the source rows are physically removed from the table. @@ -6249,7 +17737,7 @@ components: #/components/schemas/IngestionPipelineDefinitionWorkdayReportParametersQueryKeyValue description: >- (Optional) Additional custom parameters for Workday Report This - field is deprecated and should not be used. Use `parameters` + field is deprecated and should not be used. Use ``parameters`` instead. IngestionPipelineDefinitionWorkdayReportParametersQueryKeyValue: type: object @@ -6265,6 +17753,157 @@ components: date, else current_offset() 2. current_date() 3. date_sub(current_date(), x) -> subtract x (some non-negative integer) days from current date + JiraConnectorOptions: + type: object + properties: + include_jira_spaces: + type: array + items: + type: string + description: (Optional) Projects to filter Jira data on + use_audit_logs: + type: boolean + description: >- + (Optional) Whether the pipeline reads Jira's audit log to detect + changes for the tables it ingests. Reading the audit log requires a + paid Jira plan and appropriate admin permissions. For example, the + issues table uses the audit log to detect and propagate deletes: + when true, rows deleted in Jira are removed from the destination + table; when false, the destination retains all previously ingested + rows and only reflects inserts and updates. If not specified, + defaults to false. + description: Jira specific options for ingestion + JsonTransformerOptions: + type: object + properties: + as_variant: + type: boolean + schema: + type: string + description: Inline schema string for JSON parsing (Spark DDL format). + schema_evolution_mode: + $ref: '#/components/schemas/FileIngestionOptionsSchemaEvolutionMode' + description: (Optional) Schema evolution mode for schema inference. + schema_file_path: + type: string + description: Path to a schema file (.ddl). + schema_hints: + type: string + description: >- + (Optional) Schema hints as a comma-separated string of "column_name + type" pairs. + KafkaOptions: + type: object + properties: + client_config: + type: object + key_transformer: + $ref: '#/components/schemas/Transformer' + description: >- + (Optional) Transformer for the message key. If not specified, the + key is left as raw bytes. + max_offsets_per_trigger: + type: integer + description: >- + Internal option to control the maximum number of offsets to process + per trigger. + max_partitions: + type: string + description: >- + (Optional) Controls how Kafka offset ranges are bin-packed into + Spark input partitions. Defaults to "auto" when unset, which lets + the engine choose the input partition count from measured + throughput. Set to "none" to disable bin-packing, so each offset + range maps to one input partition. + starting_offset: + type: string + description: >- + (Optional) Where to begin reading when no checkpoint exists. Valid + values: "latest" and "earliest". Defaults to "latest". + topic_pattern: + type: string + description: >- + Java regex pattern to subscribe to matching topics. Only one of + topics or topic_pattern must be specified. + topics: + type: array + items: + type: string + description: >- + Topics to subscribe to. Only one of topics or topic_pattern must be + specified. + value_transformer: + $ref: '#/components/schemas/Transformer' + description: >- + (Optional) Transformer for the message value. If not specified, the + value is left as raw bytes. + LinkedInAdsOptions: + type: object + properties: + custom_report_options: + $ref: >- + #/components/schemas/LinkedInAdsOptionsLinkedInAdsCustomReportOptions + description: >- + (Optional) Custom report definition. Only valid on a table object. + When set, the table is synthesized from /rest/adAnalytics using the + finder, pivots, time granularity and metrics here. When unset, the + table must match one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Days to look back during incremental sync for + late-arriving data. If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial sync of report tables, + YYYY-MM-DD. Earliest date from which to sync historical data; + overrides the default when set. For finder attributedRevenueMetrics, + this must be between 30 and 366 days before today. If not specified, + defaults to 1 year of history. + description: >- + LinkedIn Ads specific options for ingestion. sync_start_date and + lookback_window_days apply to + both the prebuilt analytics tables and custom reports. custom_report_options defines a custom + (user-defined) adAnalytics report and is only valid on a table object. + LinkedInAdsOptionsLinkedInAdsCustomReportOptions: + type: object + properties: + finder: + $ref: >- + #/components/schemas/LinkedInAdsOptionsLinkedInAdsCustomReportOptionsLinkedInAdsFinder + description: (Required) adAnalytics finder. See LinkedInAdsFinder. + entity_granularity: + type: array + items: + $ref: >- + #/components/schemas/LinkedInAdsOptionsLinkedInAdsCustomReportOptionsLinkedInAdsEntityGranularity + metrics: + type: array + items: + type: string + description: >- + (Optional) LinkedIn metric names for the report. Open vocabulary + (not an enum): the valid set is large (~100) and evolves with the + LinkedIn adAnalytics API, so values are passed through verbatim. If + empty, a pivot-safe default core set is ingested: impressions, + clicks, costInLocalCurrency, externalWebsiteConversions (valid for + every pivot). Ignored for attributedRevenueMetrics (always returns + the full RevenueAttributionMetrics struct). + time_granularity: + $ref: >- + #/components/schemas/LinkedInAdsOptionsLinkedInAdsCustomReportOptionsLinkedInAdsTimeGranularity + description: >- + (Optional) Time aggregation. Defaults to DAILY when unspecified. + Used by analytics/statistics; ignored for attributedRevenueMetrics. + required: + - finder + description: >- + User-defined custom report for the LinkedIn Ads connector. The + destination table name comes from + the enclosing TableSpec.destination_table, the start date from the enclosing + LinkedInAdsOptions.sync_start_date, and the account it runs against from the source schema + (namespace) -- none are repeated here. ListPipelineEventsResponse: type: object properties: @@ -6305,6 +17944,119 @@ components: ManualTrigger: type: object properties: {} + MarketoOptions: + type: object + properties: + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial sync in YYYY-MM-DD format. + This determines the earliest date from which to sync historical + data. If not specified, complete history is ingested. + description: Marketo specific options for ingestion + MetaMarketingOptions: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_attribution_windows) Action attribution + windows for insights reporting (e.g. "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use custom_report_options.action_breakdowns) + Action breakdowns + action_report_time: + type: string + description: >- + (Optional, DEPRECATED — use + custom_report_options.action_report_time) Timing used to report + action statistics (impression, conversion, mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: >- + (Optional, DEPRECATED — use custom_report_options.breakdowns) + Breakdowns to configure + custom_insights_lookback_window: + type: integer + description: >- + (Optional) Window in days to revisit data during sync to capture + updated conversion data from the API, shared by prebuilt and custom + reports. + custom_report_options: + $ref: >- + #/components/schemas/MetaMarketingOptionsMetaMarketingCustomReportOptions + description: >- + (Optional) Per-table custom report definition. When set, defines the + shape of the insights call for this table + (level/fields/breakdowns/action_breakdowns/etc.). Supersedes the + deprecated flat report-shape fields above. + level: + type: string + description: >- + (Optional, DEPRECATED — use custom_report_options.level) Granularity + of data to pull (account, ad, adset, campaign) + start_date: + type: string + description: >- + (Optional) Start date in yyyy-MM-dd format (e.g. 2025-01-15). Data + added after this date will be ingested, shared by prebuilt and + custom reports. + time_increment: + type: string + description: >- + (Optional, DEPRECATED — use custom_report_options.time_increment) + Value in string by which to aggregate statistics (can take all_days, + monthly or number of days) + description: Meta Marketing (Meta Ads) specific options for ingestion + MetaMarketingOptionsMetaMarketingCustomReportOptions: + type: object + properties: + action_attribution_windows: + type: array + items: + type: string + description: >- + (Optional) Action attribution windows for insights reporting (e.g. + "28d_click", "1d_view") + action_breakdowns: + type: array + items: + type: string + description: (Optional) Action breakdowns to configure for data aggregation + action_report_time: + type: string + description: >- + (Optional) Timing used to report action statistics (impression, + conversion, mixed, or lifetime) + breakdowns: + type: array + items: + type: string + description: (Optional) Breakdowns to configure for data aggregation + level: + type: string + description: >- + (Optional) Granularity of data to pull (account, ad, adset, + campaign) + time_increment: + type: string + description: >- + (Optional) Value in string by which to aggregate statistics + (all_days, monthly or number of days) + description: >- + Defines the shape of a single Meta Ads custom report (one /insights call + shape). start_date, + custom_insights_lookback_window live on MetaMarketingOptions, not here. Metrics are not + customer-selectable; the connector returns a fixed standard metric set. NotebookLibrary: type: object properties: @@ -6373,6 +18125,11 @@ components: flow_name: type: string description: The name of the flow. Not unique. + graph_id: + type: string + description: >- + The UUID of the graph associated with this event, corresponding to a + GRAPH_UPDATED event. host: type: string description: The optional host name where the event was triggered @@ -6431,11 +18188,97 @@ components: update_id: type: string description: The id of an execution. Globally unique. + OutlookOptions: + type: object + properties: + attachment_mode: + $ref: '#/components/schemas/OutlookAttachmentMode' + description: >- + (Optional) Controls which attachments to ingest. If not specified, + defaults to ALL. + body_format: + $ref: '#/components/schemas/OutlookBodyFormat' + description: >- + (Optional) Defines how the body_content column is populated. + TEXT_HTML: Preserves full formatting, links, and styling. + TEXT_PLAIN: Converts body to plain text. Recommended for AI/RAG + pipelines to reduce token usage and noise. + folder_filter: + type: array + items: + type: string + description: Deprecated. Use include_folders instead. + include_folders: + type: array + items: + type: string + description: >- + (Optional) Filter mail folders to include in the sync. If not + specified, all folders will be synced. Examples: Inbox, Sent Items, + Custom_Folder Filter semantics: OR between different folders. + include_mailboxes: + type: array + items: + type: string + description: >- + (Optional) List of mailboxes to sync (e.g. mailbox email addresses + or identifiers). If not specified, all accessible mailboxes are + ingested. Filter semantics: OR between different mailboxes. + include_senders: + type: array + items: + type: string + description: >- + (Optional) Filter emails by sender address. Uses exact email match. + Examples: user@vendor.com, alerts@system.io, noreply@company.com If + not specified, emails from all senders will be synced. Filter + semantics: OR between different senders. + include_subjects: + type: array + items: + type: string + description: >- + (Optional) Filter emails by subject line. Values ending with "*" use + prefix match (subject starts with the part before "*"); otherwise + substring match (subject contains the value). Examples: "Invoice" + (substring), "Re:*" (prefix), "Support Ticket", "URGENT*" If not + specified, emails with all subjects will be synced. Filter + semantics: OR between different subjects. + sender_filter: + type: array + items: + type: string + description: Deprecated. Use include_senders instead. + start_date: + type: string + description: >- + (Optional) Start date for the initial sync in YYYY-MM-DD format. + Format: YYYY-MM-DD (e.g., 2024-01-01) This determines the earliest + date from which to sync historical data. If not specified, complete + history is ingested. + subject_filter: + type: array + items: + type: string + description: Deprecated. Use include_subjects instead. + description: Outlook specific options for ingestion PathPattern: type: object properties: include: type: string + PeriodicTrigger: + type: object + properties: + interval: + type: integer + description: >- + An integer value representing the interval quantity for the + schedule. + unit: + $ref: '#/components/schemas/PeriodicTriggerTimeUnit' + description: A time unit for the schedule. + description: Periodic (EVERY syntax) schedule trigger configuration for a pipeline. PipelineAccessControlRequest: type: object properties: @@ -6496,16 +18339,16 @@ components: The configuration for delivering spark logs to a long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, - the logs will be delivered to the destination every `5 mins`. The - destination of driver logs is `$destination/$clusterId/driver`, + the logs will be delivered to the destination every ``5 mins``. The + destination of driver logs is ``$destination/$clusterId/driver``, while the destination of executor logs is - `$destination/$clusterId/executor`. + ``$destination/$clusterId/executor``. custom_tags: type: object description: >- Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these - tags in addition to `default_tags`. Notes: - Currently, Databricks + tags in addition to ``default_tags``. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags driver_instance_pool_id: @@ -6519,7 +18362,7 @@ components: description: >- The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as - `node_type_id` defined above. + ``node_type_id`` defined above. enable_local_disk_encryption: type: boolean description: Whether to enable local disk encryption for the cluster. @@ -6534,17 +18377,18 @@ components: description: >- The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially - in the order provided. If `cluster_log_conf` is specified, init - script logs are sent to `//init_scripts`. + in the order provided. If ``cluster_log_conf`` is specified, init + script logs are sent to ``//init_scripts``. instance_pool_id: type: string description: The optional ID of the instance pool to which the cluster belongs. label: type: string description: >- - A label for the cluster specification, either `default` to configure - the default cluster, or `maintenance` to configure the maintenance - cluster. This field is optional. The default value is `default`. + A label for the cluster specification, either ``default`` to + configure the default cluster, or ``maintenance`` to configure the + maintenance cluster. This field is optional. The default value is + ``default``. node_type_id: type: string description: >- @@ -6557,14 +18401,14 @@ components: type: integer description: >- Number of worker nodes that this cluster should have. A cluster has - one Spark Driver and `num_workers` Executors for a total of - `num_workers` + 1 Spark nodes. Note: When reading the properties of - a cluster, this field reflects the desired number of workers rather - than the actual current number of workers. For instance, if a + one Spark Driver and ``num_workers`` Executors for a total of + ``num_workers`` + 1 Spark nodes. Note: When reading the properties + of a cluster, this field reflects the desired number of workers + rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the - workers listed in `spark_info` will gradually increase from 5 to 10 - as the new nodes are provisioned. + workers listed in ``spark_info`` will gradually increase from 5 to + 10 as the new nodes are provisioned. policy_id: type: string description: >- @@ -6581,15 +18425,15 @@ components: description: >- An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the - form (X,Y) will be exported as is (i.e., `export X='Y'`) while + form (X,Y) will be exported as is (i.e., ``export X='Y'``) while launching the driver and workers. In order to specify an additional - set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to - `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This + set of ``SPARK_DAEMON_JAVA_OPTS``, we recommend appending them to + ``$SPARK_DAEMON_JAVA_OPTS`` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: - `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": - "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": - "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` + ``{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": + "/local_disk0"}`` or ``{"SPARK_DAEMON_JAVA_OPTS": + "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}`` ssh_public_keys: type: array items: @@ -6597,7 +18441,7 @@ components: description: >- SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login - with the user name `ubuntu` on port `2200`. Up to 10 keys can be + with the user name ``ubuntu`` on port ``2200``. Up to 10 keys can be specified. PipelineClusterAutoscale: type: object @@ -6608,17 +18452,17 @@ components: type: integer description: >- The maximum number of workers to which the cluster can scale up when - overloaded. `max_workers` must be strictly greater than - `min_workers`. + overloaded. ``max_workers`` must be strictly greater than + ``min_workers``. mode: $ref: '#/components/schemas/PipelineClusterAutoscaleMode' description: >- Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your - pipelines. Enhanced Autoscaling is available for `updates` clusters - only. The legacy autoscaling feature is used for `maintenance` - clusters. + pipelines. Enhanced Autoscaling is available for ``updates`` + clusters only. The legacy autoscaling feature is used for + ``maintenance`` clusters. required: - min_workers - max_workers @@ -6627,9 +18471,21 @@ components: properties: kind: $ref: '#/components/schemas/DeploymentKind' + deployment_id: + type: string + description: >- + ID of the deployment that manages this pipeline. Only set when + ``kind`` is ``BUNDLE``. Used to look up deployment metadata from the + Deployment Metadata service. metadata_file_path: type: string description: The path to the file containing metadata about the deployment. + version_id: + type: string + description: >- + ID of the version of the deployment that produced this pipeline. + Only set when ``kind`` is ``BUNDLE``. Identifies a specific snapshot + of the deployment in the Deployment Metadata service. required: - kind PipelineEvent: @@ -6677,8 +18533,8 @@ components: $ref: '#/components/schemas/PathPattern' description: >- The unified field to include source codes. Each entry can be a - notebook path, a file path, or a folder path that ends `/**`. This - field cannot be used together with `notebook` or `file`. + notebook path, a file path, or a folder path that ends ``/**``. This + field cannot be used together with ``notebook`` or ``file``. jar: type: string description: URI of the jar to be installed. Currently only DBFS is supported. @@ -6731,13 +18587,13 @@ components: type: string description: >- A catalog in Unity Catalog to publish data from this pipeline to. If - `target` is specified, tables in this pipeline are published to a - `target` schema inside `catalog` (for example, - `catalog`.`target`.`table`). If `target` is not specified, no data - is published to Unity Catalog. + ``target`` is specified, tables in this pipeline are published to a + ``target`` schema inside ``catalog`` (for example, + ``catalog``.``target``.``table``). If ``target`` is not specified, + no data is published to Unity Catalog. channel: type: string - description: DLT Release Channel that specifies which version to use. + description: SDP Release Channel that specifies which version to use. clusters: type: array items: @@ -6750,7 +18606,7 @@ components: type: boolean description: >- Whether the pipeline is continuous or triggered. This replaces - `trigger`. + ``trigger``. deployment: $ref: '#/components/schemas/PipelineDeployment' description: Deployment type of this pipeline. @@ -6802,6 +18658,11 @@ components: restart_window: $ref: '#/components/schemas/RestartWindow' description: Restart window of this pipeline. + rewind_generation_interval: + $ref: '#/components/schemas/PeriodicTrigger' + description: >- + Interval at which rewind points are generated during pipeline + execution. root_path: type: string description: >- @@ -6817,6 +18678,11 @@ components: serverless: type: boolean description: Whether serverless compute is enabled for this pipeline. + serverless_compute_id: + type: string + description: >- + Serverless compute ID specified by the user for serverless + pipelines. storage: type: string description: DBFS root directory for storing checkpoints and tables. @@ -6830,12 +18696,14 @@ components: type: string description: >- Target schema (database) to add tables in this pipeline to. Exactly - one of `schema` or `target` must be specified. To publish to Unity - Catalog, also specify `catalog`. This legacy field is deprecated for - pipeline creation in favor of the `schema` field. + one of ``schema`` or ``target`` must be specified. To publish to + Unity Catalog, also specify ``catalog``. This legacy field is + deprecated for pipeline creation in favor of the ``schema`` field. trigger: $ref: '#/components/schemas/PipelineTrigger' - description: 'Which pipeline trigger to use. Deprecated: Use `continuous` instead.' + description: >- + Which pipeline trigger to use. Deprecated: Use ``continuous`` + instead. usage_policy_id: type: string description: Usage policy of this pipeline. @@ -6903,11 +18771,11 @@ components: packages, see https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ The value should be a string representing the environment version - number, for example: `"4"`. + number, for example: ``"4"``. description: >- The environment entity used to preserve serverless environment side panel, jobs' environment for - non-notebook task, and DLT's environment for classic and serverless pipelines. In this minimal + non-notebook task, and SDP's environment for classic and serverless pipelines. In this minimal environment spec, only pip dependencies are supported. PostgresCatalogConfig: type: object @@ -6932,6 +18800,89 @@ components: description: >- PostgresSlotConfig contains the configuration for a Postgres logical replication slot + ProtobufTransformerOptions: + type: object + properties: + desc_file_path: + type: string + message_name: + type: string + description: 'Required: fully-qualified message type name.' + parse_mode: + $ref: '#/components/schemas/ParseMode' + description: >- + (Optional) Parse mode for Protobuf data. Valid values: FAILFAST, + PERMISSIVE. Defaults to FAILFAST. + recursive_fields_max_depth: + type: integer + description: >- + (Optional) Maximum expansion depth for recursive protobuf fields. + Spark SQL does not natively support recursive types, so recursive + fields are expanded up to this depth and truncated beyond it. Valid + values: -1 (disallow recursive fields), 0 (drop), 1-10. + schema_registry: + $ref: '#/components/schemas/SchemaRegistryConfig' + description: >- + (Optional) Schema registry to resolve the Protobuf schema at runtime + instead of providing it via desc_file_path. + RabbitmqOptions: + type: object + properties: + queue: + type: string + description: (Required) RabbitMQ queue name to consume from. + required: + - queue + description: >- + RabbitMQ specific options for ingestion. Performance tuning options + (consumers_per_task, + max_messages_per_fetch, etc.) are intentionally not exposed in the public API. The managed + connector uses sensible defaults internally. These can be added later if user demand arises. + RedditAdsOptions: + type: object + properties: + custom_report_options: + $ref: '#/components/schemas/RedditAdsOptionsRedditAdsCustomReportOptions' + description: >- + (Optional) Custom report definition. When set, the table is treated + as a user-defined Reddit Ads custom report. When unset, the table + must match one of the connector's prebuilt sources. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back for report tables during + incremental sync to capture late-arriving conversions and + attribution data. If not specified, defaults to 30 days. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial sync of report tables in + YYYY-MM-DD format. This determines the earliest date from which to + sync historical data. If not specified, defaults to 2 years ago. + description: Reddit Ads specific options for ingestion + RedditAdsOptionsRedditAdsCustomReportOptions: + type: object + properties: + breakdowns: + type: array + items: + type: string + description: >- + (Optional) Breakdown dimensions to group report data by. Examples: + CAMPAIGN_ID, DATE, COUNTRY, REGION, AD_ID. Must include at least one + time dimension (DATE or HOUR). + fields: + type: array + items: + type: string + description: >- + (Optional) Fields to include in the report (maps to the Reddit Ads + API ``fields`` parameter). Examples: IMPRESSIONS, CLICKS, SPEND, + CPC, CTR. + description: >- + User-defined custom report for the Reddit Ads connector. Applies only to + the custom_report table + — prebuilt tables ignore this. ReplaceWhereOverride: type: object properties: @@ -6942,7 +18893,7 @@ components: type: string description: >- SQL predicate string to use as replace_where condition. Example: - `date = '2024-10-10' AND city = 'xyz'` + ``date = '2024-10-10' AND city = 'xyz'`` description: Specifies a replace_where predicate override for a replace where flow. ReportSpec: type: object @@ -6991,6 +18942,9 @@ components: for details. If not specified, UTC will be used. required: - start_hour + RestorePipelineRequestResponse: + type: object + properties: {} RewindDatasetSpec: type: object properties: @@ -7022,9 +18976,17 @@ components: description: >- If true, this is a dry run and we should emit the RewindSummary but not perform the rewind. + rewind_point_id: + type: string + description: >- + The ID of a previously generated rewind point to rewind to (same as + the originating event log ID). Exactly one of rewind_timestamp or + rewind_point_id must be specified. rewind_timestamp: type: string - description: The base timestamp to rewind to. Must be specified. + description: >- + The base timestamp to rewind to. Exactly one of rewind_timestamp or + rewind_point_id must be specified. description: >- Information about a rewind being requested for this pipeline or some of the datasets in it. @@ -7035,7 +18997,7 @@ components: type: string description: >- Application ID of an active service principal. Setting this field - requires the `servicePrincipal/user` role. + requires the ``servicePrincipal/user`` role. user_name: type: string description: >- @@ -7047,8 +19009,25 @@ components: principal that the pipeline runs as. If not specified, the pipeline runs as the user who created the pipeline. - Only `user_name` or `service_principal_name` can be specified. If both are specified, an error - is thrown. + Only ``user_name`` or ``service_principal_name`` can be specified. If both are specified, an + error is thrown. + SchemaRegistryConfig: + type: object + properties: + confluent_options: + $ref: '#/components/schemas/ConfluentSchemaRegistryOptions' + connection_name: + type: string + description: >- + (Optional) UC connection for registry authentication. Specify if + different from the top-level source connection. + protobuf_message_name: + type: string + description: >- + (Optional, Protobuf only) Selects a specific message from a schema + that defines multiple Protobuf messages. Simple ("Location") or + fully-qualified ("com.example.protos.Location"). Defaults to the + first message. SchemaSpec: type: object properties: @@ -7064,6 +19043,22 @@ components: same name as the source tables are created in this destination schema. The pipeline fails If a table with the same name already exists. + confluence_options: + $ref: '#/components/schemas/IngestionPipelineDefinitionConfluenceOptions' + description: 'Deprecated: Use connector_options.confluence_options instead.' + connector_options: + $ref: '#/components/schemas/ConnectorOptions' + description: (Optional) Source Specific Connector Options + fanout_options: + $ref: '#/components/schemas/IngestionPipelineDefinitionFanoutOptions' + description: >- + Fanout options for multi-table routing from streaming sources. When + set, records are routed to destination tables based on a per-record + routing key. The key value becomes the table name: + {destination_catalog}.{destination_schema}.{key_value}. + jira_options: + $ref: '#/components/schemas/IngestionPipelineDefinitionJiraOptions' + description: 'Deprecated: Use connector_options.jira_options instead.' source_catalog: type: string description: >- @@ -7101,6 +19096,30 @@ components: items: $ref: '#/components/schemas/StackFrame' description: Stack trace consisting of a list of stack frames + SharepointOptions: + type: object + properties: + entity_type: + $ref: '#/components/schemas/SharepointOptionsSharepointEntityType' + file_ingestion_options: + $ref: '#/components/schemas/FileIngestionOptions' + description: (Optional) File ingestion options for processing files. + url: + type: string + description: Required. The SharePoint URL. + SmartsheetOptions: + type: object + properties: + enforce_schema: + type: boolean + description: >- + (Optional) When true, maps each column to its Smartsheet-declared + type (Text/Number/Date/ Checkbox/etc.). Cells that do not conform to + the declared type are set to NULL. When false, all columns land as + STRING. Use false for sheets with irregular data or columns that + frequently violate their own declared type. If not specified, + defaults to true. + description: Smartsheet specific options for ingestion SourceCatalogConfig: type: object properties: @@ -7116,8 +19135,13 @@ components: SourceConfig: type: object properties: + api_source_connector_config: + $ref: '#/components/schemas/ApiSourceConnectorConfig' catalog: $ref: '#/components/schemas/SourceCatalogConfig' + description: Catalog-level source configuration parameters + google_ads_config: + $ref: '#/components/schemas/GoogleAdsConfig' StackFrame: type: object properties: @@ -7151,11 +19175,20 @@ components: destination_schema: type: string description: Required. Destination schema to store table. + confluence_options: + $ref: '#/components/schemas/IngestionPipelineDefinitionConfluenceOptions' + description: 'Deprecated: Use connector_options.confluence_options instead.' + connector_options: + $ref: '#/components/schemas/ConnectorOptions' + description: (Optional) Source Specific Connector Options destination_table: type: string description: >- Optional. Destination table name. The pipeline fails if a table with that name already exists. If not set, the source table name is used. + jira_options: + $ref: '#/components/schemas/IngestionPipelineDefinitionJiraOptions' + description: 'Deprecated: Use connector_options.jira_options instead.' source_catalog: type: string description: >- @@ -7181,6 +19214,28 @@ components: properties: auto_full_refresh_policy: $ref: '#/components/schemas/AutoFullRefreshPolicy' + clustering_columns: + type: array + items: + type: string + description: >- + List of column names to use for clustering the destination table. + When specified, the destination Delta table will be clustered by + these columns. This can improve query performance when filtering on + these columns. Note: clustering_columns in table specific + configuration will override the pipeline definition. Note: we can + only provide enable_auto_clustering or clustering_columns, added as + separate fields as we cannot have repeated field in oneof. + enable_auto_clustering: + type: boolean + description: >- + Whether to enable auto clustering on the destination table. When + enabled, Delta will automatically optimize the data layout based on + the clustering columns for improved query performance. Note: + enable_auto_clustering in table specific configuration will override + the pipeline definition. Note: we can only provide + enable_auto_clustering or clustering_columns, added as separate + fields as we cannot have repeated field in oneof. exclude_columns: type: array items: @@ -7190,7 +19245,7 @@ components: specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually - exclusive with `include_columns`. + exclusive with ``include_columns``. include_columns: type: array items: @@ -7201,7 +19256,7 @@ components: included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with - `exclude_columns`. + ``exclude_columns``. primary_keys: type: array items: @@ -7223,7 +19278,6 @@ components: ingestion. This setting is only valid for the Salesforce connector scd_type: $ref: '#/components/schemas/TableSpecificConfigScdType' - description: The SCD type to use to ingest the table. sequence_by: type: array items: @@ -7232,10 +19286,134 @@ components: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order. + source_metadata_column: + type: string + description: >- + (Optional) Name of the struct column added to each ingested record + to hold per row source metadata. + storage_mode: + $ref: '#/components/schemas/StorageMode' + description: >- + (Optional) Defines how ingested data is written and maintained in + the destination table. Mutually exclusive with scd_type — a 400 + error is returned if both are set. + table_properties: + type: object + description: >- + Table properties to set on the destination table. These are + key-value pairs that configure various Delta table behaviors or any + user defined properties. Example: {"delta.feature.variantType": + "supported", "delta.enableTypeWidening": "true"} Note: + table_properties in table specific configuration will override the + table_properties of the pipeline definition. workday_report_parameters: $ref: >- #/components/schemas/IngestionPipelineDefinitionWorkdayReportParameters description: (Optional) Additional custom parameters for Workday Report + TikTokAdsOptions: + type: object + properties: + custom_report_options: + $ref: '#/components/schemas/TikTokAdsOptionsTikTokAdsCustomReportOptions' + description: >- + (Optional) Custom report definition. When set, the table is treated + as a user-defined TikTok Ads custom report: the connector + synthesizes a report request from the dimensions, metrics, report + type, and data level specified here. Supersedes the deprecated + top-level dimensions/metrics/report_type/ data_level/query_lifetime + fields above. + data_level: + $ref: '#/components/schemas/TikTokAdsOptionsTikTokDataLevel' + description: Deprecated. Use custom_report_options.data_level instead. + dimensions: + type: array + items: + type: string + description: Deprecated. Use custom_report_options.dimensions instead. + lookback_window_days: + type: integer + description: >- + (Optional) Number of days to look back for report tables during + incremental sync to capture late-arriving conversions and + attribution data. + metrics: + type: array + items: + type: string + description: Deprecated. Use custom_report_options.metrics instead. + query_lifetime: + type: boolean + description: Deprecated. Use custom_report_options.query_lifetime instead. + report_type: + $ref: '#/components/schemas/TikTokAdsOptionsTikTokReportType' + description: Deprecated. Use custom_report_options.report_type instead. + sync_start_date: + type: string + description: >- + (Optional) Start date for the initial sync of report tables in + YYYY-MM-DD format. This determines the earliest date from which to + sync historical data. + description: TikTok Ads specific options for ingestion + TikTokAdsOptionsTikTokAdsCustomReportOptions: + type: object + properties: + data_level: + $ref: '#/components/schemas/TikTokAdsOptionsTikTokDataLevel' + description: >- + (Optional) Data level for the report. If not specified, defaults to + AUCTION_CAMPAIGN. + dimensions: + type: array + items: + type: string + description: >- + (Optional) Dimensions to include in the report (e.g. "campaign_id", + "adgroup_id", "ad_id", "stat_time_day", "stat_time_hour"). + metrics: + type: array + items: + type: string + description: >- + (Optional) Metrics to include in the report (e.g. "spend", + "impressions", "clicks", "conversion", "cpc"). + query_lifetime: + type: boolean + description: >- + (Optional) Whether to request lifetime metrics (all-time aggregated + data). When true, the report returns all-time data. If not + specified, defaults to false. + report_type: + $ref: '#/components/schemas/TikTokAdsOptionsTikTokReportType' + description: >- + (Optional) Report type for the TikTok Ads API. If not specified, + defaults to BASIC. + description: >- + User-defined custom report for the TikTok Ads connector. Groups the + dimensions + metrics + + report type + data level that define a TikTok Ads custom report request. + Transformer: + type: object + properties: + avro_options: + $ref: '#/components/schemas/AvroTransformerOptions' + format: + $ref: '#/components/schemas/TransformerFormat' + description: 'Required: the wire format of the data.' + input_column: + type: string + description: >- + Optional input column to transform. When set, the transformer reads + from this column instead of the default source column. + json_options: + $ref: '#/components/schemas/JsonTransformerOptions' + output_column: + type: string + description: >- + Optional output column name. When set, the transformed result is + written to this column instead of replacing the input column. + protobuf_options: + $ref: '#/components/schemas/ProtobufTransformerOptions' + description: Specifies how to transform binary data into structured data. Truncation: type: object properties: @@ -7284,12 +19462,27 @@ components: refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh. + mode: + $ref: '#/components/schemas/UpdateMode' + description: >- + Indicates whether the update is either part of a continuous job run, + or running in legacy continuous pipeline mode. Returned only for + GetUpdate; not populated in ListUpdates responses. parameters: type: object description: Key/value map of parameters used to initiate the update pipeline_id: type: string description: The ID of the pipeline. + refresh_flow_selection: + type: array + items: + type: string + description: >- + Flow names to selectively refresh. These are unioned with other + selective refresh options (refresh_selection, + full_refresh_selection, etc.) to determine the final set of flows to + refresh. refresh_selection: type: array items: @@ -7299,6 +19492,12 @@ components: refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh. + rewind_spec: + $ref: '#/components/schemas/RewindSpec' + description: >- + The rewind specification for this update. If set, this update is a + rewind execution (the update will rewind the targeted datasets to + the requested timestamp instead of running a normal refresh). state: $ref: '#/components/schemas/UpdateInfoState' description: The update state. @@ -7319,6 +19518,71 @@ components: $ref: '#/components/schemas/UpdateStateInfoState' update_id: type: string + ZendeskSupportOptions: + type: object + properties: + start_date: + type: string + description: >- + (Optional) Start date in YYYY-MM-DD format for the initial sync. + This determines the earliest date from which to sync historical + data. + description: Zendesk Support specific options for ingestion + BingAdsOptionsBingCustomReportOptionsBingAdsGranularity: + type: string + x-enum: + - DAILY + - HOURLY + - MONTHLY + - SUMMARY + - WEEKLY + - YEARLY + description: Time granularity for Bing Ads report aggregation. + BingAdsOptionsBingCustomReportOptionsBingAdsReportType: + type: string + x-enum: + - ACCOUNT_PERFORMANCE + - AD_DYNAMIC_TEXT_PERFORMANCE + - AD_EXTENSION_BY_AD + - AD_EXTENSION_BY_KEYWORD + - AD_EXTENSION_DETAIL + - AD_GROUP_PERFORMANCE + - AD_PERFORMANCE + - AGE_GENDER_AUDIENCE + - APPS_PERFORMANCE + - ASSET_GROUP_PERFORMANCE + - ASSET_PERFORMANCE + - AUDIENCE_PERFORMANCE + - BID_STRATEGY + - BUDGET_SUMMARY + - CALL_DETAIL + - CAMPAIGN_PERFORMANCE + - COMBINATION_PERFORMANCE + - CONVERSION_PERFORMANCE + - DESTINATION_URL_PERFORMANCE + - DSA_AUTO_TARGET_PERFORMANCE + - DSA_CATEGORY_PERFORMANCE + - DSA_SEARCH_QUERY_PERFORMANCE + - GEOGRAPHIC_PERFORMANCE + - GOALS_AND_FUNNELS + - HOTEL_DIMENSION_PERFORMANCE + - HOTEL_GROUP_PERFORMANCE + - KEYWORD_PERFORMANCE + - MS_CLICK_ID_PERFORMANCE + - NEGATIVE_KEYWORD_CONFLICT + - PRODUCT_DIMENSION_PERFORMANCE + - PRODUCT_MATCH_COUNT + - PRODUCT_PARTITION_PERFORMANCE + - PRODUCT_PARTITION_UNIT_PERFORMANCE + - PRODUCT_SEARCH_QUERY_PERFORMANCE + - PROFESSIONAL_DEMOGRAPHICS_AUDIENCE + - PUBLISHER_USAGE_PERFORMANCE + - SEARCH_CAMPAIGN_CHANGE_HISTORY + - SEARCH_INSIGHT_PERFORMANCE + - SEARCH_QUERY_PERFORMANCE + - SHARE_OF_VOICE + - USER_LOCATION_PERFORMANCE + description: Supported Microsoft Advertising report types for custom reports. CloneMode: type: string x-enum: @@ -7351,48 +19615,290 @@ components: Days of week in which the window is allowed to happen. If not specified all days of the week - will be used. - DeploymentKind: + will be used. + DeploymentKind: + type: string + x-enum: + - BUNDLE + description: |- + The deployment method that manages the pipeline: + + - BUNDLE: The pipeline is managed by a Databricks Asset Bundle. + EventLevel: + type: string + x-enum: + - ERROR + - INFO + - METRICS + - WARN + description: The severity level of the event. + FileIngestionOptionsFileFormat: + type: string + x-enum: + - AVRO + - BINARYFILE + - CSV + - EXCEL + - FILE + - JSON + - ORC + - PARQUET + - XML + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + FileIngestionOptionsSchemaEvolutionMode: type: string x-enum: - - BUNDLE + - ADD_NEW_COLUMNS + - ADD_NEW_COLUMNS_WITH_TYPE_WIDENING + - FAIL_ON_NEW_COLUMNS + - NONE + - RESCUE description: >- - The deployment method that manages the pipeline: - BUNDLE: The pipeline - is managed by a + Based on - Databricks Asset Bundle. - EventLevel: - type: string - x-enum: - - ERROR - - INFO - - METRICS - - WARN - description: The severity level of the event. + https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema#how-does-auto-loader-schema-evolution-work GetPipelineResponseHealth: type: string x-enum: - HEALTHY - UNHEALTHY description: The health of a pipeline. + GoogleDriveOptionsGoogleDriveEntityType: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - PERMISSION + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + GoogleDriveOptionsGoogleDriveIngestionScope: + type: string + x-enum: + - DOMAIN_ALL + - DOMAIN_SHARED_DRIVES + - DOMAIN_USER_DRIVES + - USER_ACCESSIBLE + - USER_DRIVE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. IngestionSourceType: type: string x-enum: + - ADOBE_CAMPAIGNS + - ADOBE_COMMERCE + - ADP_WORKFORCE_NOW + - AHA + - AIRTABLE + - AKAMAI_WAF + - AMPLITUDE + - API_SOURCE + - APPFIGURES + - APPLE_APP_STORE + - APPLE_SEARCH_ADS + - ATLASSIAN_ORGANIZATION + - AWIN + - AZURE_MONITOR_LOGS - BIGQUERY + - BING_ADS + - CERIDIAN_DAYFORCE + - COMMUNITY + - CONFLUENCE + - CROWDSTRIKE_EVENT_STREAM + - DELIGHTED - DYNAMICS365 + - EPIC_CLARITY - FOREIGN_CATALOG + - FRESHSERVICE + - FRONT - GA4_RAW_DATA + - GENESYS + - GITHUB + - GITLAB + - GMAIL + - GONG + - GOOGLE_ADS + - GOOGLE_ANALYTICS + - GOOGLE_CALENDAR + - GOOGLE_DRIVE + - GOOGLE_SEARCH_CONSOLE + - GOOGLE_WORKSPACE + - GUIDEWIRE + - GURU + - HIBOB + - HUBSPOT + - IRONCLAD + - JIRA + - KAFKA + - LINEAR + - LINKEDIN_ADS + - M365_AUDIT_LOGS - MANAGED_POSTGRESQL + - MARKETO + - META_MARKETING + - MICROSOFT_ENTRA_ID + - MICROSOFT_TEAMS + - MONDAY_COM - MYSQL + - NETSKOPE_LOGS - NETSUITE + - NOTION + - OKTA_SYSTEM_LOGS + - ONE_PASSWORD_EVENT_LOGS - ORACLE + - ORACLE_ELOQUA + - ORACLE_FUSION_CLOUD + - OUTLOOK + - PAGERDUTY + - PARTNERSTACK + - PENDO + - PINTEREST_ADS - POSTGRESQL + - PROOFPOINT_SIEM + - QUICKBOOKS + - RABBITMQ + - REDDIT_ADS + - REDSHIFT - SALESFORCE + - SALESFORCE_MARKETING_CLOUD + - SALESLOFT + - SAP_SUCCESSFACTORS + - SAS + - SENDGRID - SERVICENOW - SHAREPOINT + - SHOPIFY + - SLACK_ACCESS_AND_INTEGRATION_LOGS + - SLACK_AUDIT_LOGS + - SMARTSHEET + - SNAPCHAT_ADS + - SPLUNK + - SQLDW - SQLSERVER + - SQUARE - TERADATA + - TIKTOK_ADS + - VEEVA + - VEEVA_VAULT + - VERKADA + - WIZ_AUDIT_LOGS + - WORKDAY_ACTIVITY_LOGGING + - WORKDAY_HCM - WORKDAY_RAAS + - X_ADS + - YOUTUBE_ANALYTICS + - ZENDESK + - ZIP + - ZOHO_BOOKS + - ZOOM + - ZOOM_LOGS description: |- Create a collection of name/value pairs. @@ -7430,6 +19936,34 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + LinkedInAdsOptionsLinkedInAdsCustomReportOptionsLinkedInAdsEntityGranularity: + type: string + x-enum: + - CAMPAIGN + - CAMPAIGN_GROUP + - CREATIVE + description: Entity pivot to group by. + LinkedInAdsOptionsLinkedInAdsCustomReportOptionsLinkedInAdsFinder: + type: string + x-enum: + - ANALYTICS + - ATTRIBUTED_REVENUE_METRICS + - STATISTICS + description: >- + adAnalytics finder. Determines call shape, valid pivots, and metric + requirements. + LinkedInAdsOptionsLinkedInAdsCustomReportOptionsLinkedInAdsTimeGranularity: + type: string + x-enum: + - ALL + - DAILY + - MONTHLY + - YEARLY + description: >- + Time aggregation. Used by analytics/statistics; ignored for + attributedRevenueMetrics. Defaults + + to DAILY when unspecified. MaturityLevel: type: string x-enum: @@ -7437,6 +19971,35 @@ components: - EVOLVING - STABLE description: Maturity level for EventDetails. + OutlookAttachmentMode: + type: string + x-enum: + - ALL + - INLINE_ONLY + - NONE + - NON_INLINE_ONLY + description: Attachment behavior mode for Outlook ingestion + OutlookBodyFormat: + type: string + x-enum: + - TEXT_HTML + - TEXT_PLAIN + description: Body format for Outlook email content + ParseMode: + type: string + x-enum: + - FAILFAST + - PERMISSIVE + description: >- + Determines how errors encountered while deserializing records are + handled. + PeriodicTriggerTimeUnit: + type: string + x-enum: + - DAYS + - HOURS + - WEEKS + description: Time unit enums for different time units. PipelineClusterAutoscaleMode: type: string x-enum: @@ -7449,10 +20012,10 @@ components: cluster resources based on workload volume, with minimal impact to the data processing latency - of your pipelines. Enhanced Autoscaling is available for `updates` + of your pipelines. Enhanced Autoscaling is available for ``updates`` clusters only. The legacy - autoscaling feature is used for `maintenance` clusters. + autoscaling feature is used for ``maintenance`` clusters. PipelinePermissionLevel: type: string x-enum: @@ -7486,6 +20049,52 @@ components: - DEFAULT_PUBLISHING_MODE - LEGACY_PUBLISHING_MODE description: Enum representing the publishing mode of a pipeline. + SharepointOptionsSharepointEntityType: + type: string + x-enum: + - FILE + - FILE_METADATA + - FILE_PERMISSION + - GROUP_MEMBERSHIP + - LIST + - PERMISSION + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. StartUpdateCause: type: string x-enum: @@ -7497,6 +20106,15 @@ components: - SERVICE_UPGRADE - USER_ACTION description: What triggered this update. + StorageMode: + type: string + x-enum: + - APPEND_ONLY + - SCD_TYPE_1 + - SCD_TYPE_2 + description: >- + Defines how ingested data is written and maintained in the destination + table. TableSpecificConfigScdType: type: string x-enum: @@ -7504,6 +20122,68 @@ components: - SCD_TYPE_1 - SCD_TYPE_2 description: The SCD type to use to ingest the table. + TikTokAdsOptionsTikTokDataLevel: + type: string + x-enum: + - AUCTION_AD + - AUCTION_ADGROUP + - AUCTION_ADVERTISER + - AUCTION_CAMPAIGN + description: Data level for TikTok Ads report aggregation. + TikTokAdsOptionsTikTokReportType: + type: string + x-enum: + - AUDIENCE + - BASIC + - BUSINESS_CENTER + - DSA + - GMV_MAX + - PLAYABLE_AD + description: Report type for TikTok Ads API. + TransformerFormat: + type: string + x-enum: + - AVRO + - JSON + - PROTOBUF + - STRING + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. UpdateInfoCause: type: string x-enum: @@ -7530,6 +20210,48 @@ components: - STOPPING - WAITING_FOR_RESOURCES description: The update state. + UpdateMode: + type: string + x-enum: + - CONTINUOUS + - DEFAULT + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. UpdateStateInfoState: type: string x-enum: @@ -7551,6 +20273,13 @@ components: name: pipelines title: Pipelines methods: + apply_environment: + operation: + $ref: >- + #/paths/~1api~12.0~1pipelines~1{pipeline_id}~1environment~1apply/post + response: + mediaType: application/json + openAPIDocKey: '200' clone: config: requestBodyTranslate: @@ -7597,6 +20326,12 @@ components: response: mediaType: application/json openAPIDocKey: '200' + restore: + operation: + $ref: '#/paths/~1api~12.0~1pipelines~1{pipeline_id}~1restore/post' + response: + mediaType: application/json + openAPIDocKey: '200' stop: operation: $ref: '#/paths/~1api~12.0~1pipelines~1{pipeline_id}~1stop/post' diff --git a/providers/src/databricks_workspace/v00.00.00000/services/postgres.yaml b/providers/src/databricks_workspace/v00.00.00000/services/postgres.yaml index ad09d62c..c088da66 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/postgres.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/postgres.yaml @@ -1,60 +1,62 @@ openapi: 3.0.0 info: title: Databricks Postgres API (workspace) - description: >- - OpenAPI specification for the Databricks postgres service (workspace-level - APIs), generated from the Databricks Python SDK. + description: OpenAPI specification for the Databricks postgres service (workspace-level APIs), generated from the Databricks + Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.postgres servers: - - url: https://{deployment_name}.cloud.databricks.com - variables: - deployment_name: - description: The Databricks Workspace Deployment Name - default: dbc-abcd0123-a1bc +- url: https://{deployment_name}.cloud.databricks.com + variables: + deployment_name: + description: The Databricks Workspace Deployment Name + default: dbc-abcd0123-a1bc paths: - /api/2.0/postgres/projects/{project_id}/branches: + /api/2.0/postgres/{parent}/branches: post: operationId: postgres_create_branch summary: Creates a new database branch in the project. tags: - - postgres + - postgres description: |- Creates a new database branch in the project. - :param project_id: str + :param parent: str The Project where this Branch will be created. Format: projects/{project_id} :param branch: :class:`Branch` The Branch to create. :param branch_id: str The ID to use for the Branch. This becomes the final component of the branch's resource name. The ID is required and must be 1-63 characters long, start with a lowercase letter, and contain only - lowercase letters, numbers, and hyphens. For example, `development` becomes - `projects/my-app/branches/development`. + lowercase letters, numbers, and hyphens. For example, ``development`` becomes + ``projects/my-app/branches/development``. + :param replace_existing: bool (optional) + If true, update the branch if it already exists instead of returning an error. :returns: :class:`Operation` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The Project where this Branch will be created. Format: - projects/{project_id} - - name: branch_id - in: query - required: true - schema: - type: string - description: >- - The ID to use for the Branch. This becomes the final component of - the branch's resource name. The ID is required and must be 1-63 - characters long, start with a lowercase letter, and contain only - lowercase letters, numbers, and hyphens. For example, `development` - becomes `projects/my-app/branches/development`. + - name: parent + in: path + required: true + schema: + type: string + description: 'The Project where this Branch will be created. Format: projects/{project_id}' + - name: branch_id + in: query + required: true + schema: + type: string + description: The ID to use for the Branch. This becomes the final component of the branch's resource name. The ID + is required and must be 1-63 characters long, start with a lowercase letter, and contain only lowercase letters, + numbers, and hyphens. For example, ``development`` becomes ``projects/my-app/branches/development``. + - name: replace_existing + in: query + required: false + schema: + type: boolean + description: If true, update the branch if it already exists instead of returning an error. requestBody: content: application/json: @@ -64,22 +66,20 @@ paths: branch: type: object properties: + branch_id: + type: string create_time: type: string format: date-time + description: A timestamp indicating when the branch was created. name: type: string - description: >- - Output only. The full resource path of the branch. - Format: projects/{project_id}/branches/{branch_id} + description: 'Output only. The full resource path of the branch. Format: projects/{project_id}/branches/{branch_id}' parent: type: string - description: >- - The project containing this branch (API resource - hierarchy). Format: projects/{project_id} Note: This - field indicates where the branch exists in the resource - hierarchy. For point-in-time branching from another - branch, see `status.source_branch`. + description: 'The project containing this branch (API resource hierarchy). Format: projects/{project_id} + Note: This field indicates where the branch exists in the resource hierarchy. For point-in-time branching + from another branch, see ``status.source_branch``.' spec: type: object properties: @@ -88,64 +88,57 @@ paths: format: date-time is_protected: type: boolean - description: >- - When set to true, protects the branch from deletion - and reset. Associated compute endpoints and the - project cannot be deleted while the branch is - protected. + description: When set to true, protects the branch from deletion and reset. Associated compute endpoints + and the project cannot be deleted while the branch is protected. no_expiry: type: boolean - description: >- - Explicitly disable expiration. When set to true, the - branch will not expire. If set to false, the request - is invalid; provide either ttl or expire_time - instead. + description: Explicitly disable expiration. When set to true, the branch will not expire. If set + to false, the request is invalid; provide either ttl or expire_time instead. Mutually exclusive + with ``expire_time`` and ``ttl``. When updating, use ``spec.expiration`` in the update_mask. source_branch: type: string - description: >- - The name of the source branch from which this branch - was created (data lineage for point-in-time - recovery). If not specified, defaults to the - project's default branch. Format: - projects/{project_id}/branches/{branch_id} + description: 'The name of the source branch from which this branch was created (data lineage for + point-in-time recovery). If not specified, defaults to the project''s default branch. Format: + projects/{project_id}/branches/{branch_id}' source_branch_lsn: type: string - description: >- - The Log Sequence Number (LSN) on the source branch - from which this branch was created. + description: The Log Sequence Number (LSN) on the source branch from which this branch was created. source_branch_time: type: string format: date-time - description: >- - The point in time on the source branch from which - this branch was created. + description: The point in time on the source branch from which this branch was created. ttl: type: string - description: >- - Relative time-to-live duration. When set, the branch - will expire at creation_time + ttl. + description: Relative time-to-live duration. When set, the branch will expire at creation_time + + ttl. Mutually exclusive with ``expire_time`` and ``no_expiry``. When updating, use ``spec.expiration`` + in the update_mask. description: The spec contains the branch configuration. status: type: object properties: + branch_id: + type: string current_state: type: string x-enum: - - ARCHIVED - - IMPORTING - - INIT - - READY - - RESETTING - description: The state of the branch. + - ARCHIVED + - DELETED + - IMPORTING + - INIT + - READY + - RESETTING + description: The branch's state, indicating if it is initializing, ready for use, or archived. default: type: boolean description: Whether the branch is the project's default branch. + delete_time: + type: string + format: date-time + description: A timestamp indicating when the branch was deleted. Empty if the branch is not deleted. expire_time: type: string format: date-time - description: >- - Absolute expiration time for the branch. Empty if - expiration is disabled. + description: Absolute expiration time for the branch. Empty if expiration is disabled. is_protected: type: boolean description: Whether the branch is protected. @@ -155,37 +148,36 @@ paths: pending_state: type: string x-enum: - - ARCHIVED - - IMPORTING - - INIT - - READY - - RESETTING - description: >- - The pending state of the branch, if a state - transition is in progress. + - ARCHIVED + - DELETED + - IMPORTING + - INIT + - READY + - RESETTING + description: The pending state of the branch, if a state transition is in progress. + purge_time: + type: string + format: date-time + description: A timestamp indicating when the branch is scheduled to be purged. Empty if the branch + is not deleted, otherwise set to a timestamp in the future. source_branch: type: string - description: >- - The name of the source branch from which this branch - was created. Format: - projects/{project_id}/branches/{branch_id} + description: 'The name of the source branch from which this branch was created. Format: projects/{project_id}/branches/{branch_id}' source_branch_lsn: type: string - description: >- - The Log Sequence Number (LSN) on the source branch - from which this branch was created. + description: The Log Sequence Number (LSN) on the source branch from which this branch was created. source_branch_time: type: string format: date-time - description: >- - The point in time on the source branch from which - this branch was created. + description: The point in time on the source branch from which this branch was created. + source_recovery_branch: + type: string + description: 'If this branch is a child of a recovery branch, this field identifies that recovery + source. For non-recovery-derived branches this is unset. Format: projects/{project_id}/preview/recovery-branches/{recovery_branch_id}' state_change_time: type: string format: date-time - description: >- - A timestamp indicating when the `current_state` - began. + description: A timestamp indicating when the ``current_state`` began. description: The current status of a Branch. uid: type: string @@ -196,7 +188,7 @@ paths: description: A timestamp indicating when the branch was last updated. description: The Branch to create. required: - - branch + - branch responses: '200': description: Success @@ -220,7 +212,7 @@ paths: operationId: postgres_list_branches summary: Returns a paginated list of database branches in the project. tags: - - postgres + - postgres description: |- Returns a paginated list of database branches in the project. @@ -230,31 +222,37 @@ paths: Upper bound for items returned. Cannot be negative. :param page_token: str (optional) Page token from a previous response. If not provided, returns the first page. + :param show_deleted: bool (optional) + Whether to include soft-deleted branches in the response. When true, deleted branches are included + alongside active branches. Purged branches are never returned. :returns: Iterator over :class:`Branch` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The Project that owns this collection of branches. Format: - projects/{project_id} - - name: page_size - in: query - required: false - schema: - type: integer - description: Upper bound for items returned. Cannot be negative. - - name: page_token - in: query - required: false - schema: - type: string - description: >- - Page token from a previous response. If not provided, returns the - first page. + - name: parent + in: path + required: true + schema: + type: string + description: 'The Project that owns this collection of branches. Format: projects/{project_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. Cannot be negative. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token from a previous response. If not provided, returns the first page. + - name: show_deleted + in: query + required: false + schema: + type: boolean + description: Whether to include soft-deleted branches in the response. When true, deleted branches are included alongside + active branches. Purged branches are never returned. responses: '200': description: Success @@ -274,122 +272,190 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI - /api/2.0/postgres/projects/{project_id}/branches/{branch_id}/databases: + /api/2.0/postgres/catalogs: post: - operationId: postgres_create_database - summary: Create a Database. + operationId: postgres_create_catalog + summary: Register a Postgres database in the Unity Catalog. tags: - - postgres - description: >- - Create a Database. - - - Creates a database in the specified branch. A branch can have multiple - databases. - - - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch where this Database will be created. Format: projects/{project_id}/branches/{branch_id} - :param database: :class:`Database` - The desired specification of a Database. - :param database_id: str (optional) - The ID to use for the Database, which will become the final component of the database's resource - name. This ID becomes the database name in postgres. - - This value should be 4-63 characters, and only use characters available in DNS names, as defined by - RFC-1123 + - postgres + description: |- + Register a Postgres database in the Unity Catalog. - If database_id is not specified in the request, it is generated automatically. + :param catalog: :class:`Catalog` + :param catalog_id: str + The ID in the Unity Catalog. It becomes the full resource name, for example "my_catalog" becomes + "catalogs/my_catalog". :returns: :class:`Operation` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch where this Database will be - created. Format: projects/{project_id}/branches/{branch_id} - - name: database_id - in: query - required: false - schema: - type: string - description: >- - The ID to use for the Database, which will become the final - component of the database's resource name. This ID becomes the - database name in postgres. This value should be 4-63 characters, and - only use characters available in DNS names, as defined by RFC-1123 - If database_id is not specified in the request, it is generated - automatically. + - name: catalog_id + in: query + required: true + schema: + type: string + description: The ID in the Unity Catalog. It becomes the full resource name, for example "my_catalog" becomes "catalogs/my_catalog". requestBody: content: application/json: schema: type: object properties: - database: + catalog: type: object properties: + catalog_id: + type: string create_time: type: string format: date-time - description: A timestamp indicating when the database was created. + description: A timestamp indicating when the catalog was created. name: type: string - description: >- - The resource name of the database. Format: - projects/{project_id}/branches/{branch_id}/databases/{database_id} - parent: - type: string - description: >- - The branch containing this database. Format: - projects/{project_id}/branches/{branch_id} + description: 'Output only. The full resource path of the catalog. Format: "catalogs/{catalog_id}".' spec: type: object properties: postgres_database: type: string - role: + description: The name of the Postgres database inside the specified Lakebase project and branch + to be associated with the UC catalog. This database must already exist, unless create_database_if_missing + is set to true on creation. A database can only be registered with one UC catalog at a time. To + re-register a database with a different catalog, the existing catalog must be deleted first. A + child branch inherits the fact of parent's registration. This means the same-named database in + a child branch cannot be registered with a second catalog while the parent's registration exists. + To allow registering the database of a child branch, drop and recreate the database on the child + branch. This removes the fact of parent's registration from this branch only. Doing Point In Time + Restore (PITR) prior to the moment before the Postgres DB was registered in the Catalog drops + the fact of registration of the database. So the user should avoid doing so. + branch: type: string - description: >- - The name of the role that owns the database. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} - To change the owner, pass valid existing Role name - when updating the Database A database always has an - owner. - description: The desired state of the Database. + description: 'The resource path of the branch associated with the catalog. Format: projects/{project_id}/branches/{branch_id}.' + create_database_if_missing: + type: boolean + description: If set to true, the specified postgres_database is created on behalf of the calling + user if it does not already exist. In this case, the calling user has a role created for them + in Postgres if they do not already have one. Defaults to false, meaning that the request fails + if the specified postgres_database does not already exist. + required: + - postgres_database + description: The desired state of the Catalog. status: type: object properties: + branch: + type: string + description: 'The resource path of the branch associated with the catalog. Format: projects/{project_id}/branches/{branch_id}.' postgres_database: type: string - role: + description: The name of the Postgres database associated with the catalog. + project: type: string - description: >- - The name of the role that owns the database. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} - description: The observed state of the Database. + description: 'The resource path of the project associated with the catalog. Format: projects/{project_id}.' + description: The observed state of the Catalog. + uid: + type: string + description: System-generated unique identifier for the catalog. update_time: type: string format: date-time - description: >- - A timestamp indicating when the database was last - updated. - description: The desired specification of a Database. + description: A timestamp indicating when the catalog was last updated. + required: + - catalog + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/cdf-configs: + post: + operationId: postgres_create_cdf_config + summary: Create a CDF configuration that materializes the change data feed for all tables in a Postgres schema + tags: + - postgres + description: |- + Create a CDF configuration that materializes the change data feed for all tables in a Postgres schema + as open-format Delta tables in Unity Catalog. Once created, each table's change history is + continuously written to its corresponding Lakehouse table. + + :param parent: str + The parent database under which to create the CdfConfig. Format: + projects/{project}/branches/{branch}/databases/{database} + :param cdf_config: :class:`CdfConfig` + The CdfConfig to create. The catalog, schema, and postgres_schema fields are required; all other + fields are output only and ignored on input. + :param cdf_config_id: str (optional) + The user-specified id for the CdfConfig, forming the final segment of its resource name. Must match + the pattern ``[a-z][a-z0-9_]{0,62}``. Defaults to the Postgres schema name when omitted. + + :returns: :class:`Operation` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent database under which to create the CdfConfig. Format: projects/{project}/branches/{branch}/databases/{database}' + - name: cdf_config_id + in: query + required: false + schema: + type: string + description: The user-specified id for the CdfConfig, forming the final segment of its resource name. Must match the + pattern ``[a-z][a-z0-9_]{0,62}``. Defaults to the Postgres schema name when omitted. + requestBody: + content: + application/json: + schema: + type: object + properties: + cdf_config: + type: object + properties: + catalog: + type: string + description: The Unity Catalog catalog that replicated tables are written into. Set at creation; the + CdfConfig is immutable. + schema: + type: string + description: The Unity Catalog schema that replicated tables are written into. Set at creation; the + CdfConfig is immutable. + postgres_schema: + type: string + description: The Postgres schema this CdfConfig replicates from. Unique within the parent database. + Set at creation; the CdfConfig is immutable. + cdf_config_id: + type: string + description: The user-specified id; equals the final segment of ``name``. Defaults to the Postgres schema + name for configs without an explicit id. + create_time: + type: string + format: date-time + description: When the CdfConfig was created. + name: + type: string + description: 'Output only. The full resource name of the CdfConfig. Format: projects/{project}/branches/{branch}/databases/{database}/cdf-configs/{cdf_config}' + required: + - catalog + - schema + - postgres_schema + description: The CdfConfig to create. The catalog, schema, and postgres_schema fields are required; all + other fields are output only and ignored on input. required: - - database + - cdf_config responses: '200': description: Success @@ -410,61 +476,49 @@ paths: type: string x-stackql-sdk-source: PostgresAPI get: - operationId: postgres_list_databases - summary: List Databases. + operationId: postgres_list_cdf_configs + summary: List all CDF configurations for a Lakebase database. Each configuration maps a Postgres schema to a tags: - - postgres + - postgres description: |- - List Databases. + List all CDF configurations for a Lakebase database. Each configuration maps a Postgres schema to a + Unity Catalog schema where the change data feed is materialized. - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch that owns this collection of databases. Format: projects/{project_id}/branches/{branch_id} + :param parent: str + The parent database to list CdfConfigs for. Format: + projects/{project}/branches/{branch}/databases/{database} :param page_size: int (optional) - Upper bound for items returned. + Maximum number of CdfConfigs to return. :param page_token: str (optional) - Pagination token to go to the next page of Databases. Requests first page if absent. + Pagination token returned by a previous ListCdfConfigs call. Empty on the first page. - :returns: Iterator over :class:`Database` + :returns: Iterator over :class:`CdfConfig` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch that owns this collection of - databases. Format: projects/{project_id}/branches/{branch_id} - - name: page_size - in: query - required: false - schema: - type: integer - description: Upper bound for items returned. - - name: page_token - in: query - required: false - schema: - type: string - description: >- - Pagination token to go to the next page of Databases. Requests first - page if absent. + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent database to list CdfConfigs for. Format: projects/{project}/branches/{branch}/databases/{database}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Maximum number of CdfConfigs to return. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token returned by a previous ListCdfConfigs call. Empty on the first page. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListDatabasesResponse' + $ref: '#/components/schemas/ListCdfConfigsResponse' default: description: Error response content: @@ -477,277 +531,153 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI - /api/2.0/postgres/projects/{project_id}/branches/{branch_id}/endpoints: + /api/2.0/postgres/{parent}/data-api: post: - operationId: postgres_create_endpoint - summary: Creates a new compute endpoint in the branch. + operationId: postgres_create_data_api + summary: Enable Data API for a database. tags: - - postgres + - postgres description: |- - Creates a new compute endpoint in the branch. + Enable Data API for a database. - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch where this Endpoint will be created. Format: projects/{project_id}/branches/{branch_id} - :param endpoint: :class:`Endpoint` - The Endpoint to create. - :param endpoint_id: str - The ID to use for the Endpoint. This becomes the final component of the endpoint's resource name. - The ID is required and must be 1-63 characters long, start with a lowercase letter, and contain only - lowercase letters, numbers, and hyphens. For example, `primary` becomes - `projects/my-app/branches/development/endpoints/primary`. + :param parent: str + Parent database: projects/{project_id}/branches/{branch_id}/databases/{database_id} + :param data_api: :class:`DataApi` + The Data API configuration to create. :returns: :class:`Operation` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch where this Endpoint will be - created. Format: projects/{project_id}/branches/{branch_id} - - name: endpoint_id - in: query - required: true - schema: - type: string - description: >- - The ID to use for the Endpoint. This becomes the final component of - the endpoint's resource name. The ID is required and must be 1-63 - characters long, start with a lowercase letter, and contain only - lowercase letters, numbers, and hyphens. For example, `primary` - becomes `projects/my-app/branches/development/endpoints/primary`. + - name: parent + in: path + required: true + schema: + type: string + description: 'Parent database: projects/{project_id}/branches/{branch_id}/databases/{database_id}' requestBody: content: application/json: schema: type: object properties: - endpoint: + data_api: type: object properties: create_time: type: string format: date-time + description: A timestamp indicating when the Data API was first enabled. name: type: string - description: >- - Output only. The full resource path of the endpoint. - Format: - projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} + description: 'Resource name: projects/{project_id}/branches/{branch_id}/databases/{database_id}/data-api' parent: type: string - description: >- - The branch containing this endpoint (API resource - hierarchy). Format: - projects/{project_id}/branches/{branch_id} + description: 'The database containing this Data API configuration. Format: projects/{project_id}/branches/{branch_id}/databases/{database_id}' spec: type: object properties: - endpoint_type: + db_aggregates_enabled: + type: boolean + description: 'Enable aggregate functions (count, sum, avg, etc.) in Data API responses. Default: + true.' + db_anon_role: + type: string + description: 'The PostgreSQL role used for unauthenticated (anonymous) requests. Must be a valid + PostgreSQL role name (1-63 chars, [a-zA-Z_][a-zA-Z0-9_$]*). Default: "anonymous".' + db_extra_search_path: + type: array + items: + type: string + description: Additional schemas to include in the PostgreSQL search path. Each entry must be a valid + PostgreSQL schema name. + db_max_rows: + type: integer + description: Maximum number of rows returned in a single Data API response. Must be a positive integer. + db_schemas: + type: array + items: + type: string + description: 'Database schemas exposed through the Data API. Each entry must be a valid PostgreSQL + schema name (1-63 chars, [a-zA-Z_][a-zA-Z0-9_$]*). Maximum 100 entries. Default: ["public"].' + jwt_cache_max_lifetime: + type: string + description: Maximum lifetime for cached JWT tokens. Zero duration disables caching. + jwt_role_claim_key: + type: string + description: 'JSON path to the role claim in JWT tokens (e.g., ".sub"). Default: ".sub".' + openapi_mode: type: string x-enum: - - ENDPOINT_TYPE_READ_ONLY - - ENDPOINT_TYPE_READ_WRITE - description: >- - The compute endpoint type. Either `read_write` or - `read_only`. - autoscaling_limit_max_cu: - type: number - description: >- - The maximum number of Compute Units. Minimum value - is 0.5. - autoscaling_limit_min_cu: - type: number - description: >- - The minimum number of Compute Units. Minimum value - is 0.5. - disabled: - type: boolean - description: >- - Whether to restrict connections to the compute - endpoint. Enabling this option schedules a suspend - compute operation. A disabled compute endpoint - cannot be enabled by a connection or console action. - group: - type: object - properties: - min: - type: integer - max: - type: integer - description: >- - The maximum number of computes in the endpoint - group. Currently, this must be equal to min. Set - to 1 for single compute endpoints, to disable - HA. To manually suspend all computes in an - endpoint group, set disabled to true on the - EndpointSpec. - enable_readable_secondaries: - type: boolean - description: >- - Whether to allow read-only connections to - read-write endpoints. Only relevant for - read-write endpoints where size.max > 1. - required: - - min - - max - description: >- - Settings for optional HA configuration of the - endpoint. If unspecified, the endpoint defaults to - non HA settings, with a single compute backing the - endpoint (and no readable secondaries for Read/Write - endpoints). - no_suspension: + - OPEN_API_MODE_DISABLED + - OPEN_API_MODE_IGNORE_PRIVILEGES + description: OpenAPI documentation mode for the Data API endpoint. + server_cors_allowed_origins: + type: array + items: + type: string + description: Allowed origins for CORS requests. Each entry should be a valid origin URL, or use + "*" to allow all origins. + server_timing_enabled: type: boolean - description: >- - When set to true, explicitly disables automatic - suspension (never suspend). Should be set to true - when provided. - settings: - type: object - properties: - pg_settings: - type: object - description: A raw representation of Postgres settings. - description: A collection of settings for a compute endpoint. - suspend_timeout_duration: - type: string - description: >- - Duration of inactivity after which the compute - endpoint is automatically suspended. If specified - should be between 60s and 604800s (1 minute to 1 - week). - required: - - endpoint_type - description: >- - The spec contains the compute endpoint configuration, - including autoscaling limits, suspend timeout, and - disabled state. + description: Enable the Server-Timing header in Data API responses. + description: The desired Data API configuration. status: type: object properties: - autoscaling_limit_max_cu: - type: number - autoscaling_limit_min_cu: - type: number - description: The minimum number of Compute Units. - current_state: - type: string - x-enum: - - ACTIVE - - DEGRADED - - IDLE - - INIT - description: The state of the compute endpoint. - disabled: + available_schemas: + type: array + items: + type: string + description: Schemas available in the database (for reference when configuring db_schemas). + db_aggregates_enabled: type: boolean - description: >- - Whether to restrict connections to the compute - endpoint. Enabling this option schedules a suspend - compute operation. A disabled compute endpoint - cannot be enabled by a connection or console action. - endpoint_type: + description: Actual aggregate function setting read from the database. + db_anon_role: type: string - x-enum: - - ENDPOINT_TYPE_READ_ONLY - - ENDPOINT_TYPE_READ_WRITE - description: >- - The endpoint type. A branch can only have one - READ_WRITE endpoint. - group: - type: object - properties: - min: - type: integer - max: - type: integer - description: >- - The maximum number of computes in the endpoint - group. Currently, this must be equal to min. Set - to 1 for single compute endpoints, to disable - HA. To manually suspend all computes in an - endpoint group, set disabled to true on the - EndpointSpec. - enable_readable_secondaries: - type: boolean - description: >- - Whether read-only connections to read-write - endpoints are allowed. Only relevant if read - replicas are configured by specifying size.max > - 1. - required: - - min - - max - description: Details on the HA configuration of the endpoint. - hosts: - type: object - properties: - host: - type: string - description: >- - The hostname to connect to this endpoint. For - read-write endpoints, this is a read-write - hostname which connects to the primary compute. - For read-only endpoints, this is a read-only - hostname which allows read-only operations. - read_only_host: - type: string - description: >- - An optionally defined read-only host for the - endpoint, without pooling. For read-only - endpoints, this attribute is always defined and - is equivalent to host. For read-write endpoints, - this attribute is defined if the enclosing - endpoint is a group with greater than 1 computes - configured, and has readable secondaries - enabled. - description: >- - Contains host information for connecting to the - endpoint. - pending_state: + description: Actual anonymous role name read from the database. + db_extra_search_path: + type: array + items: + type: string + description: Actual extra search path schemas read from the database. + db_max_rows: + type: integer + description: Actual max rows setting read from the database. + db_schemas: + type: array + items: + type: string + description: Actual exposed schemas read from the database. + jwt_cache_max_lifetime: + type: string + description: Actual JWT cache max lifetime read from the database. + jwt_role_claim_key: + type: string + description: Actual JWT role claim key read from the database. + openapi_mode: type: string x-enum: - - ACTIVE - - DEGRADED - - IDLE - - INIT - description: The state of the compute endpoint. - settings: - type: object - properties: - pg_settings: - type: object - description: A raw representation of Postgres settings. - description: A collection of settings for a compute endpoint. - suspend_timeout_duration: + - OPEN_API_MODE_DISABLED + - OPEN_API_MODE_IGNORE_PRIVILEGES + description: Actual OpenAPI mode read from the database. + server_cors_allowed_origins: + type: array + items: + type: string + description: Actual CORS allowed origins read from the database. + server_timing_enabled: + type: boolean + description: Actual Server-Timing header setting read from the database. + url: type: string - description: >- - Duration of inactivity after which the compute - endpoint is automatically suspended. - description: Current operational status of the compute endpoint. - uid: - type: string - description: System-generated unique ID for the endpoint. + description: Data API endpoint URL. + description: The observed Data API state (read-only). update_time: type: string format: date-time - description: >- - A timestamp indicating when the compute endpoint was - last updated. - description: The Endpoint to create. + description: A timestamp indicating when the Data API configuration was last updated. + description: The Data API configuration to create. required: - - endpoint + - data_api responses: '200': description: Success @@ -767,331 +697,109 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI - get: - operationId: postgres_list_endpoints - summary: Returns a paginated list of compute endpoints in the branch. + /api/2.0/postgres/{parent}/databases: + post: + operationId: postgres_create_database + summary: Create a Database. tags: - - postgres + - postgres description: |- - Returns a paginated list of compute endpoints in the branch. + Create a Database. - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch that owns this collection of endpoints. Format: projects/{project_id}/branches/{branch_id} - :param page_size: int (optional) - Upper bound for items returned. Cannot be negative. - :param page_token: str (optional) - Page token from a previous response. If not provided, returns the first page. - - :returns: Iterator over :class:`Endpoint` - parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch that owns this collection of - endpoints. Format: projects/{project_id}/branches/{branch_id} - - name: page_size - in: query - required: false - schema: - type: integer - description: Upper bound for items returned. Cannot be negative. - - name: page_token - in: query - required: false - schema: - type: string - description: >- - Page token from a previous response. If not provided, returns the - first page. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListEndpointsResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PostgresAPI - /api/2.0/postgres/projects: - post: - operationId: postgres_create_project - summary: >- - Creates a new Lakebase Autoscaling Postgres database project, which - contains branches and compute - tags: - - postgres - description: >- - Creates a new Lakebase Autoscaling Postgres database project, which - contains branches and compute + Creates a database in the specified branch. A branch can have multiple databases. - endpoints. + :param parent: str + The Branch where this Database will be created. Format: projects/{project_id}/branches/{branch_id} + :param database: :class:`Database` + The desired specification of a Database. + :param database_id: str (optional) + The ID to use for the Database, which will become the final component of the database's resource + name. This ID becomes the database name in postgres. + This value should be 4-63 characters, and only use characters available in DNS names, as defined by + RFC-1123 - :param project: :class:`Project` - The Project to create. - :param project_id: str - The ID to use for the Project. This becomes the final component of the project's resource name. The - ID is required and must be 1-63 characters long, start with a lowercase letter, and contain only - lowercase letters, numbers, and hyphens. For example, `my-app` becomes `projects/my-app`. + If database_id is not specified in the request, it is generated automatically. + :param replace_existing: bool (optional) + If true, update the database if it already exists instead of returning an error. :returns: :class:`Operation` parameters: - - name: project_id - in: query - required: true - schema: - type: string - description: >- - The ID to use for the Project. This becomes the final component of - the project's resource name. The ID is required and must be 1-63 - characters long, start with a lowercase letter, and contain only - lowercase letters, numbers, and hyphens. For example, `my-app` - becomes `projects/my-app`. + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch where this Database will be created. Format: projects/{project_id}/branches/{branch_id}' + - name: database_id + in: query + required: false + schema: + type: string + description: The ID to use for the Database, which will become the final component of the database's resource name. + This ID becomes the database name in postgres. This value should be 4-63 characters, and only use characters available + in DNS names, as defined by RFC-1123 If database_id is not specified in the request, it is generated automatically. + - name: replace_existing + in: query + required: false + schema: + type: boolean + description: If true, update the database if it already exists instead of returning an error. requestBody: content: application/json: schema: type: object properties: - project: + database: type: object properties: create_time: type: string format: date-time - initial_endpoint_spec: - type: object - properties: - group: - type: object - properties: - min: - type: integer - max: - type: integer - description: >- - The maximum number of computes in the endpoint - group. Currently, this must be equal to min. Set - to 1 for single compute endpoints, to disable - HA. To manually suspend all computes in an - endpoint group, set disabled to true on the - EndpointSpec. - enable_readable_secondaries: - type: boolean - description: >- - Whether to allow read-only connections to - read-write endpoints. Only relevant for - read-write endpoints where size.max > 1. - required: - - min - - max - description: >- - Configuration settings for the initial Read/Write - endpoint created inside the default branch for a newly - created project. If omitted, the initial endpoint - created will have default settings, without high - availability configured. This field does not apply to - any endpoints created after project creation. Use - spec.default_endpoint_settings to configure default - settings for endpoints created after project creation. + description: A timestamp indicating when the database was created. + database_id: + type: string + description: The part of the name, chosen by the user when the resource was created. name: type: string - description: >- - Output only. The full resource path of the project. - Format: projects/{project_id} + description: 'The resource name of the database. Format: projects/{project_id}/branches/{branch_id}/databases/{database_id}' + parent: + type: string + description: 'The branch containing this database. Format: projects/{project_id}/branches/{branch_id}' spec: type: object properties: - budget_policy_id: - type: string - custom_tags: - type: array - items: - type: object - properties: - key: - type: string - value: - type: string - description: The value of the custom tag. - description: >- - Custom tags to associate with the project. Forwarded - to LBM for billing and cost tracking. To update - tags, provide the new tag list and include - "spec.custom_tags" in the update_mask. To clear all - tags, provide an empty list and include - "spec.custom_tags" in the update_mask. To preserve - existing tags, omit this field from the update_mask - (or use wildcard "*" which auto-excludes empty - tags). - default_endpoint_settings: - type: object - properties: - autoscaling_limit_max_cu: - type: number - description: >- - The maximum number of Compute Units. Minimum - value is 0.5. - autoscaling_limit_min_cu: - type: number - description: >- - The minimum number of Compute Units. Minimum - value is 0.5. - no_suspension: - type: boolean - description: >- - When set to true, explicitly disables automatic - suspension (never suspend). Should be set to - true when provided. - pg_settings: - type: object - description: A raw representation of Postgres settings. - suspend_timeout_duration: - type: string - description: >- - Duration of inactivity after which the compute - endpoint is automatically suspended. If - specified should be between 60s and 604800s (1 - minute to 1 week). - description: A collection of settings for a compute endpoint. - display_name: + role: type: string - description: >- - Human-readable project name. Length should be - between 1 and 256 characters. - enable_pg_native_login: - type: boolean - description: >- - Whether to enable PG native password login on all - endpoints in this project. Defaults to true. - history_retention_duration: + postgres_database: type: string - description: >- - The number of seconds to retain the shared history - for point in time recovery for all branches in this - project. Value should be between 172800s (2 days) - and 2592000s (30 days). - pg_version: - type: integer - description: >- - The major Postgres version number. Supported - versions are 16 and 17. - description: >- - The spec contains the project configuration, including - display_name, pg_version (Postgres version), - history_retention_duration, and - default_endpoint_settings. + description: The name of the Postgres database. This expects a valid Postgres identifier as specified + in the link below. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS + Required when creating the Database. To rename, pass a valid postgres identifier when updating + the Database. + required: + - role + description: The desired state of the Database. status: type: object properties: - branch_logical_size_limit_bytes: - type: integer - budget_policy_id: - type: string - description: The budget policy that is applied to the project. - custom_tags: - type: array - items: - type: object - properties: - key: - type: string - value: - type: string - description: The value of the custom tag. - description: >- - The effective custom tags associated with the - project. - default_endpoint_settings: - type: object - properties: - autoscaling_limit_max_cu: - type: number - description: >- - The maximum number of Compute Units. Minimum - value is 0.5. - autoscaling_limit_min_cu: - type: number - description: >- - The minimum number of Compute Units. Minimum - value is 0.5. - no_suspension: - type: boolean - description: >- - When set to true, explicitly disables automatic - suspension (never suspend). Should be set to - true when provided. - pg_settings: - type: object - description: A raw representation of Postgres settings. - suspend_timeout_duration: - type: string - description: >- - Duration of inactivity after which the compute - endpoint is automatically suspended. If - specified should be between 60s and 604800s (1 - minute to 1 week). - description: The effective default endpoint settings. - display_name: + database_id: type: string - description: The effective human-readable project name. - enable_pg_native_login: - type: boolean - description: >- - Whether to enable PG native password login on all - endpoints in this project. - history_retention_duration: + postgres_database: type: string - description: >- - The effective number of seconds to retain the shared - history for point in time recovery. - owner: + description: The name of the Postgres database. + role: type: string - description: The email of the project owner. - pg_version: - type: integer - description: The effective major Postgres version number. - synthetic_storage_size_bytes: - type: integer - description: >- - The current space occupied by the project in - storage. - description: The current status of a Project. - uid: - type: string - description: System-generated unique ID for the project. + description: 'The name of the role that owns the database. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id}' + description: The observed state of the Database. update_time: type: string format: date-time - description: >- - A timestamp indicating when the project was last - updated. - description: The Project to create. + description: A timestamp indicating when the database was last updated. + description: The desired specification of a Database. required: - - project + - database responses: '200': description: Success @@ -1112,47 +820,48 @@ paths: type: string x-stackql-sdk-source: PostgresAPI get: - operationId: postgres_list_projects - summary: >- - Returns a paginated list of database projects in the workspace that the - user has permission to access. + operationId: postgres_list_databases + summary: List Databases. tags: - - postgres - description: >- - Returns a paginated list of database projects in the workspace that the - user has permission to access. - + - postgres + description: |- + List Databases. + :param parent: str + The Branch that owns this collection of databases. Format: + projects/{project_id}/branches/{branch_id} :param page_size: int (optional) - Upper bound for items returned. Cannot be negative. The maximum value is 100. + Upper bound for items returned. :param page_token: str (optional) - Page token from a previous response. If not provided, returns the first page. + Pagination token to go to the next page of Databases. Requests first page if absent. - :returns: Iterator over :class:`Project` + :returns: Iterator over :class:`Database` parameters: - - name: page_size - in: query - required: false - schema: - type: integer - description: >- - Upper bound for items returned. Cannot be negative. The maximum - value is 100. - - name: page_token - in: query - required: false - schema: - type: string - description: >- - Page token from a previous response. If not provided, returns the - first page. + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch that owns this collection of databases. Format: projects/{project_id}/branches/{branch_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token to go to the next page of Databases. Requests first page if absent. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/ListProjectsResponse' + $ref: '#/components/schemas/ListDatabasesResponse' default: description: Error response content: @@ -1165,212 +874,237 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI - /api/2.0/postgres/projects/{project_id}/branches/{branch_id}/roles: + /api/2.0/postgres/{parent}/endpoints: post: - operationId: postgres_create_role - summary: Creates a new Postgres role in the branch. + operationId: postgres_create_endpoint + summary: Creates a new compute endpoint in the branch. tags: - - postgres + - postgres description: |- - Creates a new Postgres role in the branch. - - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch where this Role is created. Format: projects/{project_id}/branches/{branch_id} - :param role: :class:`Role` - The desired specification of a Role. - :param role_id: str (optional) - The ID to use for the Role, which will become the final component of the role's resource name. This - ID becomes the role in Postgres. - - This value should be 4-63 characters, and valid characters are lowercase letters, numbers, and - hyphens, as defined by RFC 1123. + Creates a new compute endpoint in the branch. - If role_id is not specified in the request, it is generated automatically. + :param parent: str + The Branch where this Endpoint will be created. Format: projects/{project_id}/branches/{branch_id} + :param endpoint: :class:`Endpoint` + The Endpoint to create. + :param endpoint_id: str + The ID to use for the Endpoint. This becomes the final component of the endpoint's resource name. + The ID is required and must be 1-63 characters long, start with a lowercase letter, and contain only + lowercase letters, numbers, and hyphens. For example, ``primary`` becomes + ``projects/my-app/branches/development/endpoints/primary``. + :param replace_existing: bool (optional) + If true, update the endpoint if it already exists instead of returning an error. :returns: :class:`Operation` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch where this Role is created. - Format: projects/{project_id}/branches/{branch_id} - - name: role_id - in: query - required: false - schema: - type: string - description: >- - The ID to use for the Role, which will become the final component of - the role's resource name. This ID becomes the role in Postgres. This - value should be 4-63 characters, and valid characters are lowercase - letters, numbers, and hyphens, as defined by RFC 1123. If role_id is - not specified in the request, it is generated automatically. + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch where this Endpoint will be created. Format: projects/{project_id}/branches/{branch_id}' + - name: endpoint_id + in: query + required: true + schema: + type: string + description: The ID to use for the Endpoint. This becomes the final component of the endpoint's resource name. The + ID is required and must be 1-63 characters long, start with a lowercase letter, and contain only lowercase letters, + numbers, and hyphens. For example, ``primary`` becomes ``projects/my-app/branches/development/endpoints/primary``. + - name: replace_existing + in: query + required: false + schema: + type: boolean + description: If true, update the endpoint if it already exists instead of returning an error. requestBody: content: application/json: schema: type: object properties: - role: + endpoint: type: object properties: create_time: type: string format: date-time + endpoint_id: + type: string + description: The part of the name, chosen by the user when the resource was created. name: type: string - description: >- - Output only. The full resource path of the role. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} + description: 'Output only. The full resource path of the endpoint. Format: projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}' parent: type: string - description: >- - The Branch where this Role exists. Format: - projects/{project_id}/branches/{branch_id} + description: 'The branch containing this endpoint (API resource hierarchy). Format: projects/{project_id}/branches/{branch_id}' spec: type: object properties: - attributes: - type: object - properties: - bypassrls: - type: boolean - createdb: - type: boolean - createrole: - type: boolean - description: >- - Attributes that can be granted to a Postgres role. - We are only implementing a subset for now, - see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres - keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical - underscores between words. - auth_method: - type: string - x-enum: - - LAKEBASE_OAUTH_V1 - - NO_LOGIN - - PG_PASSWORD_SCRAM_SHA_256 - description: >- - If auth_method is left unspecified, a meaningful - authentication method is derived from the - identity_type: * For the managed identities, OAUTH - is used. * For the regular postgres roles, - authentication based on postgres passwords is used. - NOTE: this is ignored for the Databricks identity - type GROUP, and NO_LOGIN is implicitly assumed - instead for the GROUP identity type. - identity_type: + endpoint_type: type: string x-enum: - - GROUP - - SERVICE_PRINCIPAL - - USER - description: >- - The type of role. When specifying a - managed-identity, the chosen role_id must be a - valid: * application ID for SERVICE_PRINCIPAL * user - email for USER * group name for GROUP - membership_roles: - type: array - items: - type: string - x-enum: - - DATABRICKS_SUPERUSER - description: >- - Roles that the DatabaseInstanceRole can be a - member of. - description: >- - An enum value for a standard role that this role is - a member of. - postgres_role: - type: string - description: >- - The name of the Postgres role. This expects a valid - Postgres identifier as specified in the link below. - https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS - Required when creating the Role. If you wish to - create a Postgres Role backed by a managed - Databricks identity, then postgres_role must be one - of the following: 1. user email for - IdentityType.USER 2. app ID for - IdentityType.SERVICE_PRINCIPAL 2. group name for - IdentityType.GROUP - description: >- - The spec contains the role configuration, including - identity type, authentication method, and role - attributes. - status: - type: object - properties: - attributes: + - ENDPOINT_TYPE_READ_ONLY + - ENDPOINT_TYPE_READ_WRITE + description: The compute endpoint type. Either ``read_write`` or ``read_only``. + autoscaling_limit_max_cu: + type: number + description: The maximum number of Compute Units. The maximum value is 64. The difference between + the minimum and maximum Compute Units (max - min) must not exceed 16. + autoscaling_limit_min_cu: + type: number + description: The minimum number of Compute Units. Minimum value is 0.5. + disabled: + type: boolean + description: Whether to restrict connections to the compute endpoint. Enabling this option schedules + a suspend compute operation. A disabled compute endpoint cannot be enabled by a connection or + console action. + group: type: object properties: - bypassrls: + min: + type: integer + max: + type: integer + description: The maximum number of computes in the endpoint group. Currently, this must be equal + to min. Set to 1 for single compute endpoints, to disable HA. To manually suspend all computes + in an endpoint group, set disabled to true on the EndpointSpec. + enable_readable_secondaries: type: boolean - createdb: - type: boolean - createrole: - type: boolean - description: >- - Attributes that can be granted to a Postgres role. - We are only implementing a subset for now, - see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres - keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical - underscores between words. - auth_method: + description: Whether to allow read-only connections to read-write endpoints. Only relevant for + read-write endpoints where size.max > 1. + required: + - min + - max + description: Settings for optional HA configuration of the endpoint. If unspecified, the endpoint + defaults to non HA settings, with a single compute backing the endpoint (and no readable secondaries + for Read/Write endpoints). + no_suspension: + type: boolean + description: When set to true, explicitly disables automatic suspension (never suspend). Should + be set to true when provided. Mutually exclusive with ``suspend_timeout_duration``. When updating, + use ``spec.suspension`` in the update_mask. + settings: + type: object + properties: + pg_settings: + type: object + description: A raw representation of Postgres settings. + description: A collection of settings for a compute endpoint. + suspend_timeout_duration: + type: string + description: Duration of inactivity after which the compute endpoint is automatically suspended. + If specified should be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with ``no_suspension``. + When updating, use ``spec.suspension`` in the update_mask. + required: + - endpoint_type + description: The spec contains the compute endpoint configuration, including autoscaling limits, suspend + timeout, and disabled state. + status: + type: object + properties: + autoscaling_limit_max_cu: + type: number + autoscaling_limit_min_cu: + type: number + description: The minimum number of Compute Units. + current_state: type: string x-enum: - - LAKEBASE_OAUTH_V1 - - NO_LOGIN - - PG_PASSWORD_SCRAM_SHA_256 - description: >- - How the role is authenticated when connecting to - Postgres. - identity_type: + - ACTIVE + - DEGRADED + - IDLE + - INIT + description: The state of the compute endpoint. + disabled: + type: boolean + description: Whether to restrict connections to the compute endpoint. Enabling this option schedules + a suspend compute operation. A disabled compute endpoint cannot be enabled by a connection or + console action. + endpoint_id: + type: string + description: Part of the resource name. + endpoint_type: type: string x-enum: - - GROUP - - SERVICE_PRINCIPAL - - USER - description: The type of the role. - membership_roles: - type: array - items: - type: string - x-enum: - - DATABRICKS_SUPERUSER - description: >- - Roles that the DatabaseInstanceRole can be a - member of. - description: >- - An enum value for a standard role that this role is - a member of. - postgres_role: + - ENDPOINT_TYPE_READ_ONLY + - ENDPOINT_TYPE_READ_WRITE + description: The endpoint type. A branch can only have one READ_WRITE endpoint. + group: + type: object + properties: + min: + type: integer + max: + type: integer + description: The maximum number of computes in the endpoint group. Currently, this must be equal + to min. Set to 1 for single compute endpoints, to disable HA. To manually suspend all computes + in an endpoint group, set disabled to true on the EndpointSpec. + enable_readable_secondaries: + type: boolean + description: Whether read-only connections to read-write endpoints are allowed. Only relevant + if read replicas are configured by specifying size.max > 1. + required: + - min + - max + description: Details on the HA configuration of the endpoint. + hosts: + type: object + properties: + host: + type: string + description: The hostname to connect to this endpoint. For read-write endpoints, this is a read-write + hostname which connects to the primary compute. For read-only endpoints, this is a read-only + hostname which allows read-only operations. + read_only_host: + type: string + description: An optionally defined read-only host for the endpoint, without pooling. For read-only + endpoints, this attribute is always defined and is equivalent to host. For read-write endpoints, + this attribute is defined if the enclosing endpoint is a group with greater than 1 computes + configured, and has readable secondaries enabled. + read_only_pooled_host: + type: string + description: The read-only hostname of the compute endpoint, with pooling. This attribute is + always defined for read-only endpoints, and may be defined for read-write endpoints if configured + with read replicas and allow read-only connections. + read_write_pooled_host: + type: string + description: The read-write hostname of the compute endpoint, with pooling. This attribute is + only defined for read-write endpoints. + description: Contains host information for connecting to the endpoint. + last_active_time: type: string - description: The name of the Postgres role. - description: >- - Current status of the role, including its identity type, - authentication method, and role attributes. + format: date-time + description: A timestamp indicating when the compute endpoint was last active. + pending_state: + type: string + x-enum: + - ACTIVE + - DEGRADED + - IDLE + - INIT + description: The state of the compute endpoint. + settings: + type: object + properties: + pg_settings: + type: object + description: A raw representation of Postgres settings. + description: A collection of settings for a compute endpoint. + suspend_timeout_duration: + type: string + description: Duration of inactivity after which the compute endpoint is automatically suspended. + description: Current operational status of the compute endpoint. + uid: + type: string + description: System-generated unique ID for the endpoint. update_time: type: string format: date-time - description: The desired specification of a Role. + description: A timestamp indicating when the compute endpoint was last updated. + description: The Endpoint to create. required: - - role + - endpoint responses: '200': description: Success @@ -1391,203 +1125,48 @@ paths: type: string x-stackql-sdk-source: PostgresAPI get: - operationId: postgres_list_roles - summary: Returns a paginated list of Postgres roles in the branch. + operationId: postgres_list_endpoints + summary: Returns a paginated list of compute endpoints in the branch. tags: - - postgres + - postgres description: |- - Returns a paginated list of Postgres roles in the branch. + Returns a paginated list of compute endpoints in the branch. - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch that owns this collection of roles. Format: projects/{project_id}/branches/{branch_id} + :param parent: str + The Branch that owns this collection of endpoints. Format: + projects/{project_id}/branches/{branch_id} :param page_size: int (optional) Upper bound for items returned. Cannot be negative. :param page_token: str (optional) Page token from a previous response. If not provided, returns the first page. - :returns: Iterator over :class:`Role` - parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch that owns this collection of - roles. Format: projects/{project_id}/branches/{branch_id} - - name: page_size - in: query - required: false - schema: - type: integer - description: Upper bound for items returned. Cannot be negative. - - name: page_token - in: query - required: false - schema: - type: string - description: >- - Page token from a previous response. If not provided, returns the - first page. - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/ListRolesResponse' - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PostgresAPI - /api/2.0/postgres/projects/{project_id}/branches/{branch_id}/roles/{role_id}: - delete: - operationId: postgres_delete_role - summary: Deletes the specified Postgres role. - tags: - - postgres - description: |- - Deletes the specified Postgres role. - - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch. Format: projects/{project_id}/branches/{branch_id} - :param role_id: str - The unique identifier of the role to delete. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id} - :param reassign_owned_to: str (optional) - Reassign objects. If this is set, all objects owned by the role are reassigned to the role specified - in this parameter. - - NOTE: setting this requires spinning up a compute to succeed, since it involves running SQL queries. - - :returns: :class:`Operation` - parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch. Format: - projects/{project_id}/branches/{branch_id} - - name: role_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the role to delete. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} - - name: reassign_owned_to - in: query - required: false - schema: - type: string - description: >- - Reassign objects. If this is set, all objects owned by the role are - reassigned to the role specified in this parameter. NOTE: setting - this requires spinning up a compute to succeed, since it involves - running SQL queries. - responses: - '200': - description: Success - content: - application/json: - schema: - type: object - default: - description: Error response - content: - application/json: - schema: - type: object - properties: - error: - type: string - message: - type: string - x-stackql-sdk-source: PostgresAPI - get: - operationId: postgres_get_role - summary: >- - Retrieves information about the specified Postgres role, including its - authentication method and - tags: - - postgres - description: >- - Retrieves information about the specified Postgres role, including its - authentication method and - - permissions. - - - :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch. Format: projects/{project_id}/branches/{branch_id} - :param role_id: str - The unique identifier of the role to retrieve. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id} - - :returns: :class:`Role` + :returns: Iterator over :class:`Endpoint` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch. Format: - projects/{project_id}/branches/{branch_id} - - name: role_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the role to retrieve. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch that owns this collection of endpoints. Format: projects/{project_id}/branches/{branch_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. Cannot be negative. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token from a previous response. If not provided, returns the first page. responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/Role' + $ref: '#/components/schemas/ListEndpointsResponse' default: description: Error response content: @@ -1600,156 +1179,113 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI - patch: - operationId: postgres_update_role - summary: Update a role for a branch. + /api/2.0/postgres/projects: + post: + operationId: postgres_create_project + summary: Creates a new Lakebase Autoscaling Postgres database project, which contains branches and compute tags: - - postgres + - postgres description: |- - Update a role for a branch. + Creates a new Lakebase Autoscaling Postgres database project, which contains branches and compute + endpoints. + :param project: :class:`Project` + The Project to create. :param project_id: str - The unique identifier of the Postgres project. Format: projects/{project_id} - :param branch_id: str - The unique identifier of the branch. Format: projects/{project_id}/branches/{branch_id} - :param role_id: str - The unique identifier of the role to update. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id} - :param role: :class:`Role` - The Postgres Role to update. - :param update_mask: FieldMask - The list of fields to update in Postgres Role. If unspecified, all fields will be updated when - possible. + The ID to use for the Project. This becomes the final component of the project's resource name. The + ID is required and must be 1-63 characters long, start with a lowercase letter, and contain only + lowercase letters, numbers, and hyphens. For example, ``my-app`` becomes ``projects/my-app``. :returns: :class:`Operation` parameters: - - name: project_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the Postgres project. Format: - projects/{project_id} - - name: branch_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the branch. Format: - projects/{project_id}/branches/{branch_id} - - name: role_id - in: path - required: true - schema: - type: string - description: >- - The unique identifier of the role to update. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} - - name: update_mask - in: query - required: true - schema: - type: object - description: >- - The list of fields to update in Postgres Role. If unspecified, all - fields will be updated when possible. + - name: project_id + in: query + required: true + schema: + type: string + description: The ID to use for the Project. This becomes the final component of the project's resource name. The ID + is required and must be 1-63 characters long, start with a lowercase letter, and contain only lowercase letters, + numbers, and hyphens. For example, ``my-app`` becomes ``projects/my-app``. requestBody: content: application/json: schema: type: object properties: - role: + project: type: object properties: create_time: type: string format: date-time - name: + delete_time: type: string - description: >- - Output only. The full resource path of the role. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} - parent: - type: string - description: >- - The Branch where this Role exists. Format: - projects/{project_id}/branches/{branch_id} - spec: + format: date-time + description: A timestamp indicating when the project was soft-deleted. Empty if the project is not deleted, + otherwise set to a timestamp in the past. + initial_branch_spec: type: object properties: - attributes: + is_protected: + type: boolean + description: Whether the initial default branch should be protected from deletion. + description: Configuration for the initial default branch created as part of project creation. Allows + overriding branch protection. These settings only apply at creation time and do not affect resources + created after project creation. + initial_database_spec: + type: object + properties: + postgres_database: + type: string + description: The name of the Postgres database. This expects a valid Postgres identifier as specified + in the link below. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS + description: Configuration for the initial Postgres database created inside the initial branch for this + project. If omitted, the initial branch still gets an initial database with name ``databricks_postgres``. + The initial database is always owned by the initial role (caller-provided via ``initial_role_spec`` + or defaulted to the caller's identity). This field is input-only; to change databases after project + creation, use the standalone Database API. + initial_endpoint_spec: + type: object + properties: + autoscaling_limit_max_cu: + type: number + description: The maximum number of Compute Units for the initial endpoint. + autoscaling_limit_min_cu: + type: number + description: The minimum number of Compute Units for the initial endpoint. + group: type: object properties: - bypassrls: - type: boolean - createdb: - type: boolean - createrole: + min: + type: integer + max: + type: integer + description: The maximum number of computes in the endpoint group. Currently, this must be equal + to min. Set to 1 for single compute endpoints, to disable HA. To manually suspend all computes + in an endpoint group, set disabled to true on the EndpointSpec. + enable_readable_secondaries: type: boolean - description: >- - Attributes that can be granted to a Postgres role. - We are only implementing a subset for now, - see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres - keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical - underscores between words. - auth_method: - type: string - x-enum: - - LAKEBASE_OAUTH_V1 - - NO_LOGIN - - PG_PASSWORD_SCRAM_SHA_256 - description: >- - If auth_method is left unspecified, a meaningful - authentication method is derived from the - identity_type: * For the managed identities, OAUTH - is used. * For the regular postgres roles, - authentication based on postgres passwords is used. - NOTE: this is ignored for the Databricks identity - type GROUP, and NO_LOGIN is implicitly assumed - instead for the GROUP identity type. - identity_type: - type: string - x-enum: - - GROUP - - SERVICE_PRINCIPAL - - USER - description: >- - The type of role. When specifying a - managed-identity, the chosen role_id must be a - valid: * application ID for SERVICE_PRINCIPAL * user - email for USER * group name for GROUP - membership_roles: - type: array - items: - type: string - x-enum: - - DATABRICKS_SUPERUSER - description: >- - Roles that the DatabaseInstanceRole can be a - member of. - description: >- - An enum value for a standard role that this role is - a member of. - postgres_role: + description: Whether to allow read-only connections to read-write endpoints. Only relevant for + read-write endpoints where size.max > 1. + required: + - min + - max + description: Settings for HA configuration of the endpoint. + no_suspension: + type: boolean + description: When set to true, explicitly disables automatic suspension (never suspend). Should + be set to true when provided. Mutually exclusive with ``suspend_timeout_duration``. + suspend_timeout_duration: type: string - description: >- - The name of the Postgres role. This expects a valid - Postgres identifier as specified in the link below. - https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS - Required when creating the Role. If you wish to - create a Postgres Role backed by a managed - Databricks identity, then postgres_role must be one - of the following: 1. user email for - IdentityType.USER 2. app ID for - IdentityType.SERVICE_PRINCIPAL 2. group name for - IdentityType.GROUP - description: >- - The spec contains the role configuration, including - identity type, authentication method, and role - attributes. - status: + description: Duration of inactivity after which the initial endpoint is automatically suspended. + If specified, should be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with + ``no_suspension``. + description: Configuration settings for the initial Read/Write endpoint created inside the initial branch + for a newly created project. If omitted, the initial endpoint created will have default settings, + without high availability configured. This field does not apply to any endpoints created after project + creation. Use spec.default_endpoint_settings to configure default settings for endpoints created after + project creation. + initial_role_spec: type: object properties: attributes: @@ -1761,62 +1297,231 @@ paths: type: boolean createrole: type: boolean - description: >- - Attributes that can be granted to a Postgres role. - We are only implementing a subset for now, - see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres - keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical - underscores between words. + description: The desired API-exposed Postgres role attribute to associate with the role. Optional. auth_method: type: string x-enum: - - LAKEBASE_OAUTH_V1 - - NO_LOGIN - - PG_PASSWORD_SCRAM_SHA_256 - description: >- - How the role is authenticated when connecting to - Postgres. + - LAKEBASE_OAUTH_V1 + - NO_LOGIN + - PG_PASSWORD_SCRAM_SHA_256 + description: 'Controls how the Postgres role authenticates when a client opens a database connection. + Supported values: - LAKEBASE_OAUTH_V1: the role authenticates by presenting a Databricks OAuth + access token derived from the backing managed identity (the Databricks user, service principal, + or group named by the role''s ``postgres_role``). No static password exists for roles using this + method. - PG_PASSWORD_SCRAM_SHA_256: the role authenticates with a Postgres password verified + server-side using the SCRAM-SHA-256 mechanism. Lakebase generates a password for the role. - NO_LOGIN: + the role cannot open a Postgres session at all. Useful for roles that exist only to own objects + or to aggregate privileges that are then granted to other, loginable roles. If auth_method is + left unspecified, a meaningful authentication method is derived from the identity_type: - For + the managed identities, OAUTH is used. - For the regular postgres roles, authentication based + on postgres passwords is used. NOTE: for the Databricks identity type GROUP, LAKEBASE_OAUTH_V1 + is the default auth method (group can login as well).' identity_type: type: string x-enum: - - GROUP - - SERVICE_PRINCIPAL - - USER - description: The type of the role. + - GROUP + - SERVICE_PRINCIPAL + - USER + description: 'The type of role. When specifying a managed-identity, the chosen role_id must be a + valid: - application ID for SERVICE_PRINCIPAL - user email for USER - group name for GROUP' membership_roles: type: array items: type: string x-enum: - - DATABRICKS_SUPERUSER - description: >- - Roles that the DatabaseInstanceRole can be a - member of. - description: >- - An enum value for a standard role that this role is - a member of. + - DATABRICKS_SUPERUSER + description: Roles that the DatabaseInstanceRole can be a member of. + description: An enum value for a standard role that this role is a member of. postgres_role: type: string - description: The name of the Postgres role. - description: >- - Current status of the role, including its identity type, - authentication method, and role attributes. - update_time: + description: 'The name of the Postgres role. This expects a valid Postgres identifier as specified + in the link below. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS + If you wish to create a Postgres Role backed by a managed Databricks identity, then postgres_role + must be one of the following: 1. user email for IdentityType.USER 2. app ID for IdentityType.SERVICE_PRINCIPAL + 3. group name for IdentityType.GROUP' + description: Configuration for the initial Postgres role created inside the initial branch for this + project. If omitted, the initial branch gets an initial role corresponding to the caller of the API + endpoint. This field is input-only; to change roles after project creation, use the standalone Role + API. + name: + type: string + description: 'Output only. The full resource path of the project. Format: projects/{project_id}' + project_id: + type: string + description: The part of the name, chosen by the user when the resource was created. + purge_time: type: string format: date-time - description: >- - The Postgres Role to update. The role's `name` field is used - to identify the role to update. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} - required: - - role - responses: - '200': - description: Success - content: - application/json: - schema: - type: object + description: A timestamp indicating when the project is scheduled for permanent deletion. Empty if the + project is not deleted, otherwise set to a timestamp in the future. + spec: + type: object + properties: + budget_policy_id: + type: string + compute_provisioner: + type: string + description: The compute provisioner used to provision endpoints in this project. Overrides the + default provisioner when set. + custom_tags: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: The value of the custom tag. + description: Custom tags to associate with the project. Forwarded to LBM for billing and cost tracking. + To update tags, provide the new tag list and include "spec.custom_tags" in the update_mask. To + clear all tags, provide an empty list and include "spec.custom_tags" in the update_mask. To preserve + existing tags, omit this field from the update_mask (or use wildcard "*" which auto-excludes empty + tags). + default_branch: + type: string + description: 'The full resource path for the default branch of the project Format: projects/{project_id}/branches/{branch_id}' + default_endpoint_settings: + type: object + properties: + autoscaling_limit_max_cu: + type: number + description: The maximum number of Compute Units. Minimum value is 0.5. + autoscaling_limit_min_cu: + type: number + description: The minimum number of Compute Units. Minimum value is 0.5. + no_suspension: + type: boolean + description: When set to true, explicitly disables automatic suspension (never suspend). Should + be set to true when provided. Mutually exclusive with ``suspend_timeout_duration``. When updating, + use ``spec.project_default_settings.suspension`` in the update_mask. + pg_settings: + type: object + description: A raw representation of Postgres settings. + suspend_timeout_duration: + type: string + description: Duration of inactivity after which the compute endpoint is automatically suspended. + If specified should be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with + ``no_suspension``. When updating, use ``spec.project_default_settings.suspension`` in the + update_mask. + description: A collection of settings for a compute endpoint. + display_name: + type: string + description: Human-readable project name. Length should be between 1 and 256 characters. + enable_pg_native_login: + type: boolean + description: Whether to enable PG native password login on all endpoints in this project. Defaults + to false. + history_retention_duration: + type: string + description: The number of seconds to retain the shared history for point in time recovery for all + branches in this project. Value should be between 172800s (2 days) and 3024000s (35 days). + pg_version: + type: integer + description: The major Postgres version number. The set of supported versions may vary; consult + the API documentation for currently accepted values. + description: The spec contains the project configuration, including display_name, pg_version (Postgres + version), history_retention_duration, and default_endpoint_settings. + status: + type: object + properties: + branch_logical_size_limit_bytes: + type: integer + budget_policy_id: + type: string + description: The budget policy that is applied to the project. + compute_last_active_time: + type: string + format: date-time + description: The most recent time when any endpoint of this project was active. + compute_provisioner: + type: string + description: The effective compute provisioner backing this project's endpoints. + custom_tags: + type: array + items: + type: object + properties: + key: + type: string + value: + type: string + description: The value of the custom tag. + description: The effective custom tags associated with the project. + default_branch: + type: string + description: The full resource path of the default branch of the project + default_endpoint_settings: + type: object + properties: + autoscaling_limit_max_cu: + type: number + description: The maximum number of Compute Units. Minimum value is 0.5. + autoscaling_limit_min_cu: + type: number + description: The minimum number of Compute Units. Minimum value is 0.5. + no_suspension: + type: boolean + description: When set to true, explicitly disables automatic suspension (never suspend). Should + be set to true when provided. Mutually exclusive with ``suspend_timeout_duration``. When updating, + use ``spec.project_default_settings.suspension`` in the update_mask. + pg_settings: + type: object + description: A raw representation of Postgres settings. + suspend_timeout_duration: + type: string + description: Duration of inactivity after which the compute endpoint is automatically suspended. + If specified should be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with + ``no_suspension``. When updating, use ``spec.project_default_settings.suspension`` in the + update_mask. + description: The effective default endpoint settings. + display_name: + type: string + description: The effective human-readable project name. + enable_pg_native_login: + type: boolean + description: Whether to enable PG native password login on all endpoints in this project. + history_retention_duration: + type: string + description: The effective number of seconds to retain the shared history for point in time recovery. + owner: + type: string + description: The email of the project owner. + pg_version: + type: integer + description: The effective major Postgres version number. + project_id: + type: string + description: Part of the resource name. + replication_role: + type: string + x-enum: + - REPLICATION_ROLE_PREVIEW_DEMOTING + - REPLICATION_ROLE_PREVIEW_PRIMARY + - REPLICATION_ROLE_PREVIEW_SECONDARY + description: The replication role of the project in this workspace. Populated only when cross-workspace + replication is configured. + synthetic_storage_size_bytes: + type: integer + description: The current space occupied by the project in storage. + description: The current status of a Project. + uid: + type: string + description: System-generated unique ID for the project. + update_time: + type: string + format: date-time + description: A timestamp indicating when the project was last updated. + description: The Project to create. + required: + - project + responses: + '200': + description: Success + content: + application/json: + schema: + type: object default: description: Error response content: @@ -1829,67 +1534,172 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI - /api/2.0/postgres/credentials: + get: + operationId: postgres_list_projects + summary: Returns a paginated list of database projects in the workspace that the user has permission to access. + tags: + - postgres + description: |- + Returns a paginated list of database projects in the workspace that the user has permission to access. + + :param page_size: int (optional) + Upper bound for items returned. Cannot be negative. The maximum value is 100. + :param page_token: str (optional) + Page token from a previous response. If not provided, returns the first page. + :param show_deleted: bool (optional) + Whether to include soft-deleted projects in the response. When true, soft-deleted projects are + included alongside active projects. Hard-deleted and already-purged projects are never returned. + + :returns: Iterator over :class:`Project` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. Cannot be negative. The maximum value is 100. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token from a previous response. If not provided, returns the first page. + - name: show_deleted + in: query + required: false + schema: + type: boolean + description: Whether to include soft-deleted projects in the response. When true, soft-deleted projects are included + alongside active projects. Hard-deleted and already-purged projects are never returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListProjectsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/preview/replication-groups: post: - operationId: postgres_generate_database_credential - summary: Generate OAuth credentials for a Postgres database. + operationId: postgres_create_replication_group_preview + summary: Creates a new replication group for the project. tags: - - postgres + - postgres description: |- - Generate OAuth credentials for a Postgres database. + Creates a new replication group for the project. - :param endpoint: str - This field is not yet supported. The endpoint for which this credential will be generated. Format: - projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} - :param claims: List[:class:`RequestedClaims`] (optional) - The returned token will be scoped to UC tables with the specified permissions. + :param parent: str + :param replication_group_preview: :class:`ReplicationGroupPreview` + :param replication_group_preview_id: str + :param request_id: str (optional) - :returns: :class:`DatabaseCredential` + :returns: :class:`Operation` + parameters: + - name: parent + in: path + required: true + schema: + type: string + - name: replication_group_preview_id + in: query + required: true + schema: + type: string + - name: request_id + in: query + required: false + schema: + type: string requestBody: content: application/json: schema: type: object properties: - endpoint: - type: string - description: >- - This field is not yet supported. The endpoint for which this - credential will be generated. Format: - projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} - claims: - type: array - items: - type: object - properties: - permission_set: + replication_group_preview: + type: object + properties: + replication_mode: + type: string + x-enum: + - REPLICATION_MODE_PREVIEW_LIVE + - REPLICATION_MODE_PREVIEW_PERIODIC + description: |- + How changes are propagated from the primary workspace to its secondaries in a replication group: + on a fixed schedule or continuously as they occur. + workspaces: + type: array + items: type: string - x-enum: - - READ_ONLY - description: >- - The permission scope for the generated credential. Supported values: - READ_ONLY. - resources: - type: array - items: - type: object - properties: - table_name: - type: string - unspecified_resource_name: - type: string - description: >- - The returned token will be scoped to UC tables with the - specified permissions. + description: The workspaces participating in this replication group. Phase 1 requires exactly 2 entries. + create_time: + type: string + format: date-time + description: Server-generated timestamps. + etag: + type: string + description: Optional optimistic concurrency token for update and delete. + name: + type: string + description: 'The resource name of the replication group. Format: projects/{project_id}/preview/replication-groups/{replication_group_id}' + observed_metrics: + type: object + properties: + as_of_time: + type: string + format: date-time + bytes_lag: + type: integer + description: The most recent observed byte lag. + throughput_bytes_per_second: + type: integer + description: The most recent observed replication throughput. + time_lag: + type: string + description: The most recent observed time lag. + description: The latest observed replication metrics for this group. + parent: + type: string + description: 'The parent project that owns this replication group. Format: projects/{project_id}' + primary_workspace: + type: string + description: The workspace currently serving writes. Server-owned. + state: + type: string + x-enum: + - REPLICATION_GROUP_PREVIEW_STATE_DEGRADED + - REPLICATION_GROUP_PREVIEW_STATE_DELETING + - REPLICATION_GROUP_PREVIEW_STATE_FAILING_OVER + - REPLICATION_GROUP_PREVIEW_STATE_PROVISIONING + - REPLICATION_GROUP_PREVIEW_STATE_READY + - REPLICATION_GROUP_PREVIEW_STATE_SWITCHING_OVER + description: The lifecycle state of the replication group. + update_time: + type: string + format: date-time + required: + - replication_mode + - workspaces required: - - endpoint + - replication_group_preview responses: '200': description: Success content: application/json: schema: - $ref: '#/components/schemas/DatabaseCredential' + type: object default: description: Error response content: @@ -1902,128 +1712,2337 @@ paths: message: type: string x-stackql-sdk-source: PostgresAPI -components: - schemas: - Branch: - type: object - properties: - create_time: - type: string - format: date-time - name: - type: string - description: >- - Output only. The full resource path of the branch. Format: - projects/{project_id}/branches/{branch_id} - parent: - type: string - description: >- - The project containing this branch (API resource hierarchy). Format: - projects/{project_id} Note: This field indicates where the branch - exists in the resource hierarchy. For point-in-time branching from - another branch, see `status.source_branch`. - spec: - $ref: '#/components/schemas/BranchSpec' - description: The spec contains the branch configuration. - status: - $ref: '#/components/schemas/BranchStatus' - description: The current status of a Branch. - uid: - type: string - description: System-generated unique ID for the branch. - update_time: - type: string - format: date-time - description: A timestamp indicating when the branch was last updated. - BranchOperationMetadata: - type: object - properties: {} - BranchSpec: - type: object - properties: - expire_time: - type: string - format: date-time - is_protected: - type: boolean - description: >- - When set to true, protects the branch from deletion and reset. - Associated compute endpoints and the project cannot be deleted while - the branch is protected. - no_expiry: - type: boolean - description: >- - Explicitly disable expiration. When set to true, the branch will not - expire. If set to false, the request is invalid; provide either ttl - or expire_time instead. - source_branch: - type: string - description: >- - The name of the source branch from which this branch was created - (data lineage for point-in-time recovery). If not specified, - defaults to the project's default branch. Format: - projects/{project_id}/branches/{branch_id} - source_branch_lsn: + get: + operationId: postgres_list_replication_group_previews + summary: Returns a paginated list of replication groups for the project. + tags: + - postgres + description: |- + Returns a paginated list of replication groups for the project. + + :param parent: str + :param page_size: int (optional) + :param page_token: str (optional) + + :returns: Iterator over :class:`ReplicationGroupPreview` + parameters: + - name: parent + in: path + required: true + schema: type: string - description: >- - The Log Sequence Number (LSN) on the source branch from which this - branch was created. - source_branch_time: + - name: page_size + in: query + required: false + schema: + type: integer + - name: page_token + in: query + required: false + schema: type: string - format: date-time - description: >- - The point in time on the source branch from which this branch was - created. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListReplicationGroupPreviewsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/roles: + post: + operationId: postgres_create_role + summary: Creates a new Postgres role in the branch. + tags: + - postgres + description: |- + Creates a new Postgres role in the branch. + + :param parent: str + The Branch where this Role is created. Format: projects/{project_id}/branches/{branch_id} + :param role: :class:`Role` + The desired specification of a Role. + :param replace_existing: bool (optional) + If true, update the role if it already exists instead of returning an error. + + When the role already exists, the provided ``role`` spec fully replaces the existing one: + ``membership_roles`` is overwritten, not merged. Leaving ``membership_roles`` empty clears all of + the role's existing memberships, including ``DATABRICKS_SUPERUSER``. Always send the complete + desired list of memberships when using this field. + :param role_id: str (optional) + The ID to use for the Role, which will become the final component of the role's resource name. This + ID becomes the role in Postgres. + + This value should be 4-63 characters, and valid characters are lowercase letters, numbers, and + hyphens, as defined by RFC 1123. + + If role_id is not specified in the request, it is generated automatically. + + :returns: :class:`Operation` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch where this Role is created. Format: projects/{project_id}/branches/{branch_id}' + - name: replace_existing + in: query + required: false + schema: + type: boolean + description: 'If true, update the role if it already exists instead of returning an error. When the role already exists, + the provided ``role`` spec fully replaces the existing one: ``membership_roles`` is overwritten, not merged. Leaving + ``membership_roles`` empty clears all of the role''s existing memberships, including ``DATABRICKS_SUPERUSER``. Always + send the complete desired list of memberships when using this field.' + - name: role_id + in: query + required: false + schema: + type: string + description: The ID to use for the Role, which will become the final component of the role's resource name. This ID + becomes the role in Postgres. This value should be 4-63 characters, and valid characters are lowercase letters, + numbers, and hyphens, as defined by RFC 1123. If role_id is not specified in the request, it is generated automatically. + requestBody: + content: + application/json: + schema: + type: object + properties: + role: + type: object + properties: + create_time: + type: string + format: date-time + name: + type: string + description: 'Output only. The full resource path of the role. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id}' + parent: + type: string + description: 'The Branch where this Role exists. Format: projects/{project_id}/branches/{branch_id}' + role_id: + type: string + description: The part of the name, chosen by the user when the resource was created. + spec: + type: object + properties: + attributes: + type: object + properties: + bypassrls: + type: boolean + createdb: + type: boolean + createrole: + type: boolean + description: |- + Attributes that can be granted to a Postgres role. We are only implementing a subset for now, + see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres + keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical + underscores between words. + auth_method: + type: string + x-enum: + - LAKEBASE_OAUTH_V1 + - NO_LOGIN + - PG_PASSWORD_SCRAM_SHA_256 + description: 'Controls how the Postgres role authenticates when a client opens a database connection. + Supported values: - LAKEBASE_OAUTH_V1: the role authenticates by presenting a Databricks OAuth + access token derived from the backing managed identity (the Databricks user, service principal, + or group named by the role''s ``postgres_role``). No static password exists for roles using this + method. - PG_PASSWORD_SCRAM_SHA_256: the role authenticates with a Postgres password verified + server-side using the SCRAM-SHA-256 mechanism. Lakebase generates a password for the role. - NO_LOGIN: + the role cannot open a Postgres session at all. Useful for roles that exist only to own objects + or to aggregate privileges that are then granted to other, loginable roles. If auth_method is + left unspecified, a meaningful authentication method is derived from the identity_type: - For + the managed identities, OAUTH is used. - For the regular postgres roles, authentication based + on postgres passwords is used. NOTE: for the Databricks identity type GROUP, LAKEBASE_OAUTH_V1 + is the default auth method (group can login as well).' + identity_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: 'The type of role. When specifying a managed-identity, the chosen role_id must be a + valid: - application ID for SERVICE_PRINCIPAL - user email for USER - group name for GROUP' + membership_roles: + type: array + items: + type: string + x-enum: + - DATABRICKS_SUPERUSER + description: Roles that the DatabaseInstanceRole can be a member of. + description: An enum value for a standard role that this role is a member of. + postgres_role: + type: string + description: 'The name of the Postgres role. This expects a valid Postgres identifier as specified + in the link below. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS + Required when creating the Role. If you wish to create a Postgres Role backed by a managed Databricks + identity, then postgres_role must be one of the following: 1. user email for IdentityType.USER + 2. app ID for IdentityType.SERVICE_PRINCIPAL 3. group name for IdentityType.GROUP' + description: The spec contains the role configuration, including identity type, authentication method, + and role attributes. + status: + type: object + properties: + attributes: + type: object + properties: + bypassrls: + type: boolean + createdb: + type: boolean + createrole: + type: boolean + description: |- + Attributes that can be granted to a Postgres role. We are only implementing a subset for now, + see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres + keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical + underscores between words. + auth_method: + type: string + x-enum: + - LAKEBASE_OAUTH_V1 + - NO_LOGIN + - PG_PASSWORD_SCRAM_SHA_256 + description: How the role is authenticated when connecting to Postgres. + identity_type: + type: string + x-enum: + - GROUP + - SERVICE_PRINCIPAL + - USER + description: The type of the role. + membership_roles: + type: array + items: + type: string + x-enum: + - DATABRICKS_SUPERUSER + description: Roles that the DatabaseInstanceRole can be a member of. + description: An enum value for a standard role that this role is a member of. + postgres_role: + type: string + description: The name of the Postgres role. + role_id: + type: string + description: Part of the resource name. + description: Current status of the role, including its identity type, authentication method, and role + attributes. + update_time: + type: string + format: date-time + description: The desired specification of a Role. + required: + - role + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + get: + operationId: postgres_list_roles + summary: Returns a paginated list of Postgres roles in the branch. + tags: + - postgres + description: |- + Returns a paginated list of Postgres roles in the branch. + + :param parent: str + The Branch that owns this collection of roles. Format: projects/{project_id}/branches/{branch_id} + :param page_size: int (optional) + Upper bound for items returned. Cannot be negative. + :param page_token: str (optional) + Page token from a previous response. If not provided, returns the first page. + + :returns: Iterator over :class:`Role` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch that owns this collection of roles. Format: projects/{project_id}/branches/{branch_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Upper bound for items returned. Cannot be negative. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token from a previous response. If not provided, returns the first page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListRolesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/snapshots: + post: + operationId: postgres_create_snapshot + summary: Creates a snapshot, an immutable point-in-time copy of a branch's data, within the project. + tags: + - postgres + description: |- + Creates a snapshot, an immutable point-in-time copy of a branch's data, within the project. + + :param parent: str + The project in which to create the snapshot. Format: projects/{project_id} + :param snapshot: :class:`Snapshot` + The snapshot to create. + :param snapshot_id: str (optional) + Client-chosen ID for the snapshot. If omitted, the server generates one. + + :returns: :class:`Operation` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The project in which to create the snapshot. Format: projects/{project_id}' + - name: snapshot_id + in: query + required: false + schema: + type: string + description: Client-chosen ID for the snapshot. If omitted, the server generates one. + requestBody: + content: + application/json: + schema: + type: object + properties: + snapshot: + type: object + properties: + create_time: + type: string + format: date-time + description: When the snapshot was created. + name: + type: string + description: 'The resource name of the snapshot. Format: projects/{project_id}/snapshots/{snapshot_id}' + snapshot_id: + type: string + description: The user-chosen ID; the final segment of ``name``. + spec: + type: object + properties: + source_branch: + type: string + description: 'The source branch to snapshot. Format: projects/{project_id}/branches/{branch_id}' + expire_time: + type: string + format: date-time + description: Absolute time at which the snapshot is deleted. Mutually exclusive with ``ttl`` and + ``no_expiry``. + no_expiry: + type: boolean + description: If true, the snapshot never expires. Mutually exclusive with ``ttl`` and ``expire_time``. + source_branch_lsn: + type: string + description: LSN to snapshot from, e.g. ``16/B374D848``. Mutually exclusive with ``source_branch_time``. + source_branch_time: + type: string + format: date-time + description: Timestamp to snapshot from. Mutually exclusive with ``source_branch_lsn``. + ttl: + type: string + description: Time-to-live. The snapshot expires this long after it is created. Mutually exclusive + with ``expire_time`` and ``no_expiry``. + required: + - source_branch + description: Client-provided configuration of the snapshot. + status: + type: object + properties: + current_state: + type: string + x-enum: + - AVAILABLE + - CREATING + - DELETING + - FAILED + description: The snapshot's current state. + diff_size_bytes: + type: integer + description: Incremental storage size in bytes since the previous snapshot. Unset when the snapshot + is not billed on incremental usage. + expire_time: + type: string + format: date-time + description: Absolute time at which the snapshot is deleted. + full_size_bytes: + type: integer + description: Full logical size of the snapshot, in bytes. + no_expiry: + type: boolean + description: True if the snapshot never expires. + source_branch: + type: string + description: 'The source branch the snapshot was taken from. Format: projects/{project_id}/branches/{branch_id}' + source_branch_lsn: + type: string + description: The LSN at which the snapshot was taken. + source_branch_time: + type: string + format: date-time + description: The point in time at which the snapshot was taken. + description: Server-observed state of the snapshot. + uid: + type: string + description: Unique system-generated ID for the snapshot. + update_time: + type: string + format: date-time + description: When the snapshot was last updated. + description: The snapshot to create. + required: + - snapshot + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + get: + operationId: postgres_list_snapshots + summary: Returns a paginated list of snapshots in the project. + tags: + - postgres + description: |- + Returns a paginated list of snapshots in the project. + + :param parent: str + The project that owns the snapshots. Format: projects/{project_id} + :param page_size: int (optional) + Maximum number of snapshots to return per page. + :param page_token: str (optional) + Page token from a previous response; omit for the first page. + + :returns: Iterator over :class:`Snapshot` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The project that owns the snapshots. Format: projects/{project_id}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Maximum number of snapshots to return per page. + - name: page_token + in: query + required: false + schema: + type: string + description: Page token from a previous response; omit for the first page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListSnapshotsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/synced_tables: + post: + operationId: postgres_create_synced_table + summary: Create a Synced Table. + tags: + - postgres + description: |- + Create a Synced Table. + + :param synced_table: :class:`SyncedTable` + :param synced_table_id: str + The ID to use for the Synced Table. This becomes the final component of the SyncedTable's resource + name. ID is required and is the synced table name, containing (catalog, schema, table) tuple. + Elements of the tuple are the UC entity names. + + Example: "{catalog}.{schema}.{table}" + + synced_table_id represents both of the following: + + 1. An online VIEW virtual table in the Unity Catalog accessible via the Lakehouse Federation. + 2. Postgres table named "{table}" in schema "{schema}" in the connected Postgres database + + :returns: :class:`Operation` + parameters: + - name: synced_table_id + in: query + required: true + schema: + type: string + description: 'The ID to use for the Synced Table. This becomes the final component of the SyncedTable''s resource + name. ID is required and is the synced table name, containing (catalog, schema, table) tuple. Elements of the tuple + are the UC entity names. Example: "{catalog}.{schema}.{table}" synced_table_id represents both of the following: + 1. An online VIEW virtual table in the Unity Catalog accessible via the Lakehouse Federation. 2. Postgres table + named "{table}" in schema "{schema}" in the connected Postgres database' + requestBody: + content: + application/json: + schema: + type: object + properties: + synced_table: + type: object + properties: + create_time: + type: string + format: date-time + name: + type: string + description: Output only. The Full resource name of the synced table in Postgres where (catalog, schema, + table) are the UC entity names. Format "synced_tables/{catalog}.{schema}.{table}" For the corresponding + source table in the Unity catalog look for the "source_table_full_name" attribute. + spec: + type: object + properties: + accelerated_sync: + type: boolean + branch: + type: string + description: 'The full resource name the branch associated with the table. Format: "projects/{project_id}/branches/{branch_id}".' + create_database_objects_if_missing: + type: boolean + description: If true, the synced table's logical database and schema resources in PG will be created + if they do not already exist. The request will fail if this is false and the database/schema do + not exist. Defaults to true if omitted. + existing_pipeline_id: + type: string + description: ID of an existing pipeline to bin-pack this synced table into. At most one of existing_pipeline_id + and new_pipeline_spec should be defined. The pipeline used for the synced table is returned via + the top level pipeline_id attribute. + extra_columns: + type: array + items: + type: object + properties: + column_name: + type: string + description: Name of the column. + column_type: + type: string + description: PostgreSQL type of the column, for example "tsvector" or "vector(1024)". + compute: + type: string + description: SQL expression used to compute the column's value, for example "to_tsvector('english', + content)". + maintenance: + type: string + x-enum: + - DEFAULT_VALUE + - STORED_GENERATED + description: How the column's value is populated and kept up to date. + required: + - column_name + - column_type + description: An extra PostgreSQL column to add to the synced table. + description: Extra PostgreSQL-only columns to add to the synced table. + extra_index_definitions: + type: array + items: + type: object + properties: + name: + type: string + description: Name of the index as it will appear in PostgreSQL. + definition: + type: string + description: 'The definition portion of a CREATE INDEX statement, placed after ON table_name. + For example: USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64).' + creation_point: + type: string + x-enum: + - CREATION_POINT_AFTER_DATA_LOAD + description: When the index should be created relative to the initial data load. + required: + - name + - definition + description: Definition of a secondary index to create on the synced table. + description: Secondary indexes to create on the synced table. + new_pipeline_spec: + type: object + properties: + budget_policy_id: + type: string + pipeline_channel: + type: string + x-enum: + - CURRENT + - PREVIEW + description: Release channel of the underlying pipeline's runtime. Some source table configurations + (e.g., read-time CDF) require PREVIEW. Defaults to CURRENT if not specified. + storage_catalog: + type: string + description: UC catalog for the pipeline to store intermediate files (checkpoints, event logs + etc). This needs to be a standard catalog where the user has permissions to create Delta tables. + storage_schema: + type: string + description: UC schema for the pipeline to store intermediate files (checkpoints, event logs + etc). This needs to be in the standard catalog where the user has permissions to create Delta + tables. + description: Specification for creating a new pipeline. At most one of existing_pipeline_id and + new_pipeline_spec should be defined. The pipeline used for the synced table is returned via the + top level pipeline_id attribute. + postgres_database: + type: string + description: The Postgres database name where the synced table will be created in. If this synced + table is created inside a Lakebase Catalog, this attribute can be omitted on creation and is inferred + from the postgres_database associated with the Lakebase Catalog. If specified when inside a Lakebase + Catalog, the value must match. A value must be specified when creating a synced table inside a + Standard Catalog. + primary_key_columns: + type: array + items: + type: string + description: Primary Key columns to be used for data insert/update in the destination. + scheduling_policy: + type: string + x-enum: + - CONTINUOUS + - SNAPSHOT + - TRIGGERED + description: Scheduling policy of the underlying pipeline. + source_table_full_name: + type: string + description: 'Three-part (catalog, schema, table) name of the source Delta table. For the corresponding + destination table, use any of the two: - synced_table_id used at the creation of the SyncedTable + - "name" consisting of "synced_tables/" prefix and the full name of the destination table.' + timeseries_key: + type: string + description: Time series key to deduplicate (tie-break) rows with the same primary key. + type_overrides: + type: array + items: + type: object + properties: + column_name: + type: string + description: Name of the source column whose target PostgreSQL type should be overridden. + pg_type: + type: string + x-enum: + - PG_SPECIFIC_TYPE_HALFVEC + - PG_SPECIFIC_TYPE_VARCHAR + - PG_SPECIFIC_TYPE_VECTOR + description: PostgreSQL-specific target type to use for the column. + size: + type: integer + description: Size parameter for the target type, for types that take one (e.g. vector dimension, + varchar length). Required when the chosen pg_type needs a size. + required: + - column_name + - pg_type + description: Overrides the default Delta-to-PostgreSQL type mapping for a single column. + description: Override the default Delta->PG type mapping for specific columns. A TypeOverride with + PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set. + description: Configuration details of the synced table, such as the source table, scheduling policy, + etc. This attribute is specified at creation time and most fields are returned as is on subsequent + queries. + status: + type: object + properties: + detailed_state: + type: string + x-enum: + - SYNCED_TABLE_OFFLINE + - SYNCED_TABLE_OFFLINE_FAILED + - SYNCED_TABLE_ONLINE + - SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE + - SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE + - SYNCED_TABLE_ONLINE_PIPELINE_FAILED + - SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE + - SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES + - SYNCED_TABLE_PROVISIONING + - SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT + - SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES + description: The state of a synced table. + last_processed_commit_version: + type: integer + description: The last source table Delta version that was successfully synced to the synced table. + last_sync: + type: object + properties: + delta_table_sync_info: + type: object + properties: + delta_commit_time: + type: string + format: date-time + delta_commit_version: + type: integer + description: The Delta Lake commit version that was last successfully synced. + sync_end_time: + type: string + format: date-time + description: The end timestamp of the most recent successful synchronization. This is the time + when the data is available in the synced table. + sync_start_time: + type: string + format: date-time + description: The starting timestamp of the most recent successful synchronization from the source + table to the destination (synced) table. Note this is the starting timestamp of the sync operation, + not the end time. E.g., for a batch, this is the time when the sync operation started. + description: Summary of the last successful synchronization from source to destination. + last_sync_time: + type: string + format: date-time + description: The end timestamp of the last time any data was synchronized from the source table + to the synced table. This is when the data is available in the synced table. + message: + type: string + description: A text description of the current state of the synced table. + ongoing_sync_progress: + type: object + properties: + estimated_completion_time_seconds: + type: number + description: The estimated time remaining to complete this update in seconds. + latest_version_currently_processing: + type: integer + description: The source table Delta version that was last processed by the pipeline. The pipeline + may not have completely processed this version yet. + sync_progress_completion: + type: number + description: The completion ratio of this update. This is a number between 0 and 1. + synced_row_count: + type: integer + description: The number of rows that have been synced in this update. + total_row_count: + type: integer + description: The total number of rows that need to be synced in this update. This number may + be an estimate. + description: Progress information of the Synced Table data synchronization pipeline. + pipeline_id: + type: string + description: ID of the associated pipeline. + project: + type: string + description: 'The full resource name of the project associated with the table. Format: "projects/{project_id}".' + provisioning_phase: + type: string + x-enum: + - PROVISIONING_PHASE_INDEX_SCAN + - PROVISIONING_PHASE_INDEX_SORT + - PROVISIONING_PHASE_MAIN + description: The current phase of the data synchronization pipeline. + unity_catalog_provisioning_state: + type: string + x-enum: + - ACTIVE + - DEGRADED + - DELETING + - FAILED + - PROVISIONING + - UPDATING + description: The provisioning state of the synced table entity in Unity Catalog. + description: Synced Table data synchronization status. + synced_table_id: + type: string + description: The part of the name, chosen by the user when the resource was created. + uid: + type: string + description: The Unity Catalog table ID for this synced table. + required: + - synced_table + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/tables: + post: + operationId: postgres_create_table + summary: Create a Table (non-synced database table for Autoscaling v2 Lakebase projects). + tags: + - postgres + description: |- + Create a Table (non-synced database table for Autoscaling v2 Lakebase projects). + + :param table: :class:`Table` + + :returns: :class:`Table` + requestBody: + content: + application/json: + schema: + type: object + properties: + table: + type: object + properties: + name: + type: string + description: Full three-part (catalog, schema, table) name of the table. + database: + type: string + description: 'The project and branch scoped database to which this table belongs. Of the format: projects/{project_id}/branches/{branch_id}/databases/{database_id} + where database_id is the name of the logical database in Postgres.' + branch: + type: string + description: The id of the database branch associated with the table. Of the format projects/{project_id}/branches/{branch_id}. + project: + type: string + description: The id of the database project associated with the table. Of the format projects/{project_id}. + table_serving_url: + type: string + description: REST API URL for serving data from this table. + required: + - name + - database + description: |- + Table represents a non-synced database table in a Lakebase project. Unlike SyncedTable, this + does not have a data synchronization pipeline. + required: + - table + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Table' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{name}: + delete: + operationId: postgres_delete_synced_table + summary: Delete a Synced Table. + tags: + - postgres + description: |- + Delete a Synced Table. + + :param name: str + The Full resource name of the synced table, of the format + "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the UC entity names. + + :returns: :class:`Operation` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The Full resource name of the synced table, of the format "synced_tables/{catalog}.{schema}.{table}", + where (catalog, schema, table) are the UC entity names. + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + get: + operationId: postgres_get_synced_table + summary: Get a Synced Table. + tags: + - postgres + description: |- + Get a Synced Table. + + :param name: str + The Full resource name of the synced table. Format: "synced_tables/{catalog}.{schema}.{table}", + where (catalog, schema, table) are the entity names in the Unity Catalog. + + :returns: :class:`SyncedTable` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: 'The Full resource name of the synced table. Format: "synced_tables/{catalog}.{schema}.{table}", where + (catalog, schema, table) are the entity names in the Unity Catalog.' + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SyncedTable' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + patch: + operationId: postgres_update_snapshot_schedule + summary: Sets the snapshot schedule for a branch. The ``schedule`` field is replaced wholesale; an empty + tags: + - postgres + description: |- + Sets the snapshot schedule for a branch. The ``schedule`` field is replaced wholesale; an empty + schedule disables automatic snapshots. + + :param name: str + The resource name of the branch's snapshot schedule. Format: + projects/{project_id}/branches/{branch_id}/snapshot-schedule + :param snapshot_schedule: :class:`SnapshotSchedule` + The snapshot schedule to set. Its ``name`` identifies the branch. Format: + projects/{project_id}/branches/{branch_id}/snapshot-schedule + :param update_mask: FieldMask + Fields to update. The only updatable path is ``schedule``, which replaces the entire set of + cadences. + + :returns: :class:`SnapshotSchedule` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: 'The resource name of the branch''s snapshot schedule. Format: projects/{project_id}/branches/{branch_id}/snapshot-schedule' + - name: update_mask + in: query + required: true + schema: + type: object + description: Fields to update. The only updatable path is ``schedule``, which replaces the entire set of cadences. + requestBody: + content: + application/json: + schema: + type: object + properties: + snapshot_schedule: + type: object + properties: + name: + type: string + description: 'The resource name of the branch''s snapshot schedule. Format: projects/{project_id}/branches/{branch_id}/snapshot-schedule' + schedule: + type: array + items: + type: object + properties: + retention: + type: string + description: How long snapshots from this cadence are kept before automatic deletion. Must be + at least 1 hour. Applied when a snapshot is taken; not retroactive, so changing it affects only + later snapshots. + daily_schedule: + type: object + properties: + hour: + type: integer + description: The hour of the day, in UTC, at which to take the snapshot, in [0, 23]. + description: Take a snapshot once per day. + monthly_schedule: + type: object + properties: + day: + type: integer + description: The day of the month on which to take the snapshot, in [1, 31]. In shorter months + the snapshot is taken on the last day instead (day 31 runs on Feb 28 or 29, and on Apr 30), + so every month gets exactly one snapshot. + hour: + type: integer + description: The hour of the day, in UTC, at which to take the snapshot, in [0, 23]. + required: + - day + description: Take a snapshot once per month. + weekly_schedule: + type: object + properties: + day_of_week: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: The day of the week on which to take the snapshot. + hour: + type: integer + description: The hour of the day, in UTC, at which to take the snapshot, in [0, 23]. + required: + - day_of_week + description: Take a snapshot once per week. + required: + - retention + description: One cadence at which automatic snapshots are taken. + description: The cadences at which automatic snapshots are taken. Update replaces the whole set; an + empty set disables automatic snapshots. Order is not significant. When several cadences fire together, + one snapshot is taken, retained for the longest of their retentions. + description: 'The snapshot schedule to set. Its ``name`` identifies the branch. Format: projects/{project_id}/branches/{branch_id}/snapshot-schedule' + required: + - snapshot_schedule + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SnapshotSchedule' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/forward-etl/configuration: + delete: + operationId: postgres_delete_forward_etl_configuration + summary: Hard delete a Forward ETL configuration and all associated table mappings. Unlike DisableForwardEtl, + tags: + - postgres + description: |- + Hard delete a Forward ETL configuration and all associated table mappings. Unlike DisableForwardEtl, + this permanently removes the config and mapping rows. + + :param parent: str + The Branch to delete Forward ETL configuration for. Format: + projects/{project_id}/branches/{branch_id} + :param pg_database_oid: int (optional) + PostgreSQL database OID to delete configuration for. + :param pg_schema_oid: int (optional) + PostgreSQL schema OID to delete configuration for. + :param tenant_id: str (optional) + Tenant ID (dashless UUID format). + :param timeline_id: str (optional) + Timeline ID (dashless UUID format). + + :returns: :class:`DeleteForwardEtlConfigurationResponse` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch to delete Forward ETL configuration for. Format: projects/{project_id}/branches/{branch_id}' + - name: pg_database_oid + in: query + required: false + schema: + type: integer + description: PostgreSQL database OID to delete configuration for. + - name: pg_schema_oid + in: query + required: false + schema: + type: integer + description: PostgreSQL schema OID to delete configuration for. + - name: tenant_id + in: query + required: false + schema: + type: string + description: Tenant ID (dashless UUID format). + - name: timeline_id + in: query + required: false + schema: + type: string + description: Timeline ID (dashless UUID format). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DeleteForwardEtlConfigurationResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/tables/{name}: + delete: + operationId: postgres_delete_table + summary: Delete a Table (non-synced database table for Autoscaling v2 Lakebase projects). + tags: + - postgres + description: |- + Delete a Table (non-synced database table for Autoscaling v2 Lakebase projects). + + :param name: str + Full three-part (catalog, schema, table) name of the table. + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Full three-part (catalog, schema, table) name of the table. + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + get: + operationId: postgres_get_table + summary: Get a Table (non-synced database table for Autoscaling v2 Lakebase projects). + tags: + - postgres + description: |- + Get a Table (non-synced database table for Autoscaling v2 Lakebase projects). + + :param name: str + Full three-part (catalog, schema, table) name of the table. + + :returns: :class:`Table` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Full three-part (catalog, schema, table) name of the table. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Table' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/forward-etl: + delete: + operationId: postgres_disable_forward_etl + summary: Disable Forward ETL for a branch. + tags: + - postgres + description: |- + Disable Forward ETL for a branch. + + :param parent: str + The Branch to disable Forward ETL for. Format: projects/{project_id}/branches/{branch_id} + :param pg_database_oid: int (optional) + PostgreSQL database OID to disable. + :param pg_schema_oid: int (optional) + PostgreSQL schema OID to disable. + :param tenant_id: str (optional) + Tenant ID (dashless UUID format). + :param timeline_id: str (optional) + Timeline ID (dashless UUID format). + + :returns: :class:`DisableForwardEtlResponse` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch to disable Forward ETL for. Format: projects/{project_id}/branches/{branch_id}' + - name: pg_database_oid + in: query + required: false + schema: + type: integer + description: PostgreSQL database OID to disable. + - name: pg_schema_oid + in: query + required: false + schema: + type: integer + description: PostgreSQL schema OID to disable. + - name: tenant_id + in: query + required: false + schema: + type: string + description: Tenant ID (dashless UUID format). + - name: timeline_id + in: query + required: false + schema: + type: string + description: Timeline ID (dashless UUID format). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DisableForwardEtlResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + get: + operationId: postgres_get_forward_etl_status + summary: Get Forward ETL configuration and status for a branch. + tags: + - postgres + description: |- + Get Forward ETL configuration and status for a branch. + + :param parent: str + The Branch to get Forward ETL status for. Format: projects/{project_id}/branches/{branch_id} + :param tenant_id: str (optional) + Tenant ID (dashless UUID format). + :param timeline_id: str (optional) + Timeline ID (dashless UUID format). + + :returns: :class:`ForwardEtlStatus` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch to get Forward ETL status for. Format: projects/{project_id}/branches/{branch_id}' + - name: tenant_id + in: query + required: false + schema: + type: string + description: Tenant ID (dashless UUID format). + - name: timeline_id + in: query + required: false + schema: + type: string + description: Timeline ID (dashless UUID format). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ForwardEtlStatus' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{name}/failover: + post: + operationId: postgres_failover_replication_group_preview + summary: Fails over the replication group to a target workspace, promoting the secondary to primary. + tags: + - postgres + description: |- + Fails over the replication group to a target workspace, promoting the secondary to primary. + + :param name: str + :param target_workspace: str + :param etag: str (optional) + :param request_id: str (optional) + + :returns: :class:`Operation` + parameters: + - name: name + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + target_workspace: + type: string + etag: + type: string + request_id: + type: string + required: + - target_workspace + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/credentials: + post: + operationId: postgres_generate_database_credential + summary: Generate OAuth credentials for a Postgres database. + tags: + - postgres + description: |- + Generate OAuth credentials for a Postgres database. + + :param endpoint: str + The endpoint resource name for which this credential will be generated. Format: + projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} + :param claims: List[:class:`RequestedClaims`] (optional) + The returned token will be scoped to UC tables with the specified permissions. + :param expire_time: Timestamp (optional) + Timestamp in UTC of when this credential should expire. Must be at least 300 seconds (5 minutes) and + at most 1 hour from the current time. + :param group_name: str (optional) + Databricks workspace group name. When provided, credentials are generated with permissions scoped to + this group. + :param ttl: Duration (optional) + The requested time-to-live for the generated credential token. Must be at least 300 seconds (5 + minutes) and at most 3600 seconds (1 hour). + + :returns: :class:`DatabaseCredential` + requestBody: + content: + application/json: + schema: + type: object + properties: + endpoint: + type: string + description: 'The endpoint resource name for which this credential will be generated. Format: projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}' + claims: + type: array + items: + type: object + properties: + permission_set: + type: string + x-enum: + - READ_ONLY + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + resources: + type: array + items: + type: object + properties: + table_name: + type: string + description: The returned token will be scoped to UC tables with the specified permissions. + expire_time: + type: string + format: date-time + description: Timestamp in UTC of when this credential should expire. Must be at least 300 seconds (5 minutes) + and at most 1 hour from the current time. + group_name: + type: string + description: Databricks workspace group name. When provided, credentials are generated with permissions + scoped to this group. + ttl: + type: string + description: The requested time-to-live for the generated credential token. Must be at least 300 seconds + (5 minutes) and at most 3600 seconds (1 hour). + required: + - endpoint + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/DatabaseCredential' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/forward-etl/metadata: + get: + operationId: postgres_get_forward_etl_metadata + summary: Get Forward ETL metadata (database and schema OIDs). + tags: + - postgres + description: |- + Get Forward ETL metadata (database and schema OIDs). + + :param parent: str + The Branch to get metadata for. Format: projects/{project_id}/branches/{branch_id} + :param tenant_id: str (optional) + Tenant ID (dashless UUID format). + :param timeline_id: str (optional) + Timeline ID (dashless UUID format). + + :returns: :class:`ForwardEtlMetadata` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The Branch to get metadata for. Format: projects/{project_id}/branches/{branch_id}' + - name: tenant_id + in: query + required: false + schema: + type: string + description: Tenant ID (dashless UUID format). + - name: timeline_id + in: query + required: false + schema: + type: string + description: Timeline ID (dashless UUID format). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ForwardEtlMetadata' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{name}/inspect: + post: + operationId: postgres_inspect_recovery_branch_preview + summary: Materializes a temporary inspection branch from the specified recovery branch for data examination. + tags: + - postgres + description: |- + Materializes a temporary inspection branch from the specified recovery branch for data examination. + + :param name: str + The recovery branch from which to create the inspection branch. + :param branch_id: str + Caller-supplied id for the inspection Branch this custom method materializes. + :param request_id: str (optional) + + :returns: :class:`Operation` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The recovery branch from which to create the inspection branch. + requestBody: + content: + application/json: + schema: + type: object + properties: + branch_id: + type: string + description: Caller-supplied id for the inspection Branch this custom method materializes. + request_id: + type: string + required: + - branch_id + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/cdf-statuses: + get: + operationId: postgres_list_cdf_statuses + summary: List the per-table CDF statuses within a Lakebase CDF configuration. Each status shows whether a + tags: + - postgres + description: |- + List the per-table CDF statuses within a Lakebase CDF configuration. Each status shows whether a + table's change data feed is snapshotting, streaming, or skipped. + + :param parent: str + The parent CdfConfig to list CdfStatuses for. Format: + projects/{project}/branches/{branch}/databases/{database}/cdf-configs/{cdf_config} + :param page_size: int (optional) + Maximum number of CdfStatuses to return. + :param page_token: str (optional) + Pagination token returned by a previous ListCdfStatuses call. Empty on the first page. + + :returns: Iterator over :class:`CdfStatus` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: 'The parent CdfConfig to list CdfStatuses for. Format: projects/{project}/branches/{branch}/databases/{database}/cdf-configs/{cdf_config}' + - name: page_size + in: query + required: false + schema: + type: integer + description: Maximum number of CdfStatuses to return. + - name: page_token + in: query + required: false + schema: + type: string + description: Pagination token returned by a previous ListCdfStatuses call. Empty on the first page. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListCdfStatusesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/compute-instances: + get: + operationId: postgres_list_compute_instances + summary: 'Lists all compute instances that have been created under the specified endpoint. Note:' + tags: + - postgres + description: |- + Lists all compute instances that have been created under the specified endpoint. Note: + ComputeInstances are managed via the parent Endpoint resource, and cannot be created, updated, or + deleted directly. + + :param parent: str + The parent, which owns the compute instances. + :param page_size: int (optional) + The maximum number of compute instances to return. The service may return fewer than this value. + + If unspecified, at most 50 compute instances will be returned. The maximum value is 1000; values + above 1000 will be coerced to 1000. + :param page_token: str (optional) + A page token, received from a previous ``ListInstances`` call. Provide this to retrieve the + subsequent page. + + When paginating, all other parameters provided to ``ListInstances`` must match the call that + provided the page token. + + :returns: Iterator over :class:`ComputeInstance` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: The parent, which owns the compute instances. + - name: page_size + in: query + required: false + schema: + type: integer + description: The maximum number of compute instances to return. The service may return fewer than this value. If unspecified, + at most 50 compute instances will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. + - name: page_token + in: query + required: false + schema: + type: string + description: A page token, received from a previous ``ListInstances`` call. Provide this to retrieve the subsequent + page. When paginating, all other parameters provided to ``ListInstances`` must match the call that provided the + page token. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListComputeInstancesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{parent}/preview/recovery-branches: + get: + operationId: postgres_list_recovery_branch_previews + summary: Returns a paginated list of recovery branches for the project. + tags: + - postgres + description: |- + Returns a paginated list of recovery branches for the project. + + :param parent: str + :param page_size: int (optional) + :param page_token: str (optional) + + :returns: Iterator over :class:`RecoveryBranchPreview` + parameters: + - name: parent + in: path + required: true + schema: + type: string + - name: page_size + in: query + required: false + schema: + type: integer + - name: page_token + in: query + required: false + schema: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListRecoveryBranchPreviewsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{name}/switchover: + post: + operationId: postgres_switchover_replication_group_preview + summary: Switches over the replication group to a target workspace with a coordinated failover. + tags: + - postgres + description: |- + Switches over the replication group to a target workspace with a coordinated failover. + + :param name: str + :param target_workspace: str + :param etag: str (optional) + :param request_id: str (optional) + + :returns: :class:`Operation` + parameters: + - name: name + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + properties: + target_workspace: + type: string + etag: + type: string + request_id: + type: string + required: + - target_workspace + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI + /api/2.0/postgres/{name}/undelete: + post: + operationId: postgres_undelete_project + summary: Undeletes a soft-deleted project. + tags: + - postgres + description: |- + Undeletes a soft-deleted project. + + :param name: str + The full resource path of the project to undelete. Format: projects/{project_id} + + :returns: :class:`Operation` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: 'The full resource path of the project to undelete. Format: projects/{project_id}' + responses: + '200': + description: Success + content: + application/json: + schema: + type: object + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: PostgresAPI +components: + schemas: + Branch: + type: object + properties: + branch_id: + type: string + create_time: + type: string + format: date-time + description: A timestamp indicating when the branch was created. + name: + type: string + description: 'Output only. The full resource path of the branch. Format: projects/{project_id}/branches/{branch_id}' + parent: + type: string + description: 'The project containing this branch (API resource hierarchy). Format: projects/{project_id} Note: This + field indicates where the branch exists in the resource hierarchy. For point-in-time branching from another branch, + see ``status.source_branch``.' + spec: + $ref: '#/components/schemas/BranchSpec' + description: The spec contains the branch configuration. + status: + $ref: '#/components/schemas/BranchStatus' + description: The current status of a Branch. + uid: + type: string + description: System-generated unique ID for the branch. + update_time: + type: string + format: date-time + description: A timestamp indicating when the branch was last updated. + BranchOperationMetadata: + type: object + properties: {} + BranchSpec: + type: object + properties: + expire_time: + type: string + format: date-time + is_protected: + type: boolean + description: When set to true, protects the branch from deletion and reset. Associated compute endpoints and the + project cannot be deleted while the branch is protected. + no_expiry: + type: boolean + description: Explicitly disable expiration. When set to true, the branch will not expire. If set to false, the request + is invalid; provide either ttl or expire_time instead. Mutually exclusive with ``expire_time`` and ``ttl``. When + updating, use ``spec.expiration`` in the update_mask. + source_branch: + type: string + description: 'The name of the source branch from which this branch was created (data lineage for point-in-time recovery). + If not specified, defaults to the project''s default branch. Format: projects/{project_id}/branches/{branch_id}' + source_branch_lsn: + type: string + description: The Log Sequence Number (LSN) on the source branch from which this branch was created. + source_branch_time: + type: string + format: date-time + description: The point in time on the source branch from which this branch was created. ttl: type: string - description: >- - Relative time-to-live duration. When set, the branch will expire at - creation_time + ttl. - BranchStatus: + description: Relative time-to-live duration. When set, the branch will expire at creation_time + ttl. Mutually exclusive + with ``expire_time`` and ``no_expiry``. When updating, use ``spec.expiration`` in the update_mask. + BranchStatus: + type: object + properties: + branch_id: + type: string + current_state: + $ref: '#/components/schemas/BranchStatusState' + description: The branch's state, indicating if it is initializing, ready for use, or archived. + default: + type: boolean + description: Whether the branch is the project's default branch. + delete_time: + type: string + format: date-time + description: A timestamp indicating when the branch was deleted. Empty if the branch is not deleted. + expire_time: + type: string + format: date-time + description: Absolute expiration time for the branch. Empty if expiration is disabled. + is_protected: + type: boolean + description: Whether the branch is protected. + logical_size_bytes: + type: integer + description: The logical size of the branch. + pending_state: + $ref: '#/components/schemas/BranchStatusState' + description: The pending state of the branch, if a state transition is in progress. + purge_time: + type: string + format: date-time + description: A timestamp indicating when the branch is scheduled to be purged. Empty if the branch is not deleted, + otherwise set to a timestamp in the future. + source_branch: + type: string + description: 'The name of the source branch from which this branch was created. Format: projects/{project_id}/branches/{branch_id}' + source_branch_lsn: + type: string + description: The Log Sequence Number (LSN) on the source branch from which this branch was created. + source_branch_time: + type: string + format: date-time + description: The point in time on the source branch from which this branch was created. + source_recovery_branch: + type: string + description: 'If this branch is a child of a recovery branch, this field identifies that recovery source. For non-recovery-derived + branches this is unset. Format: projects/{project_id}/preview/recovery-branches/{recovery_branch_id}' + state_change_time: + type: string + format: date-time + description: A timestamp indicating when the ``current_state`` began. + Catalog: + type: object + properties: + catalog_id: + type: string + create_time: + type: string + format: date-time + description: A timestamp indicating when the catalog was created. + name: + type: string + description: 'Output only. The full resource path of the catalog. Format: "catalogs/{catalog_id}".' + spec: + $ref: '#/components/schemas/CatalogCatalogSpec' + description: The desired state of the Catalog. + status: + $ref: '#/components/schemas/CatalogCatalogStatus' + description: The observed state of the Catalog. + uid: + type: string + description: System-generated unique identifier for the catalog. + update_time: + type: string + format: date-time + description: A timestamp indicating when the catalog was last updated. + CatalogCatalogSpec: type: object properties: - current_state: - $ref: '#/components/schemas/BranchStatusState' - default: + postgres_database: + type: string + description: The name of the Postgres database inside the specified Lakebase project and branch to be associated + with the UC catalog. This database must already exist, unless create_database_if_missing is set to true on creation. + A database can only be registered with one UC catalog at a time. To re-register a database with a different catalog, + the existing catalog must be deleted first. A child branch inherits the fact of parent's registration. This means + the same-named database in a child branch cannot be registered with a second catalog while the parent's registration + exists. To allow registering the database of a child branch, drop and recreate the database on the child branch. + This removes the fact of parent's registration from this branch only. Doing Point In Time Restore (PITR) prior + to the moment before the Postgres DB was registered in the Catalog drops the fact of registration of the database. + So the user should avoid doing so. + branch: + type: string + description: 'The resource path of the branch associated with the catalog. Format: projects/{project_id}/branches/{branch_id}.' + create_database_if_missing: type: boolean - description: Whether the branch is the project's default branch. - expire_time: + description: If set to true, the specified postgres_database is created on behalf of the calling user if it does + not already exist. In this case, the calling user has a role created for them in Postgres if they do not already + have one. Defaults to false, meaning that the request fails if the specified postgres_database does not already + exist. + required: + - postgres_database + description: The desired state of the Catalog. + CatalogCatalogStatus: + type: object + properties: + branch: + type: string + description: 'The resource path of the branch associated with the catalog. Format: projects/{project_id}/branches/{branch_id}.' + postgres_database: + type: string + description: The name of the Postgres database associated with the catalog. + project: + type: string + description: 'The resource path of the project associated with the catalog. Format: projects/{project_id}.' + description: The observed state of the Catalog. + CatalogOperationMetadata: + type: object + properties: {} + CdfConfig: + type: object + properties: + catalog: + type: string + description: The Unity Catalog catalog that replicated tables are written into. Set at creation; the CdfConfig is + immutable. + schema: + type: string + description: The Unity Catalog schema that replicated tables are written into. Set at creation; the CdfConfig is + immutable. + postgres_schema: + type: string + description: The Postgres schema this CdfConfig replicates from. Unique within the parent database. Set at creation; + the CdfConfig is immutable. + cdf_config_id: + type: string + description: The user-specified id; equals the final segment of ``name``. Defaults to the Postgres schema name for + configs without an explicit id. + create_time: type: string format: date-time - description: >- - Absolute expiration time for the branch. Empty if expiration is - disabled. - is_protected: - type: boolean - description: Whether the branch is protected. - logical_size_bytes: - type: integer - description: The logical size of the branch. + description: When the CdfConfig was created. + name: + type: string + description: 'Output only. The full resource name of the CdfConfig. Format: projects/{project}/branches/{branch}/databases/{database}/cdf-configs/{cdf_config}' + required: + - catalog + - schema + - postgres_schema + description: |- + A Lakebase CDF configuration (CdfConfig): one per Postgres schema per database, replicating that + schema's tables into a Unity Catalog schema. Immutable once created. + CdfConfigOperationMetadata: + type: object + properties: {} + description: |- + Metadata for CdfConfig long-running operations. Intentionally empty today; fields (e.g. + progress) may be added as the operation contract grows. + CdfStatus: + type: object + properties: + committed_lsn: + type: string + description: The high-watermark Log Sequence Number (LSN) committed to Delta Lake. + create_time: + type: string + format: date-time + description: When replication for this table was first established. + last_sync_time: + type: string + format: date-time + description: The last time changes for this table were written to Delta Lake. + name: + type: string + description: 'Output only. The full resource name of the CdfStatus. Format: projects/{project}/branches/{branch}/databases/{database}/cdf-configs/{cdf_config}/cdf-statuses/{cdf_status} + The {cdf_status} segment is the Postgres table name.' + postgres_table: + type: string + description: The Postgres table being replicated. + state: + $ref: '#/components/schemas/CdfState' + description: The current replication state of this table. + status_detail: + type: string + description: Human-readable detail for the current state (e.g. the skip/error reason). Empty for healthy states. + uc_table: + type: string + description: The Unity Catalog table receiving replicated data. + description: |- + The read-only replication status of a single Postgres table replicated under a CdfConfig. One + status exists per replicated table. It is created automatically and cannot be modified. + ComputeInstance: + type: object + properties: + name: + type: string + compute_instance_id: + type: string + description: The unique ID for this compute. + compute_host: + type: string + description: A host scoped directly to the enclosing compute. This host is guaranteed to resolve to the specific + compute instance. + current_state: + $ref: '#/components/schemas/ComputeInstanceComputeState' + description: The current state of the compute. pending_state: - $ref: '#/components/schemas/BranchStatusState' - description: >- - The pending state of the branch, if a state transition is in - progress. - source_branch: + $ref: '#/components/schemas/ComputeInstanceComputeState' + description: The desired pending state of the compute, if a state transition is in progress. + role: + $ref: '#/components/schemas/ComputeInstanceComputeType' + description: The role of this compute within the endpoint. + start_time: type: string - description: >- - The name of the source branch from which this branch was created. - Format: projects/{project_id}/branches/{branch_id} - source_branch_lsn: + format: date-time + description: A timestamp indicating when the compute was last started. + suspend_time: type: string - description: >- - The Log Sequence Number (LSN) on the source branch from which this - branch was created. - source_branch_time: + format: date-time + description: A timestamp indicating when the compute was last suspended. + required: + - name + - compute_instance_id + DailySchedule: + type: object + properties: + hour: + type: integer + description: The hour of the day, in UTC, at which to take the snapshot, in [0, 23]. + description: Take a snapshot once per day, at the configured hour. + DataApi: + type: object + properties: + create_time: type: string format: date-time - description: >- - The point in time on the source branch from which this branch was - created. - state_change_time: + description: A timestamp indicating when the Data API was first enabled. + name: + type: string + description: 'Resource name: projects/{project_id}/branches/{branch_id}/databases/{database_id}/data-api' + parent: + type: string + description: 'The database containing this Data API configuration. Format: projects/{project_id}/branches/{branch_id}/databases/{database_id}' + spec: + $ref: '#/components/schemas/DataApiDataApiSpec' + description: The desired Data API configuration. + status: + $ref: '#/components/schemas/DataApiDataApiStatus' + description: The observed Data API state (read-only). + update_time: type: string format: date-time - description: A timestamp indicating when the `current_state` began. + description: A timestamp indicating when the Data API configuration was last updated. + description: |- + DataApi represents the Data API (PostgREST) configuration for a Database. At most one DataApi + per database. Create enables Data API, Delete disables it. + DataApiDataApiSpec: + type: object + properties: + db_aggregates_enabled: + type: boolean + description: 'Enable aggregate functions (count, sum, avg, etc.) in Data API responses. Default: true.' + db_anon_role: + type: string + description: 'The PostgreSQL role used for unauthenticated (anonymous) requests. Must be a valid PostgreSQL role + name (1-63 chars, [a-zA-Z_][a-zA-Z0-9_$]*). Default: "anonymous".' + db_extra_search_path: + type: array + items: + type: string + description: Additional schemas to include in the PostgreSQL search path. Each entry must be a valid PostgreSQL + schema name. + db_max_rows: + type: integer + description: Maximum number of rows returned in a single Data API response. Must be a positive integer. + db_schemas: + type: array + items: + type: string + description: 'Database schemas exposed through the Data API. Each entry must be a valid PostgreSQL schema name (1-63 + chars, [a-zA-Z_][a-zA-Z0-9_$]*). Maximum 100 entries. Default: ["public"].' + jwt_cache_max_lifetime: + type: string + description: Maximum lifetime for cached JWT tokens. Zero duration disables caching. + jwt_role_claim_key: + type: string + description: 'JSON path to the role claim in JWT tokens (e.g., ".sub"). Default: ".sub".' + openapi_mode: + $ref: '#/components/schemas/OpenApiMode' + description: OpenAPI documentation mode for the Data API endpoint. + server_cors_allowed_origins: + type: array + items: + type: string + description: Allowed origins for CORS requests. Each entry should be a valid origin URL, or use "*" to allow all + origins. + server_timing_enabled: + type: boolean + description: Enable the Server-Timing header in Data API responses. + description: Desired PostgREST configuration (input). + DataApiDataApiStatus: + type: object + properties: + available_schemas: + type: array + items: + type: string + description: Schemas available in the database (for reference when configuring db_schemas). + db_aggregates_enabled: + type: boolean + description: Actual aggregate function setting read from the database. + db_anon_role: + type: string + description: Actual anonymous role name read from the database. + db_extra_search_path: + type: array + items: + type: string + description: Actual extra search path schemas read from the database. + db_max_rows: + type: integer + description: Actual max rows setting read from the database. + db_schemas: + type: array + items: + type: string + description: Actual exposed schemas read from the database. + jwt_cache_max_lifetime: + type: string + description: Actual JWT cache max lifetime read from the database. + jwt_role_claim_key: + type: string + description: Actual JWT role claim key read from the database. + openapi_mode: + $ref: '#/components/schemas/OpenApiMode' + description: Actual OpenAPI mode read from the database. + server_cors_allowed_origins: + type: array + items: + type: string + description: Actual CORS allowed origins read from the database. + server_timing_enabled: + type: boolean + description: Actual Server-Timing header setting read from the database. + url: + type: string + description: Data API endpoint URL. + description: Observed state (output-only). + DataApiOperationMetadata: + type: object + properties: {} Database: type: object properties: @@ -2031,16 +4050,15 @@ components: type: string format: date-time description: A timestamp indicating when the database was created. + database_id: + type: string + description: The part of the name, chosen by the user when the resource was created. name: type: string - description: >- - The resource name of the database. Format: - projects/{project_id}/branches/{branch_id}/databases/{database_id} + description: 'The resource name of the database. Format: projects/{project_id}/branches/{branch_id}/databases/{database_id}' parent: type: string - description: >- - The branch containing this database. Format: - projects/{project_id}/branches/{branch_id} + description: 'The branch containing this database. Format: projects/{project_id}/branches/{branch_id}' spec: $ref: '#/components/schemas/DatabaseDatabaseSpec' description: The desired state of the Database. @@ -2060,31 +4078,30 @@ components: format: date-time token: type: string - description: >- - The OAuth token that can be used as a password when connecting to a - database. + description: The OAuth token that can be used as a password when connecting to a database. DatabaseDatabaseSpec: type: object properties: - postgres_database: - type: string role: type: string - description: >- - The name of the role that owns the database. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} To change - the owner, pass valid existing Role name when updating the Database - A database always has an owner. + postgres_database: + type: string + description: The name of the Postgres database. This expects a valid Postgres identifier as specified in the link + below. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS Required when creating + the Database. To rename, pass a valid postgres identifier when updating the Database. + required: + - role DatabaseDatabaseStatus: type: object properties: + database_id: + type: string postgres_database: type: string + description: The name of the Postgres database. role: type: string - description: >- - The name of the role that owns the database. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} + description: 'The name of the role that owns the database. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id}' DatabaseOperationMetadata: type: object properties: {} @@ -2102,27 +4119,51 @@ components: stack_trace: type: string description: Databricks Error that is returned by all Databricks APIs. + DeleteForwardEtlConfigurationResponse: + type: object + properties: + deleted_configs: + type: integer + description: Number of configuration rows deleted (0 or 1). + deleted_mappings: + type: integer + description: Number of table mapping rows deleted. + description: Response to delete Forward ETL configuration. + DeltaTableSyncInfo: + type: object + properties: + delta_commit_time: + type: string + format: date-time + delta_commit_version: + type: integer + description: The Delta Lake commit version that was last successfully synced. + DisableForwardEtlResponse: + type: object + properties: + disabled: + type: boolean + description: Whether Forward ETL was successfully disabled. + description: Response to disable Forward ETL Endpoint: type: object properties: create_time: type: string format: date-time + endpoint_id: + type: string + description: The part of the name, chosen by the user when the resource was created. name: type: string - description: >- - Output only. The full resource path of the endpoint. Format: - projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} + description: 'Output only. The full resource path of the endpoint. Format: projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}' parent: type: string - description: >- - The branch containing this endpoint (API resource hierarchy). - Format: projects/{project_id}/branches/{branch_id} + description: 'The branch containing this endpoint (API resource hierarchy). Format: projects/{project_id}/branches/{branch_id}' spec: $ref: '#/components/schemas/EndpointSpec' - description: >- - The spec contains the compute endpoint configuration, including - autoscaling limits, suspend timeout, and disabled state. + description: The spec contains the compute endpoint configuration, including autoscaling limits, suspend timeout, + and disabled state. status: $ref: '#/components/schemas/EndpointStatus' description: Current operational status of the compute endpoint. @@ -2140,19 +4181,16 @@ components: type: integer max: type: integer - description: >- - The maximum number of computes in the endpoint group. Currently, - this must be equal to min. Set to 1 for single compute endpoints, to - disable HA. To manually suspend all computes in an endpoint group, - set disabled to true on the EndpointSpec. + description: The maximum number of computes in the endpoint group. Currently, this must be equal to min. Set to + 1 for single compute endpoints, to disable HA. To manually suspend all computes in an endpoint group, set disabled + to true on the EndpointSpec. enable_readable_secondaries: type: boolean - description: >- - Whether to allow read-only connections to read-write endpoints. Only - relevant for read-write endpoints where size.max > 1. + description: Whether to allow read-only connections to read-write endpoints. Only relevant for read-write endpoints + where size.max > 1. required: - - min - - max + - min + - max EndpointGroupStatus: type: object properties: @@ -2160,41 +4198,39 @@ components: type: integer max: type: integer - description: >- - The maximum number of computes in the endpoint group. Currently, - this must be equal to min. Set to 1 for single compute endpoints, to - disable HA. To manually suspend all computes in an endpoint group, - set disabled to true on the EndpointSpec. + description: The maximum number of computes in the endpoint group. Currently, this must be equal to min. Set to + 1 for single compute endpoints, to disable HA. To manually suspend all computes in an endpoint group, set disabled + to true on the EndpointSpec. enable_readable_secondaries: type: boolean - description: >- - Whether read-only connections to read-write endpoints are allowed. - Only relevant if read replicas are configured by specifying size.max - > 1. + description: Whether read-only connections to read-write endpoints are allowed. Only relevant if read replicas are + configured by specifying size.max > 1. required: - - min - - max + - min + - max EndpointHosts: type: object properties: host: type: string - description: >- - The hostname to connect to this endpoint. For read-write endpoints, - this is a read-write hostname which connects to the primary compute. - For read-only endpoints, this is a read-only hostname which allows - read-only operations. + description: The hostname to connect to this endpoint. For read-write endpoints, this is a read-write hostname which + connects to the primary compute. For read-only endpoints, this is a read-only hostname which allows read-only + operations. read_only_host: type: string - description: >- - An optionally defined read-only host for the endpoint, without - pooling. For read-only endpoints, this attribute is always defined - and is equivalent to host. For read-write endpoints, this attribute - is defined if the enclosing endpoint is a group with greater than 1 - computes configured, and has readable secondaries enabled. - description: >- - Encapsulates various hostnames (r/w or r/o, pooled or not) for an - endpoint. + description: An optionally defined read-only host for the endpoint, without pooling. For read-only endpoints, this + attribute is always defined and is equivalent to host. For read-write endpoints, this attribute is defined if + the enclosing endpoint is a group with greater than 1 computes configured, and has readable secondaries enabled. + read_only_pooled_host: + type: string + description: The read-only hostname of the compute endpoint, with pooling. This attribute is always defined for + read-only endpoints, and may be defined for read-write endpoints if configured with read replicas and allow read-only + connections. + read_write_pooled_host: + type: string + description: The read-write hostname of the compute endpoint, with pooling. This attribute is only defined for read-write + endpoints. + description: Encapsulates various hostnames (r/w or r/o, pooled or not) for an endpoint. EndpointOperationMetadata: type: object properties: {} @@ -2212,39 +4248,33 @@ components: $ref: '#/components/schemas/EndpointType' autoscaling_limit_max_cu: type: number - description: The maximum number of Compute Units. Minimum value is 0.5. + description: The maximum number of Compute Units. The maximum value is 64. The difference between the minimum and + maximum Compute Units (max - min) must not exceed 16. autoscaling_limit_min_cu: type: number description: The minimum number of Compute Units. Minimum value is 0.5. disabled: type: boolean - description: >- - Whether to restrict connections to the compute endpoint. Enabling - this option schedules a suspend compute operation. A disabled - compute endpoint cannot be enabled by a connection or console - action. + description: Whether to restrict connections to the compute endpoint. Enabling this option schedules a suspend compute + operation. A disabled compute endpoint cannot be enabled by a connection or console action. group: $ref: '#/components/schemas/EndpointGroupSpec' - description: >- - Settings for optional HA configuration of the endpoint. If - unspecified, the endpoint defaults to non HA settings, with a single - compute backing the endpoint (and no readable secondaries for - Read/Write endpoints). + description: Settings for optional HA configuration of the endpoint. If unspecified, the endpoint defaults to non + HA settings, with a single compute backing the endpoint (and no readable secondaries for Read/Write endpoints). no_suspension: type: boolean - description: >- - When set to true, explicitly disables automatic suspension (never - suspend). Should be set to true when provided. + description: When set to true, explicitly disables automatic suspension (never suspend). Should be set to true when + provided. Mutually exclusive with ``suspend_timeout_duration``. When updating, use ``spec.suspension`` in the + update_mask. settings: $ref: '#/components/schemas/EndpointSettings' suspend_timeout_duration: type: string - description: >- - Duration of inactivity after which the compute endpoint is - automatically suspended. If specified should be between 60s and - 604800s (1 minute to 1 week). + description: Duration of inactivity after which the compute endpoint is automatically suspended. If specified should + be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with ``no_suspension``. When updating, use + ``spec.suspension`` in the update_mask. required: - - endpoint_type + - endpoint_type EndpointStatus: type: object properties: @@ -2257,11 +4287,11 @@ components: $ref: '#/components/schemas/EndpointStatusState' disabled: type: boolean - description: >- - Whether to restrict connections to the compute endpoint. Enabling - this option schedules a suspend compute operation. A disabled - compute endpoint cannot be enabled by a connection or console - action. + description: Whether to restrict connections to the compute endpoint. Enabling this option schedules a suspend compute + operation. A disabled compute endpoint cannot be enabled by a connection or console action. + endpoint_id: + type: string + description: Part of the resource name. endpoint_type: $ref: '#/components/schemas/EndpointType' description: The endpoint type. A branch can only have one READ_WRITE endpoint. @@ -2271,30 +4301,247 @@ components: hosts: $ref: '#/components/schemas/EndpointHosts' description: Contains host information for connecting to the endpoint. + last_active_time: + type: string + format: date-time + description: A timestamp indicating when the compute endpoint was last active. pending_state: $ref: '#/components/schemas/EndpointStatusState' settings: $ref: '#/components/schemas/EndpointSettings' suspend_timeout_duration: type: string - description: >- - Duration of inactivity after which the compute endpoint is - automatically suspended. + description: Duration of inactivity after which the compute endpoint is automatically suspended. + ForwardEtlConfig: + type: object + properties: + create_time_millis: + type: integer + description: Configuration creation timestamp in milliseconds since epoch. + enabled: + type: boolean + description: Whether Forward ETL is enabled. + pg_database_oid: + type: integer + description: PostgreSQL database OID. + pg_schema_oid: + type: integer + description: PostgreSQL schema OID. + tenant_id: + type: string + description: Tenant ID (dashless UUID format). + timeline_id: + type: string + description: Timeline ID (dashless UUID format). + uc_catalog_id: + type: string + description: Unity Catalog catalog ID. + uc_schema_id: + type: string + description: Unity Catalog schema ID. + update_time_millis: + type: integer + description: Configuration last update timestamp in milliseconds since epoch. + workspace_id: + type: integer + description: Workspace ID. + description: Forward ETL configuration + ForwardEtlDatabase: + type: object + properties: + name: + type: string + description: Database name. + oid: + type: integer + description: PostgreSQL database OID. + description: Database metadata + ForwardEtlMetadata: + type: object + properties: + databases: + type: array + items: + $ref: '#/components/schemas/ForwardEtlDatabase' + description: List of databases with their PostgreSQL OIDs. + schemas: + type: array + items: + $ref: '#/components/schemas/ForwardEtlSchema' + description: List of schemas with their PostgreSQL OIDs. + description: Forward ETL metadata response + ForwardEtlSchema: + type: object + properties: + name: + type: string + description: Schema name. + oid: + type: integer + description: PostgreSQL schema OID. + description: Schema metadata + ForwardEtlStatus: + type: object + properties: + configurations: + type: array + items: + $ref: '#/components/schemas/ForwardEtlConfig' + description: List of Forward ETL configurations. + table_mappings: + type: array + items: + $ref: '#/components/schemas/ForwardEtlTableMapping' + description: Per-table replication mappings. + description: Forward ETL status response + ForwardEtlTableMapping: + type: object + properties: + enabled: + type: boolean + description: Whether replication is enabled for this table. + last_synced_lsn: + type: string + description: Last synced LSN (Log Sequence Number) for this table. + pg_table_name: + type: string + description: PostgreSQL table name. + pg_table_oid: + type: integer + description: PostgreSQL table OID. + uc_table_id: + type: string + description: Unity Catalog table ID. + uc_table_name: + type: string + description: Unity Catalog table name. + description: Per-table replication mapping + InitialBranchSpec: + type: object + properties: + is_protected: + type: boolean + description: Whether the initial default branch should be protected from deletion. + description: Configuration for the initial default branch created during project creation. + InitialDatabaseSpec: + type: object + properties: + postgres_database: + type: string + description: The name of the Postgres database. This expects a valid Postgres identifier as specified in the link + below. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS + description: |- + Configuration for the initial Postgres database created inside the initial branch for a newly + created project. If omitted, the initial branch still gets an initial database with name + ``databricks_postgres``. The initial database is always owned by the initial Postgres role + (whether caller-provided via ``initial_role_spec`` or defaulted to the caller's identity). InitialEndpointSpec: type: object properties: + autoscaling_limit_max_cu: + type: number + description: The maximum number of Compute Units for the initial endpoint. + autoscaling_limit_min_cu: + type: number + description: The minimum number of Compute Units for the initial endpoint. group: $ref: '#/components/schemas/EndpointGroupSpec' + description: Settings for HA configuration of the endpoint. + no_suspension: + type: boolean + description: When set to true, explicitly disables automatic suspension (never suspend). Should be set to true when + provided. Mutually exclusive with ``suspend_timeout_duration``. + suspend_timeout_duration: + type: string + description: Duration of inactivity after which the initial endpoint is automatically suspended. If specified, should + be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with ``no_suspension``. + description: Configuration for the initial Read/Write endpoint created during project creation. + InitialRoleSpec: + type: object + properties: + attributes: + $ref: '#/components/schemas/RoleAttributes' + description: The desired API-exposed Postgres role attribute to associate with the role. Optional. + auth_method: + $ref: '#/components/schemas/RoleAuthMethod' + description: 'Controls how the Postgres role authenticates when a client opens a database connection. Supported + values: - LAKEBASE_OAUTH_V1: the role authenticates by presenting a Databricks OAuth access token derived from + the backing managed identity (the Databricks user, service principal, or group named by the role''s ``postgres_role``). + No static password exists for roles using this method. - PG_PASSWORD_SCRAM_SHA_256: the role authenticates with + a Postgres password verified server-side using the SCRAM-SHA-256 mechanism. Lakebase generates a password for + the role. - NO_LOGIN: the role cannot open a Postgres session at all. Useful for roles that exist only to own + objects or to aggregate privileges that are then granted to other, loginable roles. If auth_method is left unspecified, + a meaningful authentication method is derived from the identity_type: - For the managed identities, OAUTH is used. + - For the regular postgres roles, authentication based on postgres passwords is used. NOTE: for the Databricks + identity type GROUP, LAKEBASE_OAUTH_V1 is the default auth method (group can login as well).' + identity_type: + $ref: '#/components/schemas/RoleIdentityType' + description: 'The type of role. When specifying a managed-identity, the chosen role_id must be a valid: - application + ID for SERVICE_PRINCIPAL - user email for USER - group name for GROUP' + membership_roles: + type: array + items: + $ref: '#/components/schemas/RoleMembershipRole' + description: An enum value for a standard role that this role is a member of. + postgres_role: + type: string + description: 'The name of the Postgres role. This expects a valid Postgres identifier as specified in the link below. + https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS If you wish to create a + Postgres Role backed by a managed Databricks identity, then postgres_role must be one of the following: 1. user + email for IdentityType.USER 2. app ID for IdentityType.SERVICE_PRINCIPAL 3. group name for IdentityType.GROUP' + description: |- + Configuration for the initial Postgres role created inside the initial branch for a newly + created project. If omitted, the default branch still gets an initial Postgres role + corresponding to the caller of the API endpoint. ListBranchesResponse: type: object properties: - branches: + branches: + type: array + items: + $ref: '#/components/schemas/Branch' + next_page_token: + type: string + description: Token to request the next page of branches. + ListCdfConfigsResponse: + type: object + properties: + cdf_configs: + type: array + items: + $ref: '#/components/schemas/CdfConfig' + description: The CdfConfigs under the parent database. + next_page_token: + type: string + description: Token to retrieve the next page of results; empty when there are no more. + description: |- + Response to a ListCdfConfigs request, containing a page of CdfConfigs and a token for fetching + the next page. + ListCdfStatusesResponse: + type: object + properties: + cdf_statuses: + type: array + items: + $ref: '#/components/schemas/CdfStatus' + description: The replicated tables under the parent CdfConfig. + next_page_token: + type: string + description: Token to retrieve the next page of results; empty when there are no more. + description: |- + Response to a ListCdfStatuses request, containing a page of replicated table statuses and a + token for fetching the next page. + ListComputeInstancesResponse: + type: object + properties: + compute_instances: type: array items: - $ref: '#/components/schemas/Branch' + $ref: '#/components/schemas/ComputeInstance' next_page_token: type: string - description: Token to request the next page of branches. + description: A token, which can be sent as ``page_token`` to retrieve the next page. If this field is omitted, there + are no subsequent pages. ListDatabasesResponse: type: object properties: @@ -2324,9 +4571,25 @@ components: type: array items: $ref: '#/components/schemas/Project' - description: >- - List of all projects in the workspace that the user has permission - to access. + description: List of all projects in the workspace that the user has permission to access. + ListRecoveryBranchPreviewsResponse: + type: object + properties: + next_page_token: + type: string + recovery_branch_previews: + type: array + items: + $ref: '#/components/schemas/RecoveryBranchPreview' + ListReplicationGroupPreviewsResponse: + type: object + properties: + next_page_token: + type: string + replication_group_previews: + type: array + items: + $ref: '#/components/schemas/ReplicationGroupPreview' ListRolesResponse: type: object properties: @@ -2337,66 +4600,116 @@ components: items: $ref: '#/components/schemas/Role' description: List of Postgres roles in the branch. + ListSnapshotsResponse: + type: object + properties: + next_page_token: + type: string + snapshots: + type: array + items: + $ref: '#/components/schemas/Snapshot' + description: The snapshots in the project. + MonthlySchedule: + type: object + properties: + day: + type: integer + description: The day of the month on which to take the snapshot, in [1, 31]. In shorter months the snapshot is taken + on the last day instead (day 31 runs on Feb 28 or 29, and on Apr 30), so every month gets exactly one snapshot. + hour: + type: integer + description: The hour of the day, in UTC, at which to take the snapshot, in [0, 23]. + required: + - day + description: Take a snapshot once per month, on the configured day at the configured hour. + NewPipelineSpec: + type: object + properties: + budget_policy_id: + type: string + pipeline_channel: + $ref: '#/components/schemas/NewPipelineSpecPipelineChannel' + description: Release channel of the underlying pipeline's runtime. Some source table configurations (e.g., read-time + CDF) require PREVIEW. Defaults to CURRENT if not specified. + storage_catalog: + type: string + description: UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to + be a standard catalog where the user has permissions to create Delta tables. + storage_schema: + type: string + description: UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to + be in the standard catalog where the user has permissions to create Delta tables. Operation: type: object properties: done: type: boolean - description: >- - If the value is `false`, it means the operation is still in - progress. If `true`, the operation is completed, and either `error` - or `response` is available. + description: If the value is ``false``, it means the operation is still in progress. If ``true``, the operation + is completed, and either ``error`` or ``response`` is available. error: $ref: '#/components/schemas/DatabricksServiceExceptionWithDetailsProto' - description: >- - The error result of the operation in case of failure or - cancellation. + description: The error result of the operation in case of failure or cancellation. metadata: type: object - description: >- - Service-specific metadata associated with the operation. It - typically contains progress information and common metadata such as - create time. Some services might not provide such metadata. + description: Service-specific metadata associated with the operation. It typically contains progress information + and common metadata such as create time. Some services might not provide such metadata. name: type: string - description: >- - The server-assigned name, which is only unique within the same - service that originally returns it. If you use the default HTTP - mapping, the `name` should be a resource name ending with - `operations/{unique_id}`. + description: The server-assigned name, which is only unique within the same service that originally returns it. + If you use the default HTTP mapping, the ``name`` should be a resource name ending with ``operations/{unique_id}``. response: type: object description: The normal, successful response of the operation. - description: >- - This resource represents a long-running operation that is the result of - a network API call. + description: This resource represents a long-running operation that is the result of a network API call. Project: type: object properties: create_time: type: string format: date-time + delete_time: + type: string + format: date-time + description: A timestamp indicating when the project was soft-deleted. Empty if the project is not deleted, otherwise + set to a timestamp in the past. + initial_branch_spec: + $ref: '#/components/schemas/InitialBranchSpec' + description: Configuration for the initial default branch created as part of project creation. Allows overriding + branch protection. These settings only apply at creation time and do not affect resources created after project + creation. + initial_database_spec: + $ref: '#/components/schemas/InitialDatabaseSpec' + description: Configuration for the initial Postgres database created inside the initial branch for this project. + If omitted, the initial branch still gets an initial database with name ``databricks_postgres``. The initial database + is always owned by the initial role (caller-provided via ``initial_role_spec`` or defaulted to the caller's identity). + This field is input-only; to change databases after project creation, use the standalone Database API. initial_endpoint_spec: $ref: '#/components/schemas/InitialEndpointSpec' - description: >- - Configuration settings for the initial Read/Write endpoint created - inside the default branch for a newly created project. If omitted, - the initial endpoint created will have default settings, without - high availability configured. This field does not apply to any - endpoints created after project creation. Use - spec.default_endpoint_settings to configure default settings for - endpoints created after project creation. + description: Configuration settings for the initial Read/Write endpoint created inside the initial branch for a + newly created project. If omitted, the initial endpoint created will have default settings, without high availability + configured. This field does not apply to any endpoints created after project creation. Use spec.default_endpoint_settings + to configure default settings for endpoints created after project creation. + initial_role_spec: + $ref: '#/components/schemas/InitialRoleSpec' + description: Configuration for the initial Postgres role created inside the initial branch for this project. If + omitted, the initial branch gets an initial role corresponding to the caller of the API endpoint. This field is + input-only; to change roles after project creation, use the standalone Role API. name: type: string - description: >- - Output only. The full resource path of the project. Format: - projects/{project_id} + description: 'Output only. The full resource path of the project. Format: projects/{project_id}' + project_id: + type: string + description: The part of the name, chosen by the user when the resource was created. + purge_time: + type: string + format: date-time + description: A timestamp indicating when the project is scheduled for permanent deletion. Empty if the project is + not deleted, otherwise set to a timestamp in the future. spec: $ref: '#/components/schemas/ProjectSpec' - description: >- - The spec contains the project configuration, including display_name, - pg_version (Postgres version), history_retention_duration, and - default_endpoint_settings. + description: The spec contains the project configuration, including display_name, pg_version (Postgres version), + history_retention_duration, and default_endpoint_settings. status: $ref: '#/components/schemas/ProjectStatus' description: The current status of a Project. @@ -2426,18 +4739,17 @@ components: description: The minimum number of Compute Units. Minimum value is 0.5. no_suspension: type: boolean - description: >- - When set to true, explicitly disables automatic suspension (never - suspend). Should be set to true when provided. + description: When set to true, explicitly disables automatic suspension (never suspend). Should be set to true when + provided. Mutually exclusive with ``suspend_timeout_duration``. When updating, use ``spec.project_default_settings.suspension`` + in the update_mask. pg_settings: type: object description: A raw representation of Postgres settings. suspend_timeout_duration: type: string - description: >- - Duration of inactivity after which the compute endpoint is - automatically suspended. If specified should be between 60s and - 604800s (1 minute to 1 week). + description: Duration of inactivity after which the compute endpoint is automatically suspended. If specified should + be between 60s and 604800s (1 minute to 1 week). Mutually exclusive with ``no_suspension``. When updating, use + ``spec.project_default_settings.suspension`` in the update_mask. description: A collection of settings for a compute endpoint. ProjectOperationMetadata: type: object @@ -2447,38 +4759,37 @@ components: properties: budget_policy_id: type: string + compute_provisioner: + type: string + description: The compute provisioner used to provision endpoints in this project. Overrides the default provisioner + when set. custom_tags: type: array items: $ref: '#/components/schemas/ProjectCustomTag' - description: >- - Custom tags to associate with the project. Forwarded to LBM for - billing and cost tracking. To update tags, provide the new tag list - and include "spec.custom_tags" in the update_mask. To clear all - tags, provide an empty list and include "spec.custom_tags" in the - update_mask. To preserve existing tags, omit this field from the - update_mask (or use wildcard "*" which auto-excludes empty tags). + description: Custom tags to associate with the project. Forwarded to LBM for billing and cost tracking. To update + tags, provide the new tag list and include "spec.custom_tags" in the update_mask. To clear all tags, provide an + empty list and include "spec.custom_tags" in the update_mask. To preserve existing tags, omit this field from + the update_mask (or use wildcard "*" which auto-excludes empty tags). + default_branch: + type: string + description: 'The full resource path for the default branch of the project Format: projects/{project_id}/branches/{branch_id}' default_endpoint_settings: $ref: '#/components/schemas/ProjectDefaultEndpointSettings' display_name: type: string - description: >- - Human-readable project name. Length should be between 1 and 256 - characters. + description: Human-readable project name. Length should be between 1 and 256 characters. enable_pg_native_login: type: boolean - description: >- - Whether to enable PG native password login on all endpoints in this - project. Defaults to true. + description: Whether to enable PG native password login on all endpoints in this project. Defaults to false. history_retention_duration: type: string - description: >- - The number of seconds to retain the shared history for point in time - recovery for all branches in this project. Value should be between - 172800s (2 days) and 2592000s (30 days). + description: The number of seconds to retain the shared history for point in time recovery for all branches in this + project. Value should be between 172800s (2 days) and 3024000s (35 days). pg_version: type: integer - description: The major Postgres version number. Supported versions are 16 and 17. + description: The major Postgres version number. The set of supported versions may vary; consult the API documentation + for currently accepted values. ProjectStatus: type: object properties: @@ -2487,11 +4798,21 @@ components: budget_policy_id: type: string description: The budget policy that is applied to the project. + compute_last_active_time: + type: string + format: date-time + description: The most recent time when any endpoint of this project was active. + compute_provisioner: + type: string + description: The effective compute provisioner backing this project's endpoints. custom_tags: type: array items: $ref: '#/components/schemas/ProjectCustomTag' description: The effective custom tags associated with the project. + default_branch: + type: string + description: The full resource path of the default branch of the project default_endpoint_settings: $ref: '#/components/schemas/ProjectDefaultEndpointSettings' description: The effective default endpoint settings. @@ -2500,23 +4821,130 @@ components: description: The effective human-readable project name. enable_pg_native_login: type: boolean - description: >- - Whether to enable PG native password login on all endpoints in this - project. + description: Whether to enable PG native password login on all endpoints in this project. history_retention_duration: type: string - description: >- - The effective number of seconds to retain the shared history for - point in time recovery. + description: The effective number of seconds to retain the shared history for point in time recovery. owner: type: string description: The email of the project owner. pg_version: type: integer description: The effective major Postgres version number. + project_id: + type: string + description: Part of the resource name. + replication_role: + $ref: '#/components/schemas/ReplicationRolePreview' + description: The replication role of the project in this workspace. Populated only when cross-workspace replication + is configured. synthetic_storage_size_bytes: type: integer description: The current space occupied by the project in storage. + RecoveryBranchPreview: + type: object + properties: + create_time: + type: string + format: date-time + name: + type: string + description: 'The resource name of the recovery branch. Format: projects/{project_id}/preview/recovery-branches/{recovery_branch_id}' + parent: + type: string + description: 'The project containing this recovery branch. Format: projects/{project_id}' + status: + $ref: '#/components/schemas/RecoveryBranchPreviewStatus' + uid: + type: string + update_time: + type: string + format: date-time + RecoveryBranchPreviewStatus: + type: object + properties: + current_state: + $ref: '#/components/schemas/RecoveryBranchPreviewStatusState' + divergent: + type: boolean + end_lsn: + type: string + description: The Log Sequence Number (LSN) up to which the recovery branch's timeline holds data. + expire_time: + type: string + format: date-time + failover_child_lsn: + type: string + description: The Log Sequence Number (LSN) at which a local child timeline was branched off this recovery branch's + timeline during recovery branch creation. + home_workspace: + type: string + description: 'The workspace that owns the source branch and where reconciliation completes. Format: a workspace + identifier.' + is_foreign: + type: boolean + origin_branch: + type: string + description: 'The normal branch from which this recovery branch originated. Format: projects/{project_id}/branches/{branch_id}' + ReplicationGroupPreview: + type: object + properties: + replication_mode: + $ref: '#/components/schemas/ReplicationModePreview' + workspaces: + type: array + items: + type: string + description: The workspaces participating in this replication group. Phase 1 requires exactly 2 entries. + create_time: + type: string + format: date-time + description: Server-generated timestamps. + etag: + type: string + description: Optional optimistic concurrency token for update and delete. + name: + type: string + description: 'The resource name of the replication group. Format: projects/{project_id}/preview/replication-groups/{replication_group_id}' + observed_metrics: + $ref: '#/components/schemas/ReplicationMetricsPreview' + description: The latest observed replication metrics for this group. + parent: + type: string + description: 'The parent project that owns this replication group. Format: projects/{project_id}' + primary_workspace: + type: string + description: The workspace currently serving writes. Server-owned. + state: + $ref: '#/components/schemas/ReplicationGroupPreviewState' + description: The lifecycle state of the replication group. + update_time: + type: string + format: date-time + required: + - replication_mode + - workspaces + ReplicationGroupPreviewOperationMetadata: + type: object + properties: {} + description: |- + Empty placeholder; required by every LRO ``metadata_type``. Mirrors BranchOperationMetadata / + RoleOperationMetadata. + ReplicationMetricsPreview: + type: object + properties: + as_of_time: + type: string + format: date-time + bytes_lag: + type: integer + description: The most recent observed byte lag. + throughput_bytes_per_second: + type: integer + description: The most recent observed replication throughput. + time_lag: + type: string + description: The most recent observed time lag. RequestedClaims: type: object properties: @@ -2531,8 +4959,6 @@ components: properties: table_name: type: string - unspecified_resource_name: - type: string Role: type: object properties: @@ -2541,24 +4967,20 @@ components: format: date-time name: type: string - description: >- - Output only. The full resource path of the role. Format: - projects/{project_id}/branches/{branch_id}/roles/{role_id} + description: 'Output only. The full resource path of the role. Format: projects/{project_id}/branches/{branch_id}/roles/{role_id}' parent: type: string - description: >- - The Branch where this Role exists. Format: - projects/{project_id}/branches/{branch_id} + description: 'The Branch where this Role exists. Format: projects/{project_id}/branches/{branch_id}' + role_id: + type: string + description: The part of the name, chosen by the user when the resource was created. spec: $ref: '#/components/schemas/RoleRoleSpec' - description: >- - The spec contains the role configuration, including identity type, - authentication method, and role attributes. + description: The spec contains the role configuration, including identity type, authentication method, and role + attributes. status: $ref: '#/components/schemas/RoleRoleStatus' - description: >- - Current status of the role, including its identity type, - authentication method, and role attributes. + description: Current status of the role, including its identity type, authentication method, and role attributes. update_time: type: string format: date-time @@ -2572,183 +4994,837 @@ components: type: boolean createrole: type: boolean - description: >- - Attributes that can be granted to a Postgres role. We are only - implementing a subset for now, + description: |- + Attributes that can be granted to a Postgres role. We are only implementing a subset for now, see xref: https://www.postgresql.org/docs/16/sql-createrole.html The values follow Postgres keyword naming e.g. CREATEDB, BYPASSRLS, etc. which is why they don't include typical underscores between words. RoleOperationMetadata: type: object - properties: {} - RoleRoleSpec: + properties: {} + RoleRoleSpec: + type: object + properties: + attributes: + $ref: '#/components/schemas/RoleAttributes' + auth_method: + $ref: '#/components/schemas/RoleAuthMethod' + description: 'Controls how the Postgres role authenticates when a client opens a database connection. Supported + values: - LAKEBASE_OAUTH_V1: the role authenticates by presenting a Databricks OAuth access token derived from + the backing managed identity (the Databricks user, service principal, or group named by the role''s ``postgres_role``). + No static password exists for roles using this method. - PG_PASSWORD_SCRAM_SHA_256: the role authenticates with + a Postgres password verified server-side using the SCRAM-SHA-256 mechanism. Lakebase generates a password for + the role. - NO_LOGIN: the role cannot open a Postgres session at all. Useful for roles that exist only to own + objects or to aggregate privileges that are then granted to other, loginable roles. If auth_method is left unspecified, + a meaningful authentication method is derived from the identity_type: - For the managed identities, OAUTH is used. + - For the regular postgres roles, authentication based on postgres passwords is used. NOTE: for the Databricks + identity type GROUP, LAKEBASE_OAUTH_V1 is the default auth method (group can login as well).' + identity_type: + $ref: '#/components/schemas/RoleIdentityType' + description: 'The type of role. When specifying a managed-identity, the chosen role_id must be a valid: - application + ID for SERVICE_PRINCIPAL - user email for USER - group name for GROUP' + membership_roles: + type: array + items: + $ref: '#/components/schemas/RoleMembershipRole' + description: An enum value for a standard role that this role is a member of. + postgres_role: + type: string + description: 'The name of the Postgres role. This expects a valid Postgres identifier as specified in the link below. + https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS Required when creating + the Role. If you wish to create a Postgres Role backed by a managed Databricks identity, then postgres_role must + be one of the following: 1. user email for IdentityType.USER 2. app ID for IdentityType.SERVICE_PRINCIPAL 3. group + name for IdentityType.GROUP' + RoleRoleStatus: + type: object + properties: + attributes: + $ref: '#/components/schemas/RoleAttributes' + auth_method: + $ref: '#/components/schemas/RoleAuthMethod' + identity_type: + $ref: '#/components/schemas/RoleIdentityType' + description: The type of the role. + membership_roles: + type: array + items: + $ref: '#/components/schemas/RoleMembershipRole' + description: An enum value for a standard role that this role is a member of. + postgres_role: + type: string + description: The name of the Postgres role. + role_id: + type: string + description: Part of the resource name. + ScheduleCadence: + type: object + properties: + retention: + type: string + description: How long snapshots from this cadence are kept before automatic deletion. Must be at least 1 hour. Applied + when a snapshot is taken; not retroactive, so changing it affects only later snapshots. + daily_schedule: + $ref: '#/components/schemas/DailySchedule' + description: Take a snapshot once per day. + monthly_schedule: + $ref: '#/components/schemas/MonthlySchedule' + description: Take a snapshot once per month. + weekly_schedule: + $ref: '#/components/schemas/WeeklySchedule' + description: Take a snapshot once per week. + required: + - retention + description: One cadence at which automatic snapshots are taken. + Snapshot: + type: object + properties: + create_time: + type: string + format: date-time + description: When the snapshot was created. + name: + type: string + description: 'The resource name of the snapshot. Format: projects/{project_id}/snapshots/{snapshot_id}' + snapshot_id: + type: string + description: The user-chosen ID; the final segment of ``name``. + spec: + $ref: '#/components/schemas/SnapshotSpec' + description: Client-provided configuration of the snapshot. + status: + $ref: '#/components/schemas/SnapshotStatus' + description: Server-observed state of the snapshot. + uid: + type: string + description: Unique system-generated ID for the snapshot. + update_time: + type: string + format: date-time + description: When the snapshot was last updated. + description: |- + An immutable, point-in-time copy of a branch's data within a project. It remains available after + the source branch is deleted. + SnapshotOperationMetadata: + type: object + properties: {} + description: Metadata for the long-running snapshot Create, Update, and Delete operations. + SnapshotSchedule: + type: object + properties: + name: + type: string + description: 'The resource name of the branch''s snapshot schedule. Format: projects/{project_id}/branches/{branch_id}/snapshot-schedule' + schedule: + type: array + items: + $ref: '#/components/schemas/ScheduleCadence' + description: The cadences at which automatic snapshots are taken. Update replaces the whole set; an empty set disables + automatic snapshots. Order is not significant. When several cadences fire together, one snapshot is taken, retained + for the longest of their retentions. + description: |- + The automatic snapshot cadences for a branch. There is exactly one schedule per branch + (singleton); it is configured in place, not created or deleted. + + Name: projects/{project_id}/branches/{branch_id}/snapshot-schedule + SnapshotSpec: + type: object + properties: + source_branch: + type: string + description: 'The source branch to snapshot. Format: projects/{project_id}/branches/{branch_id}' + expire_time: + type: string + format: date-time + description: Absolute time at which the snapshot is deleted. Mutually exclusive with ``ttl`` and ``no_expiry``. + no_expiry: + type: boolean + description: If true, the snapshot never expires. Mutually exclusive with ``ttl`` and ``expire_time``. + source_branch_lsn: + type: string + description: LSN to snapshot from, e.g. ``16/B374D848``. Mutually exclusive with ``source_branch_time``. + source_branch_time: + type: string + format: date-time + description: Timestamp to snapshot from. Mutually exclusive with ``source_branch_lsn``. + ttl: + type: string + description: Time-to-live. The snapshot expires this long after it is created. Mutually exclusive with ``expire_time`` + and ``no_expiry``. + required: + - source_branch + description: Client-provided configuration of the snapshot. + SnapshotStatus: + type: object + properties: + current_state: + $ref: '#/components/schemas/SnapshotStatusState' + description: The snapshot's current state. + diff_size_bytes: + type: integer + description: Incremental storage size in bytes since the previous snapshot. Unset when the snapshot is not billed + on incremental usage. + expire_time: + type: string + format: date-time + description: Absolute time at which the snapshot is deleted. + full_size_bytes: + type: integer + description: Full logical size of the snapshot, in bytes. + no_expiry: + type: boolean + description: True if the snapshot never expires. + source_branch: + type: string + description: 'The source branch the snapshot was taken from. Format: projects/{project_id}/branches/{branch_id}' + source_branch_lsn: + type: string + description: The LSN at which the snapshot was taken. + source_branch_time: + type: string + format: date-time + description: The point in time at which the snapshot was taken. + description: Server-observed state of a snapshot. + SyncedTable: + type: object + properties: + create_time: + type: string + format: date-time + name: + type: string + description: Output only. The Full resource name of the synced table in Postgres where (catalog, schema, table) + are the UC entity names. Format "synced_tables/{catalog}.{schema}.{table}" For the corresponding source table + in the Unity catalog look for the "source_table_full_name" attribute. + spec: + $ref: '#/components/schemas/SyncedTableSyncedTableSpec' + description: Configuration details of the synced table, such as the source table, scheduling policy, etc. This attribute + is specified at creation time and most fields are returned as is on subsequent queries. + status: + $ref: '#/components/schemas/SyncedTableSyncedTableStatus' + description: Synced Table data synchronization status. + synced_table_id: + type: string + description: The part of the name, chosen by the user when the resource was created. + uid: + type: string + description: The Unity Catalog table ID for this synced table. + SyncedTableOperationMetadata: + type: object + properties: {} + description: Metadata for SyncedTable long-running operations. + SyncedTablePipelineProgress: + type: object + properties: + estimated_completion_time_seconds: + type: number + description: The estimated time remaining to complete this update in seconds. + latest_version_currently_processing: + type: integer + description: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely + processed this version yet. + sync_progress_completion: + type: number + description: The completion ratio of this update. This is a number between 0 and 1. + synced_row_count: + type: integer + description: The number of rows that have been synced in this update. + total_row_count: + type: integer + description: The total number of rows that need to be synced in this update. This number may be an estimate. + description: Progress information of the Synced Table data synchronization pipeline. + SyncedTablePosition: + type: object + properties: + delta_table_sync_info: + $ref: '#/components/schemas/DeltaTableSyncInfo' + sync_end_time: + type: string + format: date-time + description: The end timestamp of the most recent successful synchronization. This is the time when the data is + available in the synced table. + sync_start_time: + type: string + format: date-time + description: The starting timestamp of the most recent successful synchronization from the source table to the destination + (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, + this is the time when the sync operation started. + SyncedTableSyncedTableSpec: + type: object + properties: + accelerated_sync: + type: boolean + branch: + type: string + description: 'The full resource name the branch associated with the table. Format: "projects/{project_id}/branches/{branch_id}".' + create_database_objects_if_missing: + type: boolean + description: If true, the synced table's logical database and schema resources in PG will be created if they do + not already exist. The request will fail if this is false and the database/schema do not exist. Defaults to true + if omitted. + existing_pipeline_id: + type: string + description: ID of an existing pipeline to bin-pack this synced table into. At most one of existing_pipeline_id + and new_pipeline_spec should be defined. The pipeline used for the synced table is returned via the top level + pipeline_id attribute. + extra_columns: + type: array + items: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecExtraColumn' + description: Extra PostgreSQL-only columns to add to the synced table. + extra_index_definitions: + type: array + items: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecSecondaryIndex' + description: Secondary indexes to create on the synced table. + new_pipeline_spec: + $ref: '#/components/schemas/NewPipelineSpec' + description: Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec + should be defined. The pipeline used for the synced table is returned via the top level pipeline_id attribute. + postgres_database: + type: string + description: The Postgres database name where the synced table will be created in. If this synced table is created + inside a Lakebase Catalog, this attribute can be omitted on creation and is inferred from the postgres_database + associated with the Lakebase Catalog. If specified when inside a Lakebase Catalog, the value must match. A value + must be specified when creating a synced table inside a Standard Catalog. + primary_key_columns: + type: array + items: + type: string + description: Primary Key columns to be used for data insert/update in the destination. + scheduling_policy: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy' + description: Scheduling policy of the underlying pipeline. + source_table_full_name: + type: string + description: 'Three-part (catalog, schema, table) name of the source Delta table. For the corresponding destination + table, use any of the two: - synced_table_id used at the creation of the SyncedTable - "name" consisting of "synced_tables/" + prefix and the full name of the destination table.' + timeseries_key: + type: string + description: Time series key to deduplicate (tie-break) rows with the same primary key. + type_overrides: + type: array + items: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecTypeOverride' + description: Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED + is rejected; a valid pg_type must be set. + SyncedTableSyncedTableSpecExtraColumn: + type: object + properties: + column_name: + type: string + description: Name of the column. + column_type: + type: string + description: PostgreSQL type of the column, for example "tsvector" or "vector(1024)". + compute: + type: string + description: SQL expression used to compute the column's value, for example "to_tsvector('english', content)". + maintenance: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecExtraColumnMaintenance' + required: + - column_name + - column_type + description: An extra PostgreSQL column to add to the synced table. + SyncedTableSyncedTableSpecSecondaryIndex: + type: object + properties: + name: + type: string + description: Name of the index as it will appear in PostgreSQL. + definition: + type: string + description: 'The definition portion of a CREATE INDEX statement, placed after ON table_name. For example: USING + hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64).' + creation_point: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecSecondaryIndexCreationPoint' + description: When the index should be created relative to the initial data load. + required: + - name + - definition + description: Definition of a secondary index to create on the synced table. + SyncedTableSyncedTableSpecTypeOverride: type: object properties: - attributes: - $ref: '#/components/schemas/RoleAttributes' - auth_method: - $ref: '#/components/schemas/RoleAuthMethod' - description: >- - If auth_method is left unspecified, a meaningful authentication - method is derived from the identity_type: * For the managed - identities, OAUTH is used. * For the regular postgres roles, - authentication based on postgres passwords is used. NOTE: this is - ignored for the Databricks identity type GROUP, and NO_LOGIN is - implicitly assumed instead for the GROUP identity type. - identity_type: - $ref: '#/components/schemas/RoleIdentityType' - description: >- - The type of role. When specifying a managed-identity, the chosen - role_id must be a valid: * application ID for SERVICE_PRINCIPAL * - user email for USER * group name for GROUP - membership_roles: - type: array - items: - $ref: '#/components/schemas/RoleMembershipRole' - description: An enum value for a standard role that this role is a member of. - postgres_role: + column_name: type: string - description: >- - The name of the Postgres role. This expects a valid Postgres - identifier as specified in the link below. - https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS - Required when creating the Role. If you wish to create a Postgres - Role backed by a managed Databricks identity, then postgres_role - must be one of the following: 1. user email for IdentityType.USER 2. - app ID for IdentityType.SERVICE_PRINCIPAL 2. group name for - IdentityType.GROUP - RoleRoleStatus: + description: Name of the source column whose target PostgreSQL type should be overridden. + pg_type: + $ref: '#/components/schemas/SyncedTableSyncedTableSpecPgSpecificType' + description: PostgreSQL-specific target type to use for the column. + size: + type: integer + description: Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). + Required when the chosen pg_type needs a size. + required: + - column_name + - pg_type + description: Overrides the default Delta-to-PostgreSQL type mapping for a single column. + SyncedTableSyncedTableStatus: type: object properties: - attributes: - $ref: '#/components/schemas/RoleAttributes' - auth_method: - $ref: '#/components/schemas/RoleAuthMethod' - identity_type: - $ref: '#/components/schemas/RoleIdentityType' - description: The type of the role. - membership_roles: - type: array - items: - $ref: '#/components/schemas/RoleMembershipRole' - description: An enum value for a standard role that this role is a member of. - postgres_role: + detailed_state: + $ref: '#/components/schemas/SyncedTableState' + last_processed_commit_version: + type: integer + description: The last source table Delta version that was successfully synced to the synced table. + last_sync: + $ref: '#/components/schemas/SyncedTablePosition' + description: Summary of the last successful synchronization from source to destination. + last_sync_time: type: string - description: The name of the Postgres role. + format: date-time + description: The end timestamp of the last time any data was synchronized from the source table to the synced table. + This is when the data is available in the synced table. + message: + type: string + description: A text description of the current state of the synced table. + ongoing_sync_progress: + $ref: '#/components/schemas/SyncedTablePipelineProgress' + pipeline_id: + type: string + description: ID of the associated pipeline. + project: + type: string + description: 'The full resource name of the project associated with the table. Format: "projects/{project_id}".' + provisioning_phase: + $ref: '#/components/schemas/ProvisioningPhase' + description: The current phase of the data synchronization pipeline. + unity_catalog_provisioning_state: + $ref: '#/components/schemas/ProvisioningInfoState' + description: The provisioning state of the synced table entity in Unity Catalog. + Table: + type: object + properties: + name: + type: string + description: Full three-part (catalog, schema, table) name of the table. + database: + type: string + description: 'The project and branch scoped database to which this table belongs. Of the format: projects/{project_id}/branches/{branch_id}/databases/{database_id} + where database_id is the name of the logical database in Postgres.' + branch: + type: string + description: The id of the database branch associated with the table. Of the format projects/{project_id}/branches/{branch_id}. + project: + type: string + description: The id of the database project associated with the table. Of the format projects/{project_id}. + table_serving_url: + type: string + description: REST API URL for serving data from this table. + required: + - name + - database + description: |- + Table represents a non-synced database table in a Lakebase project. Unlike SyncedTable, this + does not have a data synchronization pipeline. + WeeklySchedule: + type: object + properties: + day_of_week: + $ref: '#/components/schemas/DayOfWeek' + description: The day of the week on which to take the snapshot. + hour: + type: integer + description: The hour of the day, in UTC, at which to take the snapshot, in [0, 23]. + required: + - day_of_week + description: Take a snapshot once per week, on the configured day at the configured hour. BranchStatusState: type: string x-enum: - - ARCHIVED - - IMPORTING - - INIT - - READY - - RESETTING + - ARCHIVED + - DELETED + - IMPORTING + - INIT + - READY + - RESETTING description: The state of the branch. + CdfState: + type: string + x-enum: + - CDF_STATE_SKIPPED + - CDF_STATE_SNAPSHOTTING + - CDF_STATE_STREAMING + - CDF_STATE_TERMINATED + description: The replication state of a single replicated table (CdfStatus). + ComputeInstanceComputeState: + type: string + x-enum: + - ACTIVE + - IDLE + - INIT + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + ComputeInstanceComputeType: + type: string + x-enum: + - HOT_STANDBY + - READ_ONLY + - READ_WRITE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + DayOfWeek: + type: string + x-enum: + - FRIDAY + - MONDAY + - SATURDAY + - SUNDAY + - THURSDAY + - TUESDAY + - WEDNESDAY + description: The day of the week on which a weekly snapshot is taken. EndpointStatusState: type: string x-enum: - - ACTIVE - - DEGRADED - - IDLE - - INIT + - ACTIVE + - DEGRADED + - IDLE + - INIT description: The state of the compute endpoint. EndpointType: type: string x-enum: - - ENDPOINT_TYPE_READ_ONLY - - ENDPOINT_TYPE_READ_WRITE - description: The compute endpoint type. Either `read_write` or `read_only`. - ErrorCode: + - ENDPOINT_TYPE_READ_ONLY + - ENDPOINT_TYPE_READ_WRITE + description: The compute endpoint type. Either ``read_write`` or ``read_only``. + ErrorCode: + type: string + x-enum: + - ABORTED + - ALREADY_EXISTS + - BAD_REQUEST + - CANCELLED + - CATALOG_ALREADY_EXISTS + - CATALOG_DOES_NOT_EXIST + - CATALOG_NOT_EMPTY + - COULD_NOT_ACQUIRE_LOCK + - CUSTOMER_UNAUTHORIZED + - DAC_ALREADY_EXISTS + - DAC_DOES_NOT_EXIST + - DATA_LOSS + - DEADLINE_EXCEEDED + - DEPLOYMENT_TIMEOUT + - DIRECTORY_NOT_EMPTY + - DIRECTORY_PROTECTED + - DRY_RUN_FAILED + - ENDPOINT_NOT_FOUND + - EXTERNAL_LOCATION_ALREADY_EXISTS + - EXTERNAL_LOCATION_DOES_NOT_EXIST + - FEATURE_DISABLED + - GIT_CONFLICT + - GIT_REMOTE_ERROR + - GIT_SENSITIVE_TOKEN_DETECTED + - GIT_UNKNOWN_REF + - GIT_URL_NOT_ON_ALLOW_LIST + - INSECURE_PARTNER_RESPONSE + - INTERNAL_ERROR + - INVALID_PARAMETER_VALUE + - INVALID_STATE + - INVALID_STATE_TRANSITION + - IO_ERROR + - IPYNB_FILE_IN_REPO + - MALFORMED_PARTNER_RESPONSE + - MALFORMED_REQUEST + - MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST + - MAX_BLOCK_SIZE_EXCEEDED + - MAX_CHILD_NODE_SIZE_EXCEEDED + - MAX_LIST_SIZE_EXCEEDED + - MAX_NOTEBOOK_SIZE_EXCEEDED + - MAX_READ_SIZE_EXCEEDED + - METASTORE_ALREADY_EXISTS + - METASTORE_DOES_NOT_EXIST + - METASTORE_NOT_EMPTY + - NOT_FOUND + - NOT_IMPLEMENTED + - PARTIAL_DELETE + - PERMISSION_DENIED + - PERMISSION_NOT_PROPAGATED + - PRINCIPAL_DOES_NOT_EXIST + - PROJECTS_OPERATION_TIMEOUT + - PROVIDER_ALREADY_EXISTS + - PROVIDER_DOES_NOT_EXIST + - PROVIDER_SHARE_NOT_ACCESSIBLE + - QUOTA_EXCEEDED + - RECIPIENT_ALREADY_EXISTS + - RECIPIENT_DOES_NOT_EXIST + - REQUEST_LIMIT_EXCEEDED + - RESOURCE_ALREADY_EXISTS + - RESOURCE_CONFLICT + - RESOURCE_DOES_NOT_EXIST + - RESOURCE_EXHAUSTED + - RESOURCE_LIMIT_EXCEEDED + - SCHEMA_ALREADY_EXISTS + - SCHEMA_DOES_NOT_EXIST + - SCHEMA_NOT_EMPTY + - SEARCH_QUERY_TOO_LONG + - SEARCH_QUERY_TOO_SHORT + - SERVICE_UNDER_MAINTENANCE + - SHARE_ALREADY_EXISTS + - SHARE_DOES_NOT_EXIST + - STORAGE_CREDENTIAL_ALREADY_EXISTS + - STORAGE_CREDENTIAL_DOES_NOT_EXIST + - TABLE_ALREADY_EXISTS + - TABLE_DOES_NOT_EXIST + - TEMPORARILY_UNAVAILABLE + - UNAUTHENTICATED + - UNAVAILABLE + - UNKNOWN + - UNPARSEABLE_HTTP_ERROR + - WORKSPACE_TEMPORARILY_UNAVAILABLE + description: Error codes returned by Databricks APIs to indicate specific failure conditions. + NewPipelineSpecPipelineChannel: + type: string + x-enum: + - CURRENT + - PREVIEW + description: |- + Release channel of the underlying pipeline's runtime. PREVIEW provides early access to the + latest features but may be less stable. Some source table configurations (e.g., read-time CDF) + require PREVIEW. Defaults to CURRENT if not specified. + OpenApiMode: + type: string + x-enum: + - OPEN_API_MODE_DISABLED + - OPEN_API_MODE_IGNORE_PRIVILEGES + description: |- + Controls how the Data API exposes the OpenAPI documentation endpoint. Only IGNORE_PRIVILEGES and + DISABLED are supported today; "follow-privileges" is not implemented yet (it may be added later + as value 3 — adding new enum values is backward-compatible). + ProvisioningInfoState: + type: string + x-enum: + - ACTIVE + - DEGRADED + - DELETING + - FAILED + - PROVISIONING + - UPDATING + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + ProvisioningPhase: + type: string + x-enum: + - PROVISIONING_PHASE_INDEX_SCAN + - PROVISIONING_PHASE_INDEX_SORT + - PROVISIONING_PHASE_MAIN + description: The current phase of the data synchronization pipeline. + RecoveryBranchPreviewStatusState: + type: string + x-enum: + - PENDING_HOME_SYNC + - READY_FOR_INSPECTION + - RECONCILED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + ReplicationGroupPreviewState: + type: string + x-enum: + - REPLICATION_GROUP_PREVIEW_STATE_DEGRADED + - REPLICATION_GROUP_PREVIEW_STATE_DELETING + - REPLICATION_GROUP_PREVIEW_STATE_FAILING_OVER + - REPLICATION_GROUP_PREVIEW_STATE_PROVISIONING + - REPLICATION_GROUP_PREVIEW_STATE_READY + - REPLICATION_GROUP_PREVIEW_STATE_SWITCHING_OVER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + ReplicationModePreview: + type: string + x-enum: + - REPLICATION_MODE_PREVIEW_LIVE + - REPLICATION_MODE_PREVIEW_PERIODIC + description: |- + How changes are propagated from the primary workspace to its secondaries in a replication group: + on a fixed schedule or continuously as they occur. + ReplicationRolePreview: type: string x-enum: - - ABORTED - - ALREADY_EXISTS - - BAD_REQUEST - - CANCELLED - - CATALOG_ALREADY_EXISTS - - CATALOG_DOES_NOT_EXIST - - CATALOG_NOT_EMPTY - - COULD_NOT_ACQUIRE_LOCK - - CUSTOMER_UNAUTHORIZED - - DAC_ALREADY_EXISTS - - DAC_DOES_NOT_EXIST - - DATA_LOSS - - DEADLINE_EXCEEDED - - DEPLOYMENT_TIMEOUT - - DIRECTORY_NOT_EMPTY - - DIRECTORY_PROTECTED - - DRY_RUN_FAILED - - ENDPOINT_NOT_FOUND - - EXTERNAL_LOCATION_ALREADY_EXISTS - - EXTERNAL_LOCATION_DOES_NOT_EXIST - - FEATURE_DISABLED - - GIT_CONFLICT - - GIT_REMOTE_ERROR - - GIT_SENSITIVE_TOKEN_DETECTED - - GIT_UNKNOWN_REF - - GIT_URL_NOT_ON_ALLOW_LIST - - INSECURE_PARTNER_RESPONSE - - INTERNAL_ERROR - - INVALID_PARAMETER_VALUE - - INVALID_STATE - - INVALID_STATE_TRANSITION - - IO_ERROR - - IPYNB_FILE_IN_REPO - - MALFORMED_PARTNER_RESPONSE - - MALFORMED_REQUEST - - MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST - - MAX_BLOCK_SIZE_EXCEEDED - - MAX_CHILD_NODE_SIZE_EXCEEDED - - MAX_LIST_SIZE_EXCEEDED - - MAX_NOTEBOOK_SIZE_EXCEEDED - - MAX_READ_SIZE_EXCEEDED - - METASTORE_ALREADY_EXISTS - - METASTORE_DOES_NOT_EXIST - - METASTORE_NOT_EMPTY - - NOT_FOUND - - NOT_IMPLEMENTED - - PARTIAL_DELETE - - PERMISSION_DENIED - - PERMISSION_NOT_PROPAGATED - - PRINCIPAL_DOES_NOT_EXIST - - PROJECTS_OPERATION_TIMEOUT - - PROVIDER_ALREADY_EXISTS - - PROVIDER_DOES_NOT_EXIST - - PROVIDER_SHARE_NOT_ACCESSIBLE - - QUOTA_EXCEEDED - - RECIPIENT_ALREADY_EXISTS - - RECIPIENT_DOES_NOT_EXIST - - REQUEST_LIMIT_EXCEEDED - - RESOURCE_ALREADY_EXISTS - - RESOURCE_CONFLICT - - RESOURCE_DOES_NOT_EXIST - - RESOURCE_EXHAUSTED - - RESOURCE_LIMIT_EXCEEDED - - SCHEMA_ALREADY_EXISTS - - SCHEMA_DOES_NOT_EXIST - - SCHEMA_NOT_EMPTY - - SEARCH_QUERY_TOO_LONG - - SEARCH_QUERY_TOO_SHORT - - SERVICE_UNDER_MAINTENANCE - - SHARE_ALREADY_EXISTS - - SHARE_DOES_NOT_EXIST - - STORAGE_CREDENTIAL_ALREADY_EXISTS - - STORAGE_CREDENTIAL_DOES_NOT_EXIST - - TABLE_ALREADY_EXISTS - - TABLE_DOES_NOT_EXIST - - TEMPORARILY_UNAVAILABLE - - UNAUTHENTICATED - - UNAVAILABLE - - UNKNOWN - - UNPARSEABLE_HTTP_ERROR - - WORKSPACE_TEMPORARILY_UNAVAILABLE - description: >- - Error codes returned by Databricks APIs to indicate specific failure - conditions. + - REPLICATION_ROLE_PREVIEW_DEMOTING + - REPLICATION_ROLE_PREVIEW_PRIMARY + - REPLICATION_ROLE_PREVIEW_SECONDARY + description: |- + The replication role of the project in its current workspace. Populated only when + cross-workspace replication is configured for the project. RequestedClaimsPermissionSet: type: string x-enum: - - READ_ONLY + - READ_ONLY description: |- Create a collection of name/value pairs. @@ -2789,121 +5865,235 @@ components: RoleAuthMethod: type: string x-enum: - - LAKEBASE_OAUTH_V1 - - NO_LOGIN - - PG_PASSWORD_SCRAM_SHA_256 + - LAKEBASE_OAUTH_V1 + - NO_LOGIN + - PG_PASSWORD_SCRAM_SHA_256 description: How the role is authenticated when connecting to Postgres. RoleIdentityType: type: string x-enum: - - GROUP - - SERVICE_PRINCIPAL - - USER - description: >- - The type of the Databricks managed identity that this Role represents. - Leave empty if you wish - - to create a regular Postgres role not associated with a Databricks - identity. + - GROUP + - SERVICE_PRINCIPAL + - USER + description: |- + The type of the Databricks managed identity that this Role represents. Leave empty if you wish + to create a regular Postgres role not associated with a Databricks identity. RoleMembershipRole: type: string x-enum: - - DATABRICKS_SUPERUSER + - DATABRICKS_SUPERUSER description: Roles that the DatabaseInstanceRole can be a member of. + SnapshotStatusState: + type: string + x-enum: + - AVAILABLE + - CREATING + - DELETING + - FAILED + description: The state of the snapshot. + SyncedTableState: + type: string + x-enum: + - SYNCED_TABLE_OFFLINE + - SYNCED_TABLE_OFFLINE_FAILED + - SYNCED_TABLE_ONLINE + - SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE + - SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE + - SYNCED_TABLE_ONLINE_PIPELINE_FAILED + - SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE + - SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES + - SYNCED_TABLE_PROVISIONING + - SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT + - SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES + description: The state of a synced table. + SyncedTableSyncedTableSpecExtraColumnMaintenance: + type: string + x-enum: + - DEFAULT_VALUE + - STORED_GENERATED + description: How the column's value is populated and kept up to date. + SyncedTableSyncedTableSpecPgSpecificType: + type: string + x-enum: + - PG_SPECIFIC_TYPE_HALFVEC + - PG_SPECIFIC_TYPE_VARCHAR + - PG_SPECIFIC_TYPE_VECTOR + description: PostgreSQL-specific target types that can override the default Delta-to-PG mapping. + SyncedTableSyncedTableSpecSecondaryIndexCreationPoint: + type: string + x-enum: + - CREATION_POINT_AFTER_DATA_LOAD + description: Controls when the index is created relative to the initial data load. + SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy: + type: string + x-enum: + - CONTINUOUS + - SNAPSHOT + - TRIGGERED + description: Scheduling policy of the synced table's underlying pipeline. x-stackQL-resources: - branches: - id: databricks_workspace.postgres.branches - name: branches - title: Branches + postgres_branches: + id: databricks_workspace.postgres.postgres_branches + name: postgres_branches + title: Postgres Branches methods: create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches/post' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1branches/post' response: mediaType: application/json openAPIDocKey: '200' list: operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches/get' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1branches/get' response: mediaType: application/json openAPIDocKey: '200' objectKey: $.branches sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/branches/methods/list' + - $ref: '#/components/x-stackQL-resources/postgres_branches/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/postgres_branches/methods/create' + update: [] + delete: [] + replace: [] + postgres_catalogs: + id: databricks_workspace.postgres.postgres_catalogs + name: postgres_catalogs + title: Postgres Catalogs + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1catalogs/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: + - $ref: '#/components/x-stackQL-resources/postgres_catalogs/methods/create' + update: [] + delete: [] + replace: [] + postgres_cdf_configs: + id: databricks_workspace.postgres.postgres_cdf_configs + name: postgres_cdf_configs + title: Postgres Cdf Configs + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1cdf-configs/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1cdf-configs/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.cdf_configs + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_cdf_configs/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/postgres_cdf_configs/methods/create' + update: [] + delete: [] + replace: [] + postgres_data_apis: + id: databricks_workspace.postgres.postgres_data_apis + name: postgres_data_apis + title: Postgres Data Apis + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1data-api/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] insert: - - $ref: '#/components/x-stackQL-resources/branches/methods/create' + - $ref: '#/components/x-stackQL-resources/postgres_data_apis/methods/create' update: [] delete: [] replace: [] - databases: - id: databricks_workspace.postgres.databases - name: databases - title: Databases + postgres_databases: + id: databricks_workspace.postgres.postgres_databases + name: postgres_databases + title: Postgres Databases methods: - create_database: + create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1databases/post' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1databases/post' response: mediaType: application/json openAPIDocKey: '200' - list_databases: + list: operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1databases/get' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1databases/get' response: mediaType: application/json openAPIDocKey: '200' objectKey: $.databases sqlVerbs: select: - - $ref: >- - #/components/x-stackQL-resources/databases/methods/list_databases + - $ref: '#/components/x-stackQL-resources/postgres_databases/methods/list' insert: - - $ref: >- - #/components/x-stackQL-resources/databases/methods/create_database + - $ref: '#/components/x-stackQL-resources/postgres_databases/methods/create' update: [] delete: [] replace: [] - endpoints: - id: databricks_workspace.postgres.endpoints - name: endpoints - title: Endpoints + postgres_endpoints: + id: databricks_workspace.postgres.postgres_endpoints + name: postgres_endpoints + title: Postgres Endpoints methods: create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1endpoints/post' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1endpoints/post' response: mediaType: application/json openAPIDocKey: '200' list: operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1endpoints/get' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1endpoints/get' response: mediaType: application/json openAPIDocKey: '200' objectKey: $.endpoints sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/endpoints/methods/list' + - $ref: '#/components/x-stackQL-resources/postgres_endpoints/methods/list' insert: - - $ref: '#/components/x-stackQL-resources/endpoints/methods/create' + - $ref: '#/components/x-stackQL-resources/postgres_endpoints/methods/create' update: [] delete: [] replace: [] - projects: - id: databricks_workspace.postgres.projects - name: projects - title: Projects + postgres_projects: + id: databricks_workspace.postgres.postgres_projects + name: postgres_projects + title: Postgres Projects methods: create: config: @@ -2926,66 +6116,247 @@ components: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1roles~1{role_id}/patch' + $ref: '#/paths/~1api~12.0~1postgres~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + undelete: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{name}~1undelete/post' response: mediaType: application/json openAPIDocKey: '200' sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/projects/methods/list' + - $ref: '#/components/x-stackQL-resources/postgres_projects/methods/list' insert: - - $ref: '#/components/x-stackQL-resources/projects/methods/create' + - $ref: '#/components/x-stackQL-resources/postgres_projects/methods/create' update: - - $ref: '#/components/x-stackQL-resources/projects/methods/update' + - $ref: '#/components/x-stackQL-resources/postgres_projects/methods/update' + delete: [] + replace: [] + postgres_replication_group_previews: + id: databricks_workspace.postgres.postgres_replication_group_previews + name: postgres_replication_group_previews + title: Postgres Replication Group Previews + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1preview~1replication-groups/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1preview~1replication-groups/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.replication_group_previews + failover: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{name}~1failover/post' + response: + mediaType: application/json + openAPIDocKey: '200' + switchover: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{name}~1switchover/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_replication_group_previews/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/postgres_replication_group_previews/methods/create' + update: [] delete: [] replace: [] - roles: - id: databricks_workspace.postgres.roles - name: roles - title: Roles + postgres_roles: + id: databricks_workspace.postgres.postgres_roles + name: postgres_roles + title: Postgres Roles methods: create: config: requestBodyTranslate: algorithm: naive operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1roles/post' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1roles/post' response: mediaType: application/json openAPIDocKey: '200' list: operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1roles/get' + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1roles/get' response: mediaType: application/json openAPIDocKey: '200' objectKey: $.roles delete: operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1roles~1{role_id}/delete' + $ref: '#/paths/~1api~12.0~1postgres~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_roles/methods/list' + - $ref: '#/components/x-stackQL-resources/postgres_roles/methods/get' + insert: + - $ref: '#/components/x-stackQL-resources/postgres_roles/methods/create' + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/postgres_roles/methods/delete' + replace: [] + postgres_snapshots: + id: databricks_workspace.postgres.postgres_snapshots + name: postgres_snapshots + title: Postgres Snapshots + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1snapshots/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1snapshots/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.snapshots + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_snapshots/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/postgres_snapshots/methods/create' + update: [] + delete: [] + replace: [] + postgres_synced_tables: + id: databricks_workspace.postgres.postgres_synced_tables + name: postgres_synced_tables + title: Postgres Synced Tables + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1synced_tables/post' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: + - $ref: '#/components/x-stackQL-resources/postgres_synced_tables/methods/create' + update: [] + delete: [] + replace: [] + postgres_tables: + id: databricks_workspace.postgres.postgres_tables + name: postgres_tables + title: Postgres Tables + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1tables/post' + response: + mediaType: application/json + openAPIDocKey: '200' + delete: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1tables~1{name}/delete' response: mediaType: application/json openAPIDocKey: '200' get: operation: - $ref: '#/paths/~1api~12.0~1postgres~1projects~1{project_id}~1branches~1{branch_id}~1roles~1{role_id}/get' + $ref: '#/paths/~1api~12.0~1postgres~1tables~1{name}/get' response: mediaType: application/json openAPIDocKey: '200' sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/roles/methods/list' - - $ref: '#/components/x-stackQL-resources/roles/methods/get' + - $ref: '#/components/x-stackQL-resources/postgres_tables/methods/get' insert: - - $ref: '#/components/x-stackQL-resources/roles/methods/create' + - $ref: '#/components/x-stackQL-resources/postgres_tables/methods/create' + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/postgres_tables/methods/delete' + replace: [] + postgres_forward_etl_configurations: + id: databricks_workspace.postgres.postgres_forward_etl_configurations + name: postgres_forward_etl_configurations + title: Postgres Forward Etl Configurations + methods: + delete: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1forward-etl~1configuration/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: [] + update: [] + delete: + - $ref: '#/components/x-stackQL-resources/postgres_forward_etl_configurations/methods/delete' + replace: [] + postgres_forward_etl: + id: databricks_workspace.postgres.postgres_forward_etl + name: postgres_forward_etl + title: Postgres Forward Etl + methods: + disable: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1forward-etl/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1forward-etl/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_forward_etl/methods/get' + insert: [] update: [] delete: - - $ref: '#/components/x-stackQL-resources/roles/methods/delete' + - $ref: '#/components/x-stackQL-resources/postgres_forward_etl/methods/disable' replace: [] - credentials: - id: databricks_workspace.postgres.credentials - name: credentials - title: Credentials + postgres_credentials: + id: databricks_workspace.postgres.postgres_credentials + name: postgres_credentials + title: Postgres Credentials methods: generate: config: @@ -2996,9 +6367,92 @@ components: response: mediaType: application/json openAPIDocKey: '200' + sqlVerbs: + select: [] + insert: [] + update: [] + delete: [] + replace: [] + postgres_forward_etl_metadata: + id: databricks_workspace.postgres.postgres_forward_etl_metadata + name: postgres_forward_etl_metadata + title: Postgres Forward Etl Metadata + methods: + get: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1forward-etl~1metadata/get' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_forward_etl_metadata/methods/get' + insert: [] + update: [] + delete: [] + replace: [] + postgres_recovery_branch_previews: + id: databricks_workspace.postgres.postgres_recovery_branch_previews + name: postgres_recovery_branch_previews + title: Postgres Recovery Branch Previews + methods: + inspect: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{name}~1inspect/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1preview~1recovery-branches/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.recovery_branch_previews + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_recovery_branch_previews/methods/list' + insert: [] + update: [] + delete: [] + replace: [] + postgres_cdf_statuses: + id: databricks_workspace.postgres.postgres_cdf_statuses + name: postgres_cdf_statuses + title: Postgres Cdf Statuses + methods: + list: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1cdf-statuses/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.cdf_statuses + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/postgres_cdf_statuses/methods/list' + insert: [] + update: [] + delete: [] + replace: [] + postgres_compute_instances: + id: databricks_workspace.postgres.postgres_compute_instances + name: postgres_compute_instances + title: Postgres Compute Instances + methods: + list: + operation: + $ref: '#/paths/~1api~12.0~1postgres~1{parent}~1compute-instances/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.compute_instances sqlVerbs: select: - - $ref: '#/components/x-stackQL-resources/credentials/methods/generate' + - $ref: '#/components/x-stackQL-resources/postgres_compute_instances/methods/list' insert: [] update: [] delete: [] @@ -3078,7 +6532,7 @@ components: JSON_EXTRACT(status, '$.pg_version') AS pg_version, uid, update_time - FROM databricks_workspace.postgres.projects + FROM databricks_workspace.postgres.postgres_projects WHERE deployment_name = '{{ deployment_name }}' fallback: predicate: sqlDialect == "postgres" @@ -3099,7 +6553,7 @@ components: (status::jsonb)->>'pg_version' AS pg_version, uid, update_time - FROM databricks_workspace.postgres.projects + FROM databricks_workspace.postgres.postgres_projects WHERE deployment_name = '{{ deployment_name }}' vw_branches: name: vw_branches @@ -3167,7 +6621,7 @@ components: JSON_EXTRACT(status, '$.state_change_time') AS state_change_time, uid, update_time - FROM databricks_workspace.postgres.branches + FROM databricks_workspace.postgres.postgres_branches WHERE project_id = '{{ project_id }}' AND deployment_name = '{{ deployment_name }}' fallback: @@ -3186,7 +6640,7 @@ components: (status::jsonb)->>'state_change_time' AS state_change_time, uid, update_time - FROM databricks_workspace.postgres.branches + FROM databricks_workspace.postgres.postgres_branches WHERE project_id = '{{ project_id }}' AND deployment_name = '{{ deployment_name }}' vw_databases: @@ -3234,7 +6688,7 @@ components: spec, JSON_EXTRACT(status, '$.postgres_database') AS postgres_database, SPLIT_PART(JSON_EXTRACT(status, '$.role'), '/', -1) AS role - FROM databricks_workspace.postgres.databases + FROM databricks_workspace.postgres.postgres_databases WHERE deployment_name = '{{ deployment_name }}' AND project_id = '{{ project_id }}' AND branch_id = '{{ branch_id }}' @@ -3248,7 +6702,7 @@ components: spec, (status::jsonb)->>'postgres_database' AS postgres_database, SPLIT_PART((status::jsonb)->>'role', '/', -1) AS role - FROM databricks_workspace.postgres.databases + FROM databricks_workspace.postgres.postgres_databases WHERE deployment_name = '{{ deployment_name }}' AND project_id = '{{ project_id }}' AND branch_id = '{{ branch_id }}' @@ -3325,7 +6779,7 @@ components: JSON_EXTRACT(status, '$.membership_roles') AS membership_roles, JSON_EXTRACT(status, '$.postgres_role') AS postgres_role, update_time - FROM databricks_workspace.postgres.roles + FROM databricks_workspace.postgres.postgres_roles WHERE project_id = '{{ project_id }}' AND branch_id = '{{ branch_id }}' AND deployment_name = '{{ deployment_name }}' @@ -3346,7 +6800,7 @@ components: (status::jsonb)->>'membership_roles' AS membership_roles, (status::jsonb)->>'postgres_role' AS postgres_role, update_time - FROM databricks_workspace.postgres.roles + FROM databricks_workspace.postgres.postgres_roles WHERE project_id = '{{ project_id }}' AND branch_id = '{{ branch_id }}' AND deployment_name = '{{ deployment_name }}' @@ -3447,7 +6901,7 @@ components: JSON_EXTRACT(status, '$.hosts.read_write_pooled_host') AS read_write_pooled_host, uid, update_time - FROM databricks_workspace.postgres.endpoints + FROM databricks_workspace.postgres.postgres_endpoints WHERE project_id = '{{ project_id }}' AND branch_id = '{{ branch_id }}' AND deployment_name = '{{ deployment_name }}' @@ -3474,7 +6928,7 @@ components: (status::jsonb)#>>'{hosts,read_write_pooled_host}' AS read_write_pooled_host, uid, update_time - FROM databricks_workspace.postgres.endpoints + FROM databricks_workspace.postgres.postgres_endpoints WHERE project_id = '{{ project_id }}' AND branch_id = '{{ branch_id }}' AND deployment_name = '{{ deployment_name }}' diff --git a/providers/src/databricks_workspace/v00.00.00000/services/qualitymonitorv2.yaml b/providers/src/databricks_workspace/v00.00.00000/services/qualitymonitorv2.yaml index 5e5b8099..aefcbd38 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/qualitymonitorv2.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/qualitymonitorv2.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks qualitymonitorv2 service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.qualitymonitorv2 servers: - url: https://{deployment_name}.cloud.databricks.com @@ -52,10 +52,188 @@ paths: anomaly_detection_config: type: object properties: + custom_check_configurations: + type: array + items: + type: object + properties: + scalar_check: + type: object + properties: + check_name: + type: string + column_matchers: + type: array + items: + type: object + properties: + column_names: + type: array + items: + type: string + variable_name: + type: string + description: >- + Variable name within a custom sql query + that this matcher applies to. + description: >- + Column matchers to determine which tables + to apply this check to + sql_query: + type: string + description: Templated SQL query for this check + thresholds: + type: object + properties: + lower_bound: + type: object + properties: + bound_value: + type: integer + threshold_type: + type: string + x-enum: + - THRESHOLD_TYPE_AUTO + - THRESHOLD_TYPE_MANUAL + - THRESHOLD_TYPE_UNBOUNDED + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + upper_bound: + type: object + properties: + bound_value: + type: integer + threshold_type: + type: string + x-enum: + - THRESHOLD_TYPE_AUTO + - THRESHOLD_TYPE_MANUAL + - THRESHOLD_TYPE_UNBOUNDED + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + description: Upper bound threshold + description: >- + Upper/lower thresholds for the output of + the query excluded_table_full_names: type: array items: type: string + description: >- + List of fully qualified table names to exclude from + anomaly detection. + job_type: + type: string + x-enum: + - >- + ANOMALY_DETECTION_JOB_TYPE_BACKGROUND_COMPUTE_WORKER + - ANOMALY_DETECTION_JOB_TYPE_INTERNAL_HIDDEN + - ANOMALY_DETECTION_JOB_TYPE_NORMAL + description: The type of the last run of the workflow. last_run_id: type: string description: Run id of the last run of the workflow @@ -72,6 +250,74 @@ paths: - >- ANOMALY_DETECTION_RUN_STATUS_WORKSPACE_MISMATCH_ERROR description: The status of the last run of the workflow. + slicing_config: + type: object + properties: + slice_specs: + type: array + items: + type: object + properties: + column_group: + type: array + items: + type: object + properties: + name: + type: string + description: >- + The name of the column to slice on. The + top-K column values are auto-discovered. + required: + - name + description: >- + A column to slice completeness + monitoring on. + description: The columns to group by for slicing. + description: >- + A single slice dimension for per-slice + completeness monitoring. + description: The set of slice dimensions to monitor. + description: Per-schema completeness slicing configuration. + validity_check_configurations: + type: array + items: + type: object + properties: + name: + type: string + percent_null_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + upper_bound: + type: number + description: >- + Optional upper bound; we should use auto + determined bounds for now + range_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + lower_bound: + type: number + description: Lower bound for the range + upper_bound: + type: number + description: Upper bound for the range + uniqueness_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string validity_check_configurations: type: array items: @@ -347,10 +593,188 @@ paths: anomaly_detection_config: type: object properties: + custom_check_configurations: + type: array + items: + type: object + properties: + scalar_check: + type: object + properties: + check_name: + type: string + column_matchers: + type: array + items: + type: object + properties: + column_names: + type: array + items: + type: string + variable_name: + type: string + description: >- + Variable name within a custom sql query + that this matcher applies to. + description: >- + Column matchers to determine which tables + to apply this check to + sql_query: + type: string + description: Templated SQL query for this check + thresholds: + type: object + properties: + lower_bound: + type: object + properties: + bound_value: + type: integer + threshold_type: + type: string + x-enum: + - THRESHOLD_TYPE_AUTO + - THRESHOLD_TYPE_MANUAL + - THRESHOLD_TYPE_UNBOUNDED + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + upper_bound: + type: object + properties: + bound_value: + type: integer + threshold_type: + type: string + x-enum: + - THRESHOLD_TYPE_AUTO + - THRESHOLD_TYPE_MANUAL + - THRESHOLD_TYPE_UNBOUNDED + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and + know how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, + and members can have their own + + attributes -- see the documentation for + details. + description: Upper bound threshold + description: >- + Upper/lower thresholds for the output of + the query excluded_table_full_names: type: array items: type: string + description: >- + List of fully qualified table names to exclude from + anomaly detection. + job_type: + type: string + x-enum: + - >- + ANOMALY_DETECTION_JOB_TYPE_BACKGROUND_COMPUTE_WORKER + - ANOMALY_DETECTION_JOB_TYPE_INTERNAL_HIDDEN + - ANOMALY_DETECTION_JOB_TYPE_NORMAL + description: The type of the last run of the workflow. last_run_id: type: string description: Run id of the last run of the workflow @@ -367,6 +791,74 @@ paths: - >- ANOMALY_DETECTION_RUN_STATUS_WORKSPACE_MISMATCH_ERROR description: The status of the last run of the workflow. + slicing_config: + type: object + properties: + slice_specs: + type: array + items: + type: object + properties: + column_group: + type: array + items: + type: object + properties: + name: + type: string + description: >- + The name of the column to slice on. The + top-K column values are auto-discovered. + required: + - name + description: >- + A column to slice completeness + monitoring on. + description: The columns to group by for slicing. + description: >- + A single slice dimension for per-slice + completeness monitoring. + description: The set of slice dimensions to monitor. + description: Per-schema completeness slicing configuration. + validity_check_configurations: + type: array + items: + type: object + properties: + name: + type: string + percent_null_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + upper_bound: + type: number + description: >- + Optional upper bound; we should use auto + determined bounds for now + range_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string + lower_bound: + type: number + description: Lower bound for the range + upper_bound: + type: number + description: Upper bound for the range + uniqueness_validity_check: + type: object + properties: + column_names: + type: array + items: + type: string validity_check_configurations: type: array items: @@ -436,16 +928,85 @@ components: AnomalyDetectionConfig: type: object properties: + custom_check_configurations: + type: array + items: + $ref: '#/components/schemas/CustomCheckConfiguration' excluded_table_full_names: type: array items: type: string + description: >- + List of fully qualified table names to exclude from anomaly + detection. + job_type: + $ref: '#/components/schemas/AnomalyDetectionJobType' + description: The type of the last run of the workflow. last_run_id: type: string description: Run id of the last run of the workflow latest_run_status: $ref: '#/components/schemas/AnomalyDetectionRunStatus' description: The status of the last run of the workflow. + slicing_config: + $ref: '#/components/schemas/SlicingConfig' + description: Per-schema completeness slicing configuration. + validity_check_configurations: + type: array + items: + $ref: '#/components/schemas/ValidityCheckConfiguration' + ColumnMatcher: + type: object + properties: + column_names: + type: array + items: + type: string + variable_name: + type: string + description: >- + Variable name within a custom sql query that this matcher applies + to. + ColumnSpec: + type: object + properties: + name: + type: string + description: >- + The name of the column to slice on. The top-K column values are + auto-discovered. + required: + - name + description: A column to slice completeness monitoring on. + CustomCheckConfiguration: + type: object + properties: + scalar_check: + $ref: '#/components/schemas/CustomScalarCheck' + CustomCheckThresholds: + type: object + properties: + lower_bound: + $ref: '#/components/schemas/Threshold' + upper_bound: + $ref: '#/components/schemas/Threshold' + description: Upper bound threshold + CustomScalarCheck: + type: object + properties: + check_name: + type: string + column_matchers: + type: array + items: + $ref: '#/components/schemas/ColumnMatcher' + description: Column matchers to determine which tables to apply this check to + sql_query: + type: string + description: Templated SQL query for this check + thresholds: + $ref: '#/components/schemas/CustomCheckThresholds' + description: Upper/lower thresholds for the output of the query ListQualityMonitorResponse: type: object properties: @@ -496,6 +1057,31 @@ components: upper_bound: type: number description: Upper bound for the range + SliceSpec: + type: object + properties: + column_group: + type: array + items: + $ref: '#/components/schemas/ColumnSpec' + description: The columns to group by for slicing. + description: A single slice dimension for per-slice completeness monitoring. + SlicingConfig: + type: object + properties: + slice_specs: + type: array + items: + $ref: '#/components/schemas/SliceSpec' + description: The set of slice dimensions to monitor. + description: Per-schema completeness slicing configuration. + Threshold: + type: object + properties: + bound_value: + type: integer + threshold_type: + $ref: '#/components/schemas/ThresholdType' UniquenessValidityCheck: type: object properties: @@ -514,6 +1100,49 @@ components: $ref: '#/components/schemas/RangeValidityCheck' uniqueness_validity_check: $ref: '#/components/schemas/UniquenessValidityCheck' + AnomalyDetectionJobType: + type: string + x-enum: + - ANOMALY_DETECTION_JOB_TYPE_BACKGROUND_COMPUTE_WORKER + - ANOMALY_DETECTION_JOB_TYPE_INTERNAL_HIDDEN + - ANOMALY_DETECTION_JOB_TYPE_NORMAL + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. AnomalyDetectionRunStatus: type: string x-enum: @@ -526,6 +1155,49 @@ components: - ANOMALY_DETECTION_RUN_STATUS_UNKNOWN - ANOMALY_DETECTION_RUN_STATUS_WORKSPACE_MISMATCH_ERROR description: Status of Anomaly Detection Job Run + ThresholdType: + type: string + x-enum: + - THRESHOLD_TYPE_AUTO + - THRESHOLD_TYPE_MANUAL + - THRESHOLD_TYPE_UNBOUNDED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. x-stackQL-resources: quality_monitor_v2: id: databricks_workspace.qualitymonitorv2.quality_monitor_v2 diff --git a/providers/src/databricks_workspace/v00.00.00000/services/serving.yaml b/providers/src/databricks_workspace/v00.00.00000/services/serving.yaml index dcfd0ec1..e3143b18 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/serving.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/serving.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks serving service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.serving servers: - url: https://{deployment_name}.cloud.databricks.com @@ -73,9 +73,12 @@ paths: \ (optional)\n Email notification settings.\n:param rate_limits: List[:class:`RateLimit`] (optional)\n Rate limits\ \ to be applied to the serving endpoint. NOTE: this field is deprecated, please use AI\n Gateway to manage rate limits.\n\ :param route_optimized: bool (optional)\n Enable route optimization for the serving endpoint.\n:param tags: List[:class:`EndpointTag`]\ - \ (optional)\n Tags to be attached to the serving endpoint and automatically propagated to billing logs.\n\n:returns:\n\ - \ Long-running operation waiter for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating\ - \ for more details." + \ (optional)\n Tags to be attached to the serving endpoint and automatically propagated to billing logs.\n:param\ + \ telemetry_config: :class:`TelemetryConfig` (optional)\n Configuration for persisting endpoint telemetry (logs,\ + \ traces, and metrics) to Unity Catalog tables.\n:param uc_system_metrics_export_state: :class:`FirstPartyExportSetting`\ + \ (optional)\n Whether this endpoint's system metrics are exported to the system.telemetry.otel_metrics Unity\n \ + \ Catalog table. Endpoint-level property (not part of the per-version config).\n\n:returns:\n Long-running operation\ + \ waiter for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating for more details." requestBody: content: application/json: @@ -248,6 +251,8 @@ paths: properties: catalog_name: type: string + description: 'The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog + name if the inference table is already enabled.' enabled: type: boolean description: Indicates whether the inference table is enabled. @@ -259,10 +264,8 @@ paths: type: string description: 'The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.' - description: 'Configuration for Inference Tables which automatically logs requests and responses to - Unity Catalog. Note: this field is deprecated for creating new provisioned throughput endpoints, or - updating existing provisioned throughput endpoints that never have inference table configured; in - these cases please use AI Gateway to manage inference tables.' + description: 'Configuration for legacy Inference Tables which automatically log requests and responses + to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.' served_entities: type: array items: @@ -281,8 +284,8 @@ paths: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. - Example entity environment variables that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", - "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + Example entity environment variables that refer to Databricks secrets: ``{"OPENAI_API_KEY": + "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' external_model: type: object properties: @@ -320,8 +323,8 @@ paths: ai21labs_api_key_plaintext: type: string description: 'An AI21 Labs API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `ai21labs_api_key`. You must provide an API key - using one of the following fields: `ai21labs_api_key` or `ai21labs_api_key_plaintext`.' + your key using Databricks Secrets, see ``ai21labs_api_key``. You must provide an API + key using one of the following fields: ``ai21labs_api_key`` or ``ai21labs_api_key_plaintext``.' description: AI21Labs Config. Only required if the provider is 'ai21labs'. amazon_bedrock_config: type: object @@ -335,38 +338,41 @@ paths: - amazon - anthropic - cohere + - grok description: 'The underlying provider in Amazon Bedrock. Supported values (case insensitive) include: Anthropic, Cohere, AI21Labs, Amazon.' aws_access_key_id: type: string description: 'The Databricks secret key reference for an AWS access key ID with permissions to interact with Bedrock services. If you prefer to paste your API key directly, see - `aws_access_key_id_plaintext`. You must provide an API key using one of the following - fields: `aws_access_key_id` or `aws_access_key_id_plaintext`.' + ``aws_access_key_id_plaintext``. You must provide an API key using one of the following + fields: ``aws_access_key_id`` or ``aws_access_key_id_plaintext``.' aws_access_key_id_plaintext: type: string description: 'An AWS access key ID with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference your key using Databricks - Secrets, see `aws_access_key_id`. You must provide an API key using one of the following - fields: `aws_access_key_id` or `aws_access_key_id_plaintext`.' + Secrets, see ``aws_access_key_id``. You must provide an API key using one of the following + fields: ``aws_access_key_id`` or ``aws_access_key_id_plaintext``.' aws_secret_access_key: type: string description: 'The Databricks secret key reference for an AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services. If you prefer - to paste your API key directly, see `aws_secret_access_key_plaintext`. You must provide - an API key using one of the following fields: `aws_secret_access_key` or `aws_secret_access_key_plaintext`.' + to paste your API key directly, see ``aws_secret_access_key_plaintext``. You must provide + an API key using one of the following fields: ``aws_secret_access_key`` or ``aws_secret_access_key_plaintext``.' aws_secret_access_key_plaintext: type: string description: 'An AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `aws_secret_access_key`. You must provide an - API key using one of the following fields: `aws_secret_access_key` or `aws_secret_access_key_plaintext`.' + your key using Databricks Secrets, see ``aws_secret_access_key``. You must provide an + API key using one of the following fields: ``aws_secret_access_key`` or ``aws_secret_access_key_plaintext``.' instance_profile_arn: type: string description: ARN of the instance profile that the external model will use to access AWS resources. You must authenticate using an instance profile or access keys. If you prefer - to authenticate using access keys, see `aws_access_key_id`, `aws_access_key_id_plaintext`, - `aws_secret_access_key` and `aws_secret_access_key_plaintext`. + to authenticate using access keys, see ``aws_access_key_id``, ``aws_access_key_id_plaintext``, + ``aws_secret_access_key`` and ``aws_secret_access_key_plaintext``. + uc_service_credential_name: + type: string required: - aws_region - bedrock_provider @@ -379,8 +385,8 @@ paths: anthropic_api_key_plaintext: type: string description: 'The Anthropic API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `anthropic_api_key`. You must provide an API - key using one of the following fields: `anthropic_api_key` or `anthropic_api_key_plaintext`.' + your key using Databricks Secrets, see ``anthropic_api_key``. You must provide an API + key using one of the following fields: ``anthropic_api_key`` or ``anthropic_api_key_plaintext``.' description: Anthropic Config. Only required if the provider is 'anthropic'. cohere_config: type: object @@ -390,13 +396,13 @@ paths: cohere_api_key: type: string description: 'The Databricks secret key reference for a Cohere API key. If you prefer - to paste your API key directly, see `cohere_api_key_plaintext`. You must provide an - API key using one of the following fields: `cohere_api_key` or `cohere_api_key_plaintext`.' + to paste your API key directly, see ``cohere_api_key_plaintext``. You must provide an + API key using one of the following fields: ``cohere_api_key`` or ``cohere_api_key_plaintext``.' cohere_api_key_plaintext: type: string description: 'The Cohere API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `cohere_api_key`. You must provide an API key - using one of the following fields: `cohere_api_key` or `cohere_api_key_plaintext`.' + your key using Databricks Secrets, see ``cohere_api_key``. You must provide an API key + using one of the following fields: ``cohere_api_key`` or ``cohere_api_key_plaintext``.' description: Cohere Config. Only required if the provider is 'cohere'. custom_provider_config: type: object @@ -412,11 +418,11 @@ paths: value: type: string description: The Databricks secret key reference for an API Key. If you prefer to - paste your token directly, see `value_plaintext`. + paste your token directly, see ``value_plaintext``. value_plaintext: type: string description: The API Key provided as a plaintext string. If you prefer to reference - your token using Databricks Secrets, see `value`. + your token using Databricks Secrets, see ``value``. required: - key description: This is a field to provide API key authentication for the custom provider @@ -429,7 +435,7 @@ paths: token_plaintext: type: string description: The token provided as a plaintext string. If you prefer to reference - your token using Databricks Secrets, see `token`. + your token using Databricks Secrets, see ``token``. description: This is a field to provide bearer token authentication for the custom provider API. You can only specify one authentication method. required: @@ -444,16 +450,16 @@ paths: type: string description: 'The Databricks secret key reference for a Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed - to by this external model. If you prefer to paste your API key directly, see `databricks_api_token_plaintext`. - You must provide an API key using one of the following fields: `databricks_api_token` - or `databricks_api_token_plaintext`.' + to by this external model. If you prefer to paste your API key directly, see ``databricks_api_token_plaintext``. + You must provide an API key using one of the following fields: ``databricks_api_token`` + or ``databricks_api_token_plaintext``.' databricks_api_token_plaintext: type: string description: 'The Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model provided as a plaintext string. If you prefer to reference your key using Databricks - Secrets, see `databricks_api_token`. You must provide an API key using one of the following - fields: `databricks_api_token` or `databricks_api_token_plaintext`.' + Secrets, see ``databricks_api_token``. You must provide an API key using one of the + following fields: ``databricks_api_token`` or ``databricks_api_token_plaintext``.' required: - databricks_workspace_url description: Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'. @@ -464,23 +470,23 @@ paths: type: string region: type: string - description: 'This is the region for the Google Cloud Vertex AI Service. See [supported - regions] for more details. Some models are only available in specific regions. [supported - regions]: https://cloud.google.com/vertex-ai/docs/general/locations' + description: This is the region for the Google Cloud Vertex AI Service. See `supportedregions + `__ for more details. Some + models are only available in specific regions. private_key: type: string description: 'The Databricks secret key reference for a private key for the service account - which has access to the Google Cloud Vertex AI Service. See [Best practices for managing - service account keys]. If you prefer to paste your API key directly, see `private_key_plaintext`. - You must provide an API key using one of the following fields: `private_key` or `private_key_plaintext` - [Best practices for managing service account keys]: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys' + which has access to the Google Cloud Vertex AI Service. See `Bestpractices for managing + service account keys `__. + If you prefer to paste your API key directly, see ``private_key_plaintext``. You must + provide an API key using one of the following fields: ``private_key`` or ``private_key_plaintext``' private_key_plaintext: type: string description: 'The private key for the service account which has access to the Google Cloud - Vertex AI Service provided as a plaintext secret. See [Best practices for managing service - account keys]. If you prefer to reference your key using Databricks Secrets, see `private_key`. - You must provide an API key using one of the following fields: `private_key` or `private_key_plaintext`. - [Best practices for managing service account keys]: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys' + Vertex AI Service provided as a plaintext secret. See `Bestpractices for managing service + account keys `__. + If you prefer to reference your key using Databricks Secrets, see ``private_key``. You + must provide an API key using one of the following fields: ``private_key`` or ``private_key_plaintext``.' required: - project_id - region @@ -495,15 +501,15 @@ paths: microsoft_entra_client_secret: type: string description: 'The Databricks secret key reference for a client secret used for Microsoft - Entra ID authentication. If you prefer to paste your client secret directly, see `microsoft_entra_client_secret_plaintext`. - You must provide an API key using one of the following fields: `microsoft_entra_client_secret` - or `microsoft_entra_client_secret_plaintext`.' + Entra ID authentication. If you prefer to paste your client secret directly, see ``microsoft_entra_client_secret_plaintext``. + You must provide an API key using one of the following fields: ``microsoft_entra_client_secret`` + or ``microsoft_entra_client_secret_plaintext``.' microsoft_entra_client_secret_plaintext: type: string description: 'The client secret used for Microsoft Entra ID authentication provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see - `microsoft_entra_client_secret`. You must provide an API key using one of the following - fields: `microsoft_entra_client_secret` or `microsoft_entra_client_secret_plaintext`.' + ``microsoft_entra_client_secret``. You must provide an API key using one of the following + fields: ``microsoft_entra_client_secret`` or ``microsoft_entra_client_secret_plaintext``.' microsoft_entra_tenant_id: type: string description: This field is only required for Azure AD OpenAI and is the Microsoft Entra @@ -517,13 +523,15 @@ paths: openai_api_key: type: string description: 'The Databricks secret key reference for an OpenAI API key using the OpenAI - or Azure service. If you prefer to paste your API key directly, see `openai_api_key_plaintext`. - You must provide an API key using one of the following fields: `openai_api_key` or `openai_api_key_plaintext`.' + or Azure service. If you prefer to paste your API key directly, see ``openai_api_key_plaintext``. + You must provide an API key using one of the following fields: ``openai_api_key`` or + ``openai_api_key_plaintext``.' openai_api_key_plaintext: type: string description: 'The OpenAI API key using the OpenAI or Azure service provided as a plaintext - string. If you prefer to reference your key using Databricks Secrets, see `openai_api_key`. - You must provide an API key using one of the following fields: `openai_api_key` or `openai_api_key_plaintext`.' + string. If you prefer to reference your key using Databricks Secrets, see ``openai_api_key``. + You must provide an API key using one of the following fields: ``openai_api_key`` or + ``openai_api_key_plaintext``.' openai_api_type: type: string description: This is an optional field to specify the type of OpenAI API to use. For Azure @@ -552,8 +560,8 @@ paths: palm_api_key_plaintext: type: string description: 'The PaLM API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `palm_api_key`. You must provide an API key using - one of the following fields: `palm_api_key` or `palm_api_key_plaintext`.' + your key using Databricks Secrets, see ``palm_api_key``. You must provide an API key + using one of the following fields: ``palm_api_key`` or ``palm_api_key_plaintext``.' description: PaLM Config. Only required if the provider is 'palm'. required: - provider @@ -607,14 +615,18 @@ paths: type: string x-enum: - CPU + - CPU_LARGE + - CPU_MEDIUM - GPU_LARGE - GPU_MEDIUM - GPU_SMALL + - GPU_XLARGE + - GPU_XLARGE_8 - MULTIGPU_MEDIUM - description: 'The workload type of the served entity. The workload type selects which type of - compute to use in the endpoint. The default value for this parameter is "CPU". For deep learning - workloads, GPU acceleration is available by selecting workload types like GPU_SMALL and others. - See the available [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + description: The workload type of the served entity. The workload type selects which type of compute + to use in the endpoint. The default value for this parameter is "CPU". For deep learning workloads, + GPU acceleration is available by selecting workload types like GPU_SMALL and others. See the + available `GPU types `__. description: The list of served entities under the serving endpoint config. served_models: type: array @@ -636,8 +648,8 @@ paths: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. - Example entity environment variables that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", - "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + Example entity environment variables that refer to Databricks secrets: ``{"OPENAI_API_KEY": + "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' instance_profile_arn: type: string description: ARN of the instance profile that the served entity uses to access AWS resources. @@ -677,14 +689,18 @@ paths: type: string x-enum: - CPU + - CPU_LARGE + - CPU_MEDIUM - GPU_LARGE - GPU_MEDIUM - GPU_SMALL + - GPU_XLARGE + - GPU_XLARGE_8 - MULTIGPU_MEDIUM - description: 'The workload type of the served entity. The workload type selects which type of - compute to use in the endpoint. The default value for this parameter is "CPU". For deep learning - workloads, GPU acceleration is available by selecting workload types like GPU_SMALL and others. - See the available [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + description: The workload type of the served entity. The workload type selects which type of compute + to use in the endpoint. The default value for this parameter is "CPU". For deep learning workloads, + GPU acceleration is available by selecting workload types like GPU_SMALL and others. See the + available `GPU types `__. required: - scale_to_zero_enabled - model_name @@ -768,6 +784,52 @@ paths: required: - key description: Tags to be attached to the serving endpoint and automatically propagated to billing logs. + telemetry_config: + type: object + properties: + inference_table_config: + type: object + properties: + name: + type: string + description: The full name of the inference table created for this endpoint. + sampling_fraction: + type: number + description: Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 + logs all requests. + description: Inference table payload logging configuration + table_names: + type: object + properties: + annotations_table: + type: string + logs_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives + exported logs. + metrics_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives + exported metrics. + traces_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives + exported traces (spans). + description: The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. + Provide this to create a new telemetry profile for the endpoint from the given tables. + telemetry_profile_id: + type: string + description: The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse + a telemetry profile that has already been created, instead of specifying table_names. + description: Configuration for persisting endpoint telemetry (logs, traces, and metrics) to Unity Catalog + tables. + uc_system_metrics_export_state: + type: string + x-enum: + - FIRST_PARTY_EXPORT_DISABLED + - FIRST_PARTY_EXPORT_ENABLED + description: Whether this endpoint's system metrics are exported to the system.telemetry.otel_metrics Unity + Catalog table. Endpoint-level property (not part of the per-version config). required: - name responses: @@ -831,10 +893,12 @@ paths: \ dashes, and underscores.\n:param config: :class:`PtEndpointCoreConfig`\n The core config of the serving endpoint.\n\ :param ai_gateway: :class:`AiGatewayConfig` (optional)\n The AI Gateway configuration for the serving endpoint.\n\ :param budget_policy_id: str (optional)\n The budget policy associated with the endpoint.\n:param email_notifications:\ - \ :class:`EmailNotifications` (optional)\n Email notification settings.\n:param tags: List[:class:`EndpointTag`]\ - \ (optional)\n Tags to be attached to the serving endpoint and automatically propagated to billing logs.\n\n:returns:\n\ - \ Long-running operation waiter for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating\ - \ for more details." + \ :class:`EmailNotifications` (optional)\n Email notification settings.\n:param relocate_pt_commitment: :class:`PtCommitmentRelocation`\ + \ (optional)\n Relocate (\"upgrade\") an existing reserved commitment into this newly created endpoint, whose served\n\ + \ model must be a same-line upgrade of the commitment's current model.\n:param tags: List[:class:`EndpointTag`] (optional)\n\ + \ Tags to be attached to the serving endpoint and automatically propagated to billing logs.\n\n:returns:\n Long-running\ + \ operation waiter for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating for\ + \ more details." requestBody: content: application/json: @@ -871,6 +935,17 @@ paths: name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version. + pt_auto_renew: + type: boolean + description: Whether the Provisioned Throughput commitment should automatically renew at the end + of its term. Only applies when pt_term is set. + pt_term: + type: string + x-enum: + - COMMITMENT_TERM_1_MONTH + - COMMITMENT_TERM_3_MONTHS + description: Commitment term to purchase for this served entity; its presence marks the entity + as reserved (vs on-demand) PT. Valid on create and on update (purchases a commitment for a scale-up). required: - entity_name - provisioned_model_units @@ -1057,6 +1132,22 @@ paths: description: A list of email addresses to be notified when an endpoint successfully updates its configuration or state. description: Email notification settings. + relocate_pt_commitment: + type: object + properties: + source_endpoint: + type: string + description: The endpoint that currently owns the commitments (the relocation source). + commitment_ids: + type: array + items: + type: string + description: The IDs of the commitments to relocate. All move from the same source endpoint to the same + target in one transaction (all-or-nothing), so a multi-commitment upgrade is atomic. + required: + - source_endpoint + description: Relocate ("upgrade") an existing reserved commitment into this newly created endpoint, whose + served model must be a same-line upgrade of the commitment's current model. tags: type: array items: @@ -1440,12 +1531,19 @@ paths: description: "Make external services call using the credentials stored in UC Connection.\n\n:param connection_name:\ \ str\n The connection name to use. This is required to identify the external connection.\n:param method: :class:`ExternalFunctionRequestHttpMethod`\n\ \ The HTTP method to use (e.g., 'GET', 'POST').\n:param path: str\n The relative path for the API endpoint. This\ - \ is required.\n:param headers: str (optional)\n Additional headers for the request. If not provided, only auth headers\ - \ from connections would be\n passed.\n:param json: str (optional)\n The JSON payload to send in the request body.\n\ - :param params: str (optional)\n Query parameters for the request.\n:param sub_domain: str (optional)\n Optional\ - \ subdomain to prepend to the connection URL's host. If provided, this will be added as a\n prefix to the connection\ - \ URL's host. For example, if the connection URL is\n `https://api.example.com/v1` and `sub_domain` is `\"custom\"\ - `, the resulting URL will be\n `https://custom.api.example.com/v1`.\n\n:returns: :class:`HttpRequestResponse`" + \ is required.\n:param body: str (optional)\n Raw binary request body. When set, this is forwarded as the HTTP body\ + \ to the upstream connection\n without any encoding. Use this for non-text payloads (image uploads, multipart bodies)\ + \ that cannot\n be carried in the UTF-8 string ``json`` field. Mutually exclusive with ``json`` — if both are set,\n\ + \ the request is rejected. The caller is responsible for setting an appropriate ``Content-Type``\n header.\n\n \ + \ Size is bounded by the same limit as the rest of the request payload —\n ``maxRequestPayloadSizeExternalFunctions``\ + \ (default 4 MiB, configurable). Note that proto-over-JSON\n encodes ``bytes`` as base64, so the on-wire representation\ + \ is ~33% larger than the raw bytes; budget\n accordingly.\n:param headers: str (optional)\n Additional headers\ + \ for the request. If not provided, only auth headers from connections would be\n passed.\n:param json: str (optional)\n\ + \ The JSON payload to send in the request body.\n:param params: str (optional)\n Query parameters for the request.\n\ + :param sub_domain: str (optional)\n Optional subdomain to prepend to the connection URL's host. If provided, this\ + \ will be added as a\n prefix to the connection URL's host. For example, if the connection URL is\n ``https://api.example.com/v1``\ + \ and ``sub_domain`` is ``\"custom\"``, the resulting URL will be\n ``https://custom.api.example.com/v1``.\n\n:returns:\ + \ :class:`HttpRequestResponse`" requestBody: content: application/json: @@ -1467,6 +1565,15 @@ paths: path: type: string description: The relative path for the API endpoint. This is required. + body: + type: string + description: Raw binary request body. When set, this is forwarded as the HTTP body to the upstream connection + without any encoding. Use this for non-text payloads (image uploads, multipart bodies) that cannot be + carried in the UTF-8 string ``json`` field. Mutually exclusive with ``json`` — if both are set, the request + is rejected. The caller is responsible for setting an appropriate ``Content-Type`` header. Size is bounded + by the same limit as the rest of the request payload — ``maxRequestPayloadSizeExternalFunctions`` (default + 4 MiB, configurable). Note that proto-over-JSON encodes ``bytes`` as base64, so the on-wire representation + is ~33% larger than the raw bytes; budget accordingly. json: type: string description: The JSON payload to send in the request body. @@ -1476,8 +1583,8 @@ paths: sub_domain: type: string description: Optional subdomain to prepend to the connection URL's host. If provided, this will be added - as a prefix to the connection URL's host. For example, if the connection URL is `https://api.example.com/v1` - and `sub_domain` is `"custom"`, the resulting URL will be `https://custom.api.example.com/v1`. + as a prefix to the connection URL's host. For example, if the connection URL is ``https://api.example.com/v1`` + and ``sub_domain`` is ``"custom"``, the resulting URL will be ``https://custom.api.example.com/v1``. required: - connection_name - method @@ -1607,6 +1714,141 @@ paths: message: type: string x-stackql-sdk-source: ServingEndpointsAPI + /api/2.0/serving-endpoints/{name}/system-metrics-export-state: + patch: + operationId: serving_endpoints_patch_system_metrics_export_state + summary: Updates the system-metrics-export state of a serving endpoint, independently of the endpoint config + tags: + - serving + - serving_endpoints + description: "Updates the system-metrics-export state of a serving endpoint, independently of the endpoint config\n\ + (does not trigger a config-version change or redeployment).\n\n:param name: str\n The name of the serving endpoint\ + \ whose system-metrics-export state is being updated. This field is\n required.\n:param uc_system_metrics_export_state:\ + \ :class:`FirstPartyExportSetting` (optional)\n Whether this endpoint's system metrics are exported to the system.telemetry.otel_metrics\ + \ Unity\n Catalog table. Endpoint-level property (not part of the per-version config).\n\n:returns: :class:`ServingEndpointDetailed`" + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The name of the serving endpoint whose system-metrics-export state is being updated. This field is required. + requestBody: + content: + application/json: + schema: + type: object + properties: + uc_system_metrics_export_state: + type: string + x-enum: + - FIRST_PARTY_EXPORT_DISABLED + - FIRST_PARTY_EXPORT_ENABLED + description: Whether this endpoint's system metrics are exported to the system.telemetry.otel_metrics Unity + Catalog table. Endpoint-level property (not part of the per-version config). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServingEndpointDetailed' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ServingEndpointsAPI + /api/2.0/serving-endpoints/{name}/telemetry-config: + patch: + operationId: serving_endpoints_patch_telemetry_config + summary: Updates the telemetry configuration of a serving endpoint. + tags: + - serving + - serving_endpoints + description: "Updates the telemetry configuration of a serving endpoint.\n\n:param name: str\n The name of the serving\ + \ endpoint whose telemetry configuration is being updated. This field is\n required.\n:param telemetry_config: :class:`TelemetryConfig`\ + \ (optional)\n The telemetry configuration to be applied to the serving endpoint. Can specify either a\n telemetry_profile_id\ + \ to use an existing profile, or table_names to create a new profile with the\n specified Unity Catalog tables. If\ + \ not provided, the telemetry configuration will be removed from\n the endpoint.\n\n:returns: :class:`ServingEndpointDetailed`" + parameters: + - name: name + in: path + required: true + schema: + type: string + description: The name of the serving endpoint whose telemetry configuration is being updated. This field is required. + requestBody: + content: + application/json: + schema: + type: object + properties: + telemetry_config: + type: object + properties: + inference_table_config: + type: object + properties: + name: + type: string + description: The full name of the inference table created for this endpoint. + sampling_fraction: + type: number + description: Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 + logs all requests. + description: Inference table payload logging configuration + table_names: + type: object + properties: + annotations_table: + type: string + logs_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives + exported logs. + metrics_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives + exported metrics. + traces_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives + exported traces (spans). + description: The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. + Provide this to create a new telemetry profile for the endpoint from the given tables. + telemetry_profile_id: + type: string + description: The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse + a telemetry profile that has already been created, instead of specifying table_names. + description: The telemetry configuration to be applied to the serving endpoint. Can specify either a telemetry_profile_id + to use an existing profile, or table_names to create a new profile with the specified Unity Catalog tables. + If not provided, the telemetry configuration will be removed from the endpoint. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ServingEndpointDetailed' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: ServingEndpointsAPI /api/2.0/serving-endpoints/{name}/rate-limits: put: operationId: serving_endpoints_put @@ -1878,26 +2120,26 @@ paths: \ identifier that will be recorded in the inference table and the usage\n tracking table.\n:param dataframe_records:\ \ List[Any] (optional)\n Pandas Dataframe input in the records orientation.\n:param dataframe_split: :class:`DataframeSplitInput`\ \ (optional)\n Pandas Dataframe input in the split orientation.\n:param extra_params: Dict[str,str] (optional)\n\ - \ The extra parameters field used ONLY for __completions, chat,__ and __embeddings external &\n foundation model__\ + \ The extra parameters field used ONLY for **completions, chat,** and **embeddings external &\n foundation model**\ \ serving endpoints. This is a map of strings and should only be used with other\n external/foundation model query\ - \ fields.\n:param input: Any (optional)\n The input string (or array of strings) field used ONLY for __embeddings\ - \ external & foundation\n model__ serving endpoints and is the only field (along with extra_params if needed) used\ + \ fields.\n:param input: Any (optional)\n The input string (or array of strings) field used ONLY for **embeddings\ + \ external & foundation\n model** serving endpoints and is the only field (along with extra_params if needed) used\ \ by\n embeddings queries.\n:param inputs: Any (optional)\n Tensor-based input in columnar format.\n:param instances:\ \ List[Any] (optional)\n Tensor-based input in row format.\n:param max_tokens: int (optional)\n The max tokens field\ - \ used ONLY for __completions__ and __chat external & foundation model__ serving\n endpoints. This is an integer\ + \ used ONLY for **completions** and **chat external & foundation model** serving\n endpoints. This is an integer\ \ and should only be used with other chat/completions query fields.\n:param messages: List[:class:`ChatMessage`] (optional)\n\ - \ The messages field used ONLY for __chat external & foundation model__ serving endpoints. This is an\n array of\ + \ The messages field used ONLY for **chat external & foundation model** serving endpoints. This is an\n array of\ \ ChatMessage objects and should only be used with other chat query fields.\n:param n: int (optional)\n The n (number\ - \ of candidates) field used ONLY for __completions__ and __chat external & foundation\n model__ serving endpoints.\ + \ of candidates) field used ONLY for **completions** and **chat external & foundation\n model** serving endpoints.\ \ This is an integer between 1 and 5 with a default of 1 and should only be\n used with other chat/completions query\ - \ fields.\n:param prompt: Any (optional)\n The prompt string (or array of strings) field used ONLY for __completions\ - \ external & foundation\n model__ serving endpoints and should only be used with other completions query fields.\n\ - :param stop: List[str] (optional)\n The stop sequences field used ONLY for __completions__ and __chat external &\ - \ foundation model__\n serving endpoints. This is a list of strings and should only be used with other chat/completions\n\ - \ query fields.\n:param stream: bool (optional)\n The stream field used ONLY for __completions__ and __chat external\ - \ & foundation model__ serving\n endpoints. This is a boolean defaulting to false and should only be used with other\ - \ chat/completions\n query fields.\n:param temperature: float (optional)\n The temperature field used ONLY for __completions__\ - \ and __chat external & foundation model__ serving\n endpoints. This is a float between 0.0 and 2.0 with a default\ + \ fields.\n:param prompt: Any (optional)\n The prompt string (or array of strings) field used ONLY for **completions\ + \ external & foundation\n model** serving endpoints and should only be used with other completions query fields.\n\ + :param stop: List[str] (optional)\n The stop sequences field used ONLY for **completions** and **chat external &\ + \ foundation model**\n serving endpoints. This is a list of strings and should only be used with other chat/completions\n\ + \ query fields.\n:param stream: bool (optional)\n The stream field used ONLY for **completions** and **chat external\ + \ & foundation model** serving\n endpoints. This is a boolean defaulting to false and should only be used with other\ + \ chat/completions\n query fields.\n:param temperature: float (optional)\n The temperature field used ONLY for **completions**\ + \ and **chat external & foundation model** serving\n endpoints. This is a float between 0.0 and 2.0 with a default\ \ of 1.0 and should only be used with\n other chat/completions query fields.\n:param usage_context: Dict[str,str]\ \ (optional)\n Optional user-provided context that will be recorded in the usage tracking table.\n\n:returns: :class:`QueryEndpointResponse`" parameters: @@ -1942,13 +2184,13 @@ paths: description: Pandas Dataframe input in the split orientation. extra_params: type: object - description: The extra parameters field used ONLY for __completions, chat,__ and __embeddings external & - foundation model__ serving endpoints. This is a map of strings and should only be used with other external/foundation + description: The extra parameters field used ONLY for **completions, chat,** and **embeddings external & + foundation model** serving endpoints. This is a map of strings and should only be used with other external/foundation model query fields. input: type: object - description: The input string (or array of strings) field used ONLY for __embeddings external & foundation - model__ serving endpoints and is the only field (along with extra_params if needed) used by embeddings + description: The input string (or array of strings) field used ONLY for **embeddings external & foundation + model** serving endpoints and is the only field (along with extra_params if needed) used by embeddings queries. inputs: type: object @@ -1960,7 +2202,7 @@ paths: description: Tensor-based input in row format. max_tokens: type: integer - description: The max tokens field used ONLY for __completions__ and __chat external & foundation model__ + description: The max tokens field used ONLY for **completions** and **chat external & foundation model** serving endpoints. This is an integer and should only be used with other chat/completions query fields. messages: type: array @@ -1976,32 +2218,32 @@ paths: - system - user description: The role of the message. One of [system, user, assistant]. - description: The messages field used ONLY for __chat external & foundation model__ serving endpoints. This + description: The messages field used ONLY for **chat external & foundation model** serving endpoints. This is an array of ChatMessage objects and should only be used with other chat query fields. n: type: integer - description: The n (number of candidates) field used ONLY for __completions__ and __chat external & foundation - model__ serving endpoints. This is an integer between 1 and 5 with a default of 1 and should only be used + description: The n (number of candidates) field used ONLY for **completions** and **chat external & foundation + model** serving endpoints. This is an integer between 1 and 5 with a default of 1 and should only be used with other chat/completions query fields. prompt: type: object - description: The prompt string (or array of strings) field used ONLY for __completions external & foundation - model__ serving endpoints and should only be used with other completions query fields. + description: The prompt string (or array of strings) field used ONLY for **completions external & foundation + model** serving endpoints and should only be used with other completions query fields. stop: type: array items: type: string - description: The stop sequences field used ONLY for __completions__ and __chat external & foundation model__ + description: The stop sequences field used ONLY for **completions** and **chat external & foundation model** serving endpoints. This is a list of strings and should only be used with other chat/completions query fields. stream: type: boolean - description: The stream field used ONLY for __completions__ and __chat external & foundation model__ serving + description: The stream field used ONLY for **completions** and **chat external & foundation model** serving endpoints. This is a boolean defaulting to false and should only be used with other chat/completions query fields. temperature: type: number - description: The temperature field used ONLY for __completions__ and __chat external & foundation model__ + description: The temperature field used ONLY for **completions** and **chat external & foundation model** serving endpoints. This is a float between 0.0 and 2.0 with a default of 1.0 and should only be used with other chat/completions query fields. usage_context: @@ -2037,14 +2279,14 @@ paths: served entities, and the endpoint's traffic config. An endpoint that already has an update in progress\ncan not be\ \ updated until the current update completes or fails.\n\n:param name: str\n The name of the serving endpoint to\ \ update. This field is required.\n:param auto_capture_config: :class:`AutoCaptureConfigInput` (optional)\n Configuration\ - \ for Inference Tables which automatically logs requests and responses to Unity Catalog.\n Note: this field is deprecated\ - \ for creating new provisioned throughput endpoints, or updating\n existing provisioned throughput endpoints that\ - \ never have inference table configured; in these cases\n please use AI Gateway to manage inference tables.\n:param\ - \ served_entities: List[:class:`ServedEntityInput`] (optional)\n The list of served entities under the serving endpoint\ - \ config.\n:param served_models: List[:class:`ServedModelInput`] (optional)\n (Deprecated, use served_entities instead)\ - \ The list of served models under the serving endpoint\n config.\n:param traffic_config: :class:`TrafficConfig` (optional)\n\ - \ The traffic configuration associated with the serving endpoint config.\n\n:returns:\n Long-running operation waiter\ - \ for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating for more details." + \ for legacy Inference Tables which automatically log requests and responses to Unity\n Catalog. Deprecated: please\ + \ use AI Gateway inference tables instead. See\n https://docs.databricks.com/aws/en/ai-gateway/inference-tables.\n\ + :param served_entities: List[:class:`ServedEntityInput`] (optional)\n The list of served entities under the serving\ + \ endpoint config.\n:param served_models: List[:class:`ServedModelInput`] (optional)\n (Deprecated, use served_entities\ + \ instead) The list of served models under the serving endpoint\n config.\n:param traffic_config: :class:`TrafficConfig`\ + \ (optional)\n The traffic configuration associated with the serving endpoint config.\n\n:returns:\n Long-running\ + \ operation waiter for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating for\ + \ more details." parameters: - name: name in: path @@ -2063,6 +2305,8 @@ paths: properties: catalog_name: type: string + description: 'The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog + name if the inference table is already enabled.' enabled: type: boolean description: Indicates whether the inference table is enabled. @@ -2074,10 +2318,8 @@ paths: type: string description: 'The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.' - description: 'Configuration for Inference Tables which automatically logs requests and responses to Unity - Catalog. Note: this field is deprecated for creating new provisioned throughput endpoints, or updating - existing provisioned throughput endpoints that never have inference table configured; in these cases please - use AI Gateway to manage inference tables.' + description: 'Configuration for legacy Inference Tables which automatically log requests and responses to + Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.' served_entities: type: array items: @@ -2096,8 +2338,8 @@ paths: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. - Example entity environment variables that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", - "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + Example entity environment variables that refer to Databricks secrets: ``{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", + "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' external_model: type: object properties: @@ -2134,8 +2376,8 @@ paths: ai21labs_api_key_plaintext: type: string description: 'An AI21 Labs API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `ai21labs_api_key`. You must provide an API key using - one of the following fields: `ai21labs_api_key` or `ai21labs_api_key_plaintext`.' + your key using Databricks Secrets, see ``ai21labs_api_key``. You must provide an API key + using one of the following fields: ``ai21labs_api_key`` or ``ai21labs_api_key_plaintext``.' description: AI21Labs Config. Only required if the provider is 'ai21labs'. amazon_bedrock_config: type: object @@ -2149,37 +2391,41 @@ paths: - amazon - anthropic - cohere + - grok description: 'The underlying provider in Amazon Bedrock. Supported values (case insensitive) include: Anthropic, Cohere, AI21Labs, Amazon.' aws_access_key_id: type: string description: 'The Databricks secret key reference for an AWS access key ID with permissions - to interact with Bedrock services. If you prefer to paste your API key directly, see `aws_access_key_id_plaintext`. - You must provide an API key using one of the following fields: `aws_access_key_id` or `aws_access_key_id_plaintext`.' + to interact with Bedrock services. If you prefer to paste your API key directly, see ``aws_access_key_id_plaintext``. + You must provide an API key using one of the following fields: ``aws_access_key_id`` or + ``aws_access_key_id_plaintext``.' aws_access_key_id_plaintext: type: string description: 'An AWS access key ID with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see - `aws_access_key_id`. You must provide an API key using one of the following fields: `aws_access_key_id` - or `aws_access_key_id_plaintext`.' + ``aws_access_key_id``. You must provide an API key using one of the following fields: ``aws_access_key_id`` + or ``aws_access_key_id_plaintext``.' aws_secret_access_key: type: string description: 'The Databricks secret key reference for an AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services. If you prefer to - paste your API key directly, see `aws_secret_access_key_plaintext`. You must provide an - API key using one of the following fields: `aws_secret_access_key` or `aws_secret_access_key_plaintext`.' + paste your API key directly, see ``aws_secret_access_key_plaintext``. You must provide an + API key using one of the following fields: ``aws_secret_access_key`` or ``aws_secret_access_key_plaintext``.' aws_secret_access_key_plaintext: type: string description: 'An AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `aws_secret_access_key`. You must provide an API - key using one of the following fields: `aws_secret_access_key` or `aws_secret_access_key_plaintext`.' + your key using Databricks Secrets, see ``aws_secret_access_key``. You must provide an API + key using one of the following fields: ``aws_secret_access_key`` or ``aws_secret_access_key_plaintext``.' instance_profile_arn: type: string description: ARN of the instance profile that the external model will use to access AWS resources. You must authenticate using an instance profile or access keys. If you prefer to authenticate - using access keys, see `aws_access_key_id`, `aws_access_key_id_plaintext`, `aws_secret_access_key` - and `aws_secret_access_key_plaintext`. + using access keys, see ``aws_access_key_id``, ``aws_access_key_id_plaintext``, ``aws_secret_access_key`` + and ``aws_secret_access_key_plaintext``. + uc_service_credential_name: + type: string required: - aws_region - bedrock_provider @@ -2192,8 +2438,8 @@ paths: anthropic_api_key_plaintext: type: string description: 'The Anthropic API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `anthropic_api_key`. You must provide an API key - using one of the following fields: `anthropic_api_key` or `anthropic_api_key_plaintext`.' + your key using Databricks Secrets, see ``anthropic_api_key``. You must provide an API key + using one of the following fields: ``anthropic_api_key`` or ``anthropic_api_key_plaintext``.' description: Anthropic Config. Only required if the provider is 'anthropic'. cohere_config: type: object @@ -2203,13 +2449,13 @@ paths: cohere_api_key: type: string description: 'The Databricks secret key reference for a Cohere API key. If you prefer to paste - your API key directly, see `cohere_api_key_plaintext`. You must provide an API key using - one of the following fields: `cohere_api_key` or `cohere_api_key_plaintext`.' + your API key directly, see ``cohere_api_key_plaintext``. You must provide an API key using + one of the following fields: ``cohere_api_key`` or ``cohere_api_key_plaintext``.' cohere_api_key_plaintext: type: string description: 'The Cohere API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `cohere_api_key`. You must provide an API key using - one of the following fields: `cohere_api_key` or `cohere_api_key_plaintext`.' + your key using Databricks Secrets, see ``cohere_api_key``. You must provide an API key using + one of the following fields: ``cohere_api_key`` or ``cohere_api_key_plaintext``.' description: Cohere Config. Only required if the provider is 'cohere'. custom_provider_config: type: object @@ -2225,11 +2471,11 @@ paths: value: type: string description: The Databricks secret key reference for an API Key. If you prefer to paste - your token directly, see `value_plaintext`. + your token directly, see ``value_plaintext``. value_plaintext: type: string description: The API Key provided as a plaintext string. If you prefer to reference your - token using Databricks Secrets, see `value`. + token using Databricks Secrets, see ``value``. required: - key description: This is a field to provide API key authentication for the custom provider API. @@ -2242,7 +2488,7 @@ paths: token_plaintext: type: string description: The token provided as a plaintext string. If you prefer to reference your - token using Databricks Secrets, see `token`. + token using Databricks Secrets, see ``token``. description: This is a field to provide bearer token authentication for the custom provider API. You can only specify one authentication method. required: @@ -2257,16 +2503,16 @@ paths: type: string description: 'The Databricks secret key reference for a Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed - to by this external model. If you prefer to paste your API key directly, see `databricks_api_token_plaintext`. - You must provide an API key using one of the following fields: `databricks_api_token` or - `databricks_api_token_plaintext`.' + to by this external model. If you prefer to paste your API key directly, see ``databricks_api_token_plaintext``. + You must provide an API key using one of the following fields: ``databricks_api_token`` + or ``databricks_api_token_plaintext``.' databricks_api_token_plaintext: type: string description: 'The Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see - `databricks_api_token`. You must provide an API key using one of the following fields: `databricks_api_token` - or `databricks_api_token_plaintext`.' + ``databricks_api_token``. You must provide an API key using one of the following fields: + ``databricks_api_token`` or ``databricks_api_token_plaintext``.' required: - databricks_workspace_url description: Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'. @@ -2277,23 +2523,23 @@ paths: type: string region: type: string - description: 'This is the region for the Google Cloud Vertex AI Service. See [supported regions] - for more details. Some models are only available in specific regions. [supported regions]: - https://cloud.google.com/vertex-ai/docs/general/locations' + description: This is the region for the Google Cloud Vertex AI Service. See `supportedregions + `__ for more details. Some models + are only available in specific regions. private_key: type: string description: 'The Databricks secret key reference for a private key for the service account - which has access to the Google Cloud Vertex AI Service. See [Best practices for managing - service account keys]. If you prefer to paste your API key directly, see `private_key_plaintext`. - You must provide an API key using one of the following fields: `private_key` or `private_key_plaintext` - [Best practices for managing service account keys]: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys' + which has access to the Google Cloud Vertex AI Service. See `Bestpractices for managing + service account keys `__. + If you prefer to paste your API key directly, see ``private_key_plaintext``. You must provide + an API key using one of the following fields: ``private_key`` or ``private_key_plaintext``' private_key_plaintext: type: string description: 'The private key for the service account which has access to the Google Cloud - Vertex AI Service provided as a plaintext secret. See [Best practices for managing service - account keys]. If you prefer to reference your key using Databricks Secrets, see `private_key`. - You must provide an API key using one of the following fields: `private_key` or `private_key_plaintext`. - [Best practices for managing service account keys]: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys' + Vertex AI Service provided as a plaintext secret. See `Bestpractices for managing service + account keys `__. + If you prefer to reference your key using Databricks Secrets, see ``private_key``. You must + provide an API key using one of the following fields: ``private_key`` or ``private_key_plaintext``.' required: - project_id - region @@ -2308,15 +2554,15 @@ paths: microsoft_entra_client_secret: type: string description: 'The Databricks secret key reference for a client secret used for Microsoft Entra - ID authentication. If you prefer to paste your client secret directly, see `microsoft_entra_client_secret_plaintext`. - You must provide an API key using one of the following fields: `microsoft_entra_client_secret` - or `microsoft_entra_client_secret_plaintext`.' + ID authentication. If you prefer to paste your client secret directly, see ``microsoft_entra_client_secret_plaintext``. + You must provide an API key using one of the following fields: ``microsoft_entra_client_secret`` + or ``microsoft_entra_client_secret_plaintext``.' microsoft_entra_client_secret_plaintext: type: string description: 'The client secret used for Microsoft Entra ID authentication provided as a plaintext - string. If you prefer to reference your key using Databricks Secrets, see `microsoft_entra_client_secret`. - You must provide an API key using one of the following fields: `microsoft_entra_client_secret` - or `microsoft_entra_client_secret_plaintext`.' + string. If you prefer to reference your key using Databricks Secrets, see ``microsoft_entra_client_secret``. + You must provide an API key using one of the following fields: ``microsoft_entra_client_secret`` + or ``microsoft_entra_client_secret_plaintext``.' microsoft_entra_tenant_id: type: string description: This field is only required for Azure AD OpenAI and is the Microsoft Entra Tenant @@ -2330,13 +2576,13 @@ paths: openai_api_key: type: string description: 'The Databricks secret key reference for an OpenAI API key using the OpenAI or - Azure service. If you prefer to paste your API key directly, see `openai_api_key_plaintext`. - You must provide an API key using one of the following fields: `openai_api_key` or `openai_api_key_plaintext`.' + Azure service. If you prefer to paste your API key directly, see ``openai_api_key_plaintext``. + You must provide an API key using one of the following fields: ``openai_api_key`` or ``openai_api_key_plaintext``.' openai_api_key_plaintext: type: string description: 'The OpenAI API key using the OpenAI or Azure service provided as a plaintext - string. If you prefer to reference your key using Databricks Secrets, see `openai_api_key`. - You must provide an API key using one of the following fields: `openai_api_key` or `openai_api_key_plaintext`.' + string. If you prefer to reference your key using Databricks Secrets, see ``openai_api_key``. + You must provide an API key using one of the following fields: ``openai_api_key`` or ``openai_api_key_plaintext``.' openai_api_type: type: string description: This is an optional field to specify the type of OpenAI API to use. For Azure @@ -2364,8 +2610,8 @@ paths: palm_api_key_plaintext: type: string description: 'The PaLM API key provided as a plaintext string. If you prefer to reference - your key using Databricks Secrets, see `palm_api_key`. You must provide an API key using - one of the following fields: `palm_api_key` or `palm_api_key_plaintext`.' + your key using Databricks Secrets, see ``palm_api_key``. You must provide an API key using + one of the following fields: ``palm_api_key`` or ``palm_api_key_plaintext``.' description: PaLM Config. Only required if the provider is 'palm'. required: - provider @@ -2419,14 +2665,18 @@ paths: type: string x-enum: - CPU + - CPU_LARGE + - CPU_MEDIUM - GPU_LARGE - GPU_MEDIUM - GPU_SMALL + - GPU_XLARGE + - GPU_XLARGE_8 - MULTIGPU_MEDIUM - description: 'The workload type of the served entity. The workload type selects which type of compute + description: The workload type of the served entity. The workload type selects which type of compute to use in the endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is available by selecting workload types like GPU_SMALL and others. See the available - [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + `GPU types `__. description: The list of served entities under the serving endpoint config. served_models: type: array @@ -2448,8 +2698,8 @@ paths: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. - Example entity environment variables that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", - "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + Example entity environment variables that refer to Databricks secrets: ``{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", + "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' instance_profile_arn: type: string description: ARN of the instance profile that the served entity uses to access AWS resources. @@ -2489,14 +2739,18 @@ paths: type: string x-enum: - CPU + - CPU_LARGE + - CPU_MEDIUM - GPU_LARGE - GPU_MEDIUM - GPU_SMALL + - GPU_XLARGE + - GPU_XLARGE_8 - MULTIGPU_MEDIUM - description: 'The workload type of the served entity. The workload type selects which type of compute + description: The workload type of the served entity. The workload type selects which type of compute to use in the endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is available by selecting workload types like GPU_SMALL and others. See the available - [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + `GPU types `__. required: - scale_to_zero_enabled - model_name @@ -2608,6 +2862,10 @@ paths: description: "Updates any combination of the pt endpoint's served entities, the compute configuration of those\nserved\ \ entities, and the endpoint's traffic config. Updates are instantaneous and endpoint should be\nupdated instantly\n\ \n:param name: str\n The name of the pt endpoint to update. This field is required.\n:param config: :class:`PtEndpointCoreConfig`\n\ + :param pt_commitment_auto_renew: :class:`PtCommitmentAutoRenew` (optional)\n When set, updates the referenced commitment's\ + \ auto-renew flag as part of this config update.\n:param relocate_pt_commitment: :class:`PtCommitmentRelocation` (optional)\n\ + \ Relocate (\"upgrade\") an existing commitment onto this endpoint, whose served model must be a\n same-line upgrade\ + \ of the commitment's current model. Mutually exclusive with a pt_term scale-up and\n with pt_commitment_auto_renew.\n\ \n:returns:\n Long-running operation waiter for :class:`ServingEndpointDetailed`.\n See :method:wait_get_serving_endpoint_not_updating\ \ for more details." parameters: @@ -2649,6 +2907,17 @@ paths: name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version. + pt_auto_renew: + type: boolean + description: Whether the Provisioned Throughput commitment should automatically renew at the end + of its term. Only applies when pt_term is set. + pt_term: + type: string + x-enum: + - COMMITMENT_TERM_1_MONTH + - COMMITMENT_TERM_3_MONTHS + description: Commitment term to purchase for this served entity; its presence marks the entity + as reserved (vs on-demand) PT. Valid on create and on update (purchases a commitment for a scale-up). required: - entity_name - provisioned_model_units @@ -2669,6 +2938,36 @@ paths: description: The name of the served model this route configures traffic for. required: - traffic_percentage + pt_commitment_auto_renew: + type: object + properties: + commitment_id: + type: string + description: The ID of the commitment to update. + auto_renew: + type: boolean + description: The desired auto-renew setting for the commitment. + required: + - commitment_id + - auto_renew + description: When set, updates the referenced commitment's auto-renew flag as part of this config update. + relocate_pt_commitment: + type: object + properties: + source_endpoint: + type: string + description: The endpoint that currently owns the commitments (the relocation source). + commitment_ids: + type: array + items: + type: string + description: The IDs of the commitments to relocate. All move from the same source endpoint to the same + target in one transaction (all-or-nothing), so a multi-commitment upgrade is atomic. + required: + - source_endpoint + description: Relocate ("upgrade") an existing commitment onto this endpoint, whose served model must be + a same-line upgrade of the commitment's current model. Mutually exclusive with a pt_term scale-up and + with pt_commitment_auto_renew. required: - config responses: @@ -2700,8 +2999,8 @@ components: ai21labs_api_key_plaintext: type: string description: 'An AI21 Labs API key provided as a plaintext string. If you prefer to reference your key using Databricks - Secrets, see `ai21labs_api_key`. You must provide an API key using one of the following fields: `ai21labs_api_key` - or `ai21labs_api_key_plaintext`.' + Secrets, see ``ai21labs_api_key``. You must provide an API key using one of the following fields: ``ai21labs_api_key`` + or ``ai21labs_api_key_plaintext``.' AiGatewayConfig: type: object properties: @@ -2809,28 +3108,30 @@ components: aws_access_key_id: type: string description: 'The Databricks secret key reference for an AWS access key ID with permissions to interact with Bedrock - services. If you prefer to paste your API key directly, see `aws_access_key_id_plaintext`. You must provide an - API key using one of the following fields: `aws_access_key_id` or `aws_access_key_id_plaintext`.' + services. If you prefer to paste your API key directly, see ``aws_access_key_id_plaintext``. You must provide + an API key using one of the following fields: ``aws_access_key_id`` or ``aws_access_key_id_plaintext``.' aws_access_key_id_plaintext: type: string description: 'An AWS access key ID with permissions to interact with Bedrock services provided as a plaintext string. - If you prefer to reference your key using Databricks Secrets, see `aws_access_key_id`. You must provide an API - key using one of the following fields: `aws_access_key_id` or `aws_access_key_id_plaintext`.' + If you prefer to reference your key using Databricks Secrets, see ``aws_access_key_id``. You must provide an API + key using one of the following fields: ``aws_access_key_id`` or ``aws_access_key_id_plaintext``.' aws_secret_access_key: type: string description: 'The Databricks secret key reference for an AWS secret access key paired with the access key ID, with - permissions to interact with Bedrock services. If you prefer to paste your API key directly, see `aws_secret_access_key_plaintext`. - You must provide an API key using one of the following fields: `aws_secret_access_key` or `aws_secret_access_key_plaintext`.' + permissions to interact with Bedrock services. If you prefer to paste your API key directly, see ``aws_secret_access_key_plaintext``. + You must provide an API key using one of the following fields: ``aws_secret_access_key`` or ``aws_secret_access_key_plaintext``.' aws_secret_access_key_plaintext: type: string description: 'An AWS secret access key paired with the access key ID, with permissions to interact with Bedrock - services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see `aws_secret_access_key`. - You must provide an API key using one of the following fields: `aws_secret_access_key` or `aws_secret_access_key_plaintext`.' + services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see ``aws_secret_access_key``. + You must provide an API key using one of the following fields: ``aws_secret_access_key`` or ``aws_secret_access_key_plaintext``.' instance_profile_arn: type: string description: ARN of the instance profile that the external model will use to access AWS resources. You must authenticate - using an instance profile or access keys. If you prefer to authenticate using access keys, see `aws_access_key_id`, - `aws_access_key_id_plaintext`, `aws_secret_access_key` and `aws_secret_access_key_plaintext`. + using an instance profile or access keys. If you prefer to authenticate using access keys, see ``aws_access_key_id``, + ``aws_access_key_id_plaintext``, ``aws_secret_access_key`` and ``aws_secret_access_key_plaintext``. + uc_service_credential_name: + type: string required: - aws_region - bedrock_provider @@ -2842,8 +3143,8 @@ components: anthropic_api_key_plaintext: type: string description: 'The Anthropic API key provided as a plaintext string. If you prefer to reference your key using Databricks - Secrets, see `anthropic_api_key`. You must provide an API key using one of the following fields: `anthropic_api_key` - or `anthropic_api_key_plaintext`.' + Secrets, see ``anthropic_api_key``. You must provide an API key using one of the following fields: ``anthropic_api_key`` + or ``anthropic_api_key_plaintext``.' ApiKeyAuth: type: object properties: @@ -2852,11 +3153,11 @@ components: value: type: string description: The Databricks secret key reference for an API Key. If you prefer to paste your token directly, see - `value_plaintext`. + ``value_plaintext``. value_plaintext: type: string description: The API Key provided as a plaintext string. If you prefer to reference your token using Databricks - Secrets, see `value`. + Secrets, see ``value``. required: - key AutoCaptureConfigInput: @@ -2864,6 +3165,8 @@ components: properties: catalog_name: type: string + description: 'The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the + inference table is already enabled.' enabled: type: boolean description: Indicates whether the inference table is enabled. @@ -2875,11 +3178,15 @@ components: type: string description: 'The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.' + description: "Deprecated: legacy inference table configuration. Please use AI Gateway inference tables\n instead.\ + \ See https://docs.databricks.com/aws/en/ai-gateway/inference-tables." AutoCaptureConfigOutput: type: object properties: catalog_name: type: string + description: 'The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the + inference table is already enabled.' enabled: type: boolean description: Indicates whether the inference table is enabled. @@ -2893,6 +3200,8 @@ components: type: string description: 'The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.' + description: "Deprecated: legacy inference table configuration. Please use AI Gateway inference tables\n instead.\ + \ See https://docs.databricks.com/aws/en/ai-gateway/inference-tables." AutoCaptureState: type: object properties: @@ -2906,7 +3215,7 @@ components: token_plaintext: type: string description: The token provided as a plaintext string. If you prefer to reference your token using Databricks Secrets, - see `token`. + see ``token``. BuildLogsResponse: type: object properties: @@ -2930,13 +3239,13 @@ components: cohere_api_key: type: string description: 'The Databricks secret key reference for a Cohere API key. If you prefer to paste your API key directly, - see `cohere_api_key_plaintext`. You must provide an API key using one of the following fields: `cohere_api_key` - or `cohere_api_key_plaintext`.' + see ``cohere_api_key_plaintext``. You must provide an API key using one of the following fields: ``cohere_api_key`` + or ``cohere_api_key_plaintext``.' cohere_api_key_plaintext: type: string description: 'The Cohere API key provided as a plaintext string. If you prefer to reference your key using Databricks - Secrets, see `cohere_api_key`. You must provide an API key using one of the following fields: `cohere_api_key` - or `cohere_api_key_plaintext`.' + Secrets, see ``cohere_api_key``. You must provide an API key using one of the following fields: ``cohere_api_key`` + or ``cohere_api_key_plaintext``.' CustomProviderConfig: type: object properties: @@ -2973,14 +3282,14 @@ components: type: string description: 'The Databricks secret key reference for a Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model. If you prefer - to paste your API key directly, see `databricks_api_token_plaintext`. You must provide an API key using one of - the following fields: `databricks_api_token` or `databricks_api_token_plaintext`.' + to paste your API key directly, see ``databricks_api_token_plaintext``. You must provide an API key using one + of the following fields: ``databricks_api_token`` or ``databricks_api_token_plaintext``.' databricks_api_token_plaintext: type: string description: 'The Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model provided as a plaintext string. If you prefer to - reference your key using Databricks Secrets, see `databricks_api_token`. You must provide an API key using one - of the following fields: `databricks_api_token` or `databricks_api_token_plaintext`.' + reference your key using Databricks Secrets, see ``databricks_api_token``. You must provide an API key using one + of the following fields: ``databricks_api_token`` or ``databricks_api_token_plaintext``.' required: - databricks_workspace_url DataframeSplitInput: @@ -3033,10 +3342,8 @@ components: type: string auto_capture_config: $ref: '#/components/schemas/AutoCaptureConfigInput' - description: 'Configuration for Inference Tables which automatically logs requests and responses to Unity Catalog. - Note: this field is deprecated for creating new provisioned throughput endpoints, or updating existing provisioned - throughput endpoints that never have inference table configured; in these cases please use AI Gateway to manage - inference tables.' + description: 'Configuration for legacy Inference Tables which automatically log requests and responses to Unity + Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.' served_entities: type: array items: @@ -3115,6 +3422,9 @@ components: properties: config_update: $ref: '#/components/schemas/EndpointStateConfigUpdate' + pt_state: + $ref: '#/components/schemas/PtState' + description: The Provisioned Throughput state of the endpoint, including its capacity commitments. ready: $ref: '#/components/schemas/EndpointStateReady' description: The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY @@ -3235,20 +3545,20 @@ components: type: string region: type: string - description: 'This is the region for the Google Cloud Vertex AI Service. See [supported regions] for more details. - Some models are only available in specific regions. [supported regions]: https://cloud.google.com/vertex-ai/docs/general/locations' + description: This is the region for the Google Cloud Vertex AI Service. See `supportedregions `__ + for more details. Some models are only available in specific regions. private_key: type: string description: 'The Databricks secret key reference for a private key for the service account which has access to - the Google Cloud Vertex AI Service. See [Best practices for managing service account keys]. If you prefer to paste - your API key directly, see `private_key_plaintext`. You must provide an API key using one of the following fields: - `private_key` or `private_key_plaintext` [Best practices for managing service account keys]: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys' + the Google Cloud Vertex AI Service. See `Bestpractices for managing service account keys `__. + If you prefer to paste your API key directly, see ``private_key_plaintext``. You must provide an API key using + one of the following fields: ``private_key`` or ``private_key_plaintext``' private_key_plaintext: type: string description: 'The private key for the service account which has access to the Google Cloud Vertex AI Service provided - as a plaintext secret. See [Best practices for managing service account keys]. If you prefer to reference your - key using Databricks Secrets, see `private_key`. You must provide an API key using one of the following fields: - `private_key` or `private_key_plaintext`. [Best practices for managing service account keys]: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys' + as a plaintext secret. See `Bestpractices for managing service account keys `__. + If you prefer to reference your key using Databricks Secrets, see ``private_key``. You must provide an API key + using one of the following fields: ``private_key`` or ``private_key_plaintext``.' required: - project_id - region @@ -3282,13 +3592,13 @@ components: microsoft_entra_client_secret: type: string description: 'The Databricks secret key reference for a client secret used for Microsoft Entra ID authentication. - If you prefer to paste your client secret directly, see `microsoft_entra_client_secret_plaintext`. You must provide - an API key using one of the following fields: `microsoft_entra_client_secret` or `microsoft_entra_client_secret_plaintext`.' + If you prefer to paste your client secret directly, see ``microsoft_entra_client_secret_plaintext``. You must + provide an API key using one of the following fields: ``microsoft_entra_client_secret`` or ``microsoft_entra_client_secret_plaintext``.' microsoft_entra_client_secret_plaintext: type: string description: 'The client secret used for Microsoft Entra ID authentication provided as a plaintext string. If you - prefer to reference your key using Databricks Secrets, see `microsoft_entra_client_secret`. You must provide an - API key using one of the following fields: `microsoft_entra_client_secret` or `microsoft_entra_client_secret_plaintext`.' + prefer to reference your key using Databricks Secrets, see ``microsoft_entra_client_secret``. You must provide + an API key using one of the following fields: ``microsoft_entra_client_secret`` or ``microsoft_entra_client_secret_plaintext``.' microsoft_entra_tenant_id: type: string description: This field is only required for Azure AD OpenAI and is the Microsoft Entra Tenant ID. @@ -3300,13 +3610,13 @@ components: openai_api_key: type: string description: 'The Databricks secret key reference for an OpenAI API key using the OpenAI or Azure service. If you - prefer to paste your API key directly, see `openai_api_key_plaintext`. You must provide an API key using one of - the following fields: `openai_api_key` or `openai_api_key_plaintext`.' + prefer to paste your API key directly, see ``openai_api_key_plaintext``. You must provide an API key using one + of the following fields: ``openai_api_key`` or ``openai_api_key_plaintext``.' openai_api_key_plaintext: type: string description: 'The OpenAI API key using the OpenAI or Azure service provided as a plaintext string. If you prefer - to reference your key using Databricks Secrets, see `openai_api_key`. You must provide an API key using one of - the following fields: `openai_api_key` or `openai_api_key_plaintext`.' + to reference your key using Databricks Secrets, see ``openai_api_key``. You must provide an API key using one + of the following fields: ``openai_api_key`` or ``openai_api_key_plaintext``.' openai_api_type: type: string description: This is an optional field to specify the type of OpenAI API to use. For Azure OpenAI, this field is @@ -3332,8 +3642,8 @@ components: palm_api_key_plaintext: type: string description: 'The PaLM API key provided as a plaintext string. If you prefer to reference your key using Databricks - Secrets, see `palm_api_key`. You must provide an API key using one of the following fields: `palm_api_key` or - `palm_api_key_plaintext`.' + Secrets, see ``palm_api_key``. You must provide an API key using one of the following fields: ``palm_api_key`` + or ``palm_api_key_plaintext``.' PayloadTable: type: object properties: @@ -3343,6 +3653,68 @@ components: type: string status_message: type: string + PtCommitment: + type: object + properties: + auto_renew: + type: boolean + description: Whether the commitment automatically renews at the end of its term. + auto_renew_editable: + type: boolean + description: 'Whether auto_renew can still be toggled. Derived at read time, gated on the lockout window: true while + more than the auto-renew lockout window remains before expiry. The server enforces the same rule on write.' + expire_time: + type: string + format: date-time + description: The commitment's expiry (start_time + the term, in UTC calendar months/years). Stored on the commitment + and returned verbatim, so clients, the billing meter, and the purge agree on the exact instant. + id: + type: string + description: System-generated ID of the commitment. + provisioned_model_units: + type: integer + description: The number of model units reserved by this commitment. + start_time: + type: string + format: date-time + description: The time at which the commitment becomes active. + status: + $ref: '#/components/schemas/CommitmentStatus' + description: The lifecycle status of the commitment. + term: + $ref: '#/components/schemas/CommitmentTerm' + description: The commitment term length. + description: A single Provisioned Throughput capacity commitment for an endpoint. + PtCommitmentAutoRenew: + type: object + properties: + commitment_id: + type: string + description: The ID of the commitment to update. + auto_renew: + type: boolean + description: The desired auto-renew setting for the commitment. + required: + - commitment_id + - auto_renew + description: Changes the auto-renew setting of a single PT commitment. + PtCommitmentRelocation: + type: object + properties: + source_endpoint: + type: string + description: The endpoint that currently owns the commitments (the relocation source). + commitment_ids: + type: array + items: + type: string + description: The IDs of the commitments to relocate. All move from the same source endpoint to the same target in + one transaction (all-or-nothing), so a multi-commitment upgrade is atomic. + required: + - source_endpoint + description: "A request to relocate (\"upgrade\") an existing Provisioned Throughput commitment onto a\n different,\ + \ same-line endpoint. The commitment row moves intact (same commitment_id) from the\n source endpoint to the target\ + \ in a single transaction." PtEndpointCoreConfig: type: object properties: @@ -3372,9 +3744,26 @@ components: of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version. + pt_auto_renew: + type: boolean + description: Whether the Provisioned Throughput commitment should automatically renew at the end of its term. Only + applies when pt_term is set. + pt_term: + $ref: '#/components/schemas/CommitmentTerm' + description: Commitment term to purchase for this served entity; its presence marks the entity as reserved (vs on-demand) + PT. Valid on create and on update (purchases a commitment for a scale-up). required: - entity_name - provisioned_model_units + PtState: + type: object + properties: + commitments: + type: array + items: + $ref: '#/components/schemas/PtCommitment' + description: The list of capacity commitments backing the endpoint. + description: The Provisioned Throughput state for an endpoint, including all of its capacity commitments. PutAiGatewayResponse: type: object properties: @@ -3412,24 +3801,24 @@ components: $ref: '#/components/schemas/V1ResponseChoiceElement' created: type: integer - description: The timestamp in seconds when the query was created in Unix time returned by a __completions or chat - external/foundation model__ serving endpoint. + description: The timestamp in seconds when the query was created in Unix time returned by a **completions or chat + external/foundation model** serving endpoint. data: type: array items: $ref: '#/components/schemas/EmbeddingsV1ResponseEmbeddingElement' - description: The list of the embeddings returned by the __embeddings external/foundation model__ serving endpoint. + description: The list of the embeddings returned by the **embeddings external/foundation model** serving endpoint. id: type: string - description: The ID of the query that may be returned by a __completions or chat external/foundation model__ serving + description: The ID of the query that may be returned by a **completions or chat external/foundation model** serving endpoint. model: type: string - description: The name of the __external/foundation model__ used for querying. This is the name of the model that + description: The name of the **external/foundation model** used for querying. This is the name of the model that was specified in the endpoint config. object: $ref: '#/components/schemas/QueryEndpointResponseObject' - description: The type of object returned by the __external/foundation model__ serving endpoint, one of [text_completion, + description: The type of object returned by the **external/foundation model** serving endpoint, one of [text_completion, chat.completion, list (of embeddings)]. outputs: type: array @@ -3447,7 +3836,7 @@ components: behind the same endpoint with traffic split. usage: $ref: '#/components/schemas/ExternalModelUsageElement' - description: The usage object that may be returned by the __external/foundation model__ serving endpoint. This contains + description: The usage object that may be returned by the **external/foundation model** serving endpoint. This contains information about the number of tokens used in the prompt and response. RateLimit: type: object @@ -3492,7 +3881,7 @@ components: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables - that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + that refer to Databricks secrets: ``{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' external_model: $ref: '#/components/schemas/ExternalModel' description: 'The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, @@ -3539,9 +3928,9 @@ components: is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified. workload_type: $ref: '#/components/schemas/ServingModelWorkloadType' - description: 'The workload type of the served entity. The workload type selects which type of compute to use in - the endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is - available by selecting workload types like GPU_SMALL and others. See the available [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + description: The workload type of the served entity. The workload type selects which type of compute to use in the + endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is available + by selecting workload types like GPU_SMALL and others. See the available `GPU types `__. ServedEntityOutput: type: object properties: @@ -3562,7 +3951,7 @@ components: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables - that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + that refer to Databricks secrets: ``{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' external_model: $ref: '#/components/schemas/ExternalModel' description: 'The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, @@ -3613,9 +4002,9 @@ components: is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified. workload_type: $ref: '#/components/schemas/ServingModelWorkloadType' - description: 'The workload type of the served entity. The workload type selects which type of compute to use in - the endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is - available by selecting workload types like GPU_SMALL and others. See the available [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + description: The workload type of the served entity. The workload type selects which type of compute to use in the + endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is available + by selecting workload types like GPU_SMALL and others. See the available `GPU types `__. ServedEntitySpec: type: object properties: @@ -3646,7 +4035,7 @@ components: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables - that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + that refer to Databricks secrets: ``{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' instance_profile_arn: type: string description: ARN of the instance profile that the served entity uses to access AWS resources. @@ -3683,9 +4072,9 @@ components: is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified. workload_type: $ref: '#/components/schemas/ServedModelInputWorkloadType' - description: 'The workload type of the served entity. The workload type selects which type of compute to use in - the endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is - available by selecting workload types like GPU_SMALL and others. See the available [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + description: The workload type of the served entity. The workload type selects which type of compute to use in the + endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is available + by selecting workload types like GPU_SMALL and others. See the available `GPU types `__. required: - scale_to_zero_enabled - model_name @@ -3703,7 +4092,7 @@ components: type: object description: 'An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables - that refer to Databricks secrets: `{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}`' + that refer to Databricks secrets: ``{"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}``' instance_profile_arn: type: string description: ARN of the instance profile that the served entity uses to access AWS resources. @@ -3743,9 +4132,9 @@ components: is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified. workload_type: $ref: '#/components/schemas/ServingModelWorkloadType' - description: 'The workload type of the served entity. The workload type selects which type of compute to use in - the endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is - available by selecting workload types like GPU_SMALL and others. See the available [GPU types]. [GPU types]: https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html#gpu-workload-types' + description: The workload type of the served entity. The workload type selects which type of compute to use in the + endpoint. The default value for this parameter is "CPU". For deep learning workloads, GPU acceleration is available + by selecting workload types like GPU_SMALL and others. See the available `GPU types `__. ServedModelSpec: type: object properties: @@ -3810,6 +4199,13 @@ components: task: type: string description: The task type of the serving endpoint. + telemetry_config: + $ref: '#/components/schemas/TelemetryConfig' + description: Telemetry configuration for the endpoint, including inference-table payload logging. + uc_system_metrics_export_state: + $ref: '#/components/schemas/FirstPartyExportSetting' + description: Whether this endpoint's system metrics are exported to the system.telemetry.otel_metrics Unity Catalog + table. Endpoint-level property (not part of the per-version config). usage_policy_id: type: string description: The usage policy associated with serving endpoint. @@ -3903,6 +4299,13 @@ components: task: type: string description: The task type of the serving endpoint. + telemetry_config: + $ref: '#/components/schemas/TelemetryConfig' + description: Telemetry configuration for the endpoint, including inference-table payload logging. + uc_system_metrics_export_state: + $ref: '#/components/schemas/FirstPartyExportSetting' + description: Whether this endpoint's system metrics are exported to the system.telemetry.otel_metrics Unity Catalog + table. Endpoint-level property (not part of the per-version config). ServingEndpointPermission: type: object properties: @@ -3932,6 +4335,29 @@ components: type: string permission_level: $ref: '#/components/schemas/ServingEndpointPermissionLevel' + TelemetryConfig: + type: object + properties: + inference_table_config: + $ref: '#/components/schemas/TelemetryInferenceTableConfig' + table_names: + $ref: '#/components/schemas/UnityCatalogTableNames' + description: The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide + this to create a new telemetry profile for the endpoint from the given tables. + telemetry_profile_id: + type: string + description: The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry + profile that has already been created, instead of specifying table_names. + TelemetryInferenceTableConfig: + type: object + properties: + name: + type: string + description: The full name of the inference table created for this endpoint. + sampling_fraction: + type: number + description: Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests. + description: Inference table payload logging configuration TrafficConfig: type: object properties: @@ -3939,6 +4365,23 @@ components: type: array items: $ref: '#/components/schemas/Route' + UnityCatalogTableNames: + type: object + properties: + annotations_table: + type: string + logs_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported + logs. + metrics_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported + metrics. + traces_table: + type: string + description: The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported + traces (spans). UpdateInferenceEndpointNotificationsResponse: type: object properties: @@ -3953,16 +4396,16 @@ components: type: string index: type: integer - description: The index of the choice in the __chat or completions__ response. + description: The index of the choice in the **chat or completions** response. logprobs: type: integer - description: The logprobs returned only by the __completions__ endpoint. + description: The logprobs returned only by the **completions** endpoint. message: $ref: '#/components/schemas/ChatMessage' - description: The message response from the __chat__ endpoint. + description: The message response from the **chat** endpoint. text: type: string - description: The text response from the __completions__ endpoint. + description: The text response from the **completions** endpoint. AiGatewayGuardrailPiiBehaviorBehavior: type: string x-enum: @@ -4005,6 +4448,7 @@ components: - amazon - anthropic - cohere + - grok description: "Create a collection of name/value pairs.\n\nExample enumeration:\n\n>>> class Color(Enum):\n... RED\ \ = 1\n... BLUE = 2\n... GREEN = 3\n\nAccess them by:\n\n- attribute access:\n\n >>> Color.RED\n \n\n- value lookup:\n\n >>> Color(1)\n \n\n- name lookup:\n\n >>> Color['RED']\n \n\nEnumerations can be iterated over, and know how many members they have:\n\n>>> len(Color)\n3\n\n>>> list(Color)\n\ [, , ]\n\nMethods can be added to enumerations, and members can have\ \ their own\nattributes -- see the documentation for details." + FirstPartyExportSetting: + type: string + x-enum: + - FIRST_PARTY_EXPORT_DISABLED + - FIRST_PARTY_EXPORT_ENABLED + description: 'Whether first-party system-metrics export (to system.telemetry.otel_metrics) is enabled for an + + endpoint. Scoped to first-party only for now; third-party (e.g. Datadog) export is deferred.' QueryEndpointResponseObject: type: string x-enum: - chat.completion - list - text_completion - description: 'The type of object returned by the __external/foundation model__ serving endpoint, one of + description: 'The type of object returned by the **external/foundation model** serving endpoint, one of [text_completion, chat.completion, list (of embeddings)].' RateLimitKey: @@ -4113,11 +4577,15 @@ components: type: string x-enum: - CPU + - CPU_LARGE + - CPU_MEDIUM - GPU_LARGE - GPU_MEDIUM - GPU_SMALL + - GPU_XLARGE + - GPU_XLARGE_8 - MULTIGPU_MEDIUM - description: Please keep this in sync with with workload types in InferenceEndpointEntities.scala + description: Please keep this in sync with workload types in InferenceEndpointEntities.scala. ServedModelStateDeployment: type: string x-enum: @@ -4155,11 +4623,15 @@ components: type: string x-enum: - CPU + - CPU_LARGE + - CPU_MEDIUM - GPU_LARGE - GPU_MEDIUM - GPU_SMALL + - GPU_XLARGE + - GPU_XLARGE_8 - MULTIGPU_MEDIUM - description: Please keep this in sync with with workload types in InferenceEndpointEntities.scala + description: Please keep this in sync with workload types in InferenceEndpointEntities.scala. x-stackQL-resources: serving_endpoint_build_logs: id: databricks_workspace.serving.serving_endpoint_build_logs @@ -4221,6 +4693,24 @@ components: response: mediaType: application/json openAPIDocKey: '200' + patch_system_metrics_export_state: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1serving-endpoints~1{name}~1system-metrics-export-state/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + patch_telemetry_config: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1serving-endpoints~1{name}~1telemetry-config/patch' + response: + mediaType: application/json + openAPIDocKey: '200' query: config: requestBodyTranslate: diff --git a/providers/src/databricks_workspace/v00.00.00000/services/settings.yaml b/providers/src/databricks_workspace/v00.00.00000/services/settings.yaml index 2b38d3f9..13a398cc 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/settings.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/settings.yaml @@ -4,8 +4,8 @@ info: description: OpenAPI specification for the Databricks settings service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.settings servers: - url: https://{deployment_name}.cloud.databricks.com @@ -124,14 +124,14 @@ paths: :param setting: :class:`AibiDashboardEmbeddingAccessPolicySetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`AibiDashboardEmbeddingAccessPolicySetting` requestBody: @@ -212,11 +212,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -351,14 +351,14 @@ paths: :param setting: :class:`AibiDashboardEmbeddingApprovedDomainsSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`AibiDashboardEmbeddingApprovedDomainsSetting` requestBody: @@ -398,11 +398,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -482,23 +482,23 @@ paths: - automatic_cluster_update description: |- Updates the automatic cluster update setting for the workspace. A fresh etag needs to be provided in - `PATCH` requests (as part of the setting field). The etag can be retrieved by making a `GET` request - before the `PATCH` request. If the setting is updated concurrently, `PATCH` fails with 409 and the - request must be retried by using the fresh etag in the 409 response. + ``PATCH`` requests (as part of the setting field). The etag can be retrieved by making a ``GET`` + request before the ``PATCH`` request. If the setting is updated concurrently, ``PATCH`` fails with 409 + and the request must be retried by using the fresh etag in the 409 response. :param allow_missing: bool This should always be set to true for Settings API. Added for AIP compliance. :param setting: :class:`AutomaticClusterUpdateSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`AutomaticClusterUpdateSetting` requestBody: @@ -665,11 +665,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -749,23 +749,23 @@ paths: - compliance_security_profile description: |- Updates the compliance security profile setting for the workspace. A fresh etag needs to be provided - in `PATCH` requests (as part of the setting field). The etag can be retrieved by making a `GET` - request before the `PATCH` request. If the setting is updated concurrently, `PATCH` fails with 409 and - the request must be retried by using the fresh etag in the 409 response. + in ``PATCH`` requests (as part of the setting field). The etag can be retrieved by making a ``GET`` + request before the ``PATCH`` request. If the setting is updated concurrently, ``PATCH`` fails with 409 + and the request must be retried by using the fresh etag in the 409 response. :param allow_missing: bool This should always be set to true for Settings API. Added for AIP compliance. :param setting: :class:`ComplianceSecurityProfileSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`ComplianceSecurityProfileSetting` requestBody: @@ -788,6 +788,7 @@ paths: items: type: string x-enum: + - ARC_AMPE - CANADA_PROTECTED_B - CYBER_ESSENTIAL_PLUS - FEDRAMP_HIGH @@ -800,16 +801,20 @@ paths: - IRAP_PROTECTED - ISMAP - ITAR_EAR + - KSA_ECC_CCC_DCC - K_FSI - NONE - PCI_DSS description: |- Compliance standard for SHIELD customers. See README.md for how instructions of how to add new standards. - description: Set by customers when they request Compliance Security Profile (CSP) + description: Compliance standards selected by the customer for this Compliance Security Profile. is_enabled: type: boolean - description: 'SHIELD feature: CSP' + description: Whether Compliance Security Profile (CSP) is enabled on the workspace. + description: |- + SHIELD feature: CSP Compliance Security Profile (CSP) enables enhanced compliance controls on + the workspace. etag: type: string description: etag used for versioning. The response is at least as fresh as the eTag provided. This @@ -828,11 +833,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -867,6 +872,15 @@ paths: Exchange tokens with an Identity Provider to get a new access token. It allows specifying scopes to determine token permissions. + POST /exchange-tokens/token is the documented public form, expressed via ``google.api.http`` below. + GET /exchange-tokens/$exchange is a legacy alias used by the Spark driver's OAuth refresh path + (DBHttpClient#get sends a body via HttpGetWithEntity) and stays on the legacy ``option + (rpc).endpoints`` annotation: its path contains a literal ``$``, which ``google.api.http``'s LITERAL + grammar does not allow, and ``HttpPathParser`` does not percent-decode template segments (so encoding + as ``%24exchange`` would not match the literal ``$exchange`` path the Spark driver sends). + Per-endpoint ``visibility: PUBLIC_UNDOCUMENTED`` preserves the DECO-7732 intent of suppressing the GET + alias from the public API spec. + :param partition_id: :class:`PartitionId` The partition of Credentials store :param token_type: List[:class:`TokenType`] @@ -898,7 +912,7 @@ paths: - ARCLIGHT_MULTI_TENANT_AZURE_EXCHANGE_TOKEN - ARCLIGHT_MULTI_TENANT_AZURE_EXCHANGE_TOKEN_WITH_USER_DELEGATION_KEY - AZURE_ACTIVE_DIRECTORY_TOKEN - description: The type of token request. As of now, only `AZURE_ACTIVE_DIRECTORY_TOKEN` is supported. + description: The type of token request. As of now, only ``AZURE_ACTIVE_DIRECTORY_TOKEN`` is supported. description: A list of token types being requested scopes: type: array @@ -995,14 +1009,14 @@ paths: :param setting: :class:`CspEnablementAccountSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`CspEnablementAccountSetting` parameters: @@ -1031,6 +1045,7 @@ paths: items: type: string x-enum: + - ARC_AMPE - CANADA_PROTECTED_B - CYBER_ESSENTIAL_PLUS - FEDRAMP_HIGH @@ -1043,6 +1058,7 @@ paths: - IRAP_PROTECTED - ISMAP - ITAR_EAR + - KSA_ECC_CCC_DCC - K_FSI - NONE - PCI_DSS @@ -1073,11 +1089,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -1210,14 +1226,14 @@ paths: :param setting: :class:`DashboardEmailSubscriptions` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`DashboardEmailSubscriptions` requestBody: @@ -1255,11 +1271,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -1286,15 +1302,15 @@ paths: /api/2.0/settings/types/default_namespace_ws/names/default: delete: operationId: default_namespace_delete - summary: Deletes the default namespace setting for the workspace. A fresh etag needs to be provided in `DELETE` + summary: Deletes the default namespace setting for the workspace. A fresh etag needs to be provided in tags: - settings - default_namespace description: |- - Deletes the default namespace setting for the workspace. A fresh etag needs to be provided in `DELETE` - requests (as a query parameter). The etag can be retrieved by making a `GET` request before the - `DELETE` request. If the setting is updated/deleted concurrently, `DELETE` fails with 409 and the - request must be retried by using the fresh etag in the 409 response. + Deletes the default namespace setting for the workspace. A fresh etag needs to be provided in + ``DELETE`` requests (as a query parameter). The etag can be retrieved by making a ``GET`` request + before the ``DELETE`` request. If the setting is updated/deleted concurrently, ``DELETE`` fails with + 409 and the request must be retried by using the fresh etag in the 409 response. :param etag: str (optional) etag used for versioning. The response is at least as fresh as the eTag provided. This is used for @@ -1383,31 +1399,31 @@ paths: x-stackql-sdk-source: DefaultNamespaceAPI patch: operationId: default_namespace_update - summary: Updates the default namespace setting for the workspace. A fresh etag needs to be provided in `PATCH` + summary: Updates the default namespace setting for the workspace. A fresh etag needs to be provided in tags: - settings - default_namespace description: |- - Updates the default namespace setting for the workspace. A fresh etag needs to be provided in `PATCH` - requests (as part of the setting field). The etag can be retrieved by making a `GET` request before - the `PATCH` request. Note that if the setting does not exist, `GET` returns a NOT_FOUND error and the - etag is present in the error response, which should be set in the `PATCH` request. If the setting is - updated concurrently, `PATCH` fails with 409 and the request must be retried by using the fresh etag - in the 409 response. + Updates the default namespace setting for the workspace. A fresh etag needs to be provided in + ``PATCH`` requests (as part of the setting field). The etag can be retrieved by making a ``GET`` + request before the ``PATCH`` request. Note that if the setting does not exist, ``GET`` returns a + NOT_FOUND error and the etag is present in the error response, which should be set in the ``PATCH`` + request. If the setting is updated concurrently, ``PATCH`` fails with 409 and the request must be + retried by using the fresh etag in the 409 response. :param allow_missing: bool This should always be set to true for Settings API. Added for AIP compliance. :param setting: :class:`DefaultNamespaceSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`DefaultNamespaceSetting` requestBody: @@ -1453,11 +1469,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -1590,14 +1606,14 @@ paths: :param setting: :class:`DefaultWarehouseId` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`DefaultWarehouseId` requestBody: @@ -1635,11 +1651,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -1772,14 +1788,14 @@ paths: :param setting: :class:`DisableLegacyAccess` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`DisableLegacyAccess` requestBody: @@ -1817,11 +1833,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -1954,14 +1970,14 @@ paths: :param setting: :class:`DisableLegacyDbfs` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`DisableLegacyDbfs` requestBody: @@ -1999,11 +2015,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2146,14 +2162,14 @@ paths: :param setting: :class:`DisableLegacyFeatures` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`DisableLegacyFeatures` parameters: @@ -2197,11 +2213,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2271,14 +2287,14 @@ paths: :param setting: :class:`EnableExportNotebook` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`EnableExportNotebook` requestBody: @@ -2307,11 +2323,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2454,14 +2470,14 @@ paths: :param setting: :class:`AccountIpAccessEnable` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`AccountIpAccessEnable` parameters: @@ -2505,11 +2521,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2579,14 +2595,14 @@ paths: :param setting: :class:`EnableNotebookTableClipboard` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`EnableNotebookTableClipboard` requestBody: @@ -2615,11 +2631,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2689,14 +2705,14 @@ paths: :param setting: :class:`EnableResultsDownloading` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`EnableResultsDownloading` requestBody: @@ -2725,11 +2741,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2809,23 +2825,23 @@ paths: - enhanced_security_monitoring description: |- Updates the enhanced security monitoring setting for the workspace. A fresh etag needs to be provided - in `PATCH` requests (as part of the setting field). The etag can be retrieved by making a `GET` - request before the `PATCH` request. If the setting is updated concurrently, `PATCH` fails with 409 and - the request must be retried by using the fresh etag in the 409 response. + in ``PATCH`` requests (as part of the setting field). The etag can be retrieved by making a ``GET`` + request before the ``PATCH`` request. If the setting is updated concurrently, ``PATCH`` fails with 409 + and the request must be retried by using the fresh etag in the 409 response. :param allow_missing: bool This should always be set to true for Settings API. Added for AIP compliance. :param setting: :class:`EnhancedSecurityMonitoringSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`EnhancedSecurityMonitoringSetting` requestBody: @@ -2845,7 +2861,10 @@ paths: properties: is_enabled: type: boolean - description: 'SHIELD feature: ESM' + description: Whether Enhanced Security Monitoring (ESM) is enabled on the workspace. + description: |- + SHIELD feature: ESM Enhanced Security Monitoring (ESM) enables additional security monitoring on + the workspace. etag: type: string description: etag used for versioning. The response is at least as fresh as the eTag provided. This @@ -2864,11 +2883,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -2959,14 +2978,14 @@ paths: :param setting: :class:`EsmEnablementAccountSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`EsmEnablementAccountSetting` parameters: @@ -3011,11 +3030,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -3054,13 +3073,15 @@ paths: When creating or updating an IP access list: - * For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, - where one CIDR counts as a single value. Attempts to exceed that number return error 400 with - `error_code` value `QUOTA_EXCEEDED`. * If the new list would block the calling user's current IP, - error 400 is returned with `error_code` value `INVALID_STATE`. + - For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, + where one CIDR counts as a single value. Attempts to exceed that number return error 400 with + ``error_code`` value ``QUOTA_EXCEEDED``. + - If the new list would block the calling user's current IP, error 400 is returned with ``error_code`` + value ``INVALID_STATE``. It can take a few minutes for the changes to take effect. **Note**: Your new IP access list has no - effect until you enable the feature. See :method:workspaceconf/setStatus + effect until you enable the feature. See `workspaceconf/setStatus + `__ :param label: str Label for the IP access list. This **cannot** be empty. @@ -3085,8 +3106,9 @@ paths: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. ip_addresses: type: array items: @@ -3226,13 +3248,15 @@ paths: Replaces an IP access list, specified by its ID. A list can include allow lists and block lists. See the top of this file for a description of how the - server treats allow lists and block lists at run time. When replacing an IP access list: * For all - allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, where one - CIDR counts as a single value. Attempts to exceed that number return error 400 with `error_code` value - `QUOTA_EXCEEDED`. * If the resulting list would block the calling user's current IP, error 400 is - returned with `error_code` value `INVALID_STATE`. It can take a few minutes for the changes to take - effect. Note that your resulting IP access list has no effect until you enable the feature. See - :method:workspaceconf/setStatus. + server treats allow lists and block lists at run time. When replacing an IP access list: + + - For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, + where one CIDR counts as a single value. Attempts to exceed that number return error 400 with + ``error_code`` value ``QUOTA_EXCEEDED``. + - If the resulting list would block the calling user's current IP, error 400 is returned with + ``error_code`` value ``INVALID_STATE``. It can take a few minutes for the changes to take effect. + Note that your resulting IP access list has no effect until you enable the feature. See + `workspaceconf/setStatus `__. :param ip_access_list_id: str The ID for the corresponding IP access list @@ -3266,8 +3290,9 @@ paths: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. enabled: type: boolean description: Specifies whether this IP access list is enabled. @@ -3308,13 +3333,15 @@ paths: When updating an IP access list: - * For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, - where one CIDR counts as a single value. Attempts to exceed that number return error 400 with - `error_code` value `QUOTA_EXCEEDED`. * If the updated list would block the calling user's current IP, - error 400 is returned with `error_code` value `INVALID_STATE`. + - For all allow lists and block lists combined, the API supports a maximum of 1000 IP/CIDR values, + where one CIDR counts as a single value. Attempts to exceed that number return error 400 with + ``error_code`` value ``QUOTA_EXCEEDED``. + - If the updated list would block the calling user's current IP, error 400 is returned with + ``error_code`` value ``INVALID_STATE``. It can take a few minutes for the changes to take effect. Note that your resulting IP access list has - no effect until you enable the feature. See :method:workspaceconf/setStatus. + no effect until you enable the feature. See `workspaceconf/setStatus + `__. :param ip_access_list_id: str The ID for the corresponding IP access list @@ -3355,8 +3382,9 @@ paths: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. responses: '200': description: Success @@ -3439,14 +3467,14 @@ paths: :param setting: :class:`LlmProxyPartnerPoweredAccount` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`LlmProxyPartnerPoweredAccount` parameters: @@ -3490,11 +3518,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -3585,14 +3613,14 @@ paths: :param setting: :class:`LlmProxyPartnerPoweredEnforce` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`LlmProxyPartnerPoweredEnforce` parameters: @@ -3636,11 +3664,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -3773,14 +3801,14 @@ paths: :param setting: :class:`LlmProxyPartnerPoweredWorkspace` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`LlmProxyPartnerPoweredWorkspace` requestBody: @@ -3818,11 +3846,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -3881,8 +3909,41 @@ paths: generic_webhook: type: object properties: + oauth_audience: + type: string + oauth_audience_set: + type: boolean + description: '[Output-Only] Whether OAuth2 audience is set.' + oauth_client_id: + type: string + description: '[Input-Only][Optional] OAuth2 client ID for the client_credentials grant.' + oauth_client_id_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client ID is set.' + oauth_client_secret: + type: string + description: '[Input-Only][Optional] OAuth2 client secret for the client_credentials grant.' + oauth_client_secret_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client secret is set.' + oauth_scopes: + type: array + items: + type: string + description: '[Input-Only][Optional] OAuth2 scopes requested at token mint (required by some IdPs).' + oauth_scopes_set: + type: boolean + description: '[Output-Only] Whether OAuth2 scopes are set.' + oauth_token_url: + type: string + description: '[Input-Only][Optional] OAuth2 token endpoint URL (customer IdP) where access tokens + are minted.' + oauth_token_url_set: + type: boolean + description: '[Output-Only] Whether OAuth2 token URL is set.' password: type: string + description: '[Input-Only][Optional] Password for webhook.' password_set: type: boolean description: '[Output-Only] Whether password is set.' @@ -4137,8 +4198,41 @@ paths: generic_webhook: type: object properties: + oauth_audience: + type: string + oauth_audience_set: + type: boolean + description: '[Output-Only] Whether OAuth2 audience is set.' + oauth_client_id: + type: string + description: '[Input-Only][Optional] OAuth2 client ID for the client_credentials grant.' + oauth_client_id_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client ID is set.' + oauth_client_secret: + type: string + description: '[Input-Only][Optional] OAuth2 client secret for the client_credentials grant.' + oauth_client_secret_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client secret is set.' + oauth_scopes: + type: array + items: + type: string + description: '[Input-Only][Optional] OAuth2 scopes requested at token mint (required by some IdPs).' + oauth_scopes_set: + type: boolean + description: '[Output-Only] Whether OAuth2 scopes are set.' + oauth_token_url: + type: string + description: '[Input-Only][Optional] OAuth2 token endpoint URL (customer IdP) where access tokens + are minted.' + oauth_token_url_set: + type: boolean + description: '[Output-Only] Whether OAuth2 token URL is set.' password: type: string + description: '[Input-Only][Optional] Password for webhook.' password_set: type: boolean description: '[Output-Only] Whether password is set.' @@ -4356,14 +4450,14 @@ paths: :param setting: :class:`PersonalComputeSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`PersonalComputeSetting` parameters: @@ -4418,11 +4512,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -4455,9 +4549,9 @@ paths: - restrict_workspace_admins description: |- Reverts the restrict workspace admins setting status for the workspace. A fresh etag needs to be - provided in `DELETE` requests (as a query parameter). The etag can be retrieved by making a `GET` - request before the DELETE request. If the setting is updated/deleted concurrently, `DELETE` fails with - 409 and the request must be retried by using the fresh etag in the 409 response. + provided in ``DELETE`` requests (as a query parameter). The etag can be retrieved by making a ``GET`` + request before the DELETE request. If the setting is updated/deleted concurrently, ``DELETE`` fails + with 409 and the request must be retried by using the fresh etag in the 409 response. :param etag: str (optional) etag used for versioning. The response is at least as fresh as the eTag provided. This is used for @@ -4552,8 +4646,8 @@ paths: - restrict_workspace_admins description: |- Updates the restrict workspace admins setting for the workspace. A fresh etag needs to be provided in - `PATCH` requests (as part of the setting field). The etag can be retrieved by making a GET request - before the `PATCH` request. If the setting is updated concurrently, `PATCH` fails with 409 and the + ``PATCH`` requests (as part of the setting field). The etag can be retrieved by making a GET request + before the ``PATCH`` request. If the setting is updated concurrently, ``PATCH`` fails with 409 and the request must be retried by using the fresh etag in the 409 response. :param allow_missing: bool @@ -4561,14 +4655,14 @@ paths: :param setting: :class:`RestrictWorkspaceAdminsSetting` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`RestrictWorkspaceAdminsSetting` requestBody: @@ -4652,11 +4746,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -4789,14 +4883,14 @@ paths: :param setting: :class:`SqlResultsDownload` :param field_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`SqlResultsDownload` requestBody: @@ -4834,11 +4928,11 @@ paths: field_mask: type: string description: The field mask must be a single string, with multiple fields separated by commas (no spaces). - The field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., `author.given_name`). + The field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field - can be specified. Field names must exactly match the resource field names. A field mask of `*` indicates + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the fields being updated and avoid using - `*` wildcards, as it can lead to unintended results if the API changes in the future. + ``*`` wildcards, as it can lead to unintended results if the API changes in the future. required: - allow_missing - setting @@ -4874,10 +4968,13 @@ paths: :param application_id: str Application ID of the service principal. + :param autoscope_enabled: bool (optional) + Whether to enable autoscoping for this token. :param comment: str (optional) Comment that describes the purpose of the token. :param lifetime_seconds: int (optional) The number of seconds before the token expires. + :param scopes: List[str] (optional) :returns: :class:`CreateOboTokenResponse` requestBody: @@ -4889,12 +4986,19 @@ paths: application_id: type: string description: Application ID of the service principal. + autoscope_enabled: + type: boolean + description: Whether to enable autoscoping for this token. comment: type: string description: Comment that describes the purpose of the token. lifetime_seconds: type: integer description: The number of seconds before the token expires. + scopes: + type: array + items: + type: string required: - application_id responses: @@ -4989,6 +5093,123 @@ paths: message: type: string x-stackql-sdk-source: TokenManagementAPI + patch: + operationId: token_management_update_token_management + summary: Updates a token, specified by its ID. + tags: + - settings + - token_management + description: |- + Updates a token, specified by its ID. + + :param token_id: str + ID of the token. + :param token: :class:`TokenInfo` + :param update_mask: FieldMask + A list of field name under token, For example, {"update_mask": "comment,scopes"} + + The field mask must be a single string, with multiple fields separated by commas (no spaces). The + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only + the entire collection field can be specified. Field names must exactly match the resource field + names. + + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. + + :returns: :class:`TokenInfo` + parameters: + - name: token_id + in: path + required: true + schema: + type: string + description: ID of the token. + requestBody: + content: + application/json: + schema: + type: object + properties: + token: + type: object + properties: + autoscope_state: + type: string + backfill_scopes: + type: array + items: + type: string + description: Output only. Scopes inferred from offline backfill processing. + comment: + type: string + description: Comment that describes the purpose of the token, specified by the token creator. + created_by_id: + type: integer + description: User ID of the user that created the token. + created_by_username: + type: string + description: Username of the user that created the token. + creation_time: + type: integer + description: Timestamp when the token was created. + expiry_time: + type: integer + description: Timestamp when the token expires. + inferred_scopes: + type: array + items: + type: string + description: Output only. Inferred API path scopes collected for this token when autoscope is enabled. + last_used_day: + type: integer + description: Approximate timestamp for the day the token was last used. Accurate up to 1 day. + owner_id: + type: integer + description: User ID of the user that owns the token. + scopes: + type: array + items: + type: string + description: Scope of the token was created with, if applicable. + token_id: + type: string + description: ID of the token. + workspace_id: + type: integer + description: If applicable, the ID of the workspace that the token was created in. + update_mask: + type: object + description: 'A list of field name under token, For example, {"update_mask": "comment,scopes"} The field + mask must be a single string, with multiple fields separated by commas (no spaces). The field path is + relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). + Specification of elements in sequence or map fields is not allowed, as only the entire collection field + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates + full replacement. It’s recommended to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API changes in the future.' + required: + - token + - update_mask + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/TokenInfo' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: TokenManagementAPI /api/2.0/permissions/authorization/tokens/permissionLevels: get: operationId: token_management_get_permission_levels @@ -5222,12 +5443,17 @@ paths: a token with the same client ID as the authenticated token. If the user's token quota is exceeded, this call returns an error **QUOTA_EXCEEDED**. + :param autoscope_enabled: bool (optional) + Whether to enable autoscoping for this token. When true, the token will automatically collect + inferred API path scopes as it is used. :param comment: str (optional) Optional description to attach to the token. :param lifetime_seconds: int (optional) The lifetime of the token, in seconds. If the lifetime is not specified, this token remains valid for 2 years. + :param scopes: List[str] (optional) + Optional scopes of the token. :returns: :class:`CreateTokenResponse` requestBody: @@ -5236,6 +5462,10 @@ paths: schema: type: object properties: + autoscope_enabled: + type: boolean + description: Whether to enable autoscoping for this token. When true, the token will automatically collect + inferred API path scopes as it is used. comment: type: string description: Optional description to attach to the token. @@ -5243,6 +5473,11 @@ paths: type: integer description: The lifetime of the token, in seconds. If the lifetime is not specified, this token remains valid for 2 years. + scopes: + type: array + items: + type: string + description: Optional scopes of the token. responses: '200': description: Success @@ -5333,6 +5568,114 @@ paths: message: type: string x-stackql-sdk-source: TokensAPI + /api/2.0/token/{token_id}: + patch: + operationId: tokens_update + summary: Updates the comment or scopes of a token. + tags: + - settings + - tokens + description: |- + Updates the comment or scopes of a token. + + If a token with the specified ID is not valid, this call returns an error **NOT_FOUND**. + + :param token_id: str + The SHA-256 hash of the token to be updated. + :param token: :class:`PublicTokenInfo` + :param update_mask: FieldMask + A list of field name under token, For example, {"update_mask": "comment,scopes"} + + The field mask must be a single string, with multiple fields separated by commas (no spaces). The + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only + the entire collection field can be specified. Field names must exactly match the resource field + names. + + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. + + :returns: :class:`UpdateTokenResponse` + parameters: + - name: token_id + in: path + required: true + schema: + type: string + description: The SHA-256 hash of the token to be updated. + requestBody: + content: + application/json: + schema: + type: object + properties: + token: + type: object + properties: + autoscope_state: + type: string + backfill_scopes: + type: array + items: + type: string + description: Output only. Scopes inferred from offline backfill processing. + comment: + type: string + description: Comment the token was created with, if applicable. + creation_time: + type: integer + description: Server time (in epoch milliseconds) when the token was created. + expiry_time: + type: integer + description: Server time (in epoch milliseconds) when the token will expire, or -1 if not applicable. + inferred_scopes: + type: array + items: + type: string + description: Output only. Inferred API path scopes collected for this token when autoscope is enabled. + last_accessed_time: + type: integer + description: Server time (in epoch milliseconds) when the token was accessed most recently. + scopes: + type: array + items: + type: string + description: Scope of the token was created with, if applicable. + token_id: + type: string + description: The ID of this token. + update_mask: + type: object + description: 'A list of field name under token, For example, {"update_mask": "comment,scopes"} The field + mask must be a single string, with multiple fields separated by commas (no spaces). The field path is + relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., ``author.given_name``). + Specification of elements in sequence or map fields is not allowed, as only the entire collection field + can be specified. Field names must exactly match the resource field names. A field mask of ``*`` indicates + full replacement. It’s recommended to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API changes in the future.' + required: + - token + - update_mask + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateTokenResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: TokensAPI /api/2.0/workspace-conf: get: operationId: workspace_conf_get_status @@ -5587,6 +5930,13 @@ components: egress: $ref: '#/components/schemas/NetworkPolicyEgress' description: The network policies applying for egress traffic. + ingress: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicy' + description: The network policies applying for ingress traffic. + ingress_dry_run: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicy' + description: The ingress policy for dry run mode. Dry run will always run even if the request is allowed by the + ingress policy. When this field is set, the policy will be evaluated and emit logs only without blocking requests. network_policy_id: type: string description: The unique identifier for the network policy. @@ -5725,10 +6075,13 @@ components: type: array items: $ref: '#/components/schemas/ComplianceStandard' - description: Set by customers when they request Compliance Security Profile (CSP) + description: Compliance standards selected by the customer for this Compliance Security Profile. is_enabled: type: boolean - description: 'SHIELD feature: CSP' + description: Whether Compliance Security Profile (CSP) is enabled on the workspace. + description: |- + SHIELD feature: CSP Compliance Security Profile (CSP) enables enhanced compliance controls on + the workspace. ComplianceSecurityProfileSetting: type: object properties: @@ -5806,6 +6159,8 @@ components: endpoint. error_message: type: string + gcp_endpoint: + $ref: '#/components/schemas/GcpEndpoint' group_id: type: string description: Not used by customer-managed private endpoint services. The sub-resource type (group ID) of the target @@ -5865,90 +6220,398 @@ components: name is required to be 'default' if the setting only has one instance per workspace. required: - csp_enablement_account - CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRule: + CustomerFacingIngressNetworkPolicy: type: object properties: - account_id: - type: string - description: Databricks account ID. You can find your account ID from the Accounts Console. - connection_state: - $ref: '#/components/schemas/CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRulePrivateLinkConnectionState' - creation_time: - type: integer - description: Time in epoch milliseconds when this object was created. - deactivated: - type: boolean - description: Whether this private endpoint is deactivated. - deactivated_at: - type: integer - description: Time in epoch milliseconds when this object was deactivated. - domain_names: + cross_workspace_access: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceAccess' + private_access: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateAccess' + description: The network policy restrictions for private access. Configures how requests arriving over private connectivity + are governed. + public_access: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicAccess' + description: The network policy restrictions for public access to the workspace. Configures how public internet + traffic is allowed or denied access. + description: The network policies applying for ingress traffic. + CustomerFacingIngressNetworkPolicyAccountApiDestination: + type: object + properties: + excluded_scopes: type: array items: type: string - description: Only used by private endpoints towards a VPC endpoint service for customer-managed VPC endpoint service. - The target AWS resource FQDNs accessible via the VPC endpoint service. When updating this field, we perform full - update on this field. Please ensure a full list of desired domain_names is provided. - enabled: - type: boolean - description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate - this private endpoint to allow egress access from serverless compute resources. - endpoint_service: - type: string - description: The full target AWS endpoint service name that connects to the destination resources of the private - endpoint. - error_message: - type: string - network_connectivity_config_id: - type: string - description: The ID of a network connectivity configuration, which is the parent resource of this private endpoint - rule object. - resource_names: + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed here ("allow all except"). Mutually + exclusive with ``scopes`` — a single destination may set at most one of the two.' + scope_constraint: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeConstraint' + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeQualifier' + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: type: array items: type: string - description: Only used by private endpoints towards AWS S3 service. The globally unique S3 bucket names that will - be accessed via the VPC endpoint. The bucket names must be in the same region as the NCC/endpoint service. When - updating this field, we perform full update on this field. Please ensure a full list of desired resource_names - is provided. - rule_id: - type: string - description: The ID of a private endpoint rule. - updated_time: + description: The API scopes to match. Use "all-apis" to match any account-level API. + description: Matches account-level Databricks API endpoints for an ingress network policy rule. + CustomerFacingIngressNetworkPolicyAccountDatabricksOneDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingIngressNetworkPolicyAccountUiDestination: + type: object + properties: + all_destinations: + type: boolean + description: Must be set to true. + description: The account console UI destination. + CustomerFacingIngressNetworkPolicyAppsRuntimeDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingIngressNetworkPolicyAuthentication: + type: object + properties: + identities: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthenticationIdentity' + identity_type: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthenticationIdentityType' + CustomerFacingIngressNetworkPolicyAuthenticationIdentity: + type: object + properties: + principal_id: type: integer - description: Time in epoch milliseconds when this object was updated. - vpc_endpoint_id: - type: string - description: The AWS VPC endpoint ID. You can use this ID to identify VPC endpoint created by Databricks. - description: |- - Properties of the new private endpoint rule. Note that for private endpoints towards a VPC - endpoint service behind a customer-managed NLB, you must approve the endpoint in AWS console - after initialization. - DashboardEmailSubscriptions: + principal_type: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthenticationIdentityPrincipalType' + CustomerFacingIngressNetworkPolicyCrossWorkspaceAccess: type: object properties: - boolean_val: - $ref: '#/components/schemas/BooleanMessage' - etag: - type: string - description: etag used for versioning. The response is at least as fresh as the eTag provided. This is used for - optimistic concurrency control as a way to help prevent simultaneous writes of a setting overwriting each other. - It is strongly suggested that systems make use of the etag in the read -> update pattern to perform setting updates - in order to avoid race conditions. That is, get an etag from a GET request, and pass it with the PATCH request - to identify the setting version you are updating. - setting_name: - type: string - description: Name of the corresponding setting. This field is populated in the response, but it will not be respected - even if it's set in the request body. The setting name in the path parameter will be respected instead. Setting - name is required to be 'default' if the setting only has one instance per workspace. + restriction_mode: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceAccessRestrictionMode' + allow_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceIngressRule' + deny_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceIngressRule' required: - - boolean_val - DefaultNamespaceSetting: + - restriction_mode + CustomerFacingIngressNetworkPolicyCrossWorkspaceIngressRule: type: object properties: - namespace: - $ref: '#/components/schemas/StringMessage' - etag: + authentication: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthentication' + destination: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyRequestDestination' + label: + type: string + description: The label for this ingress rule. + origin: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyCrossWorkspaceRequestOrigin' + CustomerFacingIngressNetworkPolicyCrossWorkspaceRequestOrigin: + type: object + properties: + all_source_workspaces: + type: boolean + selected_workspaces: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyWorkspaceIdList' + description: Specific source workspace IDs to match. + CustomerFacingIngressNetworkPolicyEndpoints: + type: object + properties: + endpoint_ids: + type: array + items: + type: string + description: The IDs of the registered endpoints. Must contain at least one endpoint ID. + description: A set of registered endpoints, identified by their endpoint IDs. + CustomerFacingIngressNetworkPolicyIpRanges: + type: object + properties: + ip_ranges: + type: array + items: + type: string + CustomerFacingIngressNetworkPolicyLakebaseRuntimeDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingIngressNetworkPolicyManagedIpRange: + type: object + properties: + name: + type: string + description: Name of the managed IP range, e.g., "powerbi". + regions: + type: array + items: + type: string + description: Partner regions to allow, e.g., ["EastUS", "WestEurope"]. Each region must belong to the partner named + in ``name``. Empty list means all regions of the partner. Region casing matches the partner's source (e.g., Microsoft + uses "EastUS"), so values are stored verbatim, not lowercased. + description: |- + A Databricks-managed IP range identified by name (e.g., "powerbi", "dbt"), optionally scoped to + specific partner regions. Customers reference a name instead of raw CIDRs; Databricks keeps the + underlying IP list current. + CustomerFacingIngressNetworkPolicyPrivateAccess: + type: object + properties: + restriction_mode: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateAccessRestrictionMode' + description: The restriction mode for private access. + allow_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateIngressRule' + description: Allow rules are evaluated after deny rules. A request matching any allow rule is allowed; a request + matching no rule is denied by default. Only applies when restriction_mode is RESTRICTED_ACCESS. + deny_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateIngressRule' + description: Deny rules are evaluated first. A request matching any deny rule is denied, regardless of allow rules. + Only applies when restriction_mode is RESTRICTED_ACCESS. + required: + - restriction_mode + description: |- + Configures how requests arriving over private connectivity, such as registered endpoints, are + allowed or denied access. + CustomerFacingIngressNetworkPolicyPrivateIngressRule: + type: object + properties: + authentication: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthentication' + description: The authenticated identity the request must match. When unset, the rule matches all users and service + principals. On the account-level network policy, scoping to specific identities is not currently supported, so + this field must be unset (the rule matches all users and service principals). + destination: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyRequestDestination' + description: The destination the request must match — the resource being accessed, for example the workspace UI, + workspace APIs, or account-level APIs. See RequestDestination. + label: + type: string + description: The label for this ingress rule. + origin: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPrivateRequestOrigin' + description: The origin the request must match — the private connectivity the request arrives through, for example + a specific set of registered endpoints or any endpoint registered to the account. See PrivateRequestOrigin. + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + CustomerFacingIngressNetworkPolicyPrivateRequestOrigin: + type: object + properties: + all_private_access: + type: boolean + description: Matches requests arriving over any private connectivity, including registered endpoints and the workspace's + Azure Private Link (ui-api) endpoints. Can only be used in deny rules of workspace-level network policies. Must + be set to true when specified. + all_registered_endpoints: + type: boolean + description: Matches requests arriving through any endpoint registered to the account. Must be set to true when + specified. + azure_workspace_private_link: + type: boolean + description: Matches requests arriving through the workspace's Azure Private Link (ui-api) endpoints. Can only be + used in deny rules of workspace-level network policies. Must be set to true when specified. + endpoints: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyEndpoints' + description: Matches requests arriving through any of the specified registered endpoints. + description: |- + The origin of a private access request, identified by the endpoint through which the request + arrives. + CustomerFacingIngressNetworkPolicyPublicAccess: + type: object + properties: + restriction_mode: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicAccessRestrictionMode' + allow_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicIngressRule' + deny_rules: + type: array + items: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicIngressRule' + required: + - restriction_mode + CustomerFacingIngressNetworkPolicyPublicIngressRule: + type: object + properties: + authentication: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAuthentication' + destination: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyRequestDestination' + label: + type: string + description: The label for this ingress rule. + origin: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyPublicRequestOrigin' + description: |- + An ingress rule is enforced when a request satisfies all specified attributes — including + request origin, destination, and authentication. + CustomerFacingIngressNetworkPolicyPublicRequestOrigin: + type: object + properties: + all_ip_ranges: + type: boolean + excluded_ip_ranges: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyIpRanges' + description: 'Excluded means: all public IP ranges except this one.' + included_ip_ranges: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyIpRanges' + description: Will not allow IP ranges with private IPs. + managed_ip_range: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyManagedIpRange' + description: A single Databricks-maintained IP range for a well-known partner (e.g., Power BI, dbt), optionally + scoped to specific partner regions. Customers create one rule per partner. + CustomerFacingIngressNetworkPolicyRequestDestination: + type: object + properties: + account_api: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAccountApiDestination' + account_databricks_one: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAccountDatabricksOneDestination' + description: Account DatabricksOne destination is not supported. + account_ui: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAccountUiDestination' + description: Matches requests to the account console UI. Can only be used in the account-level network policy. + all_destinations: + type: boolean + description: When true, match all destinations, no other destination fields can be set. When not set or false, at + least one specific destination must be provided. + apps_runtime: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyAppsRuntimeDestination' + lakebase_runtime: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyLakebaseRuntimeDestination' + workspace_api: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyWorkspaceApiDestination' + workspace_ui: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyWorkspaceUiDestination' + CustomerFacingIngressNetworkPolicyWorkspaceApiDestination: + type: object + properties: + excluded_scopes: + type: array + items: + type: string + description: 'Inverse of ``scopes``: matches every API scope EXCEPT those listed here ("allow all except"). Mutually + exclusive with ``scopes`` — a single destination may set at most one of the two.' + scope_constraint: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeConstraint' + description: 'Deprecated: Use scope_qualifier instead.' + scope_qualifier: + $ref: '#/components/schemas/CustomerFacingIngressNetworkPolicyApiScopeQualifier' + description: Qualifies the breadth of API access for the listed scopes. See ApiScopeQualifier. + scopes: + type: array + items: + type: string + description: Matches workspace-level Databricks API endpoints for an ingress network policy rule. + CustomerFacingIngressNetworkPolicyWorkspaceIdList: + type: object + properties: + workspace_ids: + type: array + items: + type: integer + CustomerFacingIngressNetworkPolicyWorkspaceUiDestination: + type: object + properties: + all_destinations: + type: boolean + CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRule: + type: object + properties: + account_id: + type: string + description: Databricks account ID. You can find your account ID from the Accounts Console. + connection_state: + $ref: '#/components/schemas/CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRulePrivateLinkConnectionState' + creation_time: + type: integer + description: Time in epoch milliseconds when this object was created. + deactivated: + type: boolean + description: Whether this private endpoint is deactivated. + deactivated_at: + type: integer + description: Time in epoch milliseconds when this object was deactivated. + domain_names: + type: array + items: + type: string + description: Only used by private endpoints towards a VPC endpoint service for customer-managed VPC endpoint service. + The target AWS resource FQDNs accessible via the VPC endpoint service. When updating this field, we perform full + update on this field. Please ensure a full list of desired domain_names is provided. + enabled: + type: boolean + description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate + this private endpoint to allow egress access from serverless compute resources. + endpoint_service: + type: string + description: The full target AWS endpoint service name that connects to the destination resources of the private + endpoint. + error_message: + type: string + network_connectivity_config_id: + type: string + description: The ID of a network connectivity configuration, which is the parent resource of this private endpoint + rule object. + resource_names: + type: array + items: + type: string + description: Only used by private endpoints towards AWS S3 service. The globally unique S3 bucket names that will + be accessed via the VPC endpoint. The bucket names must be in the same region as the NCC/endpoint service. When + updating this field, we perform full update on this field. Please ensure a full list of desired resource_names + is provided. + rule_id: + type: string + description: The ID of a private endpoint rule. + updated_time: + type: integer + description: Time in epoch milliseconds when this object was updated. + vpc_endpoint_id: + type: string + description: The AWS VPC endpoint ID. You can use this ID to identify VPC endpoint created by Databricks. + description: |- + Properties of the new private endpoint rule. Note that for private endpoints towards a VPC + endpoint service behind a customer-managed NLB, you must approve the endpoint in AWS console + after initialization. + DashboardEmailSubscriptions: + type: object + properties: + boolean_val: + $ref: '#/components/schemas/BooleanMessage' + etag: + type: string + description: etag used for versioning. The response is at least as fresh as the eTag provided. This is used for + optimistic concurrency control as a way to help prevent simultaneous writes of a setting overwriting each other. + It is strongly suggested that systems make use of the etag in the read -> update pattern to perform setting updates + in order to avoid race conditions. That is, get an etag from a GET request, and pass it with the PATCH request + to identify the setting version you are updating. + setting_name: + type: string + description: Name of the corresponding setting. This field is populated in the response, but it will not be respected + even if it's set in the request body. The setting name in the path parameter will be respected instead. Setting + name is required to be 'default' if the setting only has one instance per workspace. + required: + - boolean_val + DefaultNamespaceSetting: + type: object + properties: + namespace: + $ref: '#/components/schemas/StringMessage' + etag: type: string description: etag used for versioning. The response is at least as fresh as the eTag provided. This is used for optimistic concurrency control as a way to help prevent simultaneous writes of a setting overwriting each other. @@ -6290,6 +6953,12 @@ components: properties: restriction_mode: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyRestrictionMode' + allowed_databricks_destinations: + type: array + items: + $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyDatabricksDestination' + description: List of Databricks workspace destinations that serverless workloads are allowed to access when in RESTRICTED_ACCESS + mode. allowed_internet_destinations: type: array items: @@ -6302,11 +6971,25 @@ components: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyStorageDestination' description: List of storage destinations that serverless workloads are allowed to access when in RESTRICTED_ACCESS mode. + blocked_internet_destinations: + type: array + items: + $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyInternetDestination' + description: List of internet destinations that serverless workloads are blocked from accessing. These destinations + are enforced when restriction mode is RESTRICTED_ACCESS or DRY_RUN. Currently supports DNS_NAME type only; IP_RANGE + support is planned. policy_enforcement: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicyPolicyEnforcement' description: Optional. When policy_enforcement is not provided, we default to ENFORCE_MODE_ALL_SERVICES required: - restriction_mode + EgressNetworkPolicyNetworkAccessPolicyDatabricksDestination: + type: object + properties: + workspace_ids: + type: array + items: + type: integer EgressNetworkPolicyNetworkAccessPolicyInternetDestination: type: object properties: @@ -6391,7 +7074,10 @@ components: properties: is_enabled: type: boolean - description: 'SHIELD feature: ESM' + description: Whether Enhanced Security Monitoring (ESM) is enabled on the workspace. + description: |- + SHIELD feature: ESM Enhanced Security Monitoring (ESM) enables additional security monitoring on + the workspace. EnhancedSecurityMonitoringSetting: type: object properties: @@ -6471,11 +7157,57 @@ components: ip_access_list: $ref: '#/components/schemas/IpAccessListInfo' description: An IP access list was successfully returned. + GcpEndpoint: + type: object + properties: + all_vpc_sc_services: + type: boolean + google_api_endpoints: + $ref: '#/components/schemas/GoogleApiEndpoints' + description: Selected Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". + psc_endpoint_uri: + type: string + description: Output only. The URI of the created PSC endpoint. + service_attachment: + type: string + description: 'The full url of the target service attachment. Example: projects/my-gcp-project/regions/us-east4/serviceAttachments/my-service-attachment' GenericWebhookConfig: type: object properties: + oauth_audience: + type: string + oauth_audience_set: + type: boolean + description: '[Output-Only] Whether OAuth2 audience is set.' + oauth_client_id: + type: string + description: '[Input-Only][Optional] OAuth2 client ID for the client_credentials grant.' + oauth_client_id_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client ID is set.' + oauth_client_secret: + type: string + description: '[Input-Only][Optional] OAuth2 client secret for the client_credentials grant.' + oauth_client_secret_set: + type: boolean + description: '[Output-Only] Whether OAuth2 client secret is set.' + oauth_scopes: + type: array + items: + type: string + description: '[Input-Only][Optional] OAuth2 scopes requested at token mint (required by some IdPs).' + oauth_scopes_set: + type: boolean + description: '[Output-Only] Whether OAuth2 scopes are set.' + oauth_token_url: + type: string + description: '[Input-Only][Optional] OAuth2 token endpoint URL (customer IdP) where access tokens are minted.' + oauth_token_url_set: + type: boolean + description: '[Output-Only] Whether OAuth2 token URL is set.' password: type: string + description: '[Input-Only][Optional] Password for webhook.' password_set: type: boolean description: '[Output-Only] Whether password is set.' @@ -6517,6 +7249,18 @@ components: token_info: $ref: '#/components/schemas/TokenInfo' description: Token with specified Token ID was successfully returned. + GoogleApiEndpoints: + type: object + properties: + endpoints: + type: array + items: + type: string + description: Google API hostnames, e.g. "storage.googleapis.com", "bigquery.googleapis.com". Use "googleapis.com" + to cover all Google APIs. + description: |- + Wrapper for a list of Google API hostnames. Wrapped in a message because proto3 oneof does not + support repeated fields directly. IpAccessListInfo: type: object properties: @@ -6817,6 +7561,8 @@ components: $ref: '#/components/schemas/NccAwsStableIpRule' azure_service_endpoint_rule: $ref: '#/components/schemas/NccAzureServiceEndpointRule' + gcp_project_id_rule: + $ref: '#/components/schemas/NetworkConnectivityConfigEgressConfigDefaultRuleGcpProjectIdRule' description: Default rules don't have specific targets. NccEgressTargetRules: type: object @@ -6865,8 +7611,8 @@ components: private link service. When updating this field, the full list of target domain_names must be specified. enabled: type: boolean - description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate - this private endpoint to allow egress access from serverless compute resources. + description: Update this field to activate/deactivate this private endpoint to allow egress access from serverless + compute resources. Only honored for first-party services on each cloud (e.g. AWS S3). endpoint_name: type: string description: The name of the Azure private endpoint resource. @@ -6876,6 +7622,8 @@ components: endpoint. error_message: type: string + gcp_endpoint: + $ref: '#/components/schemas/GcpEndpoint' group_id: type: string description: Not used by customer-managed private endpoint services. The sub-resource type (group ID) of the target @@ -6908,6 +7656,13 @@ components: description: |- Properties of the new private endpoint rule. Note that you must approve the endpoint in Azure portal after initialization. + NetworkConnectivityConfigEgressConfigDefaultRuleGcpProjectIdRule: + type: object + properties: + project_ids: + type: array + items: + type: string NetworkConnectivityConfiguration: type: object properties: @@ -6941,12 +7696,7 @@ components: network_access: $ref: '#/components/schemas/EgressNetworkPolicyNetworkAccessPolicy' description: The access policy enforced for egress traffic to the internet. - description: |- - The network policies applying for egress traffic. This message is used by the UI/REST API. We - translate this message to the format expected by the dataplane in Lakehouse Network Manager (for - the format expected by the dataplane, see networkconfig.textproto). This policy should be - consistent with [[com.databricks.api.proto.settingspolicy.EgressNetworkPolicy]]. Details see - API-design: https://docs.google.com/document/d/1DKWO_FpZMCY4cF2O62LpwII1lx8gsnDGG-qgE3t3TOA/ + description: The network policies applying for egress traffic. NotificationDestination: type: object properties: @@ -7005,14 +7755,35 @@ components: PublicTokenInfo: type: object properties: + autoscope_state: + type: string + backfill_scopes: + type: array + items: + type: string + description: Output only. Scopes inferred from offline backfill processing. comment: type: string + description: Comment the token was created with, if applicable. creation_time: type: integer description: Server time (in epoch milliseconds) when the token was created. expiry_time: type: integer description: Server time (in epoch milliseconds) when the token will expire, or -1 if not applicable. + inferred_scopes: + type: array + items: + type: string + description: Output only. Inferred API path scopes collected for this token when autoscope is enabled. + last_accessed_time: + type: integer + description: Server time (in epoch milliseconds) when the token was accessed most recently. + scopes: + type: array + items: + type: string + description: Scope of the token was created with, if applicable. token_id: type: string description: The ID of this token. @@ -7128,8 +7899,16 @@ components: TokenInfo: type: object properties: + autoscope_state: + type: string + backfill_scopes: + type: array + items: + type: string + description: Output only. Scopes inferred from offline backfill processing. comment: type: string + description: Comment that describes the purpose of the token, specified by the token creator. created_by_id: type: integer description: User ID of the user that created the token. @@ -7142,12 +7921,22 @@ components: expiry_time: type: integer description: Timestamp when the token expires. + inferred_scopes: + type: array + items: + type: string + description: Output only. Inferred API path scopes collected for this token when autoscope is enabled. last_used_day: type: integer description: Approximate timestamp for the day the token was last used. Accurate up to 1 day. owner_id: type: integer description: User ID of the user that owns the token. + scopes: + type: array + items: + type: string + description: Scope of the token was created with, if applicable. token_id: type: string description: ID of the token. @@ -7194,10 +7983,12 @@ components: private link service. When updating this field, the full list of target domain_names must be specified. enabled: type: boolean - description: Only used by private endpoints towards an AWS S3 service. Update this field to activate/deactivate - this private endpoint to allow egress access from serverless compute resources. + description: Update this field to activate/deactivate this private endpoint to allow egress access from serverless + compute resources. Only honored for first-party services on each cloud (e.g. AWS S3). error_message: type: string + gcp_endpoint: + $ref: '#/components/schemas/GcpEndpoint' resource_names: type: array items: @@ -7209,6 +8000,9 @@ components: description: |- Properties of the new private endpoint rule. Note that you must approve the endpoint in Azure portal after initialization. + UpdateTokenResponse: + type: object + properties: {} WorkspaceNetworkOption: type: object properties: @@ -7357,6 +8151,7 @@ components: ComplianceStandard: type: string x-enum: + - ARC_AMPE - CANADA_PROTECTED_B - CYBER_ESSENTIAL_PLUS - FEDRAMP_HIGH @@ -7369,12 +8164,207 @@ components: - IRAP_PROTECTED - ISMAP - ITAR_EAR + - KSA_ECC_CCC_DCC - K_FSI - NONE - PCI_DSS description: |- Compliance standard for SHIELD customers. See README.md for how instructions of how to add new standards. + CustomerFacingIngressNetworkPolicyApiScopeConstraint: + type: string + x-enum: + - ALL + - READ + description: 'Deprecated: Use ApiScopeQualifier instead.' + CustomerFacingIngressNetworkPolicyApiScopeQualifier: + type: string + x-enum: + - API_SCOPE_QUALIFIER_ALL + - API_SCOPE_QUALIFIER_READ + description: |- + Qualifies the breadth of API access permitted by an ingress network policy rule. + API_SCOPE_QUALIFIER_READ narrows matching to read-only variants of the listed scopes; + API_SCOPE_QUALIFIER_ALL matches any scope. When unset, scopes match exactly as listed. + CustomerFacingIngressNetworkPolicyAuthenticationIdentityPrincipalType: + type: string + x-enum: + - PRINCIPAL_TYPE_SERVICE_PRINCIPAL + - PRINCIPAL_TYPE_USER + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CustomerFacingIngressNetworkPolicyAuthenticationIdentityType: + type: string + x-enum: + - IDENTITY_TYPE_ALL_SERVICE_PRINCIPALS + - IDENTITY_TYPE_ALL_USERS + - IDENTITY_TYPE_SELECTED_IDENTITIES + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CustomerFacingIngressNetworkPolicyCrossWorkspaceAccessRestrictionMode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CustomerFacingIngressNetworkPolicyPrivateAccessRestrictionMode: + type: string + x-enum: + - ALLOW_ALL_REGISTERED_ENDPOINTS + - RESTRICTED_ACCESS + description: |- + The restriction mode for private access. In ALLOW_ALL_REGISTERED_ENDPOINTS mode, requests + arriving through any endpoint registered to the account are allowed, and deny rules and allow + rules cannot be set. In RESTRICTED_ACCESS mode, access is restricted based on deny rules and + allow rules; requests that do not match any allow rule are denied. + CustomerFacingIngressNetworkPolicyPublicAccessRestrictionMode: + type: string + x-enum: + - FULL_ACCESS + - RESTRICTED_ACCESS + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. CustomerFacingNetworkConnectivityConfigAwsPrivateEndpointRulePrivateLinkConnectionState: type: string x-enum: @@ -7779,8 +8769,9 @@ components: description: |- Type of IP access list. Valid values are as follows and are case-sensitive: - * `ALLOW`: An allow list. Include this IP or range. * `BLOCK`: A block list. Exclude this IP or - range. IP addresses in the block list are excluded even if they are included in an allow list. + - ``ALLOW``: An allow list. Include this IP or range. + - ``BLOCK``: A block list. Exclude this IP or range. IP addresses in the block list are excluded + even if they are included in an allow list. NccAzurePrivateEndpointRuleConnectionState: type: string x-enum: @@ -7942,7 +8933,7 @@ components: - ARCLIGHT_MULTI_TENANT_AZURE_EXCHANGE_TOKEN - ARCLIGHT_MULTI_TENANT_AZURE_EXCHANGE_TOKEN_WITH_USER_DELEGATION_KEY - AZURE_ACTIVE_DIRECTORY_TOKEN - description: The type of token request. As of now, only `AZURE_ACTIVE_DIRECTORY_TOKEN` is supported. + description: The type of token request. As of now, only ``AZURE_ACTIVE_DIRECTORY_TOKEN`` is supported. x-stackQL-resources: aibi_dashboard_embedding_access_policy: id: databricks_workspace.settings.aibi_dashboard_embedding_access_policy @@ -8842,6 +9833,15 @@ components: response: mediaType: application/json openAPIDocKey: '200' + token_management_update_token_management: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1token-management~1tokens~1{token_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' list: operation: $ref: '#/paths/~1api~12.0~1token-management~1tokens/get' @@ -8855,7 +9855,8 @@ components: - $ref: '#/components/x-stackQL-resources/token_management/methods/list' insert: - $ref: '#/components/x-stackQL-resources/token_management/methods/create' - update: [] + update: + - $ref: '#/components/x-stackQL-resources/token_management/methods/token_management_update_token_management' delete: - $ref: '#/components/x-stackQL-resources/token_management/methods/delete' replace: [] @@ -8945,12 +9946,22 @@ components: mediaType: application/json openAPIDocKey: '200' objectKey: $.token_infos + tokens_update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1token~1{token_id}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' sqlVerbs: select: - $ref: '#/components/x-stackQL-resources/tokens/methods/list' insert: - $ref: '#/components/x-stackQL-resources/tokens/methods/create' - update: [] + update: + - $ref: '#/components/x-stackQL-resources/tokens/methods/tokens_update' delete: [] replace: [] workspace_config: @@ -8987,7 +9998,7 @@ components: config: docs: fields: - - name: property + - name: key type: string description: The name of the workspace setting (only non-null values are included). - name: value @@ -9047,7 +10058,6 @@ components: 'enableJobsEmailsV2', enableJobsEmailsV2, 'enableProjectTypeInWorkspace', enableProjectTypeInWorkspace, 'mlflowModelRegistryEmailNotificationsEnabled', mlflowModelRegistryEmailNotificationsEnabled, - 'heapAnalyticsAdminConsent', heapAnalyticsAdminConsent, 'jobsListBackendPaginationEnabled', jobsListBackendPaginationEnabled, 'jobsListBackendPaginationOptOut', jobsListBackendPaginationOptOut ) AS settings_object, @@ -9055,7 +10065,7 @@ components: json_each.value FROM databricks_workspace.settings.workspace_config c, json_each(settings_object) - WHERE keys='enableWebTerminal,enableTokensConfig,maxTokenLifetimeDays,enableWorkspaceFilesystem,enableExportNotebook,enableNotebookTableClipboard,enableResultsDownloading,enableDcs,enableGp3,mlflowRunArtifactDownloadEnabled,enableUploadDataUis,storeInteractiveNotebookResultsInCustomerAccount,enableDeprecatedGlobalInitScripts,rStudioUserDefaultHomeBase,enforceUserIsolation,enableProjectsAllowList,projectsAllowListPermissions,projectsAllowList,reposIpynbResultsExportPermissions,enableDbfsFileBrowser,enableDatabricksAutologgingAdminConf,mlflowModelServingEndpointCreationEnabled,enableVerboseAuditLogs,enableFileStoreEndpoint,loginLogo,productName,homePageLogo,homePageLogoWidth,homePageWelcomeMessage,sidebarLogoText,sidebarLogoActive,sidebarLogoInactive,customReferences,loginLogoWidth,enforceWorkspaceViewAcls,enforceClusterViewAcls,enableJobViewAcls,enableHlsRuntime,enableEnforceImdsV2,enableJobsEmailsV2,enableProjectTypeInWorkspace,mlflowModelRegistryEmailNotificationsEnabled,heapAnalyticsAdminConsent,jobsListBackendPaginationEnabled,jobsListBackendPaginationOptOut' + WHERE keys='enableWebTerminal,enableTokensConfig,maxTokenLifetimeDays,enableWorkspaceFilesystem,enableExportNotebook,enableNotebookTableClipboard,enableResultsDownloading,enableDcs,enableGp3,mlflowRunArtifactDownloadEnabled,enableUploadDataUis,storeInteractiveNotebookResultsInCustomerAccount,enableDeprecatedGlobalInitScripts,rStudioUserDefaultHomeBase,enforceUserIsolation,enableProjectsAllowList,projectsAllowListPermissions,projectsAllowList,reposIpynbResultsExportPermissions,enableDbfsFileBrowser,enableDatabricksAutologgingAdminConf,mlflowModelServingEndpointCreationEnabled,enableVerboseAuditLogs,enableFileStoreEndpoint,loginLogo,productName,homePageLogo,homePageLogoWidth,homePageWelcomeMessage,sidebarLogoText,sidebarLogoActive,sidebarLogoInactive,customReferences,loginLogoWidth,enforceWorkspaceViewAcls,enforceClusterViewAcls,enableJobViewAcls,enableHlsRuntime,enableEnforceImdsV2,enableJobsEmailsV2,enableProjectTypeInWorkspace,mlflowModelRegistryEmailNotificationsEnabled,jobsListBackendPaginationEnabled,jobsListBackendPaginationOptOut' AND deployment_name='{{ deployment_name }}' ) t WHERE value IS NOT NULL; fallback: @@ -9107,12 +10117,11 @@ components: 'enableJobsEmailsV2', enableJobsEmailsV2, 'enableProjectTypeInWorkspace', enableProjectTypeInWorkspace, 'mlflowModelRegistryEmailNotificationsEnabled', mlflowModelRegistryEmailNotificationsEnabled, - 'heapAnalyticsAdminConsent', heapAnalyticsAdminConsent, - 'jobsListBackendPaginationEnabled', jobsListBackendPaginationEnabled, + 'jobsListBackendPaginationEnabled', jobsListBackendPaginationEnabled, 'jobsListBackendPaginationOptOut', jobsListBackendPaginationOptOut )) AS settings_object FROM databricks_workspace.settings.workspace_config c - WHERE keys='enableWebTerminal,enableTokensConfig,maxTokenLifetimeDays,enableWorkspaceFilesystem,enableExportNotebook,enableNotebookTableClipboard,enableResultsDownloading,enableDcs,enableGp3,mlflowRunArtifactDownloadEnabled,enableUploadDataUis,storeInteractiveNotebookResultsInCustomerAccount,enableDeprecatedGlobalInitScripts,rStudioUserDefaultHomeBase,enforceUserIsolation,enableProjectsAllowList,projectsAllowListPermissions,projectsAllowList,reposIpynbResultsExportPermissions,enableDbfsFileBrowser,enableDatabricksAutologgingAdminConf,mlflowModelServingEndpointCreationEnabled,enableVerboseAuditLogs,enableFileStoreEndpoint,loginLogo,productName,homePageLogo,homePageLogoWidth,homePageWelcomeMessage,sidebarLogoText,sidebarLogoActive,sidebarLogoInactive,customReferences,loginLogoWidth,enforceWorkspaceViewAcls,enforceClusterViewAcls,enableJobViewAcls,enableHlsRuntime,enableEnforceImdsV2,enableJobsEmailsV2,enableProjectTypeInWorkspace,mlflowModelRegistryEmailNotificationsEnabled,heapAnalyticsAdminConsent,jobsListBackendPaginationEnabled,jobsListBackendPaginationOptOut' + WHERE keys='enableWebTerminal,enableTokensConfig,maxTokenLifetimeDays,enableWorkspaceFilesystem,enableExportNotebook,enableNotebookTableClipboard,enableResultsDownloading,enableDcs,enableGp3,mlflowRunArtifactDownloadEnabled,enableUploadDataUis,storeInteractiveNotebookResultsInCustomerAccount,enableDeprecatedGlobalInitScripts,rStudioUserDefaultHomeBase,enforceUserIsolation,enableProjectsAllowList,projectsAllowListPermissions,projectsAllowList,reposIpynbResultsExportPermissions,enableDbfsFileBrowser,enableDatabricksAutologgingAdminConf,mlflowModelServingEndpointCreationEnabled,enableVerboseAuditLogs,enableFileStoreEndpoint,loginLogo,productName,homePageLogo,homePageLogoWidth,homePageWelcomeMessage,sidebarLogoText,sidebarLogoActive,sidebarLogoInactive,customReferences,loginLogoWidth,enforceWorkspaceViewAcls,enforceClusterViewAcls,enableJobViewAcls,enableHlsRuntime,enableEnforceImdsV2,enableJobsEmailsV2,enableProjectTypeInWorkspace,mlflowModelRegistryEmailNotificationsEnabled,jobsListBackendPaginationEnabled,jobsListBackendPaginationOptOut' AND deployment_name='{{ deployment_name }}' ) t, LATERAL jsonb_each_text(t.settings_object) WHERE value IS NOT NULL; diff --git a/providers/src/databricks_workspace/v00.00.00000/services/settingsv2.yaml b/providers/src/databricks_workspace/v00.00.00000/services/settingsv2.yaml index 3acb925a..9146bb09 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/settingsv2.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/settingsv2.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks settingsv2 service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.settingsv2 servers: - url: https://{deployment_name}.cloud.databricks.com @@ -183,6 +183,18 @@ paths: is the setting value set by consumers, check effective_aibi_dashboard_embedding_approved_domains for final setting value. + allowed_apps_user_api_scopes: + type: object + properties: + allowed_scopes: + type: array + items: + type: string + description: >- + Setting value for allowed_apps_user_api_scopes setting. + This is the setting value set by consumers, check + effective_allowed_apps_user_api_scopes for final setting + value. automatic_cluster_update_workspace: type: object properties: @@ -373,6 +385,75 @@ paths: Setting value for boolean type setting. This is the setting value set by consumers, check effective_boolean_val for final setting value. + collaboration_platform_connectivity: + type: object + properties: + connectivity: + type: string + x-enum: + - ALLOW_ALL + - ALLOW_SLACK + - ALLOW_TEAMS + - DENY_ALL + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + required: + - connectivity + description: >- + Setting value for collaboration_platform_connectivity + setting. This is the setting value set by consumers, + check effective_collaboration_platform_connectivity for + final setting value. effective_aibi_dashboard_embedding_access_policy: type: object properties: @@ -453,6 +534,17 @@ paths: aibi_dashboard_embedding_approved_domains setting. This is the final effective value of setting. To set a value use aibi_dashboard_embedding_approved_domains. + effective_allowed_apps_user_api_scopes: + type: object + properties: + allowed_scopes: + type: array + items: + type: string + description: >- + Effective setting value for allowed_apps_user_api_scopes + setting. This is the final effective value of setting. + To set a value use allowed_apps_user_api_scopes. effective_automatic_cluster_update_workspace: type: object properties: @@ -643,6 +735,86 @@ paths: Effective setting value for boolean type setting. This is the final effective value of setting. To set a value use boolean_val. + effective_collaboration_platform_connectivity: + type: object + properties: + connectivity: + type: string + x-enum: + - ALLOW_ALL + - ALLOW_SLACK + - ALLOW_TEAMS + - DENY_ALL + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many + members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , ] + + + Methods can be added to enumerations, and members + can have their own + + attributes -- see the documentation for details. + required: + - connectivity + description: >- + Effective setting value for + collaboration_platform_connectivity setting. This is the + final effective value of setting. To set a value use + collaboration_platform_connectivity. + effective_genie_chat_share_policy: + type: object + properties: + disable_make_chat_private: + type: boolean + disable_share_with_account_users: + type: boolean + description: >- + Effective setting value for genie_chat_share_policy + setting. This is the final effective value of setting. + To set a value use genie_chat_share_policy. effective_integer_val: type: object properties: @@ -652,6 +824,16 @@ paths: Effective setting value for integer type setting. This is the final effective value of setting. To set a value use integer_val. + effective_operational_email_custom_recipient: + type: object + properties: + email: + type: string + description: >- + Effective setting value for + operational_email_custom_recipient setting. This is the + final effective value of setting. To set a value use + operational_email_custom_recipient. effective_personal_compute: type: object properties: @@ -760,6 +942,18 @@ paths: Effective setting value for string type setting. This is the final effective value of setting. To set a value use string_val. + genie_chat_share_policy: + type: object + properties: + disable_make_chat_private: + type: boolean + disable_share_with_account_users: + type: boolean + description: >- + Setting value for genie_chat_share_policy setting. This + is the setting value set by consumers, check + effective_genie_chat_share_policy for final setting + value. integer_val: type: object properties: @@ -772,6 +966,16 @@ paths: name: type: string description: Name of the setting. + operational_email_custom_recipient: + type: object + properties: + email: + type: string + description: >- + Setting value for operational_email_custom_recipient + setting. This is the setting value set by consumers, + check effective_operational_email_custom_recipient for + final setting value. personal_compute: type: object properties: @@ -926,11 +1130,11 @@ paths: unspecified, at most 200 settings will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. :param page_token: str (optional) - A page token, received from a previous `ListWorkspaceSettingsMetadataRequest` call. Provide this to - retrieve the subsequent page. + A page token, received from a previous ``ListWorkspaceSettingsMetadataRequest`` call. Provide this + to retrieve the subsequent page. - When paginating, all other parameters provided to `ListWorkspaceSettingsMetadataRequest` must match - the call that provided the page token. + When paginating, all other parameters provided to ``ListWorkspaceSettingsMetadataRequest`` must + match the call that provided the page token. :returns: Iterator over :class:`SettingsMetadata` parameters: @@ -951,9 +1155,9 @@ paths: type: string description: >- A page token, received from a previous - `ListWorkspaceSettingsMetadataRequest` call. Provide this to + ``ListWorkspaceSettingsMetadataRequest`` call. Provide this to retrieve the subsequent page. When paginating, all other parameters - provided to `ListWorkspaceSettingsMetadataRequest` must match the + provided to ``ListWorkspaceSettingsMetadataRequest`` must match the call that provided the page token. responses: '200': @@ -991,6 +1195,13 @@ components: type: array items: type: string + AllowedAppsUserApiScopesMessage: + type: object + properties: + allowed_scopes: + type: array + items: + type: string BooleanMessage: type: object properties: @@ -1057,6 +1268,31 @@ components: type: integer minutes: type: integer + CollaborationPlatformConnectivityMessage: + type: object + properties: + connectivity: + $ref: >- + #/components/schemas/CollaborationPlatformConnectivityMessageConnectivity + required: + - connectivity + description: >- + Controls which external collaboration platforms (Slack, Microsoft Teams) + can connect to a + workspace. Defaults to ALLOW_ALL. + GenieChatSharePolicy: + type: object + properties: + disable_make_chat_private: + type: boolean + disable_share_with_account_users: + type: boolean + description: >- + Workspace controls that restrict which Genie space chat sharing modes + are available to end + users. When disable_make_chat_private is true, users cannot make a Genie chat private. When + disable_share_with_account_users is true, users cannot share a Genie chat with all account + users. Defaults preserve current behavior (both false). IntegerMessage: type: object properties: @@ -1092,6 +1328,11 @@ components: items: $ref: '#/components/schemas/SettingsMetadata' description: List of all settings available via public APIs and their metadata + OperationalEmailCustomRecipientMessage: + type: object + properties: + email: + type: string PersonalComputeMessage: type: object properties: @@ -1122,6 +1363,12 @@ components: This is the setting value set by consumers, check effective_aibi_dashboard_embedding_approved_domains for final setting value. + allowed_apps_user_api_scopes: + $ref: '#/components/schemas/AllowedAppsUserApiScopesMessage' + description: >- + Setting value for allowed_apps_user_api_scopes setting. This is the + setting value set by consumers, check + effective_allowed_apps_user_api_scopes for final setting value. automatic_cluster_update_workspace: $ref: '#/components/schemas/ClusterAutoRestartMessage' description: >- @@ -1135,6 +1382,13 @@ components: Setting value for boolean type setting. This is the setting value set by consumers, check effective_boolean_val for final setting value. + collaboration_platform_connectivity: + $ref: '#/components/schemas/CollaborationPlatformConnectivityMessage' + description: >- + Setting value for collaboration_platform_connectivity setting. This + is the setting value set by consumers, check + effective_collaboration_platform_connectivity for final setting + value. effective_aibi_dashboard_embedding_access_policy: $ref: '#/components/schemas/AibiDashboardEmbeddingAccessPolicy' description: >- @@ -1148,6 +1402,12 @@ components: aibi_dashboard_embedding_approved_domains setting. This is the final effective value of setting. To set a value use aibi_dashboard_embedding_approved_domains. + effective_allowed_apps_user_api_scopes: + $ref: '#/components/schemas/AllowedAppsUserApiScopesMessage' + description: >- + Effective setting value for allowed_apps_user_api_scopes setting. + This is the final effective value of setting. To set a value use + allowed_apps_user_api_scopes. effective_automatic_cluster_update_workspace: $ref: '#/components/schemas/ClusterAutoRestartMessage' description: >- @@ -1159,11 +1419,29 @@ components: description: >- Effective setting value for boolean type setting. This is the final effective value of setting. To set a value use boolean_val. + effective_collaboration_platform_connectivity: + $ref: '#/components/schemas/CollaborationPlatformConnectivityMessage' + description: >- + Effective setting value for collaboration_platform_connectivity + setting. This is the final effective value of setting. To set a + value use collaboration_platform_connectivity. + effective_genie_chat_share_policy: + $ref: '#/components/schemas/GenieChatSharePolicy' + description: >- + Effective setting value for genie_chat_share_policy setting. This is + the final effective value of setting. To set a value use + genie_chat_share_policy. effective_integer_val: $ref: '#/components/schemas/IntegerMessage' description: >- Effective setting value for integer type setting. This is the final effective value of setting. To set a value use integer_val. + effective_operational_email_custom_recipient: + $ref: '#/components/schemas/OperationalEmailCustomRecipientMessage' + description: >- + Effective setting value for operational_email_custom_recipient + setting. This is the final effective value of setting. To set a + value use operational_email_custom_recipient. effective_personal_compute: $ref: '#/components/schemas/PersonalComputeMessage' description: >- @@ -1181,6 +1459,12 @@ components: description: >- Effective setting value for string type setting. This is the final effective value of setting. To set a value use string_val. + genie_chat_share_policy: + $ref: '#/components/schemas/GenieChatSharePolicy' + description: >- + Setting value for genie_chat_share_policy setting. This is the + setting value set by consumers, check + effective_genie_chat_share_policy for final setting value. integer_val: $ref: '#/components/schemas/IntegerMessage' description: >- @@ -1190,6 +1474,13 @@ components: name: type: string description: Name of the setting. + operational_email_custom_recipient: + $ref: '#/components/schemas/OperationalEmailCustomRecipientMessage' + description: >- + Setting value for operational_email_custom_recipient setting. This + is the setting value set by consumers, check + effective_operational_email_custom_recipient for final setting + value. personal_compute: $ref: '#/components/schemas/PersonalComputeMessage' description: >- @@ -1212,12 +1503,22 @@ components: properties: description: type: string + display_name: + type: string + description: >- + Human-readable display name for the setting or feature preview. This + field may be unset if no display name is available. docs_link: type: string description: Link to databricks documentation for the setting name: type: string description: Name of the setting. + preview_phase: + $ref: '#/components/schemas/PreviewPhase' + description: >- + Preview phase for feature preview settings. This field is not set + for non-preview settings. type: type: string description: >- @@ -1385,6 +1686,50 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + CollaborationPlatformConnectivityMessageConnectivity: + type: string + x-enum: + - ALLOW_ALL + - ALLOW_SLACK + - ALLOW_TEAMS + - DENY_ALL + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. PersonalComputeMessagePersonalComputeMessageEnum: @@ -1406,6 +1751,25 @@ components: default policy before they will be able to create compute resources through that policy. + PreviewPhase: + type: string + x-enum: + - BETA + - GA + - GA_SOON + - PRIVATE_PREVIEW + - PUBLIC_PREVIEW + description: >- + Preview phase for settings that are feature previews. For settings that + are not feature + + previews, the preview_phase field is left unset. Mirrors only the + customer-facing phases + + surfaced in the UI; internal-only phases (DISABLED, DEV, + UNDER_MIGRATION, LAUNCHED, etc.) are + + not exposed here. RestrictWorkspaceAdminsMessageStatus: type: string x-enum: diff --git a/providers/src/databricks_workspace/v00.00.00000/services/sharing.yaml b/providers/src/databricks_workspace/v00.00.00000/services/sharing.yaml index 1db97298..c4baff9a 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/sharing.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/sharing.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks sharing service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.sharing servers: - url: https://{deployment_name}.cloud.databricks.com @@ -37,8 +37,11 @@ paths: :param comment: str (optional) Description about the provider. + :param email_recipient_id: str (optional) + The ID of the email recipient this provider is being created to accept. Only valid on DATABRICKS + CreateProvider when accepting an email invite. :param recipient_profile_str: str (optional) - This field is required when the __authentication_type__ is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** + This field is required when the **authentication_type** is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** or not provided. :returns: :class:`ProviderInfo` @@ -55,6 +58,7 @@ paths: type: string x-enum: - DATABRICKS + - EMAIL - OAUTH_CLIENT_CREDENTIALS - OIDC_FEDERATION - TOKEN @@ -62,10 +66,16 @@ paths: comment: type: string description: Description about the provider. + email_recipient_id: + type: string + description: >- + The ID of the email recipient this provider is being created + to accept. Only valid on DATABRICKS CreateProvider when + accepting an email invite. recipient_profile_str: type: string description: >- - This field is required when the __authentication_type__ is + This field is required when the **authentication_type** is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** or not provided. required: - name @@ -115,13 +125,16 @@ paths: If not provided, all providers will be returned. If no providers exist with this ID, no results will be returned. :param max_results: int (optional) - Maximum number of providers to return. - when set to 0, the page length is set to a server - configured value (recommended); - when set to a value greater than 0, the page length is the minimum - of this value and a server configured value; - when set to a value less than 0, an invalid parameter - error is returned; - If not set, all valid providers are returned (not recommended). - Note: The - number of returned providers might be less than the specified max_results size, even zero. The only - definitive indication that no further providers can be fetched is when the next_page_token is unset - from the response. + Maximum number of providers to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid providers are returned (not recommended). + - Note: The number of returned providers might be less than the specified max_results size, even + zero. The only definitive indication that no further providers can be fetched is when the + next_page_token is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -288,7 +301,7 @@ paths: :param owner: str (optional) Username of Provider owner. :param recipient_profile_str: str (optional) - This field is required when the __authentication_type__ is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** + This field is required when the **authentication_type** is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** or not provided. :returns: :class:`ProviderInfo` @@ -317,7 +330,7 @@ paths: recipient_profile_str: type: string description: >- - This field is required when the __authentication_type__ is + This field is required when the **authentication_type** is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** or not provided. responses: '200': @@ -438,19 +451,24 @@ paths: where: - * the caller is a metastore admin, or * the caller is the owner. + - the caller is a metastore admin, or + + - the caller is the owner. :param name: str Name of the provider in which to list shares. :param max_results: int (optional) - Maximum number of shares to return. - when set to 0, the page length is set to a server configured - value (recommended); - when set to a value greater than 0, the page length is the minimum of this - value and a server configured value; - when set to a value less than 0, an invalid parameter error - is returned; - If not set, all valid shares are returned (not recommended). - Note: The number of - returned shares might be less than the specified max_results size, even zero. The only definitive - indication that no further shares can be fetched is when the next_page_token is unset from the - response. + Maximum number of shares to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid shares are returned (not recommended). + - Note: The number of returned shares might be less than the specified max_results size, even zero. + The only definitive indication that no further shares can be fetched is when the next_page_token + is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -514,6 +532,9 @@ paths: :param activation_url: str The one time activation url. It also accepts activation token. + :param metastore_id: str (optional) + The provider metastore ID. Optional for now; used as a routing hint when present and principal + context parameters: - name: activation_url in: path @@ -521,6 +542,14 @@ paths: schema: type: string description: The one time activation url. It also accepts activation token. + - name: metastore_id + in: query + required: false + schema: + type: string + description: >- + The provider metastore ID. Optional for now; used as a routing hint + when present and principal context responses: '200': description: Success @@ -552,6 +581,9 @@ paths: :param activation_url: str The one time activation url. It also accepts activation token. + :param metastore_id: str (optional) + The provider metastore ID. Optional for now; used as a routing hint when present and principal + context :returns: :class:`RetrieveTokenResponse` parameters: @@ -561,6 +593,14 @@ paths: schema: type: string description: The one time activation url. It also accepts activation token. + - name: metastore_id + in: query + required: false + schema: + type: string + description: >- + The provider metastore ID. Optional for now; used as a routing hint + when present and principal context responses: '200': description: Success @@ -608,40 +648,33 @@ paths: tokens. - Supported scenarios for federation policies: 1. **User-to-Machine (U2M) - flow** (e.g., PowerBI): A user - - accesses a resource using their own identity. 2. **Machine-to-Machine - (M2M) flow** (e.g., OAuth App): - - An OAuth App accesses a resource using its own identity, typically for - tasks like running nightly + Supported scenarios for federation policies: - jobs. + 1. **User-to-Machine (U2M) flow** (e.g., PowerBI): A user accesses a + resource using their own + identity. + 2. **Machine-to-Machine (M2M) flow** (e.g., OAuth App): An OAuth App + accesses a resource using its own + identity, typically for tasks like running nightly jobs. - For an overview, refer to: - Blog post: Overview of feature: + For an overview, refer to: - https://www.databricks.com/blog/announcing-oidc-token-federation-enhanced-delta-sharing-security + - Blog post: Overview of feature: + https://www.databricks.com/blog/announcing-oidc-token-federation-enhanced-delta-sharing-security - For detailed configuration guides based on your use case: - Creating a - Federation Policy as a + For detailed configuration guides based on your use case: - provider: - https://docs.databricks.com/en/delta-sharing/create-recipient-oidc-fed - - Configuration and - - usage for Machine-to-Machine (M2M) applications (e.g., Python Delta - Sharing Client): - - https://docs.databricks.com/aws/en/delta-sharing/sharing-over-oidc-m2m - - Configuration and usage for - - User-to-Machine (U2M) applications (e.g., PowerBI): - - https://docs.databricks.com/aws/en/delta-sharing/sharing-over-oidc-u2m + - Creating a Federation Policy as a provider: + https://docs.databricks.com/en/delta-sharing/create-recipient-oidc-fed + - Configuration and usage for Machine-to-Machine (M2M) applications + (e.g., Python Delta Sharing + Client): https://docs.databricks.com/aws/en/delta-sharing/sharing-over-oidc-m2m + - Configuration and usage for User-to-Machine (U2M) applications (e.g., + PowerBI): + https://docs.databricks.com/aws/en/delta-sharing/sharing-over-oidc-u2m :param recipient_name: str Name of the recipient. This is the name of the recipient for which the policy is being created. @@ -699,15 +732,15 @@ paths: description: >- The claim that contains the subject of the token. Depending on the identity provider and the use case - (U2M or M2M), this can vary: - For Entra ID (AAD): * - U2M flow (group access): Use `groups`. * U2M flow - (user access): Use `oid`. * M2M flow (OAuth App - access): Use `azp`. - For other IdPs, refer to the + (U2M or M2M), this can vary: - For Entra ID (AAD): - + U2M flow (group access): Use ``groups``. - U2M flow + (user access): Use ``oid``. - M2M flow (OAuth App + access): Use ``azp``. - For other IdPs, refer to the specific IdP documentation. Supported - `subject_claim` values are: - `oid`: Object ID of - the user. - `azp`: Client ID of the OAuth app. - - `groups`: Object ID of the group. - `sub`: Subject - identifier for other use cases. + ``subject_claim`` values are: - ``oid``: Object ID + of the user. - ``azp``: Client ID of the OAuth app. + - ``groups``: Object ID of the group. - ``sub``: + Subject identifier for other use cases. subject: type: string description: >- @@ -716,12 +749,13 @@ paths: identifies the identity of the user or machine accessing the resource. Examples for Entra ID (AAD): - U2M flow (group access): If the subject claim is - `groups`, this must be the Object ID of the group in - Entra ID. - U2M flow (user access): If the subject - claim is `oid`, this must be the Object ID of the - user in Entra ID. - M2M flow (OAuth App access): If - the subject claim is `azp`, this must be the client - ID of the OAuth app registered in Entra ID. + ``groups``, this must be the Object ID of the group + in Entra ID. - U2M flow (user access): If the + subject claim is ``oid``, this must be the Object ID + of the user in Entra ID. - M2M flow (OAuth App + access): If the subject claim is ``azp``, this must + be the client ID of the OAuth app registered in + Entra ID. audiences: type: array items: @@ -937,6 +971,174 @@ paths: message: type: string x-stackql-sdk-source: RecipientFederationPoliciesAPI + patch: + operationId: recipient_federation_policies_update + summary: >- + Updates an existing federation policy for an OIDC_RECIPIENT. The caller + must be the owner of the + tags: + - sharing + - recipient_federation_policies + description: >- + Updates an existing federation policy for an OIDC_RECIPIENT. The caller + must be the owner of the + + recipient. + + + :param recipient_name: str + Name of the recipient. This is the name of the recipient for which the policy is being updated. + :param name: str + Name of the policy. This is the name of the current name of the policy. + :param policy: :class:`FederationPolicy` + + :param update_mask: str (optional) + The field mask specifies which fields of the policy to update. To specify multiple fields in the + field mask, use comma as the separator (no space). The special value '*' indicates that all fields + should be updated (full replacement). If unspecified, all fields that are set in the policy provided + in the update request will overwrite the corresponding fields in the existing policy. Example value: + 'comment,oidc_policy.audiences'. + + :returns: :class:`FederationPolicy` + parameters: + - name: recipient_name + in: path + required: true + schema: + type: string + description: >- + Name of the recipient. This is the name of the recipient for which + the policy is being updated. + - name: name + in: path + required: true + schema: + type: string + description: >- + Name of the policy. This is the name of the current name of the + policy. + - name: update_mask + in: query + required: false + schema: + type: string + description: >- + The field mask specifies which fields of the policy to update. To + specify multiple fields in the field mask, use comma as the + separator (no space). The special value '*' indicates that all + fields should be updated (full replacement). If unspecified, all + fields that are set in the policy provided in the update request + will overwrite the corresponding fields in the existing policy. + Example value: 'comment,oidc_policy.audiences'. + requestBody: + content: + application/json: + schema: + type: object + properties: + policy: + type: object + properties: + comment: + type: string + create_time: + type: string + description: >- + System-generated timestamp indicating when the policy + was created. + id: + type: string + description: >- + Unique, immutable system-generated identifier for the + federation policy. + name: + type: string + description: >- + Name of the federation policy. A recipient can have + multiple policies with different names. The name must + contain only lowercase alphanumeric characters, numbers, + and hyphens. + oidc_policy: + type: object + properties: + issuer: + type: string + description: >- + The required token issuer, as specified in the 'iss' + claim of federated tokens. + subject_claim: + type: string + description: >- + The claim that contains the subject of the token. + Depending on the identity provider and the use case + (U2M or M2M), this can vary: - For Entra ID (AAD): - + U2M flow (group access): Use ``groups``. - U2M flow + (user access): Use ``oid``. - M2M flow (OAuth App + access): Use ``azp``. - For other IdPs, refer to the + specific IdP documentation. Supported + ``subject_claim`` values are: - ``oid``: Object ID + of the user. - ``azp``: Client ID of the OAuth app. + - ``groups``: Object ID of the group. - ``sub``: + Subject identifier for other use cases. + subject: + type: string + description: >- + The required token subject, as specified in the + subject claim of federated tokens. The subject claim + identifies the identity of the user or machine + accessing the resource. Examples for Entra ID (AAD): + - U2M flow (group access): If the subject claim is + ``groups``, this must be the Object ID of the group + in Entra ID. - U2M flow (user access): If the + subject claim is ``oid``, this must be the Object ID + of the user in Entra ID. - M2M flow (OAuth App + access): If the subject claim is ``azp``, this must + be the client ID of the OAuth app registered in + Entra ID. + audiences: + type: array + items: + type: string + description: >- + The allowed token audiences, as specified in the + 'aud' claim of federated tokens. The audience + identifier is intended to represent the recipient of + the token. Can be any non-empty string value. As + long as the audience in the token matches at least + one audience in the policy, + required: + - issuer + - subject_claim + - subject + description: >- + Specifies the policy to use for validating OIDC claims + in the federated tokens. + update_time: + type: string + description: >- + System-generated timestamp indicating when the policy + was last updated. + required: + - policy + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/FederationPolicy' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: RecipientFederationPoliciesAPI /api/2.1/unity-catalog/recipients: post: operationId: recipients_create @@ -958,12 +1160,25 @@ paths: Name of Recipient. :param authentication_type: :class:`AuthenticationType` + :param allowed_acceptance_types: List[:class:`AuthenticationType`] + (optional) + Authentication types the invitee may choose from when activating. Each value must be TOKEN or + DATABRICKS. Empty means all supported types are allowed. This field is only present when the + **authentication_type** is **EMAIL**. + :param allowed_data_recipient_global_metastore_ids: List[str] (optional) + Optional allowlist of the data recipient's global metastore IDs permitted to accept the invite with + the DATABRICKS acceptance type. Each ID is of the form **cloud**:**region**:**metastore-uuid**. + Empty means any metastore is allowed. If non-empty, **allowed_acceptance_types** must include + **DATABRICKS**. This field is only present when the **authentication_type** is **EMAIL**. :param comment: str (optional) Description about the recipient. :param data_recipient_global_metastore_id: str (optional) The global Unity Catalog metastore id provided by the data recipient. This field is only present - when the __authentication_type__ is **DATABRICKS**. The identifier is of format - __cloud__:__region__:__metastore-uuid__. + when the **authentication_type** is **DATABRICKS**. The identifier is of format + **cloud**:**region**:**metastore-uuid**. + :param email: str (optional) + The invited email address. This field is only present when the **authentication_type** is **EMAIL**. + The 320-character cap follows RFC 5321 (64-octet local-part + ``@`` + 255-octet domain). :param expiration_time: int (optional) Expiration timestamp of the token, in epoch milliseconds. :param id: str (optional) @@ -979,7 +1194,7 @@ paths: need to perform a read-modify-write. :param sharing_code: str (optional) The one-time sharing code provided by the data recipient. This field is only present when the - __authentication_type__ is **DATABRICKS**. + **authentication_type** is **DATABRICKS**. :returns: :class:`RecipientInfo` requestBody: @@ -995,10 +1210,40 @@ paths: type: string x-enum: - DATABRICKS + - EMAIL - OAUTH_CLIENT_CREDENTIALS - OIDC_FEDERATION - TOKEN description: The delta sharing authentication type. + allowed_acceptance_types: + type: array + items: + type: string + x-enum: + - DATABRICKS + - EMAIL + - OAUTH_CLIENT_CREDENTIALS + - OIDC_FEDERATION + - TOKEN + description: The delta sharing authentication type. + description: >- + Authentication types the invitee may choose from when + activating. Each value must be TOKEN or DATABRICKS. Empty + means all supported types are allowed. This field is only + present when the **authentication_type** is **EMAIL**. + allowed_data_recipient_global_metastore_ids: + type: array + items: + type: string + description: >- + Optional allowlist of the data recipient's global metastore + IDs permitted to accept the invite with the DATABRICKS + acceptance type. Each ID is of the form + **cloud**:**region**:**metastore-uuid**. Empty means any + metastore is allowed. If non-empty, + **allowed_acceptance_types** must include **DATABRICKS**. + This field is only present when the **authentication_type** + is **EMAIL**. comment: type: string description: Description about the recipient. @@ -1007,8 +1252,15 @@ paths: description: >- The global Unity Catalog metastore id provided by the data recipient. This field is only present when the - __authentication_type__ is **DATABRICKS**. The identifier is - of format __cloud__:__region__:__metastore-uuid__. + **authentication_type** is **DATABRICKS**. The identifier is + of format **cloud**:**region**:**metastore-uuid**. + email: + type: string + description: >- + The invited email address. This field is only present when + the **authentication_type** is **EMAIL**. The 320-character + cap follows RFC 5321 (64-octet local-part + ``@`` + + 255-octet domain). expiration_time: type: integer description: Expiration timestamp of the token, in epoch milliseconds. @@ -1043,7 +1295,7 @@ paths: type: string description: >- The one-time sharing code provided by the data recipient. - This field is only present when the __authentication_type__ + This field is only present when the **authentication_type** is **DATABRICKS**. required: - name @@ -1080,25 +1332,32 @@ paths: where: - * the caller is a metastore admin, or * the caller is the owner. There - is no guarantee of a specific + - the caller is a metastore admin, or - ordering of the elements in the array. + - the caller is the owner. There is no guarantee of a specific ordering + of the elements in the array. :param data_recipient_global_metastore_id: str (optional) If not provided, all recipients will be returned. If no recipients exist with this ID, no results will be returned. :param max_results: int (optional) - Maximum number of recipients to return. - when set to 0, the page length is set to a server - configured value (recommended); - when set to a value greater than 0, the page length is the minimum - of this value and a server configured value; - when set to a value less than 0, an invalid parameter - error is returned; - If not set, all valid recipients are returned (not recommended). - Note: The - number of returned recipients might be less than the specified max_results size, even zero. The only - definitive indication that no further recipients can be fetched is when the next_page_token is unset - from the response. + Maximum number of recipients to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid recipients are returned (not recommended). + - Note: The number of returned recipients might be less than the specified max_results size, even + zero. The only definitive indication that no further recipients can be fetched is when the + next_page_token is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. + :param parent_recipient_name: str (optional) + If set, the response is scoped to the child recipients of the named parent recipient. Child + recipients are otherwise hidden from the default list response. Only applicable to email recipient + parents — non-email recipients have no children to scope to. :returns: Iterator over :class:`RecipientInfo` parameters: @@ -1132,6 +1391,16 @@ paths: schema: type: string description: Opaque pagination token to go to next page based on previous query. + - name: parent_recipient_name + in: query + required: false + schema: + type: string + description: >- + If set, the response is scoped to the child recipients of the named + parent recipient. Child recipients are otherwise hidden from the + default list response. Only applicable to email recipient parents — + non-email recipients have no children to scope to. responses: '200': description: Success @@ -1191,19 +1460,16 @@ paths: x-stackql-sdk-source: RecipientsAPI get: operationId: recipients_get - summary: >- - Gets a share recipient from the metastore. The caller must be one of: * - A user with **USE_RECIPIENT** + summary: 'Gets a share recipient from the metastore. The caller must be one of:' tags: - sharing - recipients - description: >- - Gets a share recipient from the metastore. The caller must be one of: * - A user with **USE_RECIPIENT** - - privilege on the metastore * The owner of the share recipient * A - metastore admin + description: |- + Gets a share recipient from the metastore. The caller must be one of: + - A user with **USE_RECIPIENT** privilege on the metastore + - The owner of the share recipient + - A metastore admin :param name: str Name of the recipient. @@ -1255,8 +1521,21 @@ paths: :param name: str Name of the recipient. + :param allowed_acceptance_types: List[:class:`AuthenticationType`] + (optional) + Authentication types the invitee may choose from when activating. Each value must be TOKEN or + DATABRICKS. Empty means all supported types are allowed. This field is only present when the + **authentication_type** is **EMAIL**. + :param allowed_data_recipient_global_metastore_ids: List[str] (optional) + Optional allowlist of the data recipient's global metastore IDs permitted to accept the invite with + the DATABRICKS acceptance type. Each ID is of the form **cloud**:**region**:**metastore-uuid**. + Empty means any metastore is allowed. If non-empty, **allowed_acceptance_types** must include + **DATABRICKS**. This field is only present when the **authentication_type** is **EMAIL**. :param comment: str (optional) Description about the recipient. + :param email: str (optional) + The invited email address. This field is only present when the **authentication_type** is **EMAIL**. + The 320-character cap follows RFC 5321 (64-octet local-part + ``@`` + 255-octet domain). :param expiration_time: int (optional) Expiration timestamp of the token, in epoch milliseconds. :param id: str (optional) @@ -1287,9 +1566,45 @@ paths: schema: type: object properties: + allowed_acceptance_types: + type: array + items: + type: string + x-enum: + - DATABRICKS + - EMAIL + - OAUTH_CLIENT_CREDENTIALS + - OIDC_FEDERATION + - TOKEN + description: The delta sharing authentication type. + description: >- + Authentication types the invitee may choose from when + activating. Each value must be TOKEN or DATABRICKS. Empty + means all supported types are allowed. This field is only + present when the **authentication_type** is **EMAIL**. + allowed_data_recipient_global_metastore_ids: + type: array + items: + type: string + description: >- + Optional allowlist of the data recipient's global metastore + IDs permitted to accept the invite with the DATABRICKS + acceptance type. Each ID is of the form + **cloud**:**region**:**metastore-uuid**. Empty means any + metastore is allowed. If non-empty, + **allowed_acceptance_types** must include **DATABRICKS**. + This field is only present when the **authentication_type** + is **EMAIL**. comment: type: string description: Description about the recipient. + email: + type: string + description: >- + The invited email address. This field is only present when + the **authentication_type** is **EMAIL**. The 320-character + cap follows RFC 5321 (64-octet local-part + ``@`` + + 255-octet domain). expiration_time: type: integer description: Expiration timestamp of the token, in epoch milliseconds. @@ -1427,13 +1742,16 @@ paths: :param name: str The name of the Recipient. :param max_results: int (optional) - Maximum number of permissions to return. - when set to 0, the page length is set to a server - configured value (recommended); - when set to a value greater than 0, the page length is the minimum - of this value and a server configured value; - when set to a value less than 0, an invalid parameter - error is returned; - If not set, all valid permissions are returned (not recommended). - Note: The - number of returned permissions might be less than the specified max_results size, even zero. The - only definitive indication that no further permissions can be fetched is when the next_page_token is - unset from the response. + Maximum number of permissions to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid permissions are returned (not recommended). + - Note: The number of returned permissions might be less than the specified max_results size, even + zero. The only definitive indication that no further permissions can be fetched is when the + next_page_token is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -1507,6 +1825,11 @@ paths: Name of the share. :param comment: str (optional) User-provided free-form text description. + :param replication_config: :class:`ReplicationConfig` (optional) + Configuration for share replication. + :param serverless_budget_policy_id: str (optional) + Serverless budget policy id (can only be created/updated when calling data-sharing service) + [Create,Update:IGN] :param storage_root: str (optional) Storage root URL for the share. @@ -1523,6 +1846,32 @@ paths: comment: type: string description: User-provided free-form text description. + replication_config: + type: object + properties: + enabled: + type: boolean + description: Whether replication is enabled for the share. + schedule: + type: object + properties: + interval: + type: object + properties: + duration: + type: string + description: How often the share's data should be replicated. + description: Fixed-interval replication schedule. + description: >- + Optional schedule describing how often replication runs + for this share. When unset, replication runs on a + service-default schedule. + description: Configuration for share replication. + serverless_budget_policy_id: + type: string + description: >- + Serverless budget policy id (can only be created/updated + when calling data-sharing service) [Create,Update:IGN] storage_root: type: string description: Storage root URL for the share. @@ -1565,19 +1914,39 @@ paths: is no guarantee of a specific ordering of the elements in the array. + :param filter_by_genie_space_id: str (optional) + When set, only shares that contain the Genie space with this id are returned. The filter is applied + at the data-sharing layer after MC returns the page, so the resulting page may be smaller than + ``max_results``. Intended for the provider Genie space page's "Shared with N recipients" lookup; not + optimized for providers with many shares. :param max_results: int (optional) - Maximum number of shares to return. - when set to 0, the page length is set to a server configured - value (recommended); - when set to a value greater than 0, the page length is the minimum of this - value and a server configured value; - when set to a value less than 0, an invalid parameter error - is returned; - If not set, all valid shares are returned (not recommended). - Note: The number of - returned shares might be less than the specified max_results size, even zero. The only definitive - indication that no further shares can be fetched is when the next_page_token is unset from the - response. + Maximum number of shares to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid shares are returned (not recommended). + - Note: The number of returned shares might be less than the specified max_results size, even zero. + The only definitive indication that no further shares can be fetched is when the next_page_token + is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. :returns: Iterator over :class:`ShareInfo` parameters: + - name: filter_by_genie_space_id + in: query + required: false + schema: + type: string + description: >- + When set, only shares that contain the Genie space with this id are + returned. The filter is applied at the data-sharing layer after MC + returns the page, so the resulting page may be smaller than + ``max_results``. Intended for the provider Genie space page's + "Shared with N recipients" lookup; not optimized for providers with + many shares. - name: max_results in: query required: false @@ -1724,7 +2093,7 @@ paths: the share or a metastore admin. - When the caller is a metastore admin, only the __owner__ field can be + When the caller is a metastore admin, only the **owner** field can be updated. @@ -1734,7 +2103,7 @@ paths: share and has the CREATE_SHARE privilege. - If there are notebook files in the share, the __storage_root__ field + If there are notebook files in the share, the **storage_root** field cannot be updated. @@ -1758,6 +2127,9 @@ paths: New name for the share. :param owner: str (optional) Username of current owner of share. + :param serverless_budget_policy_id: str (optional) + Serverless budget policy id (can only be created/updated when calling data-sharing service) + [Create,Update:IGN] :param storage_root: str (optional) Storage root URL for the share. :param updates: List[:class:`SharedDataObjectUpdate`] (optional) @@ -1786,6 +2158,11 @@ paths: owner: type: string description: Username of current owner of share. + serverless_budget_policy_id: + type: string + description: >- + Serverless budget policy id (can only be created/updated + when calling data-sharing service) [Create,Update:IGN] storage_root: type: string description: Storage root URL for the share. @@ -1898,6 +2275,79 @@ paths: - VIEW - VOLUME description: The type of the data object. + global_distribution_status: + type: object + properties: + eligible: + type: boolean + description: >- + Whether this asset meets the structural + requirements for global distribution (e.g. + asset type, history sharing mode, storage + scheme). + status: + type: string + x-enum: + - DISABLED + - ENABLED + description: >- + Create a collection of name/value pairs. + + + Example enumeration: + + + >>> class Color(Enum): + + ... RED = 1 + + ... BLUE = 2 + + ... GREEN = 3 + + + Access them by: + + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know + how many members they have: + + + >>> len(Color) + + 3 + + + >>> list(Color) + + [, , + ] + + + Methods can be added to enumerations, and + members can have their own + + attributes -- see the documentation for + details. + description: >- + Global Distribution status for the shared asset. + Output only. Only set for TABLE objects; absent + for all other data object types. history_data_sharing_status: type: string x-enum: @@ -1931,14 +2381,15 @@ paths: The key of a Delta Sharing recipient's property. For example "databricks-account-id". When this field - is set, field `value` can not be set. + is set, field ``value`` can not be set. value: type: string description: >- The value of the partition column. When - this value is not set, it means `null` + this value is not set, it means ``null`` value. When this field is set, field - `recipient_property_key` can not be set. + ``recipient_property_key`` can not be + set. description: Array of partitions for the shared data. shared_as: type: string @@ -1949,10 +2400,10 @@ paths: MATERIALIZED_VIEW, STREAMING_TABLE, FOREIGN_TABLE). For non-table objects (for example: VOLUME, MODEL, NOTEBOOK_FILE, FUNCTION), - use `string_shared_as` instead. Important: For + use ``string_shared_as`` instead. Important: For non-table objects, this field must be omitted entirely. Format: Must be a 2-part name - `.` (e.g., + ``.`` (e.g., "sales_schema.orders_table") - Both schema and table names must contain only alphanumeric characters and underscores - No periods, spaces, @@ -1960,11 +2411,11 @@ paths: within each part - Do not include the catalog name (use 2 parts, not 3) Behavior: - If not provided, the service automatically generates the alias as - `.` from the object's original name - - If you don't want to specify this field, omit it - entirely from the request (do not pass an empty - string) - The `shared_as` name must be unique - within the share Examples: - Valid: + ``.
`` from the object's original + name - If you don't want to specify this field, + omit it entirely from the request (do not pass an + empty string) - The ``shared_as`` name must be + unique within the share Examples: - Valid: "analytics_schema.customer_view" - Invalid: "catalog.analytics_schema.customer_view" (3 parts not allowed) - Invalid: @@ -1981,7 +2432,7 @@ paths: clients can only query starting from the version of the object at the time it was added to the share. NOTE: The start_version should be <= the - `current` version of the object. + ``current`` version of the object. status: type: string x-enum: @@ -1996,15 +2447,16 @@ paths: non-table objects (for example: VOLUME, MODEL, NOTEBOOK_FILE, FUNCTION). For table-like objects (for example: TABLE, VIEW, MATERIALIZED_VIEW, - STREAMING_TABLE, FOREIGN_TABLE), use `shared_as` + STREAMING_TABLE, FOREIGN_TABLE), use ``shared_as`` instead. Important: For table-like objects, this field must be omitted entirely. Format: - For VOLUME: Must be a 2-part name - `.` (e.g., + ``.`` (e.g., "data_schema.ml_models") - For FUNCTION: Must be a - 2-part name `.` (e.g., - "udf_schema.calculate_tax") - For MODEL: Must be a - 2-part name `.` (e.g., + 2-part name ``.`` + (e.g., "udf_schema.calculate_tax") - For MODEL: + Must be a 2-part name + ``.`` (e.g., "models.prediction_model") - For NOTEBOOK_FILE: Should be the notebook file name (e.g., "analysis_notebook.py") - All names must contain @@ -2015,7 +2467,7 @@ paths: generates the alias from the object's original name - If you don't want to specify this field, omit it entirely from the request (do not pass an - empty string) - The `string_shared_as` name must + empty string) - The ``string_shared_as`` name must be unique for objects of the same type within the share Examples: - Valid for VOLUME: "data_schema.training_data" - Valid for FUNCTION: @@ -2068,13 +2520,16 @@ paths: :param name: str The name of the Recipient. :param max_results: int (optional) - Maximum number of permissions to return. - when set to 0, the page length is set to a server - configured value (recommended); - when set to a value greater than 0, the page length is the minimum - of this value and a server configured value; - when set to a value less than 0, an invalid parameter - error is returned; - If not set, all valid permissions are returned (not recommended). - Note: The - number of returned permissions might be less than the specified max_results size, even zero. The - only definitive indication that no further permissions can be fetched is when the next_page_token is - unset from the response. + Maximum number of permissions to return. + + - when set to 0, the page length is set to a server configured value (recommended); + - when set to a value greater than 0, the page length is the minimum of this value and a server + configured value; + - when set to a value less than 0, an invalid parameter error is returned; + - If not set, all valid permissions are returned (not recommended). + - Note: The number of returned permissions might be less than the specified max_results size, even + zero. The only definitive indication that no further permissions can be fetched is when the + next_page_token is unset from the response. :param page_token: str (optional) Opaque pagination token to go to next page based on previous query. @@ -2185,6 +2640,17 @@ paths: The principal whose privileges we are changing. Only one of principal or principal_id should be specified, never both at the same time. + principal_id: + type: integer + description: >- + An opaque internal ID that identifies the principal + whose privileges should be removed. This field is + intended for removing privileges associated with a + deleted user. When set, only the entries specified in + the remove field are processed; any entries in the add + field will be rejected. Only one of principal or + principal_id should be specified, never both at the + same time. remove: type: array items: @@ -2404,6 +2870,18 @@ components: items: $ref: '#/components/schemas/PrivilegeAssignment' description: The privileges assigned to each principal + GlobalDistributionStatus: + type: object + properties: + eligible: + type: boolean + description: >- + Whether this asset meets the structural requirements for global + distribution (e.g. asset type, history sharing mode, storage + scheme). + status: + $ref: '#/components/schemas/GlobalDistributionStatusStatus' + description: Status of Global Distribution for a shared asset. IpAccessList: type: object properties: @@ -2522,24 +3000,24 @@ components: description: >- The claim that contains the subject of the token. Depending on the identity provider and the use case (U2M or M2M), this can vary: - - For Entra ID (AAD): * U2M flow (group access): Use `groups`. * U2M - flow (user access): Use `oid`. * M2M flow (OAuth App access): Use - `azp`. - For other IdPs, refer to the specific IdP documentation. - Supported `subject_claim` values are: - `oid`: Object ID of the - user. - `azp`: Client ID of the OAuth app. - `groups`: Object ID of - the group. - `sub`: Subject identifier for other use cases. + For Entra ID (AAD): - U2M flow (group access): Use ``groups``. - U2M + flow (user access): Use ``oid``. - M2M flow (OAuth App access): Use + ``azp``. - For other IdPs, refer to the specific IdP documentation. + Supported ``subject_claim`` values are: - ``oid``: Object ID of the + user. - ``azp``: Client ID of the OAuth app. - ``groups``: Object ID + of the group. - ``sub``: Subject identifier for other use cases. subject: type: string description: >- The required token subject, as specified in the subject claim of federated tokens. The subject claim identifies the identity of the user or machine accessing the resource. Examples for Entra ID (AAD): - - U2M flow (group access): If the subject claim is `groups`, this + - U2M flow (group access): If the subject claim is ``groups``, this must be the Object ID of the group in Entra ID. - U2M flow (user - access): If the subject claim is `oid`, this must be the Object ID + access): If the subject claim is ``oid``, this must be the Object ID of the user in Entra ID. - M2M flow (OAuth App access): If the - subject claim is `azp`, this must be the client ID of the OAuth app - registered in Entra ID. + subject claim is ``azp``, this must be the client ID of the OAuth + app registered in Entra ID. audiences: type: array items: @@ -2578,14 +3056,14 @@ components: type: string description: >- The key of a Delta Sharing recipient's property. For example - "databricks-account-id". When this field is set, field `value` can + "databricks-account-id". When this field is set, field ``value`` can not be set. value: type: string description: >- The value of the partition column. When this value is not set, it - means `null` value. When this field is set, field - `recipient_property_key` can not be set. + means ``null`` value. When this field is set, field + ``recipient_property_key`` can not be set. PermissionsChange: type: object properties: @@ -2599,6 +3077,15 @@ components: The principal whose privileges we are changing. Only one of principal or principal_id should be specified, never both at the same time. + principal_id: + type: integer + description: >- + An opaque internal ID that identifies the principal whose privileges + should be removed. This field is intended for removing privileges + associated with a deleted user. When set, only the entries specified + in the remove field are processed; any entries in the add field will + be rejected. Only one of principal or principal_id should be + specified, never both at the same time. remove: type: array items: @@ -2609,6 +3096,11 @@ components: properties: principal: type: string + principal_id: + type: integer + description: >- + Unique identifier of the principal. For active principals, both + ``principal`` and ``principal_id`` are present. privileges: type: array items: @@ -2623,7 +3115,7 @@ components: type: string description: >- Cloud vendor of the provider's UC metastore. This field is only - present when the __authentication_type__ is **DATABRICKS**. + present when the **authentication_type** is **DATABRICKS**. comment: type: string description: Description about the provider. @@ -2637,13 +3129,19 @@ components: type: string description: >- The global UC metastore id of the data provider. This field is only - present when the __authentication_type__ is **DATABRICKS**. The - identifier is of format __cloud__:__region__:__metastore-uuid__. + present when the **authentication_type** is **DATABRICKS**. The + identifier is of format **cloud**:**region**:**metastore-uuid**. + email_recipient_id: + type: string + description: >- + The ID of the email recipient this provider is being created to + accept. Only valid on DATABRICKS CreateProvider when accepting an + email invite. metastore_id: type: string description: >- UUID of the provider's UC metastore. This field is only present when - the __authentication_type__ is **DATABRICKS**. + the **authentication_type** is **DATABRICKS**. name: type: string description: The name of the Provider. @@ -2654,17 +3152,17 @@ components: $ref: '#/components/schemas/RecipientProfile' description: >- The recipient profile. This field is only present when the - authentication_type is `TOKEN` or `OAUTH_CLIENT_CREDENTIALS`. + authentication_type is ``TOKEN`` or ``OAUTH_CLIENT_CREDENTIALS``. recipient_profile_str: type: string description: >- - This field is required when the __authentication_type__ is + This field is required when the **authentication_type** is **TOKEN**, **OAUTH_CLIENT_CREDENTIALS** or not provided. region: type: string description: >- Cloud region of the provider's UC metastore. This field is only - present when the __authentication_type__ is **DATABRICKS**. + present when the **authentication_type** is **DATABRICKS**. updated_at: type: integer description: Time at which this Provider was created, in epoch milliseconds. @@ -2686,13 +3184,33 @@ components: description: >- Full activation url to retrieve the access token. It will be empty if the token is already retrieved. + allowed_acceptance_types: + type: array + items: + $ref: '#/components/schemas/AuthenticationType' + description: >- + Authentication types the invitee may choose from when activating. + Each value must be TOKEN or DATABRICKS. Empty means all supported + types are allowed. This field is only present when the + **authentication_type** is **EMAIL**. + allowed_data_recipient_global_metastore_ids: + type: array + items: + type: string + description: >- + Optional allowlist of the data recipient's global metastore IDs + permitted to accept the invite with the DATABRICKS acceptance type. + Each ID is of the form **cloud**:**region**:**metastore-uuid**. + Empty means any metastore is allowed. If non-empty, + **allowed_acceptance_types** must include **DATABRICKS**. This field + is only present when the **authentication_type** is **EMAIL**. authentication_type: $ref: '#/components/schemas/AuthenticationType' cloud: type: string description: >- Cloud vendor of the recipient's Unity Catalog Metastore. This field - is only present when the __authentication_type__ is **DATABRICKS**. + is only present when the **authentication_type** is **DATABRICKS**. comment: type: string description: Description about the recipient. @@ -2707,8 +3225,14 @@ components: description: >- The global Unity Catalog metastore id provided by the data recipient. This field is only present when the - __authentication_type__ is **DATABRICKS**. The identifier is of - format __cloud__:__region__:__metastore-uuid__. + **authentication_type** is **DATABRICKS**. The identifier is of + format **cloud**:**region**:**metastore-uuid**. + email: + type: string + description: >- + The invited email address. This field is only present when the + **authentication_type** is **EMAIL**. The 320-character cap follows + RFC 5321 (64-octet local-part + ``@`` + 255-octet domain). expiration_time: type: integer description: Expiration timestamp of the token, in epoch milliseconds. @@ -2722,7 +3246,7 @@ components: type: string description: >- Unique identifier of recipient's Unity Catalog Metastore. This field - is only present when the __authentication_type__ is **DATABRICKS**. + is only present when the **authentication_type** is **DATABRICKS**. name: type: string description: Name of Recipient. @@ -2740,18 +3264,18 @@ components: type: string description: >- Cloud region of the recipient's Unity Catalog Metastore. This field - is only present when the __authentication_type__ is **DATABRICKS**. + is only present when the **authentication_type** is **DATABRICKS**. sharing_code: type: string description: >- The one-time sharing code provided by the data recipient. This field - is only present when the __authentication_type__ is **DATABRICKS**. + is only present when the **authentication_type** is **DATABRICKS**. tokens: type: array items: $ref: '#/components/schemas/RecipientTokenInfo' description: >- - This field is only present when the __authentication_type__ is + This field is only present when the **authentication_type** is **TOKEN**. updated_at: type: integer @@ -2805,6 +3329,43 @@ components: version_num: type: integer description: Numeric model version that alias will reference. + ReplicationConfig: + type: object + properties: + enabled: + type: boolean + description: Whether replication is enabled for the share. + schedule: + $ref: '#/components/schemas/ReplicationSchedule' + description: >- + Optional schedule describing how often replication runs for this + share. When unset, replication runs on a service-default schedule. + description: >- + Configuration for Delta Sharing replication on a share. When enabled, + the share's data is copied + into each recipient's region so recipients can query the share locally. + ReplicationInterval: + type: object + properties: + duration: + type: string + description: How often the share's data should be replicated. + description: >- + Fixed-interval replication schedule variant. The duration must be + between 30 minutes and 24 + hours; values outside that range are rejected at the API boundary. + ReplicationSchedule: + type: object + properties: + interval: + $ref: '#/components/schemas/ReplicationInterval' + description: Fixed-interval replication schedule. + description: >- + Schedule for share replication. Today only a fixed ``interval`` variant + is supported; the + ``variant`` oneof leaves room for future variants (e.g. cron) without API churn. Each arm is + wrapped in a sub-message so variant-specific metadata (e.g. jitter, drift tolerance, timezone) + can be added later without breaking the wire format. RetrieveTokenResponse: type: object properties: @@ -2828,7 +3389,7 @@ components: required: - properties description: >- - An object with __properties__ containing map of key-value properties + An object with **properties** containing map of key-value properties attached to the securable. Share: type: object @@ -2859,6 +3420,14 @@ components: owner: type: string description: Username of current owner of share. + replication_config: + $ref: '#/components/schemas/ReplicationConfig' + description: Configuration for share replication. + serverless_budget_policy_id: + type: string + description: >- + Serverless budget policy id (can only be created/updated when + calling data-sharing service) [Create,Update:IGN] storage_location: type: string description: Storage Location URL (full path) for the share. @@ -2911,6 +3480,11 @@ components: data_object_type: $ref: '#/components/schemas/SharedDataObjectDataObjectType' description: The type of the data object. + global_distribution_status: + $ref: '#/components/schemas/GlobalDistributionStatus' + description: >- + Global Distribution status for the shared asset. Output only. Only + set for TABLE objects; absent for all other data object types. history_data_sharing_status: $ref: '#/components/schemas/SharedDataObjectHistoryDataSharingStatus' description: >- @@ -2928,21 +3502,21 @@ components: share. Use this field for table-like objects (for example: TABLE, VIEW, MATERIALIZED_VIEW, STREAMING_TABLE, FOREIGN_TABLE). For non-table objects (for example: VOLUME, MODEL, NOTEBOOK_FILE, - FUNCTION), use `string_shared_as` instead. Important: For non-table - objects, this field must be omitted entirely. Format: Must be a - 2-part name `.` (e.g., + FUNCTION), use ``string_shared_as`` instead. Important: For + non-table objects, this field must be omitted entirely. Format: Must + be a 2-part name ``.`` (e.g., "sales_schema.orders_table") - Both schema and table names must contain only alphanumeric characters and underscores - No periods, spaces, forward slashes, or control characters are allowed within each part - Do not include the catalog name (use 2 parts, not 3) Behavior: - If not provided, the service automatically generates the - alias as `.
` from the object's original name - If you - don't want to specify this field, omit it entirely from the request - (do not pass an empty string) - The `shared_as` name must be unique - within the share Examples: - Valid: "analytics_schema.customer_view" - - Invalid: "catalog.analytics_schema.customer_view" (3 parts not - allowed) - Invalid: "analytics-schema.customer-view" (hyphens not - allowed) + alias as ``.
`` from the object's original name - If + you don't want to specify this field, omit it entirely from the + request (do not pass an empty string) - The ``shared_as`` name must + be unique within the share Examples: - Valid: + "analytics_schema.customer_view" - Invalid: + "catalog.analytics_schema.customer_view" (3 parts not allowed) - + Invalid: "analytics-schema.customer-view" (hyphens not allowed) start_version: type: integer description: >- @@ -2951,7 +3525,7 @@ components: clients. If specified, clients can query snapshots or changes for versions >= start_version. If not specified, clients can only query starting from the version of the object at the time it was added to - the share. NOTE: The start_version should be <= the `current` + the share. NOTE: The start_version should be <= the ``current`` version of the object. status: $ref: '#/components/schemas/SharedDataObjectStatus' @@ -2963,20 +3537,20 @@ components: share. Use this field for non-table objects (for example: VOLUME, MODEL, NOTEBOOK_FILE, FUNCTION). For table-like objects (for example: TABLE, VIEW, MATERIALIZED_VIEW, STREAMING_TABLE, - FOREIGN_TABLE), use `shared_as` instead. Important: For table-like + FOREIGN_TABLE), use ``shared_as`` instead. Important: For table-like objects, this field must be omitted entirely. Format: - For VOLUME: - Must be a 2-part name `.` (e.g., + Must be a 2-part name ``.`` (e.g., "data_schema.ml_models") - For FUNCTION: Must be a 2-part name - `.` (e.g., "udf_schema.calculate_tax") - - For MODEL: Must be a 2-part name `.` (e.g., - "models.prediction_model") - For NOTEBOOK_FILE: Should be the + ``.`` (e.g., "udf_schema.calculate_tax") + - For MODEL: Must be a 2-part name ``.`` + (e.g., "models.prediction_model") - For NOTEBOOK_FILE: Should be the notebook file name (e.g., "analysis_notebook.py") - All names must contain only alphanumeric characters and underscores - No periods, spaces, forward slashes, or control characters are allowed within each part Behavior: - If not provided, the service automatically generates the alias from the object's original name - If you don't want to specify this field, omit it entirely from the request (do - not pass an empty string) - The `string_shared_as` name must be + not pass an empty string) - The ``string_shared_as`` name must be unique for objects of the same type within the share Examples: - Valid for VOLUME: "data_schema.training_data" - Valid for FUNCTION: "analytics.calculate_revenue" - Invalid: @@ -2998,11 +3572,19 @@ components: Table: type: object properties: + accessModes: + type: array + items: + type: string comment: type: string + description: The comment of the table. id: type: string description: The id of the table. + location: + type: string + description: The cloud storage location of the table for open sharing. materialization_namespace: type: string description: The catalog and schema of the materialized table @@ -3061,6 +3643,7 @@ components: type: string x-enum: - DATABRICKS + - EMAIL - OAUTH_CLIENT_CREDENTIALS - OIDC_FEDERATION - TOKEN @@ -3077,6 +3660,8 @@ components: - DECIMAL - DOUBLE - FLOAT + - GEOGRAPHY + - GEOMETRY - INT - INTERVAL - LONG @@ -3177,6 +3762,48 @@ components: >>> list(Color) [, , ] + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + GlobalDistributionStatusStatus: + type: string + x-enum: + - DISABLED + - ENABLED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. PartitionValueOp: @@ -3647,6 +4274,16 @@ components: response: mediaType: application/json openAPIDocKey: '200' + recipient_federation_policies_update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1data-sharing~1recipients~1{recipient_name}~1federation-policies~1{name}/patch + response: + mediaType: application/json + openAPIDocKey: '200' sqlVerbs: select: - $ref: >- @@ -3656,7 +4293,9 @@ components: insert: - $ref: >- #/components/x-stackQL-resources/recipient_federation_policies/methods/create - update: [] + update: + - $ref: >- + #/components/x-stackQL-resources/recipient_federation_policies/methods/recipient_federation_policies_update delete: - $ref: >- #/components/x-stackQL-resources/recipient_federation_policies/methods/delete diff --git a/providers/src/databricks_workspace/v00.00.00000/services/sql.yaml b/providers/src/databricks_workspace/v00.00.00000/services/sql.yaml index 09195f2c..a09ffd15 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/sql.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/sql.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks sql service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.sql servers: - url: https://{deployment_name}.cloud.databricks.com @@ -142,16 +142,17 @@ paths: type: string description: >- Custom body of alert notification, if it exists. See - [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + `here + `__ + for custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This can include email subject entries and Slack notification - headers, for example. See [here] for custom templating - instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + headers, for example. See `here + `__ + for custom templating instructions. display_name: type: string description: The display name of the alert. @@ -340,14 +341,14 @@ paths: :param id: str :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :param alert: :class:`UpdateAlertRequestAlert` (optional) :param auto_resolve_display_name: bool (optional) If true, automatically resolve alert display name conflicts. Otherwise, fail the request if the @@ -371,15 +372,15 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative - to the resource object, using a dot (`.`) to navigate - sub-fields (e.g., `author.given_name`). Specification of + to the resource object, using a dot (``.``) to navigate + sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. A field mask of - `*` indicates full replacement. It’s recommended to always - explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API - changes in the future. + ``*`` indicates full replacement. It’s recommended to always + explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. alert: type: object properties: @@ -485,16 +486,17 @@ paths: type: string description: >- Custom body of alert notification, if it exists. See - [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + `here + `__ + for custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This can include email subject entries and Slack notification - headers, for example. See [here] for custom templating - instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + headers, for example. See `here + `__ + for custom templating instructions. display_name: type: string description: The display name of the alert. @@ -563,10 +565,9 @@ paths: **Warning**: This API is deprecated. Please use :method:alerts/create - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param name: str @@ -579,7 +580,7 @@ paths: The identifier of the workspace folder containing the object. :param rearm: int (optional) Number of seconds after being triggered before the alert rearms itself and can be triggered again. - If `null`, alert will never be triggered again. + If ``null``, alert will never be triggered again. :returns: :class:`LegacyAlert` requestBody: @@ -602,8 +603,8 @@ paths: op: type: string description: >- - Operator used to compare in alert evaluation: `>`, `>=`, - `<`, `<=`, `==`, `!=` + Operator used to compare in alert evaluation: ``>``, + ``>=``, ``<``, ``<=``, ``==``, ``!=`` value: type: object description: >- @@ -614,15 +615,17 @@ paths: type: string description: >- Custom body of alert notification, if it exists. See - [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + `here + `__ + for custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This includes email subject, Slack notification header, etc. - See [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + See `here + `__ + for custom templating instructions. empty_result_state: type: string x-enum: @@ -655,7 +658,7 @@ paths: type: integer description: >- Number of seconds after being triggered before the alert - rearms itself and can be triggered again. If `null`, alert + rearms itself and can be triggered again. If ``null``, alert will never be triggered again. required: - name @@ -691,10 +694,9 @@ paths: **Warning**: This API is deprecated. Please use :method:alerts/list - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ @@ -735,10 +737,9 @@ paths: **Warning**: This API is deprecated. Please use :method:alerts/delete - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param alert_id: str @@ -774,10 +775,9 @@ paths: **Warning**: This API is deprecated. Please use :method:alerts/get - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param alert_id: str @@ -820,10 +820,9 @@ paths: **Warning**: This API is deprecated. Please use :method:alerts/update - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param alert_id: str @@ -836,7 +835,7 @@ paths: Query ID. :param rearm: int (optional) Number of seconds after being triggered before the alert rearms itself and can be triggered again. - If `null`, alert will never be triggered again. + If ``null``, alert will never be triggered again. parameters: - name: alert_id in: path @@ -863,8 +862,8 @@ paths: op: type: string description: >- - Operator used to compare in alert evaluation: `>`, `>=`, - `<`, `<=`, `==`, `!=` + Operator used to compare in alert evaluation: ``>``, + ``>=``, ``<``, ``<=``, ``==``, ``!=`` value: type: object description: >- @@ -875,15 +874,17 @@ paths: type: string description: >- Custom body of alert notification, if it exists. See - [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + `here + `__ + for custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This includes email subject, Slack notification header, etc. - See [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + See `here + `__ + for custom templating instructions. empty_result_state: type: string x-enum: @@ -911,7 +912,7 @@ paths: type: integer description: >- Number of seconds after being triggered before the alert - rearms itself and can be triggered again. If `null`, alert + rearms itself and can be triggered again. If ``null``, alert will never be triggered again. required: - name @@ -983,7 +984,7 @@ paths: - SUM description: >- If not set, the behavior is equivalent to using - `First row` in the UI. + ``First row`` in the UI. display: type: string required: @@ -1009,8 +1010,8 @@ paths: - UNKNOWN description: >- Alert state if result is empty. Please avoid setting - this field to be `UNKNOWN` because `UNKNOWN` state - is planned to be deprecated. + this field to be ``UNKNOWN`` because ``UNKNOWN`` + state is planned to be deprecated. last_evaluated_at: type: string description: Timestamp of the last evaluation. @@ -1072,7 +1073,7 @@ paths: - SUM description: >- If not set, the behavior is equivalent to - using `First row` in the UI. + using ``First row`` in the UI. display: type: string required: @@ -1105,6 +1106,14 @@ paths: quartz_cron_schedule to determine the schedule. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. + effective_pause_status: + type: string + x-enum: + - PAUSED + - UNPAUSED + description: >- + The actual pause status of the schedule. This is an + output-only field. pause_status: type: string x-enum: @@ -1125,6 +1134,11 @@ paths: custom_summary: type: string description: Custom summary for the alert. support mustache template. + effective_parent_path: + type: string + description: >- + The actual workspace path of the folder containing the + alert. This is an output-only field. effective_run_as: type: object properties: @@ -1154,6 +1168,44 @@ paths: description: >- The owner's username. This field is set to "Unavailable" if the user has been deleted. + parameters: + type: array + items: + type: object + properties: + name: + type: string + description: >- + The name of the parameter, referenced in the query + as ``:name``. + type: + type: string + description: >- + The SQL data type of the parameter, e.g. STRING, + INT, or DATE. Defaults to STRING. This is a string + rather than an enum because scalar subtypes such + as DECIMAL(10, 4) cannot be enumerated. Complex + types such as ARRAY, MAP, and STRUCT are not + supported. + value: + type: string + description: >- + The bound value for the parameter, given as a + string. If omitted, the value is interpreted as + NULL. + required: + - name + description: >- + Redash-owned copy of the internal StatementParameter + for the external AlertV2 API. The internal + ``ordinal`` and ``args`` fields are intentionally omitted: the public API supports only flat, + named scalar parameters; complex types (ARRAY, MAP, STRUCT) are not supported. This mirrors + SEA's public StatementParameter schema, see: + cmdexec/sql-exec-api/proto/sql_exec_api_service.proto:763-779 + description: >- + Query parameters bound when executing the alert query, + referenced in the query text with ``:name`` syntax. + Static values only. parent_path: type: string description: >- @@ -1173,12 +1225,12 @@ paths: Specifies the identity that will be used to run the alert. This field allows you to configure alerts to run as a specific user or service principal. - For user - identity: Set `user_name` to the email of an active + identity: Set ``user_name`` to the email of an active workspace user. Users can only set this to their own email. - For service principal: Set - `service_principal_name` to the application ID. Requires - the `servicePrincipal/user` role. If not specified, the - alert will run as the request user. + ``service_principal_name`` to the application ID. + Requires the ``servicePrincipal/user`` role. If not + specified, the alert will run as the request user. run_as_user_name: type: string description: >- @@ -1186,8 +1238,8 @@ paths: principal. On Create and Update, this field can be set to application ID of an active service principal. Setting this field requires the servicePrincipal/user - role. Deprecated: Use `run_as` field instead. This field - will be removed in a future release. + role. Deprecated: Use ``run_as`` field instead. This + field will be removed in a future release. update_time: type: string description: The timestamp indicating when the alert was updated. @@ -1364,14 +1416,14 @@ paths: :param alert: :class:`AlertV2` :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`AlertV2` parameters: @@ -1389,14 +1441,14 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. requestBody: content: application/json: @@ -1435,7 +1487,7 @@ paths: - SUM description: >- If not set, the behavior is equivalent to using - `First row` in the UI. + ``First row`` in the UI. display: type: string required: @@ -1461,8 +1513,8 @@ paths: - UNKNOWN description: >- Alert state if result is empty. Please avoid setting - this field to be `UNKNOWN` because `UNKNOWN` state - is planned to be deprecated. + this field to be ``UNKNOWN`` because ``UNKNOWN`` + state is planned to be deprecated. last_evaluated_at: type: string description: Timestamp of the last evaluation. @@ -1524,7 +1576,7 @@ paths: - SUM description: >- If not set, the behavior is equivalent to - using `First row` in the UI. + using ``First row`` in the UI. display: type: string required: @@ -1557,6 +1609,14 @@ paths: quartz_cron_schedule to determine the schedule. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. + effective_pause_status: + type: string + x-enum: + - PAUSED + - UNPAUSED + description: >- + The actual pause status of the schedule. This is an + output-only field. pause_status: type: string x-enum: @@ -1577,6 +1637,11 @@ paths: custom_summary: type: string description: Custom summary for the alert. support mustache template. + effective_parent_path: + type: string + description: >- + The actual workspace path of the folder containing the + alert. This is an output-only field. effective_run_as: type: object properties: @@ -1606,6 +1671,44 @@ paths: description: >- The owner's username. This field is set to "Unavailable" if the user has been deleted. + parameters: + type: array + items: + type: object + properties: + name: + type: string + description: >- + The name of the parameter, referenced in the query + as ``:name``. + type: + type: string + description: >- + The SQL data type of the parameter, e.g. STRING, + INT, or DATE. Defaults to STRING. This is a string + rather than an enum because scalar subtypes such + as DECIMAL(10, 4) cannot be enumerated. Complex + types such as ARRAY, MAP, and STRUCT are not + supported. + value: + type: string + description: >- + The bound value for the parameter, given as a + string. If omitted, the value is interpreted as + NULL. + required: + - name + description: >- + Redash-owned copy of the internal StatementParameter + for the external AlertV2 API. The internal + ``ordinal`` and ``args`` fields are intentionally omitted: the public API supports only flat, + named scalar parameters; complex types (ARRAY, MAP, STRUCT) are not supported. This mirrors + SEA's public StatementParameter schema, see: + cmdexec/sql-exec-api/proto/sql_exec_api_service.proto:763-779 + description: >- + Query parameters bound when executing the alert query, + referenced in the query text with ``:name`` syntax. + Static values only. parent_path: type: string description: >- @@ -1625,12 +1728,12 @@ paths: Specifies the identity that will be used to run the alert. This field allows you to configure alerts to run as a specific user or service principal. - For user - identity: Set `user_name` to the email of an active + identity: Set ``user_name`` to the email of an active workspace user. Users can only set this to their own email. - For service principal: Set - `service_principal_name` to the application ID. Requires - the `servicePrincipal/user` role. If not specified, the - alert will run as the request user. + ``service_principal_name`` to the application ID. + Requires the ``servicePrincipal/user`` role. If not + specified, the alert will run as the request user. run_as_user_name: type: string description: >- @@ -1638,8 +1741,8 @@ paths: principal. On Create and Update, this field can be set to application ID of an active service principal. Setting this field requires the servicePrincipal/user - role. Deprecated: Use `run_as` field instead. This field - will be removed in a future release. + role. Deprecated: Use ``run_as`` field instead. This + field will be removed in a future release. update_time: type: string description: The timestamp indicating when the alert was updated. @@ -1687,9 +1790,9 @@ paths: Width of a widget :param text: str (optional) If this is a textbox widget, the application displays this text. This field is ignored if the widget - contains a visualization in the `visualization` field. + contains a visualization in the ``visualization`` field. :param visualization_id: str (optional) - Query Vizualization ID returned by :method:queryvisualizations/create. + Query Visualization ID returned by :method:queryvisualizations/create. :returns: :class:`Widget` requestBody: @@ -1755,11 +1858,11 @@ paths: description: >- If this is a textbox widget, the application displays this text. This field is ignored if the widget contains a - visualization in the `visualization` field. + visualization in the ``visualization`` field. visualization_id: type: string description: >- - Query Vizualization ID returned by + Query Visualization ID returned by :method:queryvisualizations/create. required: - dashboard_id @@ -1836,9 +1939,9 @@ paths: Width of a widget :param text: str (optional) If this is a textbox widget, the application displays this text. This field is ignored if the widget - contains a visualization in the `visualization` field. + contains a visualization in the ``visualization`` field. :param visualization_id: str (optional) - Query Vizualization ID returned by :method:queryvisualizations/create. + Query Visualization ID returned by :method:queryvisualizations/create. :returns: :class:`Widget` parameters: @@ -1911,11 +2014,11 @@ paths: description: >- If this is a textbox widget, the application displays this text. This field is ignored if the widget contains a - visualization in the `visualization` field. + visualization in the ``visualization`` field. visualization_id: type: string description: >- - Query Vizualization ID returned by + Query Visualization ID returned by :method:queryvisualizations/create. required: - dashboard_id @@ -1957,10 +2060,9 @@ paths: **Warning**: This API is deprecated. Please use the AI/BI Dashboards API - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/dashboards/ + `__ :param dashboard_id: str @@ -1999,10 +2101,9 @@ paths: **Warning**: This API is deprecated. Please use the AI/BI Dashboards API - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/dashboards/ + `__ :param dashboard_id: str @@ -2053,10 +2154,9 @@ paths: **Warning**: This API is deprecated. Please use the AI/BI Dashboards API - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/dashboards/ + `__ :param dashboard_id: str @@ -2064,8 +2164,8 @@ paths: :param name: str (optional) The title of this dashboard that appears in list views and at the top of the dashboard page. :param run_as_role: :class:`RunAsRole` (optional) - Sets the **Run as** role for the object. Must be set to one of `"viewer"` (signifying "run as - viewer" behavior) or `"owner"` (signifying "run as owner" behavior) + Sets the **Run as** role for the object. Must be set to one of ``"viewer"`` (signifying "run as + viewer" behavior) or ``"owner"`` (signifying "run as owner" behavior) :param tags: List[str] (optional) @@ -2094,8 +2194,8 @@ paths: - viewer description: >- Sets the **Run as** role for the object. Must be set to one - of `"viewer"` (signifying "run as viewer" behavior) or - `"owner"` (signifying "run as owner" behavior) + of ``"viewer"`` (signifying "run as viewer" behavior) or + ``"owner"`` (signifying "run as owner" behavior) tags: type: array items: @@ -2137,10 +2237,9 @@ paths: **Warning**: This API is deprecated. Please use the AI/BI Dashboards API - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/dashboards/ + `__ :param order: :class:`ListOrder` (optional) @@ -2212,10 +2311,9 @@ paths: **Warning**: This API is deprecated. Please use the AI/BI Dashboards API - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/dashboards/ + `__ :param dashboard_id: str @@ -2254,16 +2352,15 @@ paths: fields that appear in this API response are enumerated for clarity. However, you need only a SQL - warehouse's `id` to create new + warehouse's ``id`` to create new queries against it. **Warning**: This API is deprecated. Please use :method:warehouses/list - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ @@ -2302,10 +2399,9 @@ paths: **Warning**: This API is deprecated. Please use - :method:workspace/getpermissions instead. [Learn more] - + :method:workspace/getpermissions instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param object_type: :class:`ObjectTypePlural` @@ -2367,10 +2463,9 @@ paths: **Warning**: This API is deprecated. Please use - :method:workspace/setpermissions instead. [Learn more] + :method:workspace/setpermissions instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param object_type: :class:`ObjectTypePlural` @@ -2424,9 +2519,9 @@ paths: - CAN_RUN - CAN_VIEW description: >- - * `CAN_VIEW`: Can view the query * `CAN_RUN`: Can run - the query * `CAN_EDIT`: Can edit the query * - `CAN_MANAGE`: Can manage the query + - ``CAN_VIEW``: Can view the query - ``CAN_RUN``: Can + run the query - ``CAN_EDIT``: Can edit the query - + ``CAN_MANAGE``: Can manage the query user_name: type: string required: @@ -2467,10 +2562,9 @@ paths: **Warning**: This API is deprecated. For queries and alerts, please use :method:queries/update and - :method:alerts/update respectively instead. [Learn more] - + :method:alerts/update respectively instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param object_type: :class:`OwnableObjectType` @@ -2656,7 +2750,7 @@ paths: DAY_PRECISION (YYYY-MM-DD). description: >- Date query parameter value. Can only specify one - of `dynamic_date_value` or `date_value`. + of ``dynamic_date_value`` or ``date_value``. enum_value: type: object properties: @@ -2923,14 +3017,14 @@ paths: :param id: str :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :param auto_resolve_display_name: bool (optional) If true, automatically resolve alert display name conflicts. Otherwise, fail the request if the alert's display name conflicts with an existing alert's display name. @@ -2954,15 +3048,15 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative - to the resource object, using a dot (`.`) to navigate - sub-fields (e.g., `author.given_name`). Specification of + to the resource object, using a dot (``.``) to navigate + sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. A field mask of - `*` indicates full replacement. It’s recommended to always - explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API - changes in the future. + ``*`` indicates full replacement. It’s recommended to always + explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. auto_resolve_display_name: type: boolean description: >- @@ -3068,7 +3162,7 @@ paths: DAY_PRECISION (YYYY-MM-DD). description: >- Date query parameter value. Can only specify one - of `dynamic_date_value` or `date_value`. + of ``dynamic_date_value`` or ``date_value``. enum_value: type: object properties: @@ -3256,45 +3350,42 @@ paths: making the request. - The `data_source_id` field specifies the ID of the SQL warehouse to run - this query against. You can + The ``data_source_id`` field specifies the ID of the SQL warehouse to + run this query against. You can use the Data Sources API to see a complete list of available SQL warehouses. Or you can copy the - `data_source_id` from an existing query. + ``data_source_id`` from an existing query. **Note**: You cannot add a visualization until you create the query. **Warning**: This API is deprecated. Please use :method:queries/create - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param data_source_id: str (optional) Data source ID maps to the ID of the data source used by the resource and is distinct from the - warehouse ID. [Learn more] - - [Learn more]: https://docs.databricks.com/api/workspace/datasources/list + warehouse ID. `Learn more `__ :param description: str (optional) General description that conveys additional information about this query such as usage notes. :param name: str (optional) The title of this query that appears in list views, widget headings, and on the query page. :param options: Any (optional) - Exclusively used for storing a list parameter definitions. A parameter is an object with `title`, - `name`, `type`, and `value` properties. The `value` field here is the default value. It can be - overridden at runtime. + Exclusively used for storing a list parameter definitions. A parameter is an object with ``title``, + ``name``, ``type``, and ``value`` properties. The ``value`` field here is the default value. It can + be overridden at runtime. :param parent: str (optional) The identifier of the workspace folder containing the object. :param query: str (optional) The text of the query to be run. :param run_as_role: :class:`RunAsRole` (optional) - Sets the **Run as** role for the object. Must be set to one of `"viewer"` (signifying "run as - viewer" behavior) or `"owner"` (signifying "run as owner" behavior) + Sets the **Run as** role for the object. Must be set to one of ``"viewer"`` (signifying "run as + viewer" behavior) or ``"owner"`` (signifying "run as owner" behavior) :param tags: List[str] (optional) @@ -3309,9 +3400,8 @@ paths: type: string description: >- Data source ID maps to the ID of the data source used by the - resource and is distinct from the warehouse ID. [Learn more] - [Learn more]: - https://docs.databricks.com/api/workspace/datasources/list + resource and is distinct from the warehouse ID. `Learn more + `__ description: type: string description: >- @@ -3326,9 +3416,9 @@ paths: type: object description: >- Exclusively used for storing a list parameter definitions. A - parameter is an object with `title`, `name`, `type`, and - `value` properties. The `value` field here is the default - value. It can be overridden at runtime. + parameter is an object with ``title``, ``name``, ``type``, + and ``value`` properties. The ``value`` field here is the + default value. It can be overridden at runtime. parent: type: string description: >- @@ -3344,8 +3434,8 @@ paths: - viewer description: >- Sets the **Run as** role for the object. Must be set to one - of `"viewer"` (signifying "run as viewer" behavior) or - `"owner"` (signifying "run as owner" behavior) + of ``"viewer"`` (signifying "run as viewer" behavior) or + ``"owner"`` (signifying "run as owner" behavior) tags: type: array items: @@ -3389,26 +3479,21 @@ paths: **Warning**: This API is deprecated. Please use :method:queries/list - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param order: str (optional) - Name of query attribute to order by. Default sort order is ascending. Append a dash (`-`) to order + Name of query attribute to order by. Default sort order is ascending. Append a dash (``-``) to order descending instead. - - `name`: The name of the query. - - - `created_at`: The timestamp the query was created. - - - `runtime`: The time it took to run this query. This is blank for parameterized queries. A blank - value is treated as the highest value for sorting. - - - `executed_at`: The timestamp when the query was last run. - - - `created_by`: The user name of the user that created the query. + - ``name``: The name of the query. + - ``created_at``: The timestamp the query was created. + - ``runtime``: The time it took to run this query. This is blank for parameterized queries. A blank + value is treated as the highest value for sorting. + - ``executed_at``: The timestamp when the query was last run. + - ``created_by``: The user name of the user that created the query. :param page: int (optional) Page number to retrieve. :param page_size: int (optional) @@ -3425,13 +3510,13 @@ paths: type: string description: >- Name of query attribute to order by. Default sort order is - ascending. Append a dash (`-`) to order descending instead. - - `name`: The name of the query. - `created_at`: The timestamp the - query was created. - `runtime`: The time it took to run this query. - This is blank for parameterized queries. A blank value is treated as - the highest value for sorting. - `executed_at`: The timestamp when - the query was last run. - `created_by`: The user name of the user - that created the query. + ascending. Append a dash (``-``) to order descending instead. - + ``name``: The name of the query. - ``created_at``: The timestamp the + query was created. - ``runtime``: The time it took to run this + query. This is blank for parameterized queries. A blank value is + treated as the highest value for sorting. - ``executed_at``: The + timestamp when the query was last run. - ``created_by``: The user + name of the user that created the query. - name: page in: query required: false @@ -3486,10 +3571,9 @@ paths: **Warning**: This API is deprecated. Please use :method:queries/delete - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param query_id: str @@ -3530,10 +3614,9 @@ paths: **Warning**: This API is deprecated. Please use :method:queries/get - instead. [Learn more] + instead. `Learn more - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param query_id: str @@ -3579,32 +3662,29 @@ paths: **Warning**: This API is deprecated. Please use :method:queries/update - instead. [Learn more] - + instead. `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param query_id: str :param data_source_id: str (optional) Data source ID maps to the ID of the data source used by the resource and is distinct from the - warehouse ID. [Learn more] - - [Learn more]: https://docs.databricks.com/api/workspace/datasources/list + warehouse ID. `Learn more `__ :param description: str (optional) General description that conveys additional information about this query such as usage notes. :param name: str (optional) The title of this query that appears in list views, widget headings, and on the query page. :param options: Any (optional) - Exclusively used for storing a list parameter definitions. A parameter is an object with `title`, - `name`, `type`, and `value` properties. The `value` field here is the default value. It can be - overridden at runtime. + Exclusively used for storing a list parameter definitions. A parameter is an object with ``title``, + ``name``, ``type``, and ``value`` properties. The ``value`` field here is the default value. It can + be overridden at runtime. :param query: str (optional) The text of the query to be run. :param run_as_role: :class:`RunAsRole` (optional) - Sets the **Run as** role for the object. Must be set to one of `"viewer"` (signifying "run as - viewer" behavior) or `"owner"` (signifying "run as owner" behavior) + Sets the **Run as** role for the object. Must be set to one of ``"viewer"`` (signifying "run as + viewer" behavior) or ``"owner"`` (signifying "run as owner" behavior) :param tags: List[str] (optional) @@ -3625,9 +3705,8 @@ paths: type: string description: >- Data source ID maps to the ID of the data source used by the - resource and is distinct from the warehouse ID. [Learn more] - [Learn more]: - https://docs.databricks.com/api/workspace/datasources/list + resource and is distinct from the warehouse ID. `Learn more + `__ description: type: string description: >- @@ -3642,9 +3721,9 @@ paths: type: object description: >- Exclusively used for storing a list parameter definitions. A - parameter is an object with `title`, `name`, `type`, and - `value` properties. The `value` field here is the default - value. It can be overridden at runtime. + parameter is an object with ``title``, ``name``, ``type``, + and ``value`` properties. The ``value`` field here is the + default value. It can be overridden at runtime. query: type: string description: The text of the query to be run. @@ -3655,8 +3734,8 @@ paths: - viewer description: >- Sets the **Run as** role for the object. Must be set to one - of `"viewer"` (signifying "run as viewer" behavior) or - `"owner"` (signifying "run as owner" behavior) + of ``"viewer"`` (signifying "run as viewer" behavior) or + ``"owner"`` (signifying "run as owner" behavior) tags: type: array items: @@ -3697,10 +3776,9 @@ paths: **Warning**: This API is deprecated. Please see the latest version. - [Learn more] - + `Learn more - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + `__ :param query_id: str @@ -3751,7 +3829,7 @@ paths: :param filter_by: :class:`QueryFilter` (optional) An optional filter object to limit query history results. Accepts parameters such as user IDs, endpoint IDs, and statuses to narrow the returned data. In a URL, the parameters of this filter are - specified with dot notation. For example: `filter_by.statement_ids`. + specified with dot notation. For example: ``filter_by.statement_ids``. :param include_metrics: bool (optional) Whether to include the query metrics with each query. Only use this for a small subset of queries (max_results). Defaults to false. @@ -3773,7 +3851,8 @@ paths: An optional filter object to limit query history results. Accepts parameters such as user IDs, endpoint IDs, and statuses to narrow the returned data. In a URL, the parameters of this filter are - specified with dot notation. For example: `filter_by.statement_ids`. + specified with dot notation. For example: + ``filter_by.statement_ids``. - name: include_metrics in: query required: false @@ -3933,14 +4012,14 @@ paths: :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :param visualization: :class:`UpdateVisualizationRequestVisualization` (optional) @@ -3963,15 +4042,15 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative - to the resource object, using a dot (`.`) to navigate - sub-fields (e.g., `author.given_name`). Specification of + to the resource object, using a dot (``.``) to navigate + sub-fields (e.g., ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. A field mask of - `*` indicates full replacement. It’s recommended to always - explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API - changes in the future. + ``*`` indicates full replacement. It’s recommended to always + explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. visualization: type: object properties: @@ -4029,12 +4108,9 @@ paths: **Warning**: This API is deprecated. Please use - :method:queryvisualizations/create instead. [Learn - - more] + :method:queryvisualizations/create instead. `Learn - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + more `__ :param options: Any @@ -4115,12 +4191,9 @@ paths: **Warning**: This API is deprecated. Please use - :method:queryvisualizations/delete instead. [Learn - - more] + :method:queryvisualizations/delete instead. `Learn - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + more `__ :param id: str @@ -4158,12 +4231,9 @@ paths: **Warning**: This API is deprecated. Please use - :method:queryvisualizations/update instead. [Learn - - more] + :method:queryvisualizations/update instead. `Learn - - [Learn more]: https://docs.databricks.com/en/sql/dbsql-api-latest.html + more `__ :param created_at: str (optional) @@ -4229,8 +4299,8 @@ paths: description: >- Data source ID maps to the ID of the data source used by the resource and is distinct from the warehouse ID. - [Learn more] [Learn more]: - https://docs.databricks.com/api/workspace/datasources/list + `Learn more + `__ description: type: string description: >- @@ -4244,8 +4314,8 @@ paths: description: >- Indicates whether the query is trashed. Trashed queries can't be used in dashboards, or appear in search - results. If this boolean is `true`, the `options` - property for this query includes a `moved_to_trash_at` + results. If this boolean is ``true``, the ``options`` + property for this query includes a ``moved_to_trash_at`` timestamp. Trashed queries are permanently deleted after 30 days. is_draft: @@ -4265,7 +4335,7 @@ paths: description: >- Text parameter types are not safe from SQL injection for all types of data source. Set this Boolean parameter to - `true` if a query either does not use any text type + ``true`` if a query either does not use any text type parameters or uses a data source type where text type parameters are handled safely. last_modified_by: @@ -4300,8 +4370,9 @@ paths: type: string description: >- The timestamp when this query was moved to trash. - Only present when the `is_archived` property is - `true`. Trashed items are deleted after thirty days. + Only present when the ``is_archived`` property is + ``true``. Trashed items are deleted after thirty + days. parameters: type: array items: @@ -4373,9 +4444,9 @@ paths: - CAN_RUN - CAN_VIEW description: >- - * `CAN_VIEW`: Can view the query * `CAN_RUN`: Can run - the query * `CAN_EDIT`: Can edit the query * - `CAN_MANAGE`: Can manage the query + - ``CAN_VIEW``: Can view the query - ``CAN_RUN``: Can + run the query - ``CAN_EDIT``: Can edit the query - + ``CAN_MANAGE``: Can manage the query query: type: string description: The text of the query to be run. @@ -4391,8 +4462,9 @@ paths: - viewer description: >- Sets the **Run as** role for the object. Must be set to - one of `"viewer"` (signifying "run as viewer" behavior) - or `"owner"` (signifying "run as owner" behavior) + one of ``"viewer"`` (signifying "run as viewer" + behavior) or ``"owner"`` (signifying "run as owner" + behavior) tags: type: array items: @@ -4451,7 +4523,7 @@ paths: description: >- The visualization description API changes frequently and is unsupported. You can duplicate a - visualization by copying description objects received _from the API_ and then using them to + visualization by copying description objects received *from the API* and then using them to create a new one with a POST request to the same endpoint. Databricks does not recommend constructing ad-hoc visualizations entirely in JSON. type: @@ -4573,25 +4645,25 @@ paths: For flows that generate small and predictable result sets (<= 25 MiB), - `INLINE` responses of + ``INLINE`` responses of - `JSON_ARRAY` result data are typically the simplest way to execute and + ``JSON_ARRAY`` result data are typically the simplest way to execute and fetch result data. **Use case: large result sets with EXTERNAL_LINKS** - Using `EXTERNAL_LINKS` to fetch result data allows you to fetch large + Using ``EXTERNAL_LINKS`` to fetch result data allows you to fetch large result sets efficiently. The - main differences from using `INLINE` disposition are that the result + main differences from using ``INLINE`` disposition are that the result data is accessed with URLs, and - that there are 3 supported formats: `JSON_ARRAY`, `ARROW_STREAM` and - `CSV` compared to only + that there are 3 supported formats: ``JSON_ARRAY``, ``ARROW_STREAM`` and + ``CSV`` compared to only - `JSON_ARRAY` with `INLINE`. + ``JSON_ARRAY`` with ``INLINE``. ** URLs** @@ -4601,24 +4673,21 @@ paths: storage, in the form of a URL. The URLs are valid for only a short period, <= 15 minutes. Alongside - each `external_link` is an + each ``external_link`` is an expiration field indicating the time at which the URL is no longer - valid. In `EXTERNAL_LINKS` mode, + valid. In ``EXTERNAL_LINKS`` mode, chunks can be resolved and fetched multiple times and in parallel. - ---- - + **Warning: Databricks strongly recommends that you protect the URLs that + are returned by the - ### **Warning: Databricks strongly recommends that you protect the URLs - that are returned by the + ``EXTERNAL_LINKS`` disposition.** - `EXTERNAL_LINKS` disposition.** - - When you use the `EXTERNAL_LINKS` disposition, a short-lived, URL is + When you use the ``EXTERNAL_LINKS`` disposition, a short-lived, URL is generated, which can be used to download the results directly from . As a short-lived is embedded in @@ -4628,173 +4697,176 @@ paths: Because URLs are already generated with embedded temporary s, you must - not set an `Authorization` + not set an ``Authorization`` header in the download requests. - The `EXTERNAL_LINKS` disposition can be disabled upon request by + The ``EXTERNAL_LINKS`` disposition can be disabled upon request by creating a support case. - See also [Security best practices]. - + See also `Security best practices - ---- + `__. - StatementResponse contains `statement_id` and `status`; other fields + StatementResponse contains ``statement_id`` and ``status``; other fields might be absent or present depending on context. If the SQL warehouse fails to execute the provided statement, a 200 response is - returned with `status.state` set to `FAILED` (in contrast to a failure - when accepting the request, + returned with ``status.state`` set to ``FAILED`` (in contrast to a + failure when accepting the request, which results in a non-200 response). Details of the error can be found - at `status.error` in case of + at ``status.error`` in case of execution failures. - [Security best practices]: - https://docs.databricks.com/sql/admin/sql-execution-tutorial.html#security-best-practices - - :param statement: str - The SQL statement to execute. The statement can optionally be parameterized, see `parameters`. The + The SQL statement to execute. The statement can optionally be parameterized, see ``parameters``. The maximum query text size is 16 MiB. :param warehouse_id: str - Warehouse upon which to execute a statement. See also [What are SQL warehouses?] - - [What are SQL warehouses?]: https://docs.databricks.com/sql/admin/warehouse-type.html + Warehouse upon which to execute a statement. See also `What are SQL warehouses? + `__ :param byte_limit: int (optional) Applies the given byte limit to the statement's result size. Byte counts are based on internal data - representations and might not match the final size in the requested `format`. If the result was - truncated due to the byte limit, then `truncated` in the response is set to `true`. When using - `EXTERNAL_LINKS` disposition, a default `byte_limit` of 100 GiB is applied if `byte_limit` is not - explicitly set. + representations and might not match the final size in the requested ``format``. If the result was + truncated due to the byte limit, then ``truncated`` in the response is set to ``true``. When using + ``EXTERNAL_LINKS`` disposition, a default ``byte_limit`` of 100 GiB is applied if ``byte_limit`` is + not explicitly set. :param catalog: str (optional) - Sets default catalog for statement execution, similar to [`USE CATALOG`] in SQL. - - [`USE CATALOG`]: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-use-catalog.html + Sets default catalog for statement execution, similar to `USE CATALOG + `__ in SQL. :param disposition: :class:`Disposition` (optional) - The fetch disposition provides two modes of fetching results: `INLINE` and `EXTERNAL_LINKS`. + The fetch disposition provides two modes of fetching results: ``INLINE`` and ``EXTERNAL_LINKS``. - Statements executed with `INLINE` disposition will return result data inline, in `JSON_ARRAY` + Statements executed with ``INLINE`` disposition will return result data inline, in ``JSON_ARRAY`` format, in a series of chunks. If a given statement produces a result set with a size larger than 25 MiB, that statement execution is aborted, and no result set will be available. **NOTE** Byte limits are computed based upon internal representations of the result set data, and might not match the sizes visible in JSON responses. - Statements executed with `EXTERNAL_LINKS` disposition will return result data as external links: - URLs that point to cloud storage internal to the workspace. Using `EXTERNAL_LINKS` disposition + Statements executed with ``EXTERNAL_LINKS`` disposition will return result data as external links: + URLs that point to cloud storage internal to the workspace. Using ``EXTERNAL_LINKS`` disposition allows statements to generate arbitrarily sized result sets for fetching up to 100 GiB. The resulting links have two important properties: - 1. They point to resources _external_ to the Databricks compute; therefore any associated - authentication information (typically a personal access token, OAuth token, or similar) _must be - removed_ when fetching from these links. - + 1. They point to resources *external* to the Databricks compute; therefore any associated + authentication information (typically a personal access token, OAuth token, or similar) *must be + removed* when fetching from these links. 2. These are URLs with a specific expiration, indicated in the response. The behavior when - attempting to use an expired link is cloud specific. + attempting to use an expired link is cloud specific. :param format: :class:`Format` (optional) - Statement execution supports three result formats: `JSON_ARRAY` (default), `ARROW_STREAM`, and - `CSV`. + Statement execution supports three result formats: ``JSON_ARRAY`` (default), ``ARROW_STREAM``, and + ``CSV``. - Important: The formats `ARROW_STREAM` and `CSV` are supported only with `EXTERNAL_LINKS` - disposition. `JSON_ARRAY` is supported in `INLINE` and `EXTERNAL_LINKS` disposition. + Important: The formats ``ARROW_STREAM`` and ``CSV`` are supported only with ``EXTERNAL_LINKS`` + disposition. ``JSON_ARRAY`` is supported in ``INLINE`` and ``EXTERNAL_LINKS`` disposition. - When specifying `format=JSON_ARRAY`, result data will be formatted as an array of arrays of values, - where each value is either the *string representation* of a value, or `null`. For example, the - output of `SELECT concat('id-', id) AS strCol, id AS intCol, null AS nullCol FROM range(3)` would - look like this: + When specifying ``format=JSON_ARRAY``, result data will be formatted as an array of arrays of + values, where each value is either the *string representation* of a value, or ``null``. For example, + the output of ``SELECT concat('id-', id) AS strCol, id AS intCol, null AS nullCol FROM range(3)`` + would look like this: - ``` [ [ "id-1", "1", null ], [ "id-2", "2", null ], [ "id-3", "3", null ], ] ``` + .. code-block:: - When specifying `format=JSON_ARRAY` and `disposition=EXTERNAL_LINKS`, each chunk in the result + [ + [ "id-1", "1", null ], + [ "id-2", "2", null ], + [ "id-3", "3", null ], + ] + + When specifying ``format=JSON_ARRAY`` and ``disposition=EXTERNAL_LINKS``, each chunk in the result contains compact JSON with no indentation or extra whitespace. - When specifying `format=ARROW_STREAM` and `disposition=EXTERNAL_LINKS`, each chunk in the result - will be formatted as Apache Arrow Stream. See the [Apache Arrow streaming format]. + When specifying ``format=ARROW_STREAM`` and ``disposition=EXTERNAL_LINKS``, each chunk in the result + will be formatted as Apache Arrow Stream. See the `Apache Arrow streaming format + `__. - When specifying `format=CSV` and `disposition=EXTERNAL_LINKS`, each chunk in the result will be a - CSV according to [RFC 4180] standard. All the columns values will have *string representation* - similar to the `JSON_ARRAY` format, and `null` values will be encoded as “null”. Only the first - chunk in the result would contain a header row with column names. For example, the output of `SELECT - concat('id-', id) AS strCol, id AS intCol, null as nullCol FROM range(3)` would look like this: + When specifying ``format=CSV`` and ``disposition=EXTERNAL_LINKS``, each chunk in the result will be + a CSV according to `RFC 4180 `__ standard. All the columns + values will have *string representation* similar to the ``JSON_ARRAY`` format, and ``null`` values + will be encoded as “null”. Only the first chunk in the result would contain a header row with + column names. For example, the output of ``SELECT concat('id-', id) AS strCol, id AS intCol, null as + nullCol FROM range(3)`` would look like this: - ``` strCol,intCol,nullCol id-1,1,null id-2,2,null id-3,3,null ``` + .. code-block:: - [Apache Arrow streaming format]: https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format - [RFC 4180]: https://www.rfc-editor.org/rfc/rfc4180 + strCol,intCol,nullCol + id-1,1,null + id-2,2,null + id-3,3,null :param on_wait_timeout: :class:`ExecuteStatementRequestOnWaitTimeout` (optional) - When `wait_timeout > 0s`, the call will block up to the specified time. If the statement execution - doesn't finish within this time, `on_wait_timeout` determines whether the execution should continue - or be canceled. When set to `CONTINUE`, the statement execution continues asynchronously and the - call returns a statement ID which can be used for polling with - :method:statementexecution/getStatement. When set to `CANCEL`, the statement execution is canceled - and the call returns with a `CANCELED` state. + When ``wait_timeout > 0s``, the call will block up to the specified time. If the statement execution + doesn't finish within this time, ``on_wait_timeout`` determines whether the execution should + continue or be canceled. When set to ``CONTINUE``, the statement execution continues asynchronously + and the call returns a statement ID which can be used for polling with + :method:statementexecution/getStatement. When set to ``CANCEL``, the statement execution is canceled + and the call returns with a ``CANCELED`` state. :param parameters: List[:class:`StatementParameterListItem`] (optional) A list of parameters to pass into a SQL statement containing parameter markers. A parameter consists - of a name, a value, and optionally a type. To represent a NULL value, the `value` field may be - omitted or set to `null` explicitly. If the `type` field is omitted, the value is interpreted as a - string. + of a name, a value, and optionally a type. To represent a NULL value, the ``value`` field may be + omitted or set to ``null`` explicitly. If the ``type`` field is omitted, the value is interpreted as + a string. If the type is given, parameters will be checked for type correctness according to the given type. A - value is correct if the provided string can be converted to the requested type using the `cast` - function. The exact semantics are described in the section [`cast` function] of the SQL language + value is correct if the provided string can be converted to the requested type using the ``cast`` + function. The exact semantics are described in the section `cast function + `__ of the SQL language reference. - For example, the following statement contains two parameters, `my_name` and `my_date`: + For example, the following statement contains two parameters, ``my_name`` and ``my_date``: + + .. code-block:: - ``` SELECT * FROM my_table WHERE name = :my_name AND date = :my_date ``` + SELECT * FROM my_table WHERE name = :my_name AND date = :my_date The parameters can be passed in the request body as follows: - ` { ..., "statement": "SELECT * FROM my_table WHERE name = :my_name AND date = :my_date", + ``{ ..., "statement": "SELECT * FROM my_table WHERE name = :my_name AND date = :my_date", "parameters": [ { "name": "my_name", "value": "the name" }, { "name": "my_date", "value": - "2020-01-01", "type": "DATE" } ] } ` + "2020-01-01", "type": "DATE" } ] }`` - Currently, positional parameters denoted by a `?` marker are not supported by the Databricks SQL + Currently, positional parameters denoted by a ``?`` marker are not supported by the Databricks SQL Statement Execution API. - Also see the section [Parameter markers] of the SQL language reference. - - [Parameter markers]: https://docs.databricks.com/sql/language-manual/sql-ref-parameter-marker.html - [`cast` function]: https://docs.databricks.com/sql/language-manual/functions/cast.html + Also see the section `Parameter markers + `__ of the SQL + language reference. :param query_tags: List[:class:`QueryTag`] (optional) An array of query tags to annotate a SQL statement. A query tag consists of a non-empty key and, - optionally, a value. To represent a NULL value, either omit the `value` field or manually set it to - `null` or white space. Refer to the SQL language reference for the format specification of query - tags. There's no significance to the order of tags. Only one value per key will be recorded. A + optionally, a value. To represent a NULL value, either omit the ``value`` field or manually set it + to ``null`` or white space. Refer to the SQL language reference for the format specification of + query tags. There's no significance to the order of tags. Only one value per key will be recorded. A sequence in excess of 20 query tags will be coerced to 20. Example: { ..., "query_tags": [ { "key": "team", "value": "eng" }, { "key": "some key only tag" } ] } :param row_limit: int (optional) - Applies the given row limit to the statement's result set, but unlike the `LIMIT` clause in SQL, it - also sets the `truncated` field in the response to indicate whether the result was trimmed due to - the limit or not. + Applies the given row limit to the statement's result set, but unlike the ``LIMIT`` clause in SQL, + it also sets the ``truncated`` field in the response to indicate whether the result was trimmed due + to the limit or not. :param schema: str (optional) - Sets default schema for statement execution, similar to [`USE SCHEMA`] in SQL. - - [`USE SCHEMA`]: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-use-schema.html + Sets default schema for statement execution, similar to `USE SCHEMA + `__ in SQL. :param wait_timeout: str (optional) - The time in seconds the call will wait for the statement's result set as `Ns`, where `N` can be set - to 0 or to a value between 5 and 50. + The time in seconds the call will wait for the statement's result set as ``Ns``, where ``N`` can be + set to 0 or to a value between 5 and 50. - When set to `0s`, the statement will execute in asynchronous mode and the call will not wait for the - execution to finish. In this case, the call returns directly with `PENDING` state and a statement ID - which can be used for polling with :method:statementexecution/getStatement. + When set to ``0s``, the statement will execute in asynchronous mode and the call will not wait for + the execution to finish. In this case, the call returns directly with ``PENDING`` state and a + statement ID which can be used for polling with :method:statementexecution/getStatement. When set between 5 and 50 seconds, the call will behave synchronously up to this timeout and wait for the statement execution to finish. If the execution finishes within this time, the call returns - immediately with a manifest and result data (or a `FAILED` state in case of an execution error). If - the statement takes longer to execute, `on_wait_timeout` determines what should happen after the - timeout is reached. + immediately with a manifest and result data (or a ``FAILED`` state in case of an execution error). + If the statement takes longer to execute, ``on_wait_timeout`` determines what should happen after + the timeout is reached. :returns: :class:`StatementResponse` requestBody: @@ -4807,30 +4879,31 @@ paths: type: string description: >- The SQL statement to execute. The statement can optionally - be parameterized, see `parameters`. The maximum query text + be parameterized, see ``parameters``. The maximum query text size is 16 MiB. warehouse_id: type: string description: >- - Warehouse upon which to execute a statement. See also [What - are SQL warehouses?] [What are SQL warehouses?]: - https://docs.databricks.com/sql/admin/warehouse-type.html + Warehouse upon which to execute a statement. See also `What + are SQL warehouses? + `__ byte_limit: type: integer description: >- Applies the given byte limit to the statement's result size. Byte counts are based on internal data representations and - might not match the final size in the requested `format`. If - the result was truncated due to the byte limit, then - `truncated` in the response is set to `true`. When using - `EXTERNAL_LINKS` disposition, a default `byte_limit` of 100 - GiB is applied if `byte_limit` is not explicitly set. + might not match the final size in the requested ``format``. + If the result was truncated due to the byte limit, then + ``truncated`` in the response is set to ``true``. When using + ``EXTERNAL_LINKS`` disposition, a default ``byte_limit`` of + 100 GiB is applied if ``byte_limit`` is not explicitly set. catalog: type: string description: >- Sets default catalog for statement execution, similar to - [`USE CATALOG`] in SQL. [`USE CATALOG`]: - https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-use-catalog.html + `USE CATALOG + `__ + in SQL. disposition: type: string x-enum: @@ -4838,24 +4911,24 @@ paths: - INLINE description: >- The fetch disposition provides two modes of fetching - results: `INLINE` and `EXTERNAL_LINKS`. Statements executed - with `INLINE` disposition will return result data inline, in - `JSON_ARRAY` format, in a series of chunks. If a given - statement produces a result set with a size larger than 25 - MiB, that statement execution is aborted, and no result set - will be available. **NOTE** Byte limits are computed based - upon internal representations of the result set data, and - might not match the sizes visible in JSON responses. - Statements executed with `EXTERNAL_LINKS` disposition will - return result data as external links: URLs that point to - cloud storage internal to the workspace. Using - `EXTERNAL_LINKS` disposition allows statements to generate + results: ``INLINE`` and ``EXTERNAL_LINKS``. Statements + executed with ``INLINE`` disposition will return result data + inline, in ``JSON_ARRAY`` format, in a series of chunks. If + a given statement produces a result set with a size larger + than 25 MiB, that statement execution is aborted, and no + result set will be available. **NOTE** Byte limits are + computed based upon internal representations of the result + set data, and might not match the sizes visible in JSON + responses. Statements executed with ``EXTERNAL_LINKS`` + disposition will return result data as external links: URLs + that point to cloud storage internal to the workspace. Using + ``EXTERNAL_LINKS`` disposition allows statements to generate arbitrarily sized result sets for fetching up to 100 GiB. The resulting links have two important properties: 1. They - point to resources _external_ to the Databricks compute; + point to resources *external* to the Databricks compute; therefore any associated authentication information (typically a personal access token, OAuth token, or similar) - _must be removed_ when fetching from these links. 2. These + *must be removed* when fetching from these links. 2. These are URLs with a specific expiration, indicated in the response. The behavior when attempting to use an expired link is cloud specific. @@ -4867,51 +4940,54 @@ paths: - JSON_ARRAY description: >- Statement execution supports three result formats: - `JSON_ARRAY` (default), `ARROW_STREAM`, and `CSV`. - Important: The formats `ARROW_STREAM` and `CSV` are - supported only with `EXTERNAL_LINKS` disposition. - `JSON_ARRAY` is supported in `INLINE` and `EXTERNAL_LINKS` - disposition. When specifying `format=JSON_ARRAY`, result - data will be formatted as an array of arrays of values, - where each value is either the *string representation* of a - value, or `null`. For example, the output of `SELECT - concat('id-', id) AS strCol, id AS intCol, null AS nullCol - FROM range(3)` would look like this: ``` [ [ "id-1", "1", - null ], [ "id-2", "2", null ], [ "id-3", "3", null ], ] ``` - When specifying `format=JSON_ARRAY` and - `disposition=EXTERNAL_LINKS`, each chunk in the result + ``JSON_ARRAY`` (default), ``ARROW_STREAM``, and ``CSV``. + Important: The formats ``ARROW_STREAM`` and ``CSV`` are + supported only with ``EXTERNAL_LINKS`` disposition. + ``JSON_ARRAY`` is supported in ``INLINE`` and + ``EXTERNAL_LINKS`` disposition. When specifying + ``format=JSON_ARRAY``, result data will be formatted as an + array of arrays of values, where each value is either the + *string representation* of a value, or ``null``. For + example, the output of ``SELECT concat('id-', id) AS strCol, + id AS intCol, null AS nullCol FROM range(3)`` would look + like this: .. code-block:: [ [ "id-1", "1", null ], [ + "id-2", "2", null ], [ "id-3", "3", null ], ] When + specifying ``format=JSON_ARRAY`` and + ``disposition=EXTERNAL_LINKS``, each chunk in the result contains compact JSON with no indentation or extra - whitespace. When specifying `format=ARROW_STREAM` and - `disposition=EXTERNAL_LINKS`, each chunk in the result will - be formatted as Apache Arrow Stream. See the [Apache Arrow - streaming format]. When specifying `format=CSV` and - `disposition=EXTERNAL_LINKS`, each chunk in the result will - be a CSV according to [RFC 4180] standard. All the columns - values will have *string representation* similar to the - `JSON_ARRAY` format, and `null` values will be encoded as - “null”. Only the first chunk in the result would contain a - header row with column names. For example, the output of - `SELECT concat('id-', id) AS strCol, id AS intCol, null as - nullCol FROM range(3)` would look like this: ``` - strCol,intCol,nullCol id-1,1,null id-2,2,null id-3,3,null - ``` [Apache Arrow streaming format]: - https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format - [RFC 4180]: https://www.rfc-editor.org/rfc/rfc4180 + whitespace. When specifying ``format=ARROW_STREAM`` and + ``disposition=EXTERNAL_LINKS``, each chunk in the result + will be formatted as Apache Arrow Stream. See the `Apache + Arrow streaming format + `__. + When specifying ``format=CSV`` and + ``disposition=EXTERNAL_LINKS``, each chunk in the result + will be a CSV according to `RFC 4180 + `__ standard. All + the columns values will have *string representation* similar + to the ``JSON_ARRAY`` format, and ``null`` values will be + encoded as “null”. Only the first chunk in the result would + contain a header row with column names. For example, the + output of ``SELECT concat('id-', id) AS strCol, id AS + intCol, null as nullCol FROM range(3)`` would look like + this: .. code-block:: strCol,intCol,nullCol id-1,1,null + id-2,2,null id-3,3,null on_wait_timeout: type: string x-enum: - CANCEL - CONTINUE description: >- - When `wait_timeout > 0s`, the call will block up to the + When ``wait_timeout > 0s``, the call will block up to the specified time. If the statement execution doesn't finish - within this time, `on_wait_timeout` determines whether the + within this time, ``on_wait_timeout`` determines whether the execution should continue or be canceled. When set to - `CONTINUE`, the statement execution continues asynchronously - and the call returns a statement ID which can be used for - polling with :method:statementexecution/getStatement. When - set to `CANCEL`, the statement execution is canceled and the - call returns with a `CANCELED` state. + ``CONTINUE``, the statement execution continues + asynchronously and the call returns a statement ID which can + be used for polling with + :method:statementexecution/getStatement. When set to + ``CANCEL``, the statement execution is canceled and the call + returns with a ``CANCELED`` state. parameters: type: array items: @@ -4922,13 +4998,14 @@ paths: type: type: string description: >- - The data type, given as a string. For example: `INT`, - `STRING`, `DECIMAL(10,2)`. If no type is given the - type is assumed to be `STRING`. Complex types, such as - `ARRAY`, `MAP`, and `STRUCT` are not supported. For - valid types, refer to the section [Data types] of the - SQL language reference. [Data types]: - https://docs.databricks.com/sql/language-manual/functions/cast.html + The data type, given as a string. For example: + ``INT``, ``STRING``, ``DECIMAL(10,2)``. If no type is + given the type is assumed to be ``STRING``. Complex + types, such as ``ARRAY``, ``MAP``, and ``STRUCT`` are + not supported. For valid types, refer to the section + `Data types + `__ + of the SQL language reference. value: type: string description: >- @@ -4940,28 +5017,29 @@ paths: A list of parameters to pass into a SQL statement containing parameter markers. A parameter consists of a name, a value, and optionally a type. To represent a NULL value, the - `value` field may be omitted or set to `null` explicitly. If - the `type` field is omitted, the value is interpreted as a - string. If the type is given, parameters will be checked for - type correctness according to the given type. A value is - correct if the provided string can be converted to the - requested type using the `cast` function. The exact - semantics are described in the section [`cast` function] of - the SQL language reference. For example, the following - statement contains two parameters, `my_name` and `my_date`: - ``` SELECT * FROM my_table WHERE name = :my_name AND date = - :my_date ``` The parameters can be passed in the request - body as follows: ` { ..., "statement": "SELECT * FROM - my_table WHERE name = :my_name AND date = :my_date", - "parameters": [ { "name": "my_name", "value": "the name" }, - { "name": "my_date", "value": "2020-01-01", "type": "DATE" } - ] } ` Currently, positional parameters denoted by a `?` - marker are not supported by the Databricks SQL Statement - Execution API. Also see the section [Parameter markers] of - the SQL language reference. [Parameter markers]: - https://docs.databricks.com/sql/language-manual/sql-ref-parameter-marker.html - [`cast` function]: - https://docs.databricks.com/sql/language-manual/functions/cast.html + ``value`` field may be omitted or set to ``null`` + explicitly. If the ``type`` field is omitted, the value is + interpreted as a string. If the type is given, parameters + will be checked for type correctness according to the given + type. A value is correct if the provided string can be + converted to the requested type using the ``cast`` function. + The exact semantics are described in the section `cast + function + `__ + of the SQL language reference. For example, the following + statement contains two parameters, ``my_name`` and + ``my_date``: .. code-block:: SELECT * FROM my_table WHERE + name = :my_name AND date = :my_date The parameters can be + passed in the request body as follows: ``{ ..., "statement": + "SELECT * FROM my_table WHERE name = :my_name AND date = + :my_date", "parameters": [ { "name": "my_name", "value": + "the name" }, { "name": "my_date", "value": "2020-01-01", + "type": "DATE" } ] }`` Currently, positional parameters + denoted by a ``?`` marker are not supported by the + Databricks SQL Statement Execution API. Also see the section + `Parameter markers + `__ + of the SQL language reference. query_tags: type: array items: @@ -4974,15 +5052,15 @@ paths: required: - key description: >- - * A query execution can be annotated with an optional + - A query execution can be annotated with an optional key-value pair to allow users to attribute the executions by key and optional value to filter by. QueryTag is the user-facing representation. description: >- An array of query tags to annotate a SQL statement. A query tag consists of a non-empty key and, optionally, a value. To - represent a NULL value, either omit the `value` field or - manually set it to `null` or white space. Refer to the SQL + represent a NULL value, either omit the ``value`` field or + manually set it to ``null`` or white space. Refer to the SQL language reference for the format specification of query tags. There's no significance to the order of tags. Only one value per key will be recorded. A sequence in excess of 20 @@ -4993,33 +5071,34 @@ paths: type: integer description: >- Applies the given row limit to the statement's result set, - but unlike the `LIMIT` clause in SQL, it also sets the - `truncated` field in the response to indicate whether the + but unlike the ``LIMIT`` clause in SQL, it also sets the + ``truncated`` field in the response to indicate whether the result was trimmed due to the limit or not. schema: type: string description: >- - Sets default schema for statement execution, similar to - [`USE SCHEMA`] in SQL. [`USE SCHEMA`]: - https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-use-schema.html + Sets default schema for statement execution, similar to `USE + SCHEMA + `__ + in SQL. wait_timeout: type: string description: >- The time in seconds the call will wait for the statement's - result set as `Ns`, where `N` can be set to 0 or to a value - between 5 and 50. When set to `0s`, the statement will - execute in asynchronous mode and the call will not wait for - the execution to finish. In this case, the call returns - directly with `PENDING` state and a statement ID which can + result set as ``Ns``, where ``N`` can be set to 0 or to a + value between 5 and 50. When set to ``0s``, the statement + will execute in asynchronous mode and the call will not wait + for the execution to finish. In this case, the call returns + directly with ``PENDING`` state and a statement ID which can be used for polling with :method:statementexecution/getStatement. When set between 5 and 50 seconds, the call will behave synchronously up to this timeout and wait for the statement execution to finish. If the execution finishes within this time, the call returns - immediately with a manifest and result data (or a `FAILED` + immediately with a manifest and result data (or a ``FAILED`` state in case of an execution error). If the statement takes - longer to execute, `on_wait_timeout` determines what should - happen after the timeout is reached. + longer to execute, ``on_wait_timeout`` determines what + should happen after the timeout is reached. required: - statement - warehouse_id @@ -5047,27 +5126,28 @@ paths: operationId: statement_execution_get_statement summary: >- This request can be used to poll for the statement's status. - StatementResponse contains `statement_id` + StatementResponse contains tags: - sql - statement_execution description: >- This request can be used to poll for the statement's status. - StatementResponse contains `statement_id` + StatementResponse contains - and `status`; other fields might be absent or present depending on - context. When the `status.state` + ``statement_id`` and ``status``; other fields might be absent or present + depending on context. When - field is `SUCCEEDED` it will also return the result manifest and the - first chunk of the result data. + the ``status.state`` field is ``SUCCEEDED`` it will also return the + result manifest and the first - When the statement is in the terminal states `CANCELED`, `CLOSED` or - `FAILED`, it returns HTTP 200 + chunk of the result data. When the statement is in the terminal states + ``CANCELED``, ``CLOSED`` or - with the state set. After at least 12 hours in terminal state, the - statement is removed from the + ``FAILED``, it returns HTTP 200 with the state set. After at least 12 + hours in terminal state, the - warehouse and further calls will receive an HTTP 404 response. + statement is removed from the warehouse and further calls will receive + an HTTP 404 response. **NOTE** This call currently might take up to 5 seconds to get the @@ -5111,31 +5191,31 @@ paths: get: operationId: statement_execution_get_statement_result_chunk_n summary: >- - After the statement execution has `SUCCEEDED`, this request can be used - to fetch any chunk by index. + After the statement execution has ``SUCCEEDED``, this request can be + used to fetch any chunk by index. tags: - sql - statement_execution description: >- - After the statement execution has `SUCCEEDED`, this request can be used - to fetch any chunk by index. + After the statement execution has ``SUCCEEDED``, this request can be + used to fetch any chunk by index. - Whereas the first chunk with `chunk_index=0` is typically fetched with + Whereas the first chunk with ``chunk_index=0`` is typically fetched with :method:statementexecution/executeStatement or :method:statementexecution/getStatement, this request can be used to fetch subsequent chunks. The response structure is - identical to the nested `result` + identical to the nested ``result`` element described in the :method:statementexecution/getStatement request, and similarly includes the - `next_chunk_index` and `next_chunk_internal_link` fields for simple - iteration through the result set. + ``next_chunk_index`` and ``next_chunk_internal_link`` fields for simple + iteration through the result - Depending on `disposition`, the response returns chunks of data either - inline, or as links. + set. Depending on ``disposition``, the response returns chunks of data + either inline, or as links. :param statement_id: str @@ -5193,7 +5273,10 @@ paths: The amount of time in minutes that a SQL warehouse must be idle (i.e., no RUNNING queries) before it is automatically stopped. - Supported values: - Must be == 0 or >= 10 mins - 0 indicates no autostop. + Supported values: + + - Must be == 0 or >= 10 mins + - 0 indicates no autostop. Defaults to 120 mins :param channel: :class:`Channel` (optional) @@ -5203,8 +5286,18 @@ paths: to run larger queries on it. If you want to increase the number of concurrent queries, please tune max_num_clusters. - Supported values: - 2X-Small - X-Small - Small - Medium - Large - X-Large - 2X-Large - 3X-Large - - 4X-Large - 5X-Large + Supported values: + + - 2X-Small + - X-Small + - Small + - Medium + - Large + - X-Large + - 2X-Large + - 3X-Large + - 4X-Large + - 5X-Large :param creator_name: str (optional) warehouse creator name :param enable_photon: bool (optional) @@ -5218,7 +5311,10 @@ paths: :param max_num_clusters: int (optional) Maximum number of clusters that the autoscaler will create to handle concurrent queries. - Supported values: - Must be >= min_num_clusters - Must be <= 40. + Supported values: + + - Must be >= min_num_clusters + - Must be <= 40. Defaults to min_clusters if unset. :param min_num_clusters: int (optional) @@ -5226,24 +5322,32 @@ paths: will ensure that a larger number of clusters are always running and therefore may reduce the cold start time for new queries. This is similar to reserved vs. revocable cores in a resource manager. - Supported values: - Must be > 0 - Must be <= min(max_num_clusters, 30) + Supported values: + + - Must be > 0 + - Must be <= min(max_num_clusters, 30) Defaults to 1 :param name: str (optional) Logical name for the cluster. - Supported values: - Must be unique within an org. - Must be less than 100 characters. + Supported values: + + - Must be unique within an org. + - Must be less than 100 characters. :param spot_instance_policy: :class:`SpotInstancePolicy` (optional) Configurations whether the endpoint should use spot instances. :param tags: :class:`EndpointTags` (optional) A set of key-value pairs that will be tagged on all resources (e.g., AWS instances and EBS volumes) associated with this SQL warehouse. - Supported values: - Number of tags < 45. + Supported values: + + - Number of tags < 45. :param warehouse_type: :class:`CreateWarehouseRequestWarehouseType` (optional) - Warehouse type: `PRO` or `CLASSIC`. If you want to use serverless compute, you must set to `PRO` and - also set the field `enable_serverless_compute` to `true`. + Warehouse type: ``PRO`` or ``CLASSIC``. If you want to use serverless compute, you must set to + ``PRO`` and also set the field ``enable_serverless_compute`` to ``true``. :returns: Long-running operation waiter for :class:`GetWarehouseResponse`. @@ -5406,11 +5510,13 @@ paths: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: >- - Warehouse type: `PRO` or `CLASSIC`. If you want to use - serverless compute, you must set to `PRO` and also set the - field `enable_serverless_compute` to `true`. + Warehouse type: ``PRO`` or ``CLASSIC``. If you want to use + serverless compute, you must set to ``PRO`` and also set the + field ``enable_serverless_compute`` to ``true``. responses: '200': description: Success @@ -5442,11 +5548,11 @@ paths: :param page_size: int (optional) The max number of warehouses to return. :param page_token: str (optional) - A page token, received from a previous `ListWarehouses` call. Provide this to retrieve the + A page token, received from a previous ``ListWarehouses`` call. Provide this to retrieve the subsequent page; otherwise the first will be retrieved. - When paginating, all other parameters provided to `ListWarehouses` must match the call that provided - the page token. + When paginating, all other parameters provided to ``ListWarehouses`` must match the call that + provided the page token. :param run_as_user_id: int (optional) Deprecated: this field is ignored by the server. Service Principal which will be used to fetch the list of endpoints. If not specified, SQL Gateway will use the user from the session header. @@ -5465,10 +5571,10 @@ paths: schema: type: string description: >- - A page token, received from a previous `ListWarehouses` call. + A page token, received from a previous ``ListWarehouses`` call. Provide this to retrieve the subsequent page; otherwise the first will be retrieved. When paginating, all other parameters provided to - `ListWarehouses` must match the call that provided the page token. + ``ListWarehouses`` must match the call that provided the page token. - name: run_as_user_id in: query required: false @@ -5603,10 +5709,10 @@ paths: unspecified, at most 100 overrides will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. :param page_token: str (optional) - A page token, received from a previous `ListDefaultWarehouseOverrides` call. Provide this to + A page token, received from a previous ``ListDefaultWarehouseOverrides`` call. Provide this to retrieve the subsequent page. - When paginating, all other parameters provided to `ListDefaultWarehouseOverrides` must match the + When paginating, all other parameters provided to ``ListDefaultWarehouseOverrides`` must match the call that provided the page token. :returns: Iterator over :class:`DefaultWarehouseOverride` @@ -5628,9 +5734,9 @@ paths: type: string description: >- A page token, received from a previous - `ListDefaultWarehouseOverrides` call. Provide this to retrieve the + ``ListDefaultWarehouseOverrides`` call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to - `ListDefaultWarehouseOverrides` must match the call that provided + ``ListDefaultWarehouseOverrides`` must match the call that provided the page token. responses: '200': @@ -5855,7 +5961,7 @@ paths: fields are applied. :param allow_missing: bool (optional) If set to true, and the override is not found, a new override will be created. In this situation, - `update_mask` is ignored and all fields are applied. Defaults to false. + ``update_mask`` is ignored and all fields are applied. Defaults to false. :returns: :class:`DefaultWarehouseOverride` parameters: @@ -5884,7 +5990,7 @@ paths: type: boolean description: >- If set to true, and the override is not found, a new override will - be created. In this situation, `update_mask` is ignored and all + be created. In this situation, ``update_mask`` is ignored and all fields are applied. Defaults to false. requestBody: content: @@ -5961,7 +6067,10 @@ paths: The amount of time in minutes that a SQL warehouse must be idle (i.e., no RUNNING queries) before it is automatically stopped. - Supported values: - Must be == 0 or >= 10 mins - 0 indicates no autostop. + Supported values: + + - Must be == 0 or >= 10 mins + - 0 indicates no autostop. Defaults to 120 mins :param channel: :class:`Channel` (optional) @@ -5971,8 +6080,18 @@ paths: to run larger queries on it. If you want to increase the number of concurrent queries, please tune max_num_clusters. - Supported values: - 2X-Small - X-Small - Small - Medium - Large - X-Large - 2X-Large - 3X-Large - - 4X-Large - 5X-Large + Supported values: + + - 2X-Small + - X-Small + - Small + - Medium + - Large + - X-Large + - 2X-Large + - 3X-Large + - 4X-Large + - 5X-Large :param creator_name: str (optional) warehouse creator name :param enable_photon: bool (optional) @@ -5986,7 +6105,10 @@ paths: :param max_num_clusters: int (optional) Maximum number of clusters that the autoscaler will create to handle concurrent queries. - Supported values: - Must be >= min_num_clusters - Must be <= 40. + Supported values: + + - Must be >= min_num_clusters + - Must be <= 40. Defaults to min_clusters if unset. :param min_num_clusters: int (optional) @@ -5994,24 +6116,32 @@ paths: will ensure that a larger number of clusters are always running and therefore may reduce the cold start time for new queries. This is similar to reserved vs. revocable cores in a resource manager. - Supported values: - Must be > 0 - Must be <= min(max_num_clusters, 30) + Supported values: + + - Must be > 0 + - Must be <= min(max_num_clusters, 30) Defaults to 1 :param name: str (optional) Logical name for the cluster. - Supported values: - Must be unique within an org. - Must be less than 100 characters. + Supported values: + + - Must be unique within an org. + - Must be less than 100 characters. :param spot_instance_policy: :class:`SpotInstancePolicy` (optional) Configurations whether the endpoint should use spot instances. :param tags: :class:`EndpointTags` (optional) A set of key-value pairs that will be tagged on all resources (e.g., AWS instances and EBS volumes) associated with this SQL warehouse. - Supported values: - Number of tags < 45. + Supported values: + + - Number of tags < 45. :param warehouse_type: :class:`EditWarehouseRequestWarehouseType` (optional) - Warehouse type: `PRO` or `CLASSIC`. If you want to use serverless compute, you must set to `PRO` and - also set the field `enable_serverless_compute` to `true`. + Warehouse type: ``PRO`` or ``CLASSIC``. If you want to use serverless compute, you must set to + ``PRO`` and also set the field ``enable_serverless_compute`` to ``true``. :returns: Long-running operation waiter for :class:`GetWarehouseResponse`. @@ -6181,11 +6311,13 @@ paths: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: >- - Warehouse type: `PRO` or `CLASSIC`. If you want to use - serverless compute, you must set to `PRO` and also set the - field `enable_serverless_compute` to `true`. + Warehouse type: ``PRO`` or ``CLASSIC``. If you want to use + serverless compute, you must set to ``PRO`` and also set the + field ``enable_serverless_compute`` to ``true``. responses: '200': description: Success @@ -6637,13 +6769,15 @@ paths: type: boolean description: >- If set to false the specific warehouse type will not - be be allowed as a value for warehouse_type in + be allowed as a value for warehouse_type in CreateWarehouse and EditWarehouse warehouse_type: type: string x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: >- Create a collection of name/value pairs. @@ -6698,7 +6832,7 @@ paths: attributes -- see the documentation for details. description: >- - * Configuration values to enable or disable the access to + Configuration values to enable or disable the access to specific warehouse types in the workspace. description: >- @@ -6880,8 +7014,9 @@ components: permission_level: $ref: '#/components/schemas/PermissionLevel' description: >- - * `CAN_VIEW`: Can view the query * `CAN_RUN`: Can run the query * - `CAN_EDIT`: Can edit the query * `CAN_MANAGE`: Can manage the query + - ``CAN_VIEW``: Can view the query - ``CAN_RUN``: Can run the query + - ``CAN_EDIT``: Can edit the query - ``CAN_MANAGE``: Can manage the + query user_name: type: string Alert: @@ -6895,16 +7030,17 @@ components: custom_body: type: string description: >- - Custom body of alert notification, if it exists. See [here] for - custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + Custom body of alert notification, if it exists. See `here + `__ for + custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This can include email subject entries and Slack notification headers, for example. - See [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + See `here + `__ for + custom templating instructions. display_name: type: string description: The display name of the alert. @@ -6999,8 +7135,8 @@ components: op: type: string description: >- - Operator used to compare in alert evaluation: `>`, `>=`, `<`, `<=`, - `==`, `!=` + Operator used to compare in alert evaluation: ``>``, ``>=``, ``<``, + ``<=``, ``==``, ``!=`` value: type: object description: >- @@ -7009,16 +7145,16 @@ components: custom_body: type: string description: >- - Custom body of alert notification, if it exists. See [here] for - custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + Custom body of alert notification, if it exists. See `here + `__ for + custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This includes - email subject, Slack notification header, etc. See [here] for custom - templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + email subject, Slack notification header, etc. See `here + `__ for + custom templating instructions. empty_result_state: $ref: '#/components/schemas/AlertOptionsEmptyResultState' description: State that alert evaluates to when query result is empty. @@ -7041,8 +7177,8 @@ components: type: string description: >- Data source ID maps to the ID of the data source used by the - resource and is distinct from the warehouse ID. [Learn more] [Learn - more]: https://docs.databricks.com/api/workspace/datasources/list + resource and is distinct from the warehouse ID. `Learn more + `__ description: type: string description: >- @@ -7056,8 +7192,8 @@ components: description: >- Indicates whether the query is trashed. Trashed queries can't be used in dashboards, or appear in search results. If this boolean is - `true`, the `options` property for this query includes a - `moved_to_trash_at` timestamp. Trashed queries are permanently + ``true``, the ``options`` property for this query includes a + ``moved_to_trash_at`` timestamp. Trashed queries are permanently deleted after 30 days. is_draft: type: boolean @@ -7069,7 +7205,7 @@ components: type: boolean description: >- Text parameter types are not safe from SQL injection for all types - of data source. Set this Boolean parameter to `true` if a query + of data source. Set this Boolean parameter to ``true`` if a query either does not use any text type parameters or uses a data source type where text type parameters are handled safely. name: @@ -7092,6 +7228,33 @@ components: user_id: type: integer description: The ID of the user who owns the query. + AlertStatementParameter: + type: object + properties: + name: + type: string + description: The name of the parameter, referenced in the query as ``:name``. + type: + type: string + description: >- + The SQL data type of the parameter, e.g. STRING, INT, or DATE. + Defaults to STRING. This is a string rather than an enum because + scalar subtypes such as DECIMAL(10, 4) cannot be enumerated. Complex + types such as ARRAY, MAP, and STRUCT are not supported. + value: + type: string + description: >- + The bound value for the parameter, given as a string. If omitted, + the value is interpreted as NULL. + required: + - name + description: >- + Redash-owned copy of the internal StatementParameter for the external + AlertV2 API. The internal + ``ordinal`` and ``args`` fields are intentionally omitted: the public API supports only flat, + named scalar parameters; complex types (ARRAY, MAP, STRUCT) are not supported. This mirrors + SEA's public StatementParameter schema, see: + cmdexec/sql-exec-api/proto/sql_exec_api_service.proto:763-779 AlertV2: type: object properties: @@ -7116,6 +7279,11 @@ components: custom_summary: type: string description: Custom summary for the alert. support mustache template. + effective_parent_path: + type: string + description: >- + The actual workspace path of the folder containing the alert. This + is an output-only field. effective_run_as: $ref: '#/components/schemas/AlertV2RunAs' description: >- @@ -7133,6 +7301,13 @@ components: description: >- The owner's username. This field is set to "Unavailable" if the user has been deleted. + parameters: + type: array + items: + $ref: '#/components/schemas/AlertStatementParameter' + description: >- + Query parameters bound when executing the alert query, referenced in + the query text with ``:name`` syntax. Static values only. parent_path: type: string description: >- @@ -7143,19 +7318,19 @@ components: description: >- Specifies the identity that will be used to run the alert. This field allows you to configure alerts to run as a specific user or - service principal. - For user identity: Set `user_name` to the email - of an active workspace user. Users can only set this to their own - email. - For service principal: Set `service_principal_name` to the - application ID. Requires the `servicePrincipal/user` role. If not - specified, the alert will run as the request user. + service principal. - For user identity: Set ``user_name`` to the + email of an active workspace user. Users can only set this to their + own email. - For service principal: Set ``service_principal_name`` + to the application ID. Requires the ``servicePrincipal/user`` role. + If not specified, the alert will run as the request user. run_as_user_name: type: string description: >- The run as username or application ID of service principal. On Create and Update, this field can be set to application ID of an active service principal. Setting this field requires the - servicePrincipal/user role. Deprecated: Use `run_as` field instead. - This field will be removed in a future release. + servicePrincipal/user role. Deprecated: Use ``run_as`` field + instead. This field will be removed in a future release. update_time: type: string description: The timestamp indicating when the alert was updated. @@ -7177,7 +7352,8 @@ components: $ref: '#/components/schemas/AlertEvaluationState' description: >- Alert state if result is empty. Please avoid setting this field to - be `UNKNOWN` because `UNKNOWN` state is planned to be deprecated. + be ``UNKNOWN`` because ``UNKNOWN`` state is planned to be + deprecated. last_evaluated_at: type: string description: Timestamp of the last evaluation. @@ -7226,7 +7402,7 @@ components: aggregation: $ref: '#/components/schemas/Aggregation' description: >- - If not set, the behavior is equivalent to using `First row` in the + If not set, the behavior is equivalent to using ``First row`` in the UI. display: type: string @@ -7282,7 +7458,7 @@ components: description: >- Configures the channel name and DBSQL version of the warehouse. CHANNEL_NAME_CUSTOM should be - chosen only when `dbsql_version` is specified. + chosen only when ``dbsql_version`` is specified. ChannelInfo: type: object properties: @@ -7353,16 +7529,17 @@ components: custom_body: type: string description: >- - Custom body of alert notification, if it exists. See [here] for - custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + Custom body of alert notification, if it exists. See `here + `__ for + custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This can include email subject entries and Slack notification headers, for example. - See [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + See `here + `__ for + custom templating instructions. display_name: type: string description: The display name of the alert. @@ -7466,6 +7643,11 @@ components: determine the schedule. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. + effective_pause_status: + $ref: '#/components/schemas/SchedulePauseStatus' + description: >- + The actual pause status of the schedule. This is an output-only + field. pause_status: $ref: '#/components/schemas/SchedulePauseStatus' description: Indicate whether this schedule is paused or not. @@ -7485,7 +7667,7 @@ components: type: boolean description: >- In the web application, query filters that share a name are coupled - to a single selection box if this value is `true`. + to a single selection box if this value is ``true``. id: type: string description: The ID for this dashboard. @@ -7493,8 +7675,8 @@ components: type: boolean description: >- Indicates whether a dashboard is trashed. Trashed dashboards won't - appear in list views. If this boolean is `true`, the `options` - property for this dashboard includes a `moved_to_trash_at` + appear in list views. If this boolean is ``true``, the ``options`` + property for this dashboard includes a ``moved_to_trash_at`` timestamp. Items in trash are permanently deleted after 30 days. is_draft: type: boolean @@ -7520,12 +7702,13 @@ components: permission_tier: $ref: '#/components/schemas/PermissionLevel' description: >- - * `CAN_VIEW`: Can view the query * `CAN_RUN`: Can run the query * - `CAN_EDIT`: Can edit the query * `CAN_MANAGE`: Can manage the query + - ``CAN_VIEW``: Can view the query - ``CAN_RUN``: Can run the query + - ``CAN_EDIT``: Can edit the query - ``CAN_MANAGE``: Can manage the + query slug: type: string description: >- - URL slug. Usually mirrors the query name with dashes (`-`) instead + URL slug. Usually mirrors the query name with dashes (``-``) instead of spaces. Appears in the URL for this query. tags: type: array @@ -7558,8 +7741,8 @@ components: type: string description: >- Data source ID maps to the ID of the data source used by the - resource and is distinct from the warehouse ID. [Learn more] [Learn - more]: https://docs.databricks.com/api/workspace/datasources/list + resource and is distinct from the warehouse ID. `Learn more + `__ name: type: string description: >- @@ -7581,7 +7764,7 @@ components: type: string description: >- The type of data source. For SQL warehouses, this will be - `databricks_internal`. + ``databricks_internal``. view_only: type: boolean description: Reserved for internal use. @@ -7785,9 +7968,9 @@ components: warehouse_type: $ref: '#/components/schemas/EndpointInfoWarehouseType' description: >- - Warehouse type: `PRO` or `CLASSIC`. If you want to use serverless - compute, you must set to `PRO` and also set the field - `enable_serverless_compute` to `true`. + Warehouse type: ``PRO`` or ``CLASSIC``. If you want to use + serverless compute, you must set to ``PRO`` and also set the field + ``enable_serverless_compute`` to ``true``. EndpointTagPair: type: object properties: @@ -7829,7 +8012,7 @@ components: type: string description: >- Indicates the date-time that the given external link will expire and - becomes invalid, after which point a new `external_link` must be + becomes invalid, after which point a new ``external_link`` must be requested. external_link: type: string @@ -7842,24 +8025,25 @@ components: type: object description: >- HTTP headers that must be included with a GET request to the - `external_link`. Each header is provided as a key-value pair. + ``external_link``. Each header is provided as a key-value pair. Headers are typically used to pass a decryption key to the external service. The values of these headers should be considered sensitive and the client should not expose these values in a log. next_chunk_index: type: integer description: >- - When fetching, provides the `chunk_index` for the _next_ chunk. If + When fetching, provides the ``chunk_index`` for the *next* chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a :method:statementexecution/getstatementresultchunkn request. next_chunk_internal_link: type: string description: >- - When fetching, provides a link to fetch the _next_ chunk. If absent, - indicates there are no more chunks. This link is an absolute `path` - to be joined with your `$DATABRICKS_HOST`, and should be treated as - an opaque link. This is an alternative to using `next_chunk_index`. + When fetching, provides a link to fetch the *next* chunk. If absent, + indicates there are no more chunks. This link is an absolute + ``path`` to be joined with your ``$DATABRICKS_HOST``, and should be + treated as an opaque link. This is an alternative to using + ``next_chunk_index``. row_count: type: integer description: The number of rows within the result chunk. @@ -8008,9 +8192,9 @@ components: warehouse_type: $ref: '#/components/schemas/GetWarehouseResponseWarehouseType' description: >- - Warehouse type: `PRO` or `CLASSIC`. If you want to use serverless - compute, you must set to `PRO` and also set the field - `enable_serverless_compute` to `true`. + Warehouse type: ``PRO`` or ``CLASSIC``. If you want to use + serverless compute, you must set to ``PRO`` and also set the field + ``enable_serverless_compute`` to ``true``. GetWorkspaceWarehouseConfigResponse: type: object properties: @@ -8088,14 +8272,14 @@ components: type: integer description: >- Number of seconds after being triggered before the alert rearms - itself and can be triggered again. If `null`, alert will never be + itself and can be triggered again. If ``null``, alert will never be triggered again. state: $ref: '#/components/schemas/LegacyAlertState' description: >- - State of the alert. Possible values are: `unknown` (yet to be - evaluated), `triggered` (evaluated and fulfilled trigger - conditions), or `ok` (evaluated and did not fulfill trigger + State of the alert. Possible values are: ``unknown`` (yet to be + evaluated), ``triggered`` (evaluated and fulfilled trigger + conditions), or ``ok`` (evaluated and did not fulfill trigger conditions). updated_at: type: string @@ -8114,8 +8298,8 @@ components: type: string description: >- Data source ID maps to the ID of the data source used by the - resource and is distinct from the warehouse ID. [Learn more] [Learn - more]: https://docs.databricks.com/api/workspace/datasources/list + resource and is distinct from the warehouse ID. `Learn more + `__ description: type: string description: >- @@ -8129,8 +8313,8 @@ components: description: >- Indicates whether the query is trashed. Trashed queries can't be used in dashboards, or appear in search results. If this boolean is - `true`, the `options` property for this query includes a - `moved_to_trash_at` timestamp. Trashed queries are permanently + ``true``, the ``options`` property for this query includes a + ``moved_to_trash_at`` timestamp. Trashed queries are permanently deleted after 30 days. is_draft: type: boolean @@ -8148,7 +8332,7 @@ components: type: boolean description: >- Text parameter types are not safe from SQL injection for all types - of data source. Set this Boolean parameter to `true` if a query + of data source. Set this Boolean parameter to ``true`` if a query either does not use any text type parameters or uses a data source type where text type parameters are handled safely. last_modified_by: @@ -8175,8 +8359,9 @@ components: permission_tier: $ref: '#/components/schemas/PermissionLevel' description: >- - * `CAN_VIEW`: Can view the query * `CAN_RUN`: Can run the query * - `CAN_EDIT`: Can edit the query * `CAN_MANAGE`: Can manage the query + - ``CAN_VIEW``: Can view the query - ``CAN_RUN``: Can run the query + - ``CAN_EDIT``: Can edit the query - ``CAN_MANAGE``: Can manage the + query query: type: string description: The text of the query to be run. @@ -8189,7 +8374,7 @@ components: $ref: '#/components/schemas/RunAsRole' description: >- Sets the **Run as** role for the object. Must be set to one of - `"viewer"` (signifying "run as viewer" behavior) or `"owner"` + ``"viewer"`` (signifying "run as viewer" behavior) or ``"owner"`` (signifying "run as owner" behavior) tags: type: array @@ -8241,7 +8426,7 @@ components: description: >- The visualization description API changes frequently and is unsupported. You can duplicate a - visualization by copying description objects received _from the API_ and then using them to + visualization by copying description objects received *from the API* and then using them to create a new one with a POST request to the same endpoint. Databricks does not recommend constructing ad-hoc visualizations entirely in JSON. ListAlertsResponse: @@ -8264,16 +8449,17 @@ components: custom_body: type: string description: >- - Custom body of alert notification, if it exists. See [here] for - custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + Custom body of alert notification, if it exists. See `here + `__ for + custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This can include email subject entries and Slack notification headers, for example. - See [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + See `here + `__ for + custom templating instructions. display_name: type: string description: The display name of the alert. @@ -8336,7 +8522,7 @@ components: next_page_token: type: string description: >- - A token, which can be sent as `page_token` to retrieve the next + A token, which can be sent as ``page_token`` to retrieve the next page. If this field is omitted, there are no subsequent pages. description: Response message for ListDefaultWarehouseOverrides. ListQueriesResponse: @@ -8600,7 +8786,7 @@ components: $ref: '#/components/schemas/QueryStatus' description: >- A list of statuses (QUEUED, RUNNING, CANCELED, FAILED, FINISHED) to - match query results. Corresponds to the `status` field in the + match query results. Corresponds to the ``status`` field in the response. Filtering for multiple statuses is not recommended. Instead, opt to filter by a single status multiple times and then combine the results. @@ -8638,7 +8824,7 @@ components: time. endpoint_id: type: string - description: Alias for `warehouse_id`. + description: Alias for ``warehouse_id``. error_message: type: string description: Message describing why the query could not complete. @@ -8708,10 +8894,10 @@ components: status: $ref: '#/components/schemas/QueryStatus' description: >- - Query status with one the following values: - `QUEUED`: Query has - been received and queued. - `RUNNING`: Query has started. - - `CANCELED`: Query has been cancelled by the user. - `FAILED`: Query - has failed. - `FINISHED`: Query has completed. + Query status with one the following values: - ``QUEUED``: Query has + been received and queued. - ``RUNNING``: Query has started. - + ``CANCELED``: Query has been cancelled by the user. - ``FAILED``: + Query has failed. - ``FINISHED``: Query has completed. user_id: type: integer description: The ID of the user who ran the query. @@ -8825,7 +9011,7 @@ components: result_from_cache: type: boolean description: >- - `true` if the query result was fetched from cache, `false` + ``true`` if the query result was fetched from cache, ``false`` otherwise. rows_produced_count: type: integer @@ -8881,7 +9067,7 @@ components: type: string description: >- The timestamp when this query was moved to trash. Only present when - the `is_archived` property is `true`. Trashed items are deleted + the ``is_archived`` property is ``true``. Trashed items are deleted after thirty days. parameters: type: array @@ -8899,7 +9085,7 @@ components: $ref: '#/components/schemas/DateValue' description: >- Date query parameter value. Can only specify one of - `dynamic_date_value` or `date_value`. + ``dynamic_date_value`` or ``date_value``. enum_value: $ref: '#/components/schemas/EnumValue' description: Dropdown query parameter value. @@ -8930,7 +9116,7 @@ components: required: - key description: >- - * A query execution can be annotated with an optional key-value pair to + - A query execution can be annotated with an optional key-value pair to allow users to attribute the executions by key and optional value to filter by. QueryTag is the user-facing representation. @@ -8955,7 +9141,7 @@ components: type: integer description: >- The number of bytes in the result chunk. This field is not available - when using `INLINE` disposition. + when using ``INLINE`` disposition. chunk_index: type: integer description: The position within the sequence of result set chunks. @@ -8966,9 +9152,9 @@ components: items: type: string description: >- - The `JSON_ARRAY` format is an array of arrays of values, where each - non-null value is formatted as a string. Null values are encoded as - JSON `null`. + The ``JSON_ARRAY`` format is an array of arrays of values, where + each non-null value is formatted as a string. Null values are + encoded as JSON ``null``. external_links: type: array items: @@ -8976,17 +9162,18 @@ components: next_chunk_index: type: integer description: >- - When fetching, provides the `chunk_index` for the _next_ chunk. If + When fetching, provides the ``chunk_index`` for the *next* chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a :method:statementexecution/getstatementresultchunkn request. next_chunk_internal_link: type: string description: >- - When fetching, provides a link to fetch the _next_ chunk. If absent, - indicates there are no more chunks. This link is an absolute `path` - to be joined with your `$DATABRICKS_HOST`, and should be treated as - an opaque link. This is an alternative to using `next_chunk_index`. + When fetching, provides a link to fetch the *next* chunk. If absent, + indicates there are no more chunks. This link is an absolute + ``path`` to be joined with your ``$DATABRICKS_HOST``, and should be + treated as an opaque link. This is an alternative to using + ``next_chunk_index``. row_count: type: integer description: The number of rows within the result chunk. @@ -8994,12 +9181,12 @@ components: type: integer description: The starting row offset within the result set. description: >- - Contains the result data of a single chunk when using `INLINE` + Contains the result data of a single chunk when using ``INLINE`` disposition. When using - `EXTERNAL_LINKS` disposition, the array `external_links` is used instead to provide URLs to the - result data in cloud storage. Exactly one of these alternatives is used. (While the - `external_links` array prepares the API to return multiple links in a single response. Currently - only a single link is returned.) + ``EXTERNAL_LINKS`` disposition, the array ``external_links`` is used instead to provide URLs to + the result data in cloud storage. Exactly one of these alternatives is used. (While the + ``external_links`` array prepares the API to return multiple links in a single response. + Currently only a single link is returned.) ResultManifest: type: object properties: @@ -9016,7 +9203,7 @@ components: type: integer description: >- The total number of bytes in the result set. This field is not - available when using `INLINE` disposition. + available when using ``INLINE`` disposition. total_chunk_count: type: integer description: >- @@ -9028,8 +9215,8 @@ components: truncated: type: boolean description: >- - Indicates whether the result is truncated due to `row_limit` or - `byte_limit`. + Indicates whether the result is truncated due to ``row_limit`` or + ``byte_limit``. description: The result manifest provides schema and metadata for the result set. ResultSchema: type: object @@ -9078,12 +9265,13 @@ components: type: type: string description: >- - The data type, given as a string. For example: `INT`, `STRING`, - `DECIMAL(10,2)`. If no type is given the type is assumed to be - `STRING`. Complex types, such as `ARRAY`, `MAP`, and `STRUCT` are - not supported. For valid types, refer to the section [Data types] of - the SQL language reference. [Data types]: - https://docs.databricks.com/sql/language-manual/functions/cast.html + The data type, given as a string. For example: ``INT``, ``STRING``, + ``DECIMAL(10,2)``. If no type is given the type is assumed to be + ``STRING``. Complex types, such as ``ARRAY``, ``MAP``, and + ``STRUCT`` are not supported. For valid types, refer to the section + `Data types + `__ + of the SQL language reference. value: type: string description: >- @@ -9110,16 +9298,22 @@ components: properties: error: $ref: '#/components/schemas/ServiceError' + sql_state: + type: string + description: >- + SQLSTATE error code returned when the statement execution fails. + Only populated when the statement status is ``FAILED``. state: $ref: '#/components/schemas/StatementState' description: >- - Statement execution state: - `PENDING`: waiting for warehouse - - `RUNNING`: running - `SUCCEEDED`: execution was successful, result - data available for fetch - `FAILED`: execution failed; reason for - failure described in accompanying error message - `CANCELED`: user - canceled; can come from explicit cancel call, or timeout with - `on_wait_timeout=CANCEL` - `CLOSED`: execution successful, and - statement closed; result no longer available for fetch + Statement execution state: - ``PENDING``: waiting for warehouse - + ``RUNNING``: running - ``SUCCEEDED``: execution was successful, + result data available for fetch - ``FAILED``: execution failed; + reason for failure described in accompanying error message - + ``CANCELED``: user canceled; can come from explicit cancel call, or + timeout with ``on_wait_timeout=CANCEL`` - ``CLOSED``: execution + successful, and statement closed; result no longer available for + fetch description: >- The status response includes execution state and if relevant, error information. @@ -9188,16 +9382,17 @@ components: custom_body: type: string description: >- - Custom body of alert notification, if it exists. See [here] for - custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + Custom body of alert notification, if it exists. See `here + `__ for + custom templating instructions. custom_subject: type: string description: >- Custom subject of alert notification, if it exists. This can include email subject entries and Slack notification headers, for example. - See [here] for custom templating instructions. [here]: - https://docs.databricks.com/sql/user/alerts/index.html + See `here + `__ for + custom templating instructions. display_name: type: string description: The display name of the alert. @@ -9393,12 +9588,12 @@ components: enabled: type: boolean description: >- - If set to false the specific warehouse type will not be be allowed - as a value for warehouse_type in CreateWarehouse and EditWarehouse + If set to false the specific warehouse type will not be allowed as a + value for warehouse_type in CreateWarehouse and EditWarehouse warehouse_type: $ref: '#/components/schemas/WarehouseTypePairWarehouseType' description: >- - * Configuration values to enable or disable the access to specific + Configuration values to enable or disable the access to specific warehouse types in the workspace. Widget: @@ -9413,7 +9608,7 @@ components: description: >- The visualization description API changes frequently and is unsupported. You can duplicate a visualization by copying - description objects received _from the API_ and then using them to + description objects received *from the API* and then using them to create a new one with a POST request to the same endpoint. Databricks does not recommend constructing ad-hoc visualizations entirely in JSON. @@ -9796,6 +9991,8 @@ components: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: |- Create a collection of name/value pairs. @@ -10029,6 +10226,8 @@ components: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: |- Create a collection of name/value pairs. @@ -10072,6 +10271,8 @@ components: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: |- Create a collection of name/value pairs. @@ -10116,22 +10317,22 @@ components: - CANCEL - CONTINUE description: >- - When `wait_timeout > 0s`, the call will block up to the specified time. - If the statement + When ``wait_timeout > 0s``, the call will block up to the specified + time. If the statement - execution doesn't finish within this time, `on_wait_timeout` determines - whether the execution + execution doesn't finish within this time, ``on_wait_timeout`` + determines whether the execution - should continue or be canceled. When set to `CONTINUE`, the statement + should continue or be canceled. When set to ``CONTINUE``, the statement execution continues asynchronously and the call returns a statement ID which can be used for polling with - :method:statementexecution/getStatement. When set to `CANCEL`, the + :method:statementexecution/getStatement. When set to ``CANCEL``, the statement execution is - canceled and the call returns with a `CANCELED` state. + canceled and the call returns with a ``CANCELED`` state. Format: type: string x-enum: @@ -10180,6 +10381,8 @@ components: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: |- Create a collection of name/value pairs. @@ -10463,11 +10666,11 @@ components: - CAN_MANAGE - CAN_RUN - CAN_VIEW - description: >- - * `CAN_VIEW`: Can view the query * `CAN_RUN`: Can run the query * - `CAN_EDIT`: Can edit the query - - * `CAN_MANAGE`: Can manage the query + description: |- + - ``CAN_VIEW``: Can view the query + - ``CAN_RUN``: Can run the query + - ``CAN_EDIT``: Can edit the query + - ``CAN_MANAGE``: Can manage the query PlansState: type: string x-enum: @@ -10483,6 +10686,7 @@ components: x-enum: - ALTER - ANALYZE + - CALL - COPY - CREATE - DELETE @@ -10551,7 +10755,11 @@ components: - QUEUED - RUNNING - STARTED - description: Statuses which are also used by OperationStatus in runtime + description: >- + Statuses which are also used by OperationStatus in runtime. When adding + a new QueryStatus, make + + sure to update com.databricks.sqlgateway.history.QueryStatusOrdering RunAsMode: type: string x-enum: @@ -10768,15 +10976,6 @@ components: and Executors | On Demand Driver and Executors | +-------+--------------------------------------+--------------------------------+ - - - While including "spot" in the enum name may limit the the future - extensibility of this field - - because it limits this enum to denoting "spot or not", this is the field - that PM recommends - - after discussion with customers per SC-48783. State: type: string x-enum: @@ -10786,7 +10985,7 @@ components: - STARTING - STOPPED - STOPPING - description: '* State of a warehouse.' + description: State of a warehouse. StatementState: type: string x-enum: @@ -10958,6 +11157,7 @@ components: - BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG - BUDGET_POLICY_LIMIT_ENFORCEMENT_ACTIVATED - BUDGET_POLICY_RESOLUTION_FAILURE + - CERT_ROTATION - CLOUD_ACCOUNT_POD_QUOTA_EXCEEDED - CLOUD_ACCOUNT_SETUP_FAILURE - CLOUD_OPERATION_CANCELLED @@ -10976,6 +11176,7 @@ components: - CONTROL_PLANE_CONNECTION_FAILURE_DUE_TO_MISCONFIG - CONTROL_PLANE_REQUEST_FAILURE - CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG + - COST_CONTROL_ENTITLEMENT_DENIED - DATABASE_CONNECTION_FAILURE - DATA_ACCESS_CONFIG_CHANGED - DBFS_COMPONENT_UNHEALTHY @@ -10986,6 +11187,7 @@ components: - DOCKER_IMAGE_PULL_FAILURE - DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION - DOCKER_INVALID_OS_EXCEPTION + - DRIVER_DNS_RESOLUTION_FAILURE - DRIVER_EVICTION - DRIVER_LAUNCH_TIMEOUT - DRIVER_NODE_UNREACHABLE @@ -11062,6 +11264,8 @@ components: - NETWORK_CHECK_STORAGE_FAILURE_DUE_TO_MISCONFIG - NETWORK_CONFIGURATION_FAILURE - NFS_MOUNT_FAILURE + - NO_ACTIVATED_K8S + - NO_ACTIVATED_K8S_TESTING_TAG - NO_MATCHED_K8S - NO_MATCHED_K8S_TESTING_TAG - NPIP_TUNNEL_SETUP_FAILURE @@ -11072,9 +11276,11 @@ components: - REQUEST_REJECTED - REQUEST_THROTTLED - RESOURCE_USAGE_BLOCKED + - SECRET_CREATION_ACCESS_DENIED - SECRET_CREATION_FAILURE - SECRET_PERMISSION_DENIED - SECRET_RESOLUTION_ERROR + - SECURITY_AGENTS_FAILED_INITIAL_VERIFICATION - SECURITY_DAEMON_REGISTRATION_EXCEPTION - SELF_BOOTSTRAP_FAILURE - SERVERLESS_LONG_RUNNING_TERMINATED @@ -11106,6 +11312,7 @@ components: - WORKER_SETUP_FAILURE - WORKSPACE_CANCELLED_ERROR - WORKSPACE_CONFIGURATION_ERROR + - WORKSPACE_DELEGATION_KEY_MISCONFIGURED - WORKSPACE_UPDATE description: The status code indicating why the cluster was terminated TerminationReasonType: @@ -11130,6 +11337,8 @@ components: x-enum: - CLASSIC - PRO + - REALTIME + - REYDEN - TYPE_UNSPECIFIED description: |- Create a collection of name/value pairs. @@ -11646,6 +11855,7 @@ components: response: mediaType: application/json openAPIDocKey: '200' + objectKey: $.res sqlVerbs: select: - $ref: '#/components/x-stackQL-resources/query_history/methods/list' @@ -11751,12 +11961,12 @@ components: #/components/x-stackQL-resources/statement_execution/methods/get_result_chunk - $ref: '#/components/x-stackQL-resources/statement_execution/methods/get' insert: - - $ref: >- - #/components/x-stackQL-resources/statement_execution/methods/cancel - $ref: >- #/components/x-stackQL-resources/statement_execution/methods/execute update: [] - delete: [] + delete: + - $ref: >- + #/components/x-stackQL-resources/statement_execution/methods/cancel replace: [] warehouses: id: databricks_workspace.sql.warehouses diff --git a/providers/src/databricks_workspace/v00.00.00000/services/supervisoragents.yaml b/providers/src/databricks_workspace/v00.00.00000/services/supervisoragents.yaml new file mode 100644 index 00000000..52608d67 --- /dev/null +++ b/providers/src/databricks_workspace/v00.00.00000/services/supervisoragents.yaml @@ -0,0 +1,1846 @@ +openapi: 3.0.0 +info: + title: Databricks Supervisoragents API (workspace) + description: >- + OpenAPI specification for the Databricks supervisoragents service + (workspace-level APIs), generated from the Databricks Python SDK. + version: 0.1.0 + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' + x-stackql-sdk-namespace: databricks.sdk.service.supervisoragents +servers: + - url: https://{deployment_name}.cloud.databricks.com + variables: + deployment_name: + description: The Databricks Workspace Deployment Name + default: dbc-abcd0123-a1bc +paths: + /api/2.1/{parent}/examples: + post: + operationId: supervisor_agents_create_example + summary: Creates an example for a Supervisor Agent. + tags: + - supervisoragents + - supervisor_agents + description: |- + Creates an example for a Supervisor Agent. + + :param parent: str + Parent resource where this example will be created. Format: supervisor-agents/{supervisor_agent_id} + :param example: :class:`Example` + The example to create under the parent Supervisor Agent. + + :returns: :class:`Example` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Parent resource where this example will be created. Format: + supervisor-agents/{supervisor_agent_id} + requestBody: + content: + application/json: + schema: + type: object + properties: + example: + type: object + properties: + question: + type: string + description: The example question. + guidelines: + type: array + items: + type: string + description: Guidelines for answering the question. + example_id: + type: string + description: The universally unique identifier (UUID) of the example. + name: + type: string + description: >- + Full resource name: + supervisor-agents/{supervisor_agent_id}/examples/{example_id} + required: + - question + - guidelines + description: The example to create under the parent Supervisor Agent. + required: + - example + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Example' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + get: + operationId: supervisor_agents_list_examples + summary: Lists examples under a Supervisor Agent. + tags: + - supervisoragents + - supervisor_agents + description: |- + Lists examples under a Supervisor Agent. + + :param parent: str + Parent resource to list from. Format: supervisor-agents/{supervisor_agent_id} + :param page_size: int (optional) + The maximum number of examples to return. If unspecified, at most 100 examples will be returned. The + maximum value is 100; values above 100 will be coerced to 100. + :param page_token: str (optional) + A page token, received from a previous ``ListExamples`` call. Provide this to retrieve the + subsequent page. If unspecified, the first page will be returned. + + :returns: Iterator over :class:`Example` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Parent resource to list from. Format: + supervisor-agents/{supervisor_agent_id} + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of examples to return. If unspecified, at most + 100 examples will be returned. The maximum value is 100; values + above 100 will be coerced to 100. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ``ListExamples`` call. + Provide this to retrieve the subsequent page. If unspecified, the + first page will be returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListExamplesResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + /api/2.1/supervisor-agents: + post: + operationId: supervisor_agents_create_supervisor_agent + summary: Creates a new Supervisor Agent. + tags: + - supervisoragents + - supervisor_agents + description: |- + Creates a new Supervisor Agent. + + :param supervisor_agent: :class:`SupervisorAgent` + The Supervisor Agent to create. + + :returns: :class:`SupervisorAgent` + requestBody: + content: + application/json: + schema: + type: object + properties: + supervisor_agent: + type: object + properties: + display_name: + type: string + create_time: + type: string + format: date-time + description: Creation timestamp. + creator: + type: string + description: The creator of the Supervisor Agent. + description: + type: string + description: Description of what this agent can do (user-facing). + endpoint_name: + type: string + description: The name of the supervisor agent's serving endpoint. + experiment_id: + type: string + description: The MLflow experiment ID. + id: + type: string + description: 'Deprecated: Use supervisor_agent_id instead.' + instructions: + type: string + description: >- + Optional natural-language instructions for the + supervisor agent. + name: + type: string + description: >- + The resource name of the SupervisorAgent. Format: + supervisor-agents/{supervisor_agent_id} + supervisor_agent_id: + type: string + description: >- + The universally unique identifier (UUID) of the + Supervisor Agent. + required: + - display_name + description: The Supervisor Agent to create. + required: + - supervisor_agent + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SupervisorAgent' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + get: + operationId: supervisor_agents_list_supervisor_agents + summary: Lists Supervisor Agents. + tags: + - supervisoragents + - supervisor_agents + description: |- + Lists Supervisor Agents. + + :param page_size: int (optional) + The maximum number of supervisor agents to return. If unspecified, at most 100 supervisor agents + will be returned. The maximum value is 100; values above 100 will be coerced to 100. + :param page_token: str (optional) + A page token, received from a previous ``ListSupervisorAgents`` call. Provide this to retrieve the + subsequent page. If unspecified, the first page will be returned. + + :returns: Iterator over :class:`SupervisorAgent` + parameters: + - name: page_size + in: query + required: false + schema: + type: integer + description: >- + The maximum number of supervisor agents to return. If unspecified, + at most 100 supervisor agents will be returned. The maximum value is + 100; values above 100 will be coerced to 100. + - name: page_token + in: query + required: false + schema: + type: string + description: >- + A page token, received from a previous ``ListSupervisorAgents`` + call. Provide this to retrieve the subsequent page. If unspecified, + the first page will be returned. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListSupervisorAgentsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + /api/2.1/{parent}/tools: + post: + operationId: supervisor_agents_create_tool + summary: >- + Creates a Tool under a Supervisor Agent. Specify one of "genie_space", + "knowledge_assistant", + tags: + - supervisoragents + - supervisor_agents + description: >- + Creates a Tool under a Supervisor Agent. Specify one of "genie_space", + "knowledge_assistant", + + "uc_function", "uc_connection", "app", "volume", "dashboard", "table", + "vector_search_index", + + "catalog", "schema", "supervisor_agent", "databricks_web_search", + "skill" in the request body. The + + legacy values "lakeview_dashboard", "uc_table", and "web_search" are + also accepted and remain + + equivalent to "dashboard", "table", and "databricks_web_search" + respectively. The + + "databricks_web_search" tool_type maps to the ``web_search`` spec field. + + + :param parent: str + Parent resource where this tool will be created. Format: supervisor-agents/{supervisor_agent_id} + :param tool: :class:`Tool` + + :param tool_id: str + The ID to use for the tool, which will become the final component of the tool's resource name. + + :returns: :class:`Tool` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Parent resource where this tool will be created. Format: + supervisor-agents/{supervisor_agent_id} + - name: tool_id + in: query + required: true + schema: + type: string + description: >- + The ID to use for the tool, which will become the final component of + the tool's resource name. + requestBody: + content: + application/json: + schema: + type: object + properties: + tool: + type: object + properties: + tool_type: + type: string + app: + type: object + properties: + name: + type: string + description: App name + required: + - name + description: 'Databricks app. Supported app: custom mcp, custom agent.' + catalog: + type: object + properties: + name: + type: string + description: >- + Bare UC catalog name this tool is authorized to + search (no ``.``). + required: + - name + description: Configuration for a UC catalog asset_search scope tool. + dashboard: + type: object + properties: + dashboard_id: + type: string + description: The unique identifier of the Lakeview dashboard. + required: + - dashboard_id + description: >- + Lakeview dashboard tool. Replaces the deprecated + ``lakeview_dashboard`` field. + description: + type: string + description: Description of what this tool does (user-facing). + genie_space: + type: object + properties: + id: + type: string + space_id: + type: string + description: The ID of the genie space. + required: + - id + id: + type: string + description: 'Deprecated: Use tool_id instead.' + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + serving_endpoint_name: + type: string + description: 'Deprecated: use knowledge_assistant_id instead.' + required: + - knowledge_assistant_id + lakeview_dashboard: + type: object + properties: + dashboard_id: + type: string + description: The unique identifier of the Lakeview dashboard. + required: + - dashboard_id + description: 'Deprecated: use ``dashboard`` instead.' + name: + type: string + description: >- + Full resource name: + supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + schema: + type: object + properties: + name: + type: string + description: >- + Full UC schema name (catalog.schema) this tool is + authorized to search. + required: + - name + description: Configuration for a UC schema asset_search scope tool. + serving_endpoint: + type: object + properties: + name: + type: string + required: + - name + skill: + type: object + properties: + path: + type: string + description: >- + Absolute WSFS path to a folder containing skill + subdirectories. Example: + /Workspace/Users/creator@company.com/.assistant/skills + required: + - path + description: >- + Skill tool. Points to a folder containing skill + subdirectories with SKILL.md files. Skills are + discovered from the folder at runtime and loaded on + demand via a read_skill tool. + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: The ID of the supervisor agent (tile ID). + required: + - supervisor_agent_id + description: Nested Supervisor Agent tool. + table: + type: object + properties: + name: + type: string + description: >- + Full UC table name (catalog.schema.table) this tool + is authorized to access. + required: + - name + description: >- + Unity Catalog table tool. Replaces the deprecated + ``uc_table`` field. + tool_id: + type: string + description: User specified id of the Tool. + uc_connection: + type: object + properties: + name: + type: string + required: + - name + description: >- + Databricks UC connection. Supported connection: external + mcp server. + uc_function: + type: object + properties: + name: + type: string + required: + - name + uc_mcp: + type: object + properties: + name: + type: string + description: >- + Three-level UC FQN of the registered MCP service + (catalog.schema.mcp_service). + required: + - name + description: >- + UC-registered MCP service tool. The ``name`` field on + UcMcpService is the three-level UC FQN + (catalog.schema.mcp_service); the supervisor resolves it + at request build time, calls ``tools/list`` against the + AI Gateway mcp-services proxy, and dynamically registers + every discovered MCP sub-tool as a separately-callable + tool. + uc_table: + type: object + properties: + name: + type: string + description: >- + Full UC table name (catalog.schema.table) this tool + is authorized to access. + required: + - name + description: 'Deprecated: use ``table`` instead.' + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Full Vector Search index name + (catalog.schema.index). + columns: + type: array + items: + type: string + description: >- + Optional columns to return from the index. If unset, + discovered from index schema at query time. + required: + - name + description: Configuration for a Vector Search index tool. + volume: + type: object + properties: + name: + type: string + required: + - name + web_search: + type: object + properties: {} + description: >- + Configuration for a public-web search tool. The + supervisor collapses multiple web_search tools on the + same agent into a single registered ``web_search`` tool + at runtime. + required: + - tool_type + required: + - tool + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Tool' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + get: + operationId: supervisor_agents_list_tools + summary: Lists Tools under a Supervisor Agent. + tags: + - supervisoragents + - supervisor_agents + description: |- + Lists Tools under a Supervisor Agent. + + :param parent: str + Parent resource to list from. Format: supervisor-agents/{supervisor_agent_id} + :param page_size: int (optional) + :param page_token: str (optional) + + :returns: Iterator over :class:`Tool` + parameters: + - name: parent + in: path + required: true + schema: + type: string + description: >- + Parent resource to list from. Format: + supervisor-agents/{supervisor_agent_id} + - name: page_size + in: query + required: false + schema: + type: integer + - name: page_token + in: query + required: false + schema: + type: string + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/ListToolsResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + /api/2.1/{name}: + delete: + operationId: supervisor_agents_delete_tool + summary: Deletes a Tool. + tags: + - supervisoragents + - supervisor_agents + description: |- + Deletes a Tool. + + :param name: str + The resource name of the Tool. Format: supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The resource name of the Tool. Format: + supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + responses: + '200': + description: Success + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + get: + operationId: supervisor_agents_get_tool + summary: Gets a Tool. + tags: + - supervisoragents + - supervisor_agents + description: |- + Gets a Tool. + + :param name: str + The resource name of the Tool. Format: supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + + :returns: :class:`Tool` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + The resource name of the Tool. Format: + supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Tool' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + patch: + operationId: supervisor_agents_update_tool + summary: >- + Updates a Tool. Only the ``description`` field can be updated. To change + immutable fields such as tool + tags: + - supervisoragents + - supervisor_agents + description: >- + Updates a Tool. Only the ``description`` field can be updated. To change + immutable fields such as tool + + type, spec, or tool ID, delete the tool and recreate it. + + + :param name: str + Full resource name: supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + :param tool: :class:`Tool` + The Tool to update. + :param update_mask: FieldMask + Field mask for fields to be updated. + + :returns: :class:`Tool` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: >- + Full resource name: + supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + - name: update_mask + in: query + required: true + schema: + type: object + description: Field mask for fields to be updated. + requestBody: + content: + application/json: + schema: + type: object + properties: + tool: + type: object + properties: + tool_type: + type: string + app: + type: object + properties: + name: + type: string + description: App name + required: + - name + description: 'Databricks app. Supported app: custom mcp, custom agent.' + catalog: + type: object + properties: + name: + type: string + description: >- + Bare UC catalog name this tool is authorized to + search (no ``.``). + required: + - name + description: Configuration for a UC catalog asset_search scope tool. + dashboard: + type: object + properties: + dashboard_id: + type: string + description: The unique identifier of the Lakeview dashboard. + required: + - dashboard_id + description: >- + Lakeview dashboard tool. Replaces the deprecated + ``lakeview_dashboard`` field. + description: + type: string + description: Description of what this tool does (user-facing). + genie_space: + type: object + properties: + id: + type: string + space_id: + type: string + description: The ID of the genie space. + required: + - id + id: + type: string + description: 'Deprecated: Use tool_id instead.' + knowledge_assistant: + type: object + properties: + knowledge_assistant_id: + type: string + serving_endpoint_name: + type: string + description: 'Deprecated: use knowledge_assistant_id instead.' + required: + - knowledge_assistant_id + lakeview_dashboard: + type: object + properties: + dashboard_id: + type: string + description: The unique identifier of the Lakeview dashboard. + required: + - dashboard_id + description: 'Deprecated: use ``dashboard`` instead.' + name: + type: string + description: >- + Full resource name: + supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + schema: + type: object + properties: + name: + type: string + description: >- + Full UC schema name (catalog.schema) this tool is + authorized to search. + required: + - name + description: Configuration for a UC schema asset_search scope tool. + serving_endpoint: + type: object + properties: + name: + type: string + required: + - name + skill: + type: object + properties: + path: + type: string + description: >- + Absolute WSFS path to a folder containing skill + subdirectories. Example: + /Workspace/Users/creator@company.com/.assistant/skills + required: + - path + description: >- + Skill tool. Points to a folder containing skill + subdirectories with SKILL.md files. Skills are + discovered from the folder at runtime and loaded on + demand via a read_skill tool. + supervisor_agent: + type: object + properties: + supervisor_agent_id: + type: string + description: The ID of the supervisor agent (tile ID). + required: + - supervisor_agent_id + description: Nested Supervisor Agent tool. + table: + type: object + properties: + name: + type: string + description: >- + Full UC table name (catalog.schema.table) this tool + is authorized to access. + required: + - name + description: >- + Unity Catalog table tool. Replaces the deprecated + ``uc_table`` field. + tool_id: + type: string + description: User specified id of the Tool. + uc_connection: + type: object + properties: + name: + type: string + required: + - name + description: >- + Databricks UC connection. Supported connection: external + mcp server. + uc_function: + type: object + properties: + name: + type: string + required: + - name + uc_mcp: + type: object + properties: + name: + type: string + description: >- + Three-level UC FQN of the registered MCP service + (catalog.schema.mcp_service). + required: + - name + description: >- + UC-registered MCP service tool. The ``name`` field on + UcMcpService is the three-level UC FQN + (catalog.schema.mcp_service); the supervisor resolves it + at request build time, calls ``tools/list`` against the + AI Gateway mcp-services proxy, and dynamically registers + every discovered MCP sub-tool as a separately-callable + tool. + uc_table: + type: object + properties: + name: + type: string + description: >- + Full UC table name (catalog.schema.table) this tool + is authorized to access. + required: + - name + description: 'Deprecated: use ``table`` instead.' + vector_search_index: + type: object + properties: + name: + type: string + description: >- + Full Vector Search index name + (catalog.schema.index). + columns: + type: array + items: + type: string + description: >- + Optional columns to return from the index. If unset, + discovered from index schema at query time. + required: + - name + description: Configuration for a Vector Search index tool. + volume: + type: object + properties: + name: + type: string + required: + - name + web_search: + type: object + properties: {} + description: >- + Configuration for a public-web search tool. The + supervisor collapses multiple web_search tools on the + same agent into a single registered ``web_search`` tool + at runtime. + required: + - tool_type + description: The Tool to update. + required: + - tool + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/Tool' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + /api/2.0/permissions/supervisor-agents/{supervisor_agent_id}/permissionLevels: + get: + operationId: supervisor_agents_get_permission_levels + summary: Gets the permission levels that a user can have on an object. + tags: + - supervisoragents + - supervisor_agents + description: |- + Gets the permission levels that a user can have on an object. + + :param supervisor_agent_id: str + The supervisor agent for which to get or manage permissions. + + :returns: :class:`GetSupervisorAgentPermissionLevelsResponse` + parameters: + - name: supervisor_agent_id + in: path + required: true + schema: + type: string + description: The supervisor agent for which to get or manage permissions. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: >- + #/components/schemas/GetSupervisorAgentPermissionLevelsResponse + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + /api/2.0/permissions/supervisor-agents/{supervisor_agent_id}: + get: + operationId: supervisor_agents_get_permissions + summary: >- + Gets the permissions of a supervisor agent. Supervisor agents can + inherit permissions from their root + tags: + - supervisoragents + - supervisor_agents + description: >- + Gets the permissions of a supervisor agent. Supervisor agents can + inherit permissions from their root + + object. + + + :param supervisor_agent_id: str + The supervisor agent for which to get or manage permissions. + + :returns: :class:`SupervisorAgentPermissions` + parameters: + - name: supervisor_agent_id + in: path + required: true + schema: + type: string + description: The supervisor agent for which to get or manage permissions. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SupervisorAgentPermissions' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + put: + operationId: supervisor_agents_set_permissions + summary: >- + Sets permissions on an object, replacing existing permissions if they + exist. Deletes all direct + tags: + - supervisoragents + - supervisor_agents + description: >- + Sets permissions on an object, replacing existing permissions if they + exist. Deletes all direct + + permissions if none are specified. Objects can inherit permissions from + their root object. + + + :param supervisor_agent_id: str + The supervisor agent for which to get or manage permissions. + :param access_control_list: + List[:class:`SupervisorAgentAccessControlRequest`] (optional) + + + :returns: :class:`SupervisorAgentPermissions` + parameters: + - name: supervisor_agent_id + in: path + required: true + schema: + type: string + description: The supervisor agent for which to get or manage permissions. + requestBody: + content: + application/json: + schema: + type: object + properties: + access_control_list: + type: array + items: + type: object + properties: + group_name: + type: string + permission_level: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + description: Permission level + service_principal_name: + type: string + description: application ID of a service principal + user_name: + type: string + description: name of the user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SupervisorAgentPermissions' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI + patch: + operationId: supervisor_agents_update_permissions + summary: >- + Updates the permissions on a supervisor agent. Supervisor agents can + inherit permissions from their + tags: + - supervisoragents + - supervisor_agents + description: >- + Updates the permissions on a supervisor agent. Supervisor agents can + inherit permissions from their + + root object. + + + :param supervisor_agent_id: str + The supervisor agent for which to get or manage permissions. + :param access_control_list: + List[:class:`SupervisorAgentAccessControlRequest`] (optional) + + + :returns: :class:`SupervisorAgentPermissions` + parameters: + - name: supervisor_agent_id + in: path + required: true + schema: + type: string + description: The supervisor agent for which to get or manage permissions. + requestBody: + content: + application/json: + schema: + type: object + properties: + access_control_list: + type: array + items: + type: object + properties: + group_name: + type: string + permission_level: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + description: Permission level + service_principal_name: + type: string + description: application ID of a service principal + user_name: + type: string + description: name of the user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/SupervisorAgentPermissions' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: SupervisorAgentsAPI +components: + schemas: + App: + type: object + properties: + name: + type: string + description: App name + required: + - name + description: 'Databricks app. Supported app: custom mcp, custom agent.' + Catalog: + type: object + properties: + name: + type: string + description: Bare UC catalog name this tool is authorized to search (no ``.``). + required: + - name + description: >- + UC catalog asset_search scope. One tool authorizes asset_search over one + catalog. Multiple + catalog tools widen the scope; the backend merges them into a single CATALOG entry in + asset_search's scoped_assets. + Example: + type: object + properties: + question: + type: string + description: The example question. + guidelines: + type: array + items: + type: string + description: Guidelines for answering the question. + example_id: + type: string + description: The universally unique identifier (UUID) of the example. + name: + type: string + description: >- + Full resource name: + supervisor-agents/{supervisor_agent_id}/examples/{example_id} + required: + - question + - guidelines + description: >- + An example associated with a Supervisor Agent. Contains a question and + guidelines for how the + agent should respond. + GenieSpace: + type: object + properties: + id: + type: string + space_id: + type: string + description: The ID of the genie space. + required: + - id + GetSupervisorAgentPermissionLevelsResponse: + type: object + properties: + permission_levels: + type: array + items: + $ref: '#/components/schemas/SupervisorAgentPermissionsDescription' + KnowledgeAssistant: + type: object + properties: + knowledge_assistant_id: + type: string + serving_endpoint_name: + type: string + description: 'Deprecated: use knowledge_assistant_id instead.' + required: + - knowledge_assistant_id + LakeviewDashboard: + type: object + properties: + dashboard_id: + type: string + description: The unique identifier of the Lakeview dashboard. + required: + - dashboard_id + description: Lakeview Dashboard tool scoped to a specific published dashboard. + ListExamplesResponse: + type: object + properties: + examples: + type: array + items: + $ref: '#/components/schemas/Example' + next_page_token: + type: string + description: A list of Supervisor Agent examples. + ListSupervisorAgentsResponse: + type: object + properties: + next_page_token: + type: string + supervisor_agents: + type: array + items: + $ref: '#/components/schemas/SupervisorAgent' + ListToolsResponse: + type: object + properties: + next_page_token: + type: string + tools: + type: array + items: + $ref: '#/components/schemas/Tool' + Schema: + type: object + properties: + name: + type: string + description: >- + Full UC schema name (catalog.schema) this tool is authorized to + search. + required: + - name + description: >- + UC schema asset_search scope. One tool authorizes asset_search over one + schema. Multiple schema + tools widen the scope. + ServingEndpoint: + type: object + properties: + name: + type: string + required: + - name + Skill: + type: object + properties: + path: + type: string + description: >- + Absolute WSFS path to a folder containing skill subdirectories. + Example: /Workspace/Users/creator@company.com/.assistant/skills + required: + - path + description: >- + Skill tool. Points to a folder containing skill subdirectories with + SKILL.md files (YAML + frontmatter with name + description, followed by markdown instructions). Skills are discovered + from the folder at runtime and loaded on demand via a read_skill tool registered by the + supervisor. + SupervisorAgent: + type: object + properties: + display_name: + type: string + create_time: + type: string + format: date-time + description: Creation timestamp. + creator: + type: string + description: The creator of the Supervisor Agent. + description: + type: string + description: Description of what this agent can do (user-facing). + endpoint_name: + type: string + description: The name of the supervisor agent's serving endpoint. + experiment_id: + type: string + description: The MLflow experiment ID. + id: + type: string + description: 'Deprecated: Use supervisor_agent_id instead.' + instructions: + type: string + description: Optional natural-language instructions for the supervisor agent. + name: + type: string + description: >- + The resource name of the SupervisorAgent. Format: + supervisor-agents/{supervisor_agent_id} + supervisor_agent_id: + type: string + description: The universally unique identifier (UUID) of the Supervisor Agent. + required: + - display_name + SupervisorAgentAccessControlRequest: + type: object + properties: + group_name: + type: string + permission_level: + $ref: '#/components/schemas/SupervisorAgentPermissionLevel' + service_principal_name: + type: string + description: application ID of a service principal + user_name: + type: string + description: name of the user + SupervisorAgentAccessControlResponse: + type: object + properties: + all_permissions: + type: array + items: + $ref: '#/components/schemas/SupervisorAgentPermission' + display_name: + type: string + description: Display name of the user or service principal. + group_name: + type: string + description: name of the group + service_principal_name: + type: string + description: Name of the service principal. + user_name: + type: string + description: name of the user + SupervisorAgentPermission: + type: object + properties: + inherited: + type: boolean + inherited_from_object: + type: array + items: + type: string + permission_level: + $ref: '#/components/schemas/SupervisorAgentPermissionLevel' + SupervisorAgentPermissions: + type: object + properties: + access_control_list: + type: array + items: + $ref: '#/components/schemas/SupervisorAgentAccessControlResponse' + object_id: + type: string + object_type: + type: string + SupervisorAgentPermissionsDescription: + type: object + properties: + description: + type: string + permission_level: + $ref: '#/components/schemas/SupervisorAgentPermissionLevel' + SupervisorAgentTool: + type: object + properties: + supervisor_agent_id: + type: string + description: The ID of the supervisor agent (tile ID). + required: + - supervisor_agent_id + description: Nested Supervisor Agent tool. + Tool: + type: object + properties: + tool_type: + type: string + app: + $ref: '#/components/schemas/App' + catalog: + $ref: '#/components/schemas/Catalog' + description: Configuration for a UC catalog asset_search scope tool. + dashboard: + $ref: '#/components/schemas/LakeviewDashboard' + description: >- + Lakeview dashboard tool. Replaces the deprecated + ``lakeview_dashboard`` field. + description: + type: string + description: Description of what this tool does (user-facing). + genie_space: + $ref: '#/components/schemas/GenieSpace' + id: + type: string + description: 'Deprecated: Use tool_id instead.' + knowledge_assistant: + $ref: '#/components/schemas/KnowledgeAssistant' + lakeview_dashboard: + $ref: '#/components/schemas/LakeviewDashboard' + description: 'Deprecated: use ``dashboard`` instead.' + name: + type: string + description: >- + Full resource name: + supervisor-agents/{supervisor_agent_id}/tools/{tool_id} + schema: + $ref: '#/components/schemas/Schema' + description: Configuration for a UC schema asset_search scope tool. + serving_endpoint: + $ref: '#/components/schemas/ServingEndpoint' + skill: + $ref: '#/components/schemas/Skill' + description: >- + Skill tool. Points to a folder containing skill subdirectories with + SKILL.md files. Skills are discovered from the folder at runtime and + loaded on demand via a read_skill tool. + supervisor_agent: + $ref: '#/components/schemas/SupervisorAgentTool' + table: + $ref: '#/components/schemas/UcTable' + description: >- + Unity Catalog table tool. Replaces the deprecated ``uc_table`` + field. + tool_id: + type: string + description: User specified id of the Tool. + uc_connection: + $ref: '#/components/schemas/UcConnection' + uc_function: + $ref: '#/components/schemas/UcFunction' + uc_mcp: + $ref: '#/components/schemas/UcMcpService' + description: >- + UC-registered MCP service tool. The ``name`` field on UcMcpService + is the three-level UC FQN (catalog.schema.mcp_service); the + supervisor resolves it at request build time, calls ``tools/list`` + against the AI Gateway mcp-services proxy, and dynamically registers + every discovered MCP sub-tool as a separately-callable tool. + uc_table: + $ref: '#/components/schemas/UcTable' + description: 'Deprecated: use ``table`` instead.' + vector_search_index: + $ref: '#/components/schemas/VectorSearchIndex' + description: Configuration for a Vector Search index tool. + volume: + $ref: '#/components/schemas/Volume' + web_search: + $ref: '#/components/schemas/WebSearch' + description: >- + Configuration for a public-web search tool. The supervisor collapses + multiple web_search tools on the same agent into a single registered + ``web_search`` tool at runtime. + required: + - tool_type + UcConnection: + type: object + properties: + name: + type: string + required: + - name + description: 'Databricks UC connection. Supported connection: external mcp server.' + UcFunction: + type: object + properties: + name: + type: string + required: + - name + UcMcpService: + type: object + properties: + name: + type: string + description: >- + Three-level UC FQN of the registered MCP service + (catalog.schema.mcp_service). + required: + - name + description: >- + UC-registered MCP service tool. The ``name`` field is the three-level UC + FQN of the MCP service + (``catalog.schema.mcp_service``). At request build time the supervisor calls ``tools/list`` + against the AI Gateway mcp-services proxy and dynamically registers every discovered MCP + sub-tool as a separately-callable tool. Per-sub-tool config is not stored here — discovery is + dynamic so newly published MCP functions are picked up without redeploying the agent. + UcTable: + type: object + properties: + name: + type: string + description: >- + Full UC table name (catalog.schema.table) this tool is authorized to + access. + required: + - name + description: >- + Unity Catalog table. One tool represents one authorized table; the + backend collapses all + uc_table tools on a supervisor agent into a single subagent that can access the union of their + tables. + VectorSearchIndex: + type: object + properties: + name: + type: string + description: Full Vector Search index name (catalog.schema.index). + columns: + type: array + items: + type: string + description: >- + Optional columns to return from the index. If unset, discovered from + index schema at query time. + required: + - name + description: Vector Search index tool authorizing access to a single index. + Volume: + type: object + properties: + name: + type: string + required: + - name + WebSearch: + type: object + properties: {} + description: >- + Public-web search tool. Empty body — backend, model registration, and + client_id are not + customer-tunable. The display name and description for this tool come from the parent + ``Tool.name`` / ``Tool.description`` fields. Reserved for future scoping (allowed domains, + region overrides). + SupervisorAgentPermissionLevel: + type: string + x-enum: + - CAN_MANAGE + - CAN_QUERY + description: Permission level + x-stackQL-resources: + examples: + id: databricks_workspace.supervisoragents.examples + name: examples + title: Examples + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1{parent}~1examples/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1{parent}~1examples/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.examples + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/examples/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/examples/methods/create' + update: [] + delete: [] + replace: [] + supervisor_agents: + id: databricks_workspace.supervisoragents.supervisor_agents + name: supervisor_agents + title: Supervisor Agents + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1supervisor-agents/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1supervisor-agents/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.supervisor_agents + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/supervisor_agents/methods/list' + insert: + - $ref: '#/components/x-stackQL-resources/supervisor_agents/methods/create' + update: [] + delete: [] + replace: [] + tools: + id: databricks_workspace.supervisoragents.tools + name: tools + title: Tools + methods: + create: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1{parent}~1tools/post' + response: + mediaType: application/json + openAPIDocKey: '200' + list: + operation: + $ref: '#/paths/~1api~12.1~1{parent}~1tools/get' + response: + mediaType: application/json + openAPIDocKey: '200' + objectKey: $.tools + delete: + operation: + $ref: '#/paths/~1api~12.1~1{name}/delete' + response: + mediaType: application/json + openAPIDocKey: '200' + get: + operation: + $ref: '#/paths/~1api~12.1~1{name}/get' + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.1~1{name}/patch' + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: '#/components/x-stackQL-resources/tools/methods/list' + - $ref: '#/components/x-stackQL-resources/tools/methods/get' + insert: + - $ref: '#/components/x-stackQL-resources/tools/methods/create' + update: + - $ref: '#/components/x-stackQL-resources/tools/methods/update' + delete: + - $ref: '#/components/x-stackQL-resources/tools/methods/delete' + replace: [] + supervisor_agent_permission_levels: + id: databricks_workspace.supervisoragents.supervisor_agent_permission_levels + name: supervisor_agent_permission_levels + title: Supervisor Agent Permission Levels + methods: + get: + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1supervisor-agents~1{supervisor_agent_id}~1permissionLevels/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/supervisor_agent_permission_levels/methods/get + insert: [] + update: [] + delete: [] + replace: [] + supervisor_agent_permissions: + id: databricks_workspace.supervisoragents.supervisor_agent_permissions + name: supervisor_agent_permissions + title: Supervisor Agent Permissions + methods: + get: + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1supervisor-agents~1{supervisor_agent_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + set: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1supervisor-agents~1{supervisor_agent_id}/put + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1supervisor-agents~1{supervisor_agent_id}/patch + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/supervisor_agent_permissions/methods/get + insert: [] + update: + - $ref: >- + #/components/x-stackQL-resources/supervisor_agent_permissions/methods/update + delete: [] + replace: + - $ref: >- + #/components/x-stackQL-resources/supervisor_agent_permissions/methods/set +x-stackQL-config: + pagination: + requestToken: + key: page_token + location: query + responseToken: + key: next_page_token + location: body diff --git a/providers/src/databricks_workspace/v00.00.00000/services/tags.yaml b/providers/src/databricks_workspace/v00.00.00000/services/tags.yaml index 799e2714..63146523 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/tags.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/tags.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks tags service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.tags servers: - url: https://{deployment_name}.cloud.databricks.com @@ -20,25 +20,22 @@ paths: operationId: tag_policies_create_tag_policy summary: >- Creates a new tag policy, making the associated tag key governed. For - Terraform usage, see the [Tag + Terraform usage, see the `Tag tags: - tags - tag_policies description: >- Creates a new tag policy, making the associated tag key governed. For - Terraform usage, see the [Tag + Terraform usage, see the `Tag - Policy Terraform documentation]. To manage permissions for tag policies, - use the [Account Access + Policy Terraform documentation - Control Proxy API]. + `__. + To manage permissions for tag policies, use the `Account Access Control + Proxy API - [Account Access Control Proxy API]: - https://docs.databricks.com/api/workspace/accountaccesscontrolproxy - - [Tag Policy Terraform documentation]: - https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/tag_policy + `__. :param tag_policy: :class:`TagPolicy` @@ -56,6 +53,9 @@ paths: properties: tag_key: type: string + account_id: + type: string + description: The account ID that owns this tag policy. create_time: type: string description: Timestamp when the tag policy was created @@ -63,6 +63,35 @@ paths: type: string id: type: string + propagation_config: + type: object + properties: + conflict_resolution: + type: object + properties: + default_value_override: + type: object + properties: + default_value: + type: string + description: >- + The tag value to apply when conflicts are + detected. This value must be one of the + allowed values defined in the tag policy. + description: >- + Uses a specified default value to override when + conflicts happen. + description: >- + Policy that determines how to resolve conflicts when + multiple upstream sources have different tag values. + enabled: + type: boolean + description: >- + Determines whether this tag should automatically + propagate through lineage. + description: >- + Configuration that controls how tags are automatically + propagated through data lineage. update_time: type: string description: Timestamp when the tag policy was last updated @@ -102,25 +131,22 @@ paths: operationId: tag_policies_list_tag_policies summary: >- Lists the tag policies for all governed tags in the account. For - Terraform usage, see the [Tag Policy + Terraform usage, see the `Tag Policy tags: - tags - tag_policies description: >- Lists the tag policies for all governed tags in the account. For - Terraform usage, see the [Tag Policy + Terraform usage, see the `Tag Policy - Terraform documentation]. To list granted permissions for tag policies, - use the [Account Access + Terraform documentation - Control Proxy API]. + `__. + To list granted permissions for tag policies, use the `Account Access + Control Proxy API - [Account Access Control Proxy API]: - https://docs.databricks.com/api/workspace/accountaccesscontrolproxy - - [Tag Policy Terraform documentation]: - https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/tag_policies + `__. :param page_size: int (optional) @@ -182,11 +208,9 @@ paths: Deletes a tag policy by its associated governed tag's key, leaving that tag key ungoverned. For - Terraform usage, see the [Tag Policy Terraform documentation]. - + Terraform usage, see the `Tag Policy Terraform documentation - [Tag Policy Terraform documentation]: - https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/tag_policy + `__. :param tag_key: str @@ -215,25 +239,22 @@ paths: operationId: tag_policies_get_tag_policy summary: >- Gets a single tag policy by its associated governed tag's key. For - Terraform usage, see the [Tag + Terraform usage, see the `Tag tags: - tags - tag_policies description: >- Gets a single tag policy by its associated governed tag's key. For - Terraform usage, see the [Tag - - Policy Terraform documentation]. To list granted permissions for tag - policies, use the [Account Access + Terraform usage, see the `Tag - Control Proxy API]. + Policy Terraform documentation + `__. - [Account Access Control Proxy API]: - https://docs.databricks.com/api/workspace/accountaccesscontrolproxy + To list granted permissions for tag policies, use the `Account Access + Control Proxy API - [Tag Policy Terraform documentation]: - https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/tag_policy + `__. :param tag_key: str @@ -269,25 +290,22 @@ paths: operationId: tag_policies_update_tag_policy summary: >- Updates an existing tag policy for a single governed tag. For Terraform - usage, see the [Tag Policy + usage, see the `Tag Policy tags: - tags - tag_policies description: >- Updates an existing tag policy for a single governed tag. For Terraform - usage, see the [Tag Policy + usage, see the `Tag Policy - Terraform documentation]. To manage permissions for tag policies, use - the [Account Access Control + Terraform documentation - Proxy API]. + `__. + To manage permissions for tag policies, use the `Account Access Control + Proxy API - [Account Access Control Proxy API]: - https://docs.databricks.com/api/workspace/accountaccesscontrolproxy - - [Tag Policy Terraform documentation]: - https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/tag_policy + `__. :param tag_key: str @@ -296,14 +314,14 @@ paths: :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`TagPolicy` parameters: @@ -320,14 +338,14 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. requestBody: content: application/json: @@ -339,6 +357,9 @@ paths: properties: tag_key: type: string + account_id: + type: string + description: The account ID that owns this tag policy. create_time: type: string description: Timestamp when the tag policy was created @@ -346,6 +367,35 @@ paths: type: string id: type: string + propagation_config: + type: object + properties: + conflict_resolution: + type: object + properties: + default_value_override: + type: object + properties: + default_value: + type: string + description: >- + The tag value to apply when conflicts are + detected. This value must be one of the + allowed values defined in the tag policy. + description: >- + Uses a specified default value to override when + conflicts happen. + description: >- + Policy that determines how to resolve conflicts when + multiple upstream sources have different tag values. + enabled: + type: boolean + description: >- + Determines whether this tag should automatically + propagate through lineage. + description: >- + Configuration that controls how tags are automatically + propagated through data lineage. update_time: type: string description: Timestamp when the tag policy was last updated @@ -581,14 +631,14 @@ paths: :param tag_assignment: :class:`TagAssignment` :param update_mask: str The field mask must be a single string, with multiple fields separated by commas (no spaces). The - field path is relative to the resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map fields is not allowed, as only + field path is relative to the resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to always explicitly list the - fields being updated and avoid using `*` wildcards, as it can lead to unintended results if the API - changes in the future. + A field mask of ``*`` indicates full replacement. It’s recommended to always explicitly list the + fields being updated and avoid using ``*`` wildcards, as it can lead to unintended results if the + API changes in the future. :returns: :class:`TagAssignment` parameters: @@ -624,14 +674,14 @@ paths: description: >- The field mask must be a single string, with multiple fields separated by commas (no spaces). The field path is relative to the - resource object, using a dot (`.`) to navigate sub-fields (e.g., - `author.given_name`). Specification of elements in sequence or map + resource object, using a dot (``.``) to navigate sub-fields (e.g., + ``author.given_name``). Specification of elements in sequence or map fields is not allowed, as only the entire collection field can be specified. Field names must exactly match the resource field names. - A field mask of `*` indicates full replacement. It’s recommended to - always explicitly list the fields being updated and avoid using `*` - wildcards, as it can lead to unintended results if the API changes - in the future. + A field mask of ``*`` indicates full replacement. It’s recommended + to always explicitly list the fields being updated and avoid using + ``*`` wildcards, as it can lead to unintended results if the API + changes in the future. requestBody: content: application/json: @@ -756,6 +806,27 @@ paths: x-stackql-sdk-source: WorkspaceEntityTagAssignmentsAPI components: schemas: + ConflictResolutionPolicy: + type: object + properties: + default_value_override: + $ref: '#/components/schemas/DefaultValueOverridePolicy' + description: Uses a specified default value to override when conflicts happen. + description: >- + Policy that determines how to resolve conflicts when multiple upstream + sources have different + tag values. + DefaultValueOverridePolicy: + type: object + properties: + default_value: + type: string + description: >- + The tag value to apply when conflicts are detected. This value must + be one of the allowed values defined in the tag policy. + description: >- + Policy that specifies a default value to use when resolving tag + conflicts during propagation. ListTagAssignmentsResponse: type: object properties: @@ -774,6 +845,22 @@ components: type: array items: $ref: '#/components/schemas/TagPolicy' + PropagationConfig: + type: object + properties: + conflict_resolution: + $ref: '#/components/schemas/ConflictResolutionPolicy' + description: >- + Policy that determines how to resolve conflicts when multiple + upstream sources have different tag values. + enabled: + type: boolean + description: >- + Determines whether this tag should automatically propagate through + lineage. + description: >- + Configuration that controls how tags are automatically propagated + through data lineage. TagAssignment: type: object properties: @@ -801,6 +888,9 @@ components: properties: tag_key: type: string + account_id: + type: string + description: The account ID that owns this tag policy. create_time: type: string description: Timestamp when the tag policy was created @@ -808,6 +898,11 @@ components: type: string id: type: string + propagation_config: + $ref: '#/components/schemas/PropagationConfig' + description: >- + Configuration that controls how tags are automatically propagated + through data lineage. update_time: type: string description: Timestamp when the tag policy was last updated diff --git a/providers/src/databricks_workspace/v00.00.00000/services/vectorsearch.yaml b/providers/src/databricks_workspace/v00.00.00000/services/vectorsearch.yaml index 8dc443b6..d9ff6f1a 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/vectorsearch.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/vectorsearch.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks vectorsearch service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.vectorsearch servers: - url: https://{deployment_name}.cloud.databricks.com @@ -26,14 +26,19 @@ paths: Create a new endpoint. :param name: str - Name of the vector search endpoint + Name of the AI Search endpoint :param endpoint_type: :class:`EndpointType` Type of endpoint :param budget_policy_id: str (optional) The budget policy id to be applied - :param min_qps: int (optional) - Min QPS for the endpoint. Mutually exclusive with num_replicas. The actual replica count is - calculated at index creation/sync time based on this value. + :param num_replicas: int (optional) + Initial number of replicas for the endpoint. If not specified, defaults to 1. + :param target_qps: int (optional) + Target QPS for the endpoint. Mutually exclusive with num_replicas. The actual replica count is + calculated at index creation/sync time based on this value. Best-effort target; the system does not + guarantee this QPS will be achieved. + :param usage_policy_id: str (optional) + The usage policy id to be applied once we've migrated to usage policies :returns: Long-running operation waiter for :class:`EndpointInfo`. @@ -46,21 +51,34 @@ paths: properties: name: type: string - description: Name of the vector search endpoint + description: Name of the AI Search endpoint endpoint_type: type: string x-enum: - STANDARD + - STORAGE_OPTIMIZED description: Type of endpoint budget_policy_id: type: string description: The budget policy id to be applied - min_qps: + num_replicas: type: integer description: >- - Min QPS for the endpoint. Mutually exclusive with + Initial number of replicas for the endpoint. If not + specified, defaults to 1. + target_qps: + type: integer + description: >- + Target QPS for the endpoint. Mutually exclusive with num_replicas. The actual replica count is calculated at - index creation/sync time based on this value. + index creation/sync time based on this value. Best-effort + target; the system does not guarantee this QPS will be + achieved. + usage_policy_id: + type: string + description: >- + The usage policy id to be applied once we've migrated to + usage policies required: - name - endpoint_type @@ -85,12 +103,12 @@ paths: x-stackql-sdk-source: VectorSearchEndpointsAPI get: operationId: vector_search_endpoints_list_endpoints - summary: List all vector search endpoints in the workspace. + summary: List all AI Search endpoints in the workspace. tags: - vectorsearch - vector_search_endpoints description: |- - List all vector search endpoints in the workspace. + List all AI Search endpoints in the workspace. :param page_token: str (optional) Token for pagination @@ -125,22 +143,22 @@ paths: /api/2.0/vector-search/endpoints/{endpoint_name}: delete: operationId: vector_search_endpoints_delete_endpoint - summary: Delete a vector search endpoint. + summary: Delete an AI Search endpoint. tags: - vectorsearch - vector_search_endpoints description: |- - Delete a vector search endpoint. + Delete an AI Search endpoint. :param endpoint_name: str - Name of the vector search endpoint + Name of the AI Search endpoint parameters: - name: endpoint_name in: path required: true schema: type: string - description: Name of the vector search endpoint + description: Name of the AI Search endpoint responses: '200': description: Success @@ -158,12 +176,12 @@ paths: x-stackql-sdk-source: VectorSearchEndpointsAPI get: operationId: vector_search_endpoints_get_endpoint - summary: Get details for a single vector search endpoint. + summary: Get details for a single AI Search endpoint. tags: - vectorsearch - vector_search_endpoints description: |- - Get details for a single vector search endpoint. + Get details for a single AI Search endpoint. :param endpoint_name: str Name of the endpoint @@ -205,9 +223,15 @@ paths: Update an endpoint :param endpoint_name: str - Name of the vector search endpoint - :param min_qps: int (optional) - Min QPS for the endpoint. Positive integer sets QPS target; -1 resets to default scaling behavior. + Name of the AI Search endpoint + :param replication_factor: int (optional) + OpenSearch replication factor. Directly sets userThroughputSettings.replicationFactor. Mutually + exclusive with target_qps (and the deprecated min_qps alias). Must be non-negative (0 = no + replication). The autoscaler caps the effective value based on endpoint scaling settings. Note: This + is the raw replication factor, not "total data copies". For the user-facing replica count (which + uses total-copies semantics), see PatchEndpointThroughputRequest.num_replicas. + :param target_qps: int (optional) + Target QPS for the endpoint. Best-effort; the system does not guarantee this QPS will be achieved. :returns: :class:`EndpointInfo` parameters: @@ -216,18 +240,29 @@ paths: required: true schema: type: string - description: Name of the vector search endpoint + description: Name of the AI Search endpoint requestBody: content: application/json: schema: type: object properties: - min_qps: + replication_factor: type: integer description: >- - Min QPS for the endpoint. Positive integer sets QPS target; - -1 resets to default scaling behavior. + OpenSearch replication factor. Directly sets + userThroughputSettings.replicationFactor. Mutually exclusive + with target_qps (and the deprecated min_qps alias). Must be + non-negative (0 = no replication). The autoscaler caps the + effective value based on endpoint scaling settings. Note: + This is the raw replication factor, not "total data copies". + For the user-facing replica count (which uses total-copies + semantics), see PatchEndpointThroughputRequest.num_replicas. + target_qps: + type: integer + description: >- + Target QPS for the endpoint. Best-effort; the system does + not guarantee this QPS will be achieved. responses: '200': description: Success @@ -247,6 +282,331 @@ paths: message: type: string x-stackql-sdk-source: VectorSearchEndpointsAPI + /api/2.0/permissions/vector-search-endpoints/{endpoint_id}/permissionLevels: + get: + operationId: vector_search_endpoints_get_permission_levels + summary: Gets the permission levels that a user can have on an object. + tags: + - vectorsearch + - vector_search_endpoints + description: |- + Gets the permission levels that a user can have on an object. + + :param endpoint_id: str + The vector search endpoint for which to get or manage permissions. + + :returns: :class:`GetVectorSearchEndpointPermissionLevelsResponse` + parameters: + - name: endpoint_id + in: path + required: true + schema: + type: string + description: The vector search endpoint for which to get or manage permissions. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: >- + #/components/schemas/GetVectorSearchEndpointPermissionLevelsResponse + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchEndpointsAPI + /api/2.0/permissions/vector-search-endpoints/{endpoint_id}: + get: + operationId: vector_search_endpoints_get_permissions + summary: >- + Gets the permissions of a vector search endpoint. Vector search + endpoints can inherit permissions from + tags: + - vectorsearch + - vector_search_endpoints + description: >- + Gets the permissions of a vector search endpoint. Vector search + endpoints can inherit permissions from + + their root object. + + + :param endpoint_id: str + The vector search endpoint for which to get or manage permissions. + + :returns: :class:`VectorSearchEndpointPermissions` + parameters: + - name: endpoint_id + in: path + required: true + schema: + type: string + description: The vector search endpoint for which to get or manage permissions. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/VectorSearchEndpointPermissions' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchEndpointsAPI + put: + operationId: vector_search_endpoints_set_permissions + summary: >- + Sets permissions on an object, replacing existing permissions if they + exist. Deletes all direct + tags: + - vectorsearch + - vector_search_endpoints + description: >- + Sets permissions on an object, replacing existing permissions if they + exist. Deletes all direct + + permissions if none are specified. Objects can inherit permissions from + their root object. + + + :param endpoint_id: str + The vector search endpoint for which to get or manage permissions. + :param access_control_list: + List[:class:`VectorSearchEndpointAccessControlRequest`] (optional) + + + :returns: :class:`VectorSearchEndpointPermissions` + parameters: + - name: endpoint_id + in: path + required: true + schema: + type: string + description: The vector search endpoint for which to get or manage permissions. + requestBody: + content: + application/json: + schema: + type: object + properties: + access_control_list: + type: array + items: + type: object + properties: + group_name: + type: string + permission_level: + type: string + x-enum: + - CAN_CREATE + - CAN_MANAGE + - CAN_USE + description: Permission level + service_principal_name: + type: string + description: application ID of a service principal + user_name: + type: string + description: name of the user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/VectorSearchEndpointPermissions' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchEndpointsAPI + patch: + operationId: vector_search_endpoints_update_permissions + summary: >- + Updates the permissions on a vector search endpoint. Vector search + endpoints can inherit permissions + tags: + - vectorsearch + - vector_search_endpoints + description: >- + Updates the permissions on a vector search endpoint. Vector search + endpoints can inherit permissions + + from their root object. + + + :param endpoint_id: str + The vector search endpoint for which to get or manage permissions. + :param access_control_list: + List[:class:`VectorSearchEndpointAccessControlRequest`] (optional) + + + :returns: :class:`VectorSearchEndpointPermissions` + parameters: + - name: endpoint_id + in: path + required: true + schema: + type: string + description: The vector search endpoint for which to get or manage permissions. + requestBody: + content: + application/json: + schema: + type: object + properties: + access_control_list: + type: array + items: + type: object + properties: + group_name: + type: string + permission_level: + type: string + x-enum: + - CAN_CREATE + - CAN_MANAGE + - CAN_USE + description: Permission level + service_principal_name: + type: string + description: application ID of a service principal + user_name: + type: string + description: name of the user + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/VectorSearchEndpointPermissions' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchEndpointsAPI + /api/2.0/vector-search/endpoints/{endpoint_name}/throughput: + patch: + operationId: vector_search_endpoints_patch_endpoint_throughput + summary: Update the throughput (concurrency) of an endpoint + tags: + - vectorsearch + - vector_search_endpoints + description: |- + Update the throughput (concurrency) of an endpoint + + :param endpoint_name: str + Name of the AI Search endpoint + :param all_or_nothing: bool (optional) + If true, the request will fail if the requested concurrency or limits cannot be exactly met. If + false, the request will be adjusted to the closest possible value. + :param concurrency: float (optional) + Requested concurrency (total CPU) for the endpoint. If not specified, the current concurrency is + maintained. + :param maximum_concurrency_allowed: float (optional) + Maximum concurrency allowed for the endpoint. If not specified, the current maximum is maintained. + :param minimal_concurrency_allowed: float (optional) + Minimum concurrency allowed for the endpoint. If not specified, the current minimum is maintained. + :param num_replicas: int (optional) + Requested number of data copies for the endpoint (including primary). For example: num_replicas=2 + means 2 total copies of the data (1 primary + 1 replica). If not specified, the current replication + factor is maintained. Valid range: 1-6 (where 1 = no replication, 6 = 1 primary + 5 replicas). + + :returns: :class:`PatchEndpointThroughputResponse` + parameters: + - name: endpoint_name + in: path + required: true + schema: + type: string + description: Name of the AI Search endpoint + requestBody: + content: + application/json: + schema: + type: object + properties: + all_or_nothing: + type: boolean + description: >- + If true, the request will fail if the requested concurrency + or limits cannot be exactly met. If false, the request will + be adjusted to the closest possible value. + concurrency: + type: number + description: >- + Requested concurrency (total CPU) for the endpoint. If not + specified, the current concurrency is maintained. + maximum_concurrency_allowed: + type: number + description: >- + Maximum concurrency allowed for the endpoint. If not + specified, the current maximum is maintained. + minimal_concurrency_allowed: + type: number + description: >- + Minimum concurrency allowed for the endpoint. If not + specified, the current minimum is maintained. + num_replicas: + type: integer + description: >- + Requested number of data copies for the endpoint (including + primary). For example: num_replicas=2 means 2 total copies + of the data (1 primary + 1 replica). If not specified, the + current replication factor is maintained. Valid range: 1-6 + (where 1 = no replication, 6 = 1 primary + 5 replicas). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/PatchEndpointThroughputResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchEndpointsAPI /api/2.0/vector-search/endpoints/{name}/metrics: post: operationId: vector_search_endpoints_retrieve_user_visible_metrics @@ -258,7 +618,7 @@ paths: Retrieve user-visible metrics for an endpoint :param name: str - Vector search endpoint name + AI Search endpoint name :param end_time: str (optional) End time for metrics query :param granularity_in_seconds: int (optional) @@ -277,7 +637,7 @@ paths: required: true schema: type: string - description: Vector search endpoint name + description: AI Search endpoint name requestBody: content: application/json: @@ -352,7 +712,7 @@ paths: Update the budget policy of an endpoint :param endpoint_name: str - Name of the vector search endpoint + Name of the AI Search endpoint :param budget_policy_id: str The budget policy id to be applied @@ -363,7 +723,7 @@ paths: required: true schema: type: string - description: Name of the vector search endpoint + description: Name of the AI Search endpoint requestBody: content: application/json: @@ -405,9 +765,9 @@ paths: Update the custom tags of an endpoint. :param endpoint_name: str - Name of the vector search endpoint + Name of the AI Search endpoint :param custom_tags: List[:class:`CustomTag`] - The new custom tags for the vector search endpoint + The new custom tags for the AI Search endpoint :returns: :class:`UpdateEndpointCustomTagsResponse` parameters: @@ -416,7 +776,7 @@ paths: required: true schema: type: string - description: Name of the vector search endpoint + description: Name of the AI Search endpoint requestBody: content: application/json: @@ -432,12 +792,10 @@ paths: type: string value: type: string - description: >- - [Optional] Value field for a vector search endpoint - tag. + description: '[Optional] Value field for an AI Search endpoint tag.' required: - key - description: The new custom tags for the vector search endpoint + description: The new custom tags for the AI Search endpoint required: - custom_tags responses: @@ -480,10 +838,12 @@ paths: :param delta_sync_index_spec: :class:`DeltaSyncVectorIndexSpecRequest` (optional) - Specification for Delta Sync Index. Required if `index_type` is `DELTA_SYNC`. + Specification for Delta Sync Index. Required if ``index_type`` is ``DELTA_SYNC``. :param direct_access_index_spec: :class:`DirectAccessVectorIndexSpec` (optional) - Specification for Direct Vector Access Index. Required if `index_type` is `DIRECT_ACCESS`. + Specification for Direct Vector Access Index. Required if ``index_type`` is ``DIRECT_ACCESS``. + :param index_subtype: :class:`IndexSubtype` (optional) + The subtype of the index. Use ``HYBRID`` or ``FULL_TEXT``. ``VECTOR`` is not supported. :returns: :class:`VectorIndex` requestBody: @@ -507,26 +867,37 @@ paths: - DELTA_SYNC - DIRECT_ACCESS description: >- - There are 2 types of Vector Search indexes: - `DELTA_SYNC`: - An index that automatically syncs + There are 2 types of AI Search indexes: - with a source Delta Table, automatically and incrementally - updating the index as the underlying - data in the Delta Table changes. - `DIRECT_ACCESS`: An index - that supports direct read and write - - of vectors and metadata through our REST and SDK APIs. With - this model, the user manages index - - updates. + - ``DELTA_SYNC``: An index that automatically syncs with a + source Delta Table, automatically and + incrementally updating the index as the underlying data in the Delta Table changes. + - ``DIRECT_ACCESS``: An index that supports direct read and + write of vectors and metadata + through our REST and SDK APIs. With this model, the user manages index updates. delta_sync_index_spec: type: object properties: + columns_to_index: + type: array + items: + type: string columns_to_sync: type: array items: type: string + description: >- + [Optional] Select the columns to sync with the vector + index. If you leave this field blank, all columns from + the source table are synced with the index. The primary + key column and embedding source column or embedding + vector column are always synced. + effective_budget_policy_id: + type: string + description: The budget policy id applied to the AI Search index + effective_usage_policy_id: + type: string embedding_source_columns: type: array items: @@ -565,20 +936,21 @@ paths: - CONTINUOUS - TRIGGERED description: >- - Pipeline execution mode. - `TRIGGERED`: If the pipeline - uses the triggered execution mode, the system stops - processing after successfully refreshing the source - table in the pipeline once, ensuring the table is + Pipeline execution mode. - ``TRIGGERED``: If the + pipeline uses the triggered execution mode, the system + stops processing after successfully refreshing the + source table in the pipeline once, ensuring the table is updated based on the data available when the update - started. - `CONTINUOUS`: If the pipeline uses continuous - execution, the pipeline processes new data as it arrives - in the source table to keep vector index fresh. + started. - ``CONTINUOUS``: If the pipeline uses + continuous execution, the pipeline processes new data as + it arrives in the source table to keep vector index + fresh. source_table: type: string description: The name of the source table. description: >- - Specification for Delta Sync Index. Required if `index_type` - is `DELTA_SYNC`. + Specification for Delta Sync Index. Required if + ``index_type`` is ``DELTA_SYNC``. direct_access_index_spec: type: object properties: @@ -610,16 +982,33 @@ paths: description: >- The columns that contain the embedding vectors. The format should be array[double]. + requested_schema_json: + type: string + description: >- + The index schema exactly as the user supplied it on + create, preserving the original type spellings (e.g. + ``integer``) rather than Unity Catalog's canonical names + (e.g. ``int``) that ``schema_json`` returns. schema_json: type: string description: >- The schema of the index in JSON format. Supported types - are `integer`, `long`, `float`, `double`, `boolean`, - `string`, `date`, `timestamp`. Supported types for - vector column: `array`, `array`,`. + are ``integer``, ``long``, ``float``, ``double``, + ``boolean``, ``string``, ``date``, ``timestamp``. + Supported types for vector column: ``array``, + ``array``,`. description: >- Specification for Direct Vector Access Index. Required if - `index_type` is `DIRECT_ACCESS`. + ``index_type`` is ``DIRECT_ACCESS``. + index_subtype: + type: string + x-enum: + - FULL_TEXT + - HYBRID + - VECTOR + description: >- + The subtype of the index. Use ``HYBRID`` or ``FULL_TEXT``. + ``VECTOR`` is not supported. required: - name - endpoint_name @@ -831,6 +1220,140 @@ paths: message: type: string x-stackql-sdk-source: VectorSearchIndexesAPI + /api/2.0/vector-search/indexes/{name}/autoeval: + get: + operationId: vector_search_indexes_get_auto_eval_status + summary: Returns the status of the latest autoeval run for a vector index. + tags: + - vectorsearch + - vector_search_indexes + description: |- + Returns the status of the latest autoeval run for a vector index. + + :param name: str + Fully qualified index name (catalog.schema.index). + + :returns: :class:`GetAutoEvalStatusResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Fully qualified index name (catalog.schema.index). + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/GetAutoEvalStatusResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchIndexesAPI + post: + operationId: vector_search_indexes_run_auto_eval + summary: Triggers an autoeval quality evaluation for a vector index. + tags: + - vectorsearch + - vector_search_indexes + description: |- + Triggers an autoeval quality evaluation for a vector index. + + :param name: str + Fully qualified index name (catalog.schema.index). + :param num_queries: int (optional) + Number of queries to generate for evaluation (default: 50). + :param num_results: int (optional) + Number of results to fetch per query (default: 10). + :param query_types: List[str] (optional) + Query types to evaluate (default: FULL_TEXT, ANN, HYBRID). + :param queryset_query_column: str (optional) + Column in ``queryset_table`` holding the query text. Required when ``queryset_table`` is set; + ignored otherwise. + :param queryset_relevant_docs_column: str (optional) + Optional column in ``queryset_table`` holding the ground-truth relevant document IDs for each query + (STRING or ARRAY). When set, recall@k is reported against these labels; when unset, + evaluation falls back to LLM-judged metrics only. Ignored when ``queryset_table`` is unset. + :param queryset_table: str (optional) + Fully qualified Unity Catalog table (catalog.schema.table) of evaluation queries to run against the + index. When set, queries are read from this table and synthetic query generation is skipped. The + table takes precedence over any automatically detected query source. + + :returns: :class:`RunAutoEvalResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Fully qualified index name (catalog.schema.index). + requestBody: + content: + application/json: + schema: + type: object + properties: + num_queries: + type: integer + description: 'Number of queries to generate for evaluation (default: 50).' + num_results: + type: integer + description: 'Number of results to fetch per query (default: 10).' + query_types: + type: array + items: + type: string + description: 'Query types to evaluate (default: FULL_TEXT, ANN, HYBRID).' + queryset_query_column: + type: string + description: >- + Column in ``queryset_table`` holding the query text. + Required when ``queryset_table`` is set; ignored otherwise. + queryset_relevant_docs_column: + type: string + description: >- + Optional column in ``queryset_table`` holding the + ground-truth relevant document IDs for each query (STRING or + ARRAY). When set, recall@k is reported against these + labels; when unset, evaluation falls back to LLM-judged + metrics only. Ignored when ``queryset_table`` is unset. + queryset_table: + type: string + description: >- + Fully qualified Unity Catalog table (catalog.schema.table) + of evaluation queries to run against the index. When set, + queries are read from this table and synthetic query + generation is skipped. The table takes precedence over any + automatically detected query source. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/RunAutoEvalResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchIndexesAPI /api/2.0/vector-search/indexes/{index_name}/query: post: operationId: vector_search_indexes_query_index @@ -847,31 +1370,43 @@ paths: List of column names to include in the response. :param columns_to_rerank: List[str] (optional) Column names used to retrieve data to send to the reranker. + :param facets: List[str] (optional) + Facets to compute over the matched results. Each entry has one of these forms: ``""`` - top + 10 distinct values by count ``" TOP "`` - top n distinct values, where n > 0 ``" + BUCKETS [[from,to],...]"`` - inclusive numeric ranges ``TOP`` and ``BUCKETS`` are case-insensitive. + A column may appear at most once. :param filters_json: str (optional) JSON string representing query filters. Example filters: - - `{"id <": 5}`: Filter for id less than 5. - `{"id >": 5}`: Filter for id greater than 5. - `{"id - <=": 5}`: Filter for id less than equal to 5. - `{"id >=": 5}`: Filter for id greater than equal to - 5. - `{"id": 5}`: Filter for id equal to 5. + - ``{"id <": 5}``: Filter for id less than 5. + - ``{"id >": 5}``: Filter for id greater than 5. + - ``{"id <=": 5}``: Filter for id less than equal to 5. + - ``{"id >=": 5}``: Filter for id greater than equal to 5. + - ``{"id": 5}``: Filter for id equal to 5. :param num_results: int (optional) Number of results to return. Defaults to 10. + :param query_columns: List[str] (optional) + Text columns to search for ``query_text``. When empty, all text columns are searched. :param query_text: str (optional) Query text. Required for Delta Sync Index using model endpoint. :param query_type: str (optional) - The query type to use. Choices are `ANN` and `HYBRID` and `FULL_TEXT`. Defaults to `ANN`. + The query type to use. Choices are ``ANN`` and ``HYBRID`` and ``FULL_TEXT``. Defaults to ``ANN``. :param query_vector: List[float] (optional) Query vector. Required for Direct Vector Access Index and Delta Sync Index using self-managed vectors. :param reranker: :class:`RerankerConfig` (optional) If set, the top 50 results are reranked with the Databricks Reranker model before returning the - `num_results` results to the user. The setting `columns_to_rerank` selects which columns are used - for reranking. For each datapoint, the columns selected are concatenated before being sent to the - reranking model. See https://docs.databricks.com/aws/en/vector-search/query-vector-search#rerank for - more information. + ``num_results`` results to the user. The setting ``columns_to_rerank`` selects which columns are + used for reranking. For each datapoint, the columns selected are concatenated before being sent to + the reranking model. See https://docs.databricks.com/aws/en/vector-search/query-vector-search#rerank + for more information. :param score_threshold: float (optional) Threshold for the approximate nearest neighbor search. Defaults to 0.0. + :param sort_columns: List[str] (optional) + Sort results by column values instead of the default relevance ordering. Each clause has the form + ``" ASC"`` or ``" DESC"``, for example ``["rating DESC", "price ASC"]``. :returns: :class:`QueryVectorIndexResponse` parameters: @@ -897,18 +1432,36 @@ paths: items: type: string description: Column names used to retrieve data to send to the reranker. + facets: + type: array + items: + type: string + description: >- + Facets to compute over the matched results. Each entry has + one of these forms: ``""`` - top 10 distinct values + by count ``" TOP "`` - top n distinct values, + where n > 0 ``" BUCKETS [[from,to],...]"`` - + inclusive numeric ranges ``TOP`` and ``BUCKETS`` are + case-insensitive. A column may appear at most once. filters_json: type: string description: >- JSON string representing query filters. Example filters: - - `{"id <": 5}`: Filter for id less than 5. - `{"id >": 5}`: - Filter for id greater than 5. - `{"id <=": 5}`: Filter for - id less than equal to 5. - `{"id >=": 5}`: Filter for id - greater than equal to 5. - `{"id": 5}`: Filter for id equal - to 5. + ``{"id <": 5}``: Filter for id less than 5. - ``{"id >": + 5}``: Filter for id greater than 5. - ``{"id <=": 5}``: + Filter for id less than equal to 5. - ``{"id >=": 5}``: + Filter for id greater than equal to 5. - ``{"id": 5}``: + Filter for id equal to 5. num_results: type: integer description: Number of results to return. Defaults to 10. + query_columns: + type: array + items: + type: string + description: >- + Text columns to search for ``query_text``. When empty, all + text columns are searched. query_text: type: string description: >- @@ -917,8 +1470,8 @@ paths: query_type: type: string description: >- - The query type to use. Choices are `ANN` and `HYBRID` and - `FULL_TEXT`. Defaults to `ANN`. + The query type to use. Choices are ``ANN`` and ``HYBRID`` + and ``FULL_TEXT``. Defaults to ``ANN``. query_vector: type: array items: @@ -931,6 +1484,18 @@ paths: properties: model: type: string + model_type: + type: string + x-enum: + - MODEL_TYPE_BASE + - MODEL_TYPE_FINETUNED + description: >- + EXPERIMENTAL. Discriminator for how the ``model`` field + is interpreted: BASE/UNSPECIFIED expects the literal + "databricks_reranker"; FINETUNED treats ``model`` as a + Model Serving endpoint name in the caller's workspace. + See the doc comment on ``model`` for the per-case + contract. parameters: type: object properties: @@ -938,10 +1503,13 @@ paths: type: array items: type: string + description: >- + Parameters that control how the reranker processes the + query results. description: >- If set, the top 50 results are reranked with the Databricks - Reranker model before returning the `num_results` results to - the user. The setting `columns_to_rerank` selects which + Reranker model before returning the ``num_results`` results + to the user. The setting ``columns_to_rerank`` selects which columns are used for reranking. For each datapoint, the columns selected are concatenated before being sent to the reranking model. See @@ -952,6 +1520,15 @@ paths: description: >- Threshold for the approximate nearest neighbor search. Defaults to 0.0. + sort_columns: + type: array + items: + type: string + description: >- + Sort results by column values instead of the default + relevance ordering. Each clause has the form ``" + ASC"`` or ``" DESC"``, for example ``["rating DESC", + "price ASC"]``. required: - columns responses: @@ -977,16 +1554,16 @@ paths: post: operationId: vector_search_indexes_query_next_page summary: >- - Use `next_page_token` returned from previous `QueryVectorIndex` or - `QueryVectorIndexNextPage` request + Use ``next_page_token`` returned from previous ``QueryVectorIndex`` or + ``QueryVectorIndexNextPage`` tags: - vectorsearch - vector_search_indexes description: >- - Use `next_page_token` returned from previous `QueryVectorIndex` or - `QueryVectorIndexNextPage` request + Use ``next_page_token`` returned from previous ``QueryVectorIndex`` or + ``QueryVectorIndexNextPage`` - to fetch next page of results. + request to fetch next page of results. :param index_name: str @@ -994,7 +1571,7 @@ paths: :param endpoint_name: str (optional) Name of the endpoint. :param page_token: str (optional) - Page token returned from previous `QueryVectorIndex` or `QueryVectorIndexNextPage` API. + Page token returned from previous ``QueryVectorIndex`` or ``QueryVectorIndexNextPage`` API. :returns: :class:`QueryVectorIndexResponse` parameters: @@ -1016,8 +1593,8 @@ paths: page_token: type: string description: >- - Page token returned from previous `QueryVectorIndex` or - `QueryVectorIndexNextPage` API. + Page token returned from previous ``QueryVectorIndex`` or + ``QueryVectorIndexNextPage`` API. responses: '200': description: Success @@ -1037,20 +1614,118 @@ paths: message: type: string x-stackql-sdk-source: VectorSearchIndexesAPI + /api/2.0/vector-search/indexes/{name}/reranker-finetuning: + post: + operationId: vector_search_indexes_run_reranker_finetuning + summary: Triggers reranker finetuning for a vector index. + tags: + - vectorsearch + - vector_search_indexes + description: |- + Triggers reranker finetuning for a vector index. + + :param name: str + Fully qualified index name (catalog.schema.index). + :param embedding_model: str (optional) + Model-serving endpoint name. Reranker finetuning only supports managed Delta Sync indices + (Databricks-computed embeddings), so this field is informational — it is auto-derived from the + index and not used to embed training queries locally. + :param model_name: str (optional) + Fully qualified UC name for the registered finetuned model (catalog.schema.model). When unset, the + handler derives a default of ``..reranker_`` from ``name``. + :param num_queries: int (optional) + Cap on the number of queries sampled from ``query_table`` (or generated when ``query_table`` is + unset). Use -1 to process all queries (the data-gen default). Lower values cut LLM-judge cost and + run time. + :param query_column: str (optional) + Column in ``query_table`` containing the query text. Ignored when ``query_table`` is unset. Defaults + to "query_text" when omitted. + :param query_table: str (optional) + Optional fully qualified UC Delta table holding training queries (catalog.schema.table). When unset, + the data-gen job synthesises queries from the index corpus via an LLM. + + :returns: :class:`RunRerankerFinetuningResponse` + parameters: + - name: name + in: path + required: true + schema: + type: string + description: Fully qualified index name (catalog.schema.index). + requestBody: + content: + application/json: + schema: + type: object + properties: + embedding_model: + type: string + description: >- + Model-serving endpoint name. Reranker finetuning only + supports managed Delta Sync indices (Databricks-computed + embeddings), so this field is informational — it is + auto-derived from the index and not used to embed training + queries locally. + model_name: + type: string + description: >- + Fully qualified UC name for the registered finetuned model + (catalog.schema.model). When unset, the handler derives a + default of + ``..reranker_`` from + ``name``. + num_queries: + type: integer + description: >- + Cap on the number of queries sampled from ``query_table`` + (or generated when ``query_table`` is unset). Use -1 to + process all queries (the data-gen default). Lower values cut + LLM-judge cost and run time. + query_column: + type: string + description: >- + Column in ``query_table`` containing the query text. Ignored + when ``query_table`` is unset. Defaults to "query_text" when + omitted. + query_table: + type: string + description: >- + Optional fully qualified UC Delta table holding training + queries (catalog.schema.table). When unset, the data-gen job + synthesises queries from the index corpus via an LLM. + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/RunRerankerFinetuningResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchIndexesAPI /api/2.0/vector-search/indexes/{index_name}/scan: post: operationId: vector_search_indexes_scan_index summary: >- - Scan the specified vector index and return the first `num_results` + Scan the specified vector index and return the first ``num_results`` entries after the exclusive tags: - vectorsearch - vector_search_indexes description: >- - Scan the specified vector index and return the first `num_results` + Scan the specified vector index and return the first ``num_results`` entries after the exclusive - `primary_key`. + ``primary_key``. :param index_name: str @@ -1133,6 +1808,57 @@ paths: message: type: string x-stackql-sdk-source: VectorSearchIndexesAPI + /api/2.0/vector-search/indexes/{index_name}/usage-policy: + patch: + operationId: vector_search_indexes_update_index_budget_policy + summary: Update the budget policy of an index + tags: + - vectorsearch + - vector_search_indexes + description: |- + Update the budget policy of an index + + :param index_name: str + Name of the AI Search index + :param usage_policy_id: str (optional) + The usage policy id to be applied + + :returns: :class:`UpdateVectorIndexUsagePolicyResponse` + parameters: + - name: index_name + in: path + required: true + schema: + type: string + description: Name of the AI Search index + requestBody: + content: + application/json: + schema: + type: object + properties: + usage_policy_id: + type: string + description: The usage policy id to be applied + responses: + '200': + description: Success + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateVectorIndexUsagePolicyResponse' + default: + description: Error response + content: + application/json: + schema: + type: object + properties: + error: + type: string + message: + type: string + x-stackql-sdk-source: VectorSearchIndexesAPI /api/2.0/vector-search/indexes/{index_name}/upsert-data: post: operationId: vector_search_indexes_upsert_data_vector_index @@ -1190,6 +1916,65 @@ paths: x-stackql-sdk-source: VectorSearchIndexesAPI components: schemas: + AdjustedThroughputRequest: + type: object + properties: + concurrency: + type: number + description: Adjusted concurrency (total CPU) for the endpoint + maximum_concurrency_allowed: + type: number + description: Adjusted maximum concurrency allowed for the endpoint + minimal_concurrency_allowed: + type: number + description: Adjusted minimum concurrency allowed for the endpoint + description: Adjusted throughput request parameters + AutoEvalJob: + type: object + properties: + current_stage: + $ref: '#/components/schemas/AutoEvalStage' + description: Pipeline stage currently in progress. + dashboard_url: + type: string + description: Lakeview dashboard URL for the latest run's results. + metrics_table_full_name: + type: string + description: >- + Fully qualified Delta table name where per-run metrics are persisted + (``autoeval_metrics_``). + mlflow_experiment_id: + type: string + description: >- + MLflow experiment_id used by the autoeval wheel. Stable per index. + The UI uses this to construct an "Open in MLflow" deep link without + an additional MLflow tag fetch. + mlflow_run_id: + type: string + description: MLflow run_id of the latest autoeval run. Per-run, latest only. + overall_progress: + type: number + description: >- + Overall progress across all stages, in the range [0.0, 1.0]. Capped + at 0.99 while the run is RUNNING — the bar only reaches 1.0 when + status flips to AUTO_EVAL_DISPLAY_STATUS_SUCCEEDED. + progress_on_current_stage: + type: integer + description: >- + Items completed within the current stage (e.g., queries generated, + (query_type, reranker, phase) tuples evaluated, results saved). + results_table_full_name: + type: string + description: >- + Fully qualified Delta table name where per-query results are + persisted (``autoeval_results_``). + total_for_current_stage: + type: integer + description: Total items expected within the current stage. + description: >- + State of the most recent autoeval Databricks Jobs background-compute run + for an index. The UI + uses this to render the staged progress bar. ColumnInfo: type: object properties: @@ -1205,7 +1990,7 @@ components: type: string value: type: string - description: '[Optional] Value field for a vector search endpoint tag.' + description: '[Optional] Value field for an AI Search endpoint tag.' required: - key DeleteDataResult: @@ -1235,10 +2020,24 @@ components: DeltaSyncVectorIndexSpecRequest: type: object properties: + columns_to_index: + type: array + items: + type: string columns_to_sync: type: array items: type: string + description: >- + [Optional] Select the columns to sync with the vector index. If you + leave this field blank, all columns from the source table are synced + with the index. The primary key column and embedding source column + or embedding vector column are always synced. + effective_budget_policy_id: + type: string + description: The budget policy id applied to the AI Search index + effective_usage_policy_id: + type: string embedding_source_columns: type: array items: @@ -1257,11 +2056,11 @@ components: pipeline_type: $ref: '#/components/schemas/PipelineType' description: >- - Pipeline execution mode. - `TRIGGERED`: If the pipeline uses the + Pipeline execution mode. - ``TRIGGERED``: If the pipeline uses the triggered execution mode, the system stops processing after successfully refreshing the source table in the pipeline once, ensuring the table is updated based on the data available when the - update started. - `CONTINUOUS`: If the pipeline uses continuous + update started. - ``CONTINUOUS``: If the pipeline uses continuous execution, the pipeline processes new data as it arrives in the source table to keep vector index fresh. source_table: @@ -1270,10 +2069,29 @@ components: DeltaSyncVectorIndexSpecResponse: type: object properties: + columns_to_index: + type: array + items: + type: string + columns_to_sync: + type: array + items: + type: string + description: >- + [Optional] Select the columns to sync with the vector index. If you + leave this field blank, all columns from the source table are synced + with the index. The primary key column and embedding source column + or embedding vector column are always synced. + effective_budget_policy_id: + type: string + description: The budget policy id applied to the AI Search index + effective_usage_policy_id: + type: string embedding_source_columns: type: array items: $ref: '#/components/schemas/EmbeddingSourceColumn' + description: The columns that contain the embedding source. embedding_vector_columns: type: array items: @@ -1290,11 +2108,11 @@ components: pipeline_type: $ref: '#/components/schemas/PipelineType' description: >- - Pipeline execution mode. - `TRIGGERED`: If the pipeline uses the + Pipeline execution mode. - ``TRIGGERED``: If the pipeline uses the triggered execution mode, the system stops processing after successfully refreshing the source table in the pipeline once, ensuring the table is updated based on the data available when the - update started. - `CONTINUOUS`: If the pipeline uses continuous + update started. - ``CONTINUOUS``: If the pipeline uses continuous execution, the pipeline processes new data as it arrives in the source table to keep vector index fresh. source_table: @@ -1314,13 +2132,20 @@ components: description: >- The columns that contain the embedding vectors. The format should be array[double]. + requested_schema_json: + type: string + description: >- + The index schema exactly as the user supplied it on create, + preserving the original type spellings (e.g. ``integer``) rather + than Unity Catalog's canonical names (e.g. ``int``) that + ``schema_json`` returns. schema_json: type: string description: >- The schema of the index in JSON format. Supported types are - `integer`, `long`, `float`, `double`, `boolean`, `string`, `date`, - `timestamp`. Supported types for vector column: `array`, - `array`,`. + ``integer``, ``long``, ``float``, ``double``, ``boolean``, + ``string``, ``date``, ``timestamp``. Supported types for vector + column: ``array``, ``array``,`. EmbeddingSourceColumn: type: object properties: @@ -1345,8 +2170,11 @@ components: EndpointInfo: type: object properties: + budget_policy_id: + type: string creation_timestamp: type: integer + description: Timestamp of endpoint creation creator: type: string description: Creator of the endpoint @@ -1375,17 +2203,20 @@ components: description: User who last updated the endpoint name: type: string - description: Name of the vector search endpoint + description: Name of the AI Search endpoint num_indexes: type: integer description: Number of indexes on the endpoint scaling_info: $ref: '#/components/schemas/EndpointScalingInfo' description: Scaling information for the endpoint + throughput_info: + $ref: '#/components/schemas/EndpointThroughputInfo' + description: Throughput information for the endpoint EndpointScalingInfo: type: object properties: - requested_min_qps: + requested_target_qps: type: integer state: $ref: '#/components/schemas/ScalingChangeState' @@ -1400,6 +2231,92 @@ components: $ref: '#/components/schemas/EndpointStatusState' description: Current state of the endpoint description: Status information of an endpoint + EndpointThroughputInfo: + type: object + properties: + change_request_message: + type: string + description: Additional information about the throughput change request + change_request_state: + $ref: '#/components/schemas/ThroughputChangeRequestState' + description: The state of the most recent throughput change request + current_concurrency: + type: number + description: The current concurrency (total CPU) allocated to the endpoint + current_concurrency_utilization_percentage: + type: number + description: The current utilization of concurrency as a percentage (0-100) + current_num_replicas: + type: integer + description: The current number of replicas allocated to the endpoint + maximum_concurrency_allowed: + type: number + description: The maximum concurrency allowed for this endpoint + minimal_concurrency_allowed: + type: number + description: The minimum concurrency allowed for this endpoint + requested_concurrency: + type: number + description: The requested concurrency (total CPU) for the endpoint + requested_num_replicas: + type: integer + description: The requested number of replicas for the endpoint + description: Throughput information for an endpoint + FacetResultData: + type: object + properties: + facet_array: + type: array + items: + type: array + items: + type: string + description: >- + Facet rows. Each row is ``[facet_column_name, value_or_range, + count]``. + facet_row_count: + type: integer + description: Number of facet rows returned. + description: Facet aggregation rows returned by a query. + GetAutoEvalStatusResponse: + type: object + properties: + end_time_ms: + type: integer + job_id: + type: string + description: >- + Databricks Jobs job_id of the autoeval background-compute job for + this index, so the UI can surface a link to the job. Unset when no + autoeval job exists for the index yet. + latest_run: + $ref: '#/components/schemas/AutoEvalJob' + description: >- + State of the latest autoeval run, including stage progress. + Populated only while status is AUTO_EVAL_DISPLAY_STATUS_RUNNING and + the running wheel has reported at least one stage update. Absent for + terminal states. + run_as_user: + type: string + description: >- + The user the latest job run was created as. Used by the UI to + construct the per-run MLflow dashboard URL. + state_message: + type: string + description: >- + Free-form failure copy from the underlying job. Populated only when + status is AUTO_EVAL_DISPLAY_STATUS_FAILED. Capped server-side to + bound payload size when the job emits long stack traces. + status: + $ref: '#/components/schemas/AutoEvalDisplayStatus' + description: Current display status of the latest autoeval run. + GetVectorSearchEndpointPermissionLevelsResponse: + type: object + properties: + permission_levels: + type: array + items: + $ref: '#/components/schemas/VectorSearchEndpointPermissionsDescription' ListEndpointResponse: type: object properties: @@ -1490,9 +2407,15 @@ components: properties: creator: type: string + endpoint_id: + type: string + description: ID of the endpoint associated with the index. endpoint_name: type: string description: Name of the endpoint associated with the index + index_subtype: + $ref: '#/components/schemas/IndexSubtype' + description: The subtype of the index. index_type: $ref: '#/components/schemas/VectorIndexType' name: @@ -1504,19 +2427,36 @@ components: PatchEndpointBudgetPolicyResponse: type: object properties: + budget_policy_id: + type: string effective_budget_policy_id: type: string + description: The budget policy applied to the AI Search endpoint. + PatchEndpointThroughputResponse: + type: object + properties: + adjusted_request: + $ref: '#/components/schemas/AdjustedThroughputRequest' + message: + type: string + description: Message explaining the status or any adjustments made + status: + $ref: '#/components/schemas/ThroughputPatchStatus' + description: The status of the throughput change request QueryVectorIndexResponse: type: object properties: + facet_result: + $ref: '#/components/schemas/FacetResultData' manifest: $ref: '#/components/schemas/ResultManifest' + description: Metadata about the result set. next_page_token: type: string description: >- - [Optional] Token that can be used in `QueryVectorIndexNextPage` API - to get next page of results. If more than 1000 results satisfy the - query, they are returned in groups of 1000. Empty value means no + [Optional] Token that can be used in ``QueryVectorIndexNextPage`` + API to get next page of results. If more than 1000 results satisfy + the query, they are returned in groups of 1000. Empty value means no more results. The maximum number of results that can be returned is 10,000. result: @@ -1527,8 +2467,19 @@ components: properties: model: type: string + model_type: + $ref: '#/components/schemas/RerankerConfigModelType' + description: >- + EXPERIMENTAL. Discriminator for how the ``model`` field is + interpreted: BASE/UNSPECIFIED expects the literal + "databricks_reranker"; FINETUNED treats ``model`` as a Model Serving + endpoint name in the caller's workspace. See the doc comment on + ``model`` for the per-case contract. parameters: $ref: '#/components/schemas/RerankerConfigRerankerParameters' + description: >- + Parameters that control how the reranker processes the query + results. RerankerConfigRerankerParameters: type: object properties: @@ -1561,6 +2512,14 @@ components: items: $ref: '#/components/schemas/ColumnInfo' description: Information about each column in the result set. + facet_column_count: + type: integer + description: Number of columns in ``facet_result``. + facet_columns: + type: array + items: + $ref: '#/components/schemas/ColumnInfo' + description: Information about each column in ``facet_result``. description: Metadata about the result set. RetrieveUserVisibleMetricsResponse: type: object @@ -1576,6 +2535,12 @@ components: A token that can be used to get the next page of results. If not present, there are no more results to show. description: Response containing user-visible metrics + RunAutoEvalResponse: + type: object + properties: {} + RunRerankerFinetuningResponse: + type: object + properties: {} ScanVectorIndexResponse: type: object properties: @@ -1607,9 +2572,15 @@ components: $ref: '#/components/schemas/CustomTag' name: type: string - description: >- - The name of the vector search endpoint whose custom tags were - updated. + description: The name of the AI Search endpoint whose custom tags were updated. + UpdateVectorIndexUsagePolicyResponse: + type: object + properties: + effective_usage_policy_id: + type: string + usage_policy_id: + type: string + description: The updated usage policy id UpsertDataResult: type: object properties: @@ -1650,9 +2621,15 @@ components: $ref: '#/components/schemas/DeltaSyncVectorIndexSpecResponse' direct_access_index_spec: $ref: '#/components/schemas/DirectAccessVectorIndexSpec' + endpoint_id: + type: string + description: ID of the endpoint associated with the index. endpoint_name: type: string description: Name of the endpoint associated with the index + index_subtype: + $ref: '#/components/schemas/IndexSubtype' + description: The subtype of the index. index_type: $ref: '#/components/schemas/VectorIndexType' name: @@ -1677,6 +2654,123 @@ components: ready: type: boolean description: Whether the index is ready for search + VectorSearchEndpointAccessControlRequest: + type: object + properties: + group_name: + type: string + permission_level: + $ref: '#/components/schemas/VectorSearchEndpointPermissionLevel' + service_principal_name: + type: string + description: application ID of a service principal + user_name: + type: string + description: name of the user + VectorSearchEndpointAccessControlResponse: + type: object + properties: + all_permissions: + type: array + items: + $ref: '#/components/schemas/VectorSearchEndpointPermission' + display_name: + type: string + description: Display name of the user or service principal. + group_name: + type: string + description: name of the group + service_principal_name: + type: string + description: Name of the service principal. + user_name: + type: string + description: name of the user + VectorSearchEndpointPermission: + type: object + properties: + inherited: + type: boolean + inherited_from_object: + type: array + items: + type: string + permission_level: + $ref: '#/components/schemas/VectorSearchEndpointPermissionLevel' + VectorSearchEndpointPermissions: + type: object + properties: + access_control_list: + type: array + items: + $ref: '#/components/schemas/VectorSearchEndpointAccessControlResponse' + object_id: + type: string + object_type: + type: string + VectorSearchEndpointPermissionsDescription: + type: object + properties: + description: + type: string + permission_level: + $ref: '#/components/schemas/VectorSearchEndpointPermissionLevel' + AutoEvalDisplayStatus: + type: string + x-enum: + - AUTO_EVAL_DISPLAY_STATUS_FAILED + - AUTO_EVAL_DISPLAY_STATUS_PENDING + - AUTO_EVAL_DISPLAY_STATUS_RUNNING + - AUTO_EVAL_DISPLAY_STATUS_SUCCEEDED + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. + AutoEvalStage: + type: string + x-enum: + - AUTO_EVAL_STAGE_FEW_SHOT_QUERIES + - AUTO_EVAL_STAGE_GENERATE_QUERIES + - AUTO_EVAL_STAGE_GENERATE_RESULTS + - AUTO_EVAL_STAGE_METRICS_COMPUTATION + description: >- + Pipeline stages within a single autoeval run, in execution order. Used + together with AutoEvalJob + + to drive a staged progress bar in the UI. DeleteDataStatus: type: string x-enum: @@ -1734,26 +2828,48 @@ components: type: string x-enum: - STANDARD + - STORAGE_OPTIMIZED description: Type of endpoint. + IndexSubtype: + type: string + x-enum: + - FULL_TEXT + - HYBRID + - VECTOR + description: >- + The subtype of the AI Search index, determining the indexing and + retrieval strategy. + + + - ``VECTOR``: Not supported. Use ``HYBRID`` instead. + + - ``FULL_TEXT``: An index that uses full-text search without vector + embeddings. + + - ``HYBRID``: An index that uses vector embeddings for similarity search + and hybrid search. PipelineType: type: string x-enum: - CONTINUOUS - TRIGGERED description: >- - Pipeline execution mode. - `TRIGGERED`: If the pipeline uses the - triggered execution mode, the - - system stops processing after successfully refreshing the source table - in the pipeline once, - - ensuring the table is updated based on the data available when the - update started. - + Pipeline execution mode. - `CONTINUOUS`: If the pipeline uses continuous execution, the pipeline - processes new data as it - arrives in the source table to keep vector index fresh. + - ``TRIGGERED``: If the pipeline uses the triggered execution mode, the + system stops processing + after successfully refreshing the source table in the pipeline once, ensuring the table is + updated based on the data available when the update started. + - ``CONTINUOUS``: If the pipeline uses continuous execution, the + pipeline processes new data as + it arrives in the source table to keep vector index fresh. + RerankerConfigModelType: + type: string + x-enum: + - MODEL_TYPE_BASE + - MODEL_TYPE_FINETUNED + description: EXPERIMENTAL. Selects how ``model`` is interpreted. ScalingChangeState: type: string x-enum: @@ -1797,6 +2913,23 @@ components: Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. + ThroughputChangeRequestState: + type: string + x-enum: + - CHANGE_ADJUSTED + - CHANGE_FAILED + - CHANGE_IN_PROGRESS + - CHANGE_REACHED_MAXIMUM + - CHANGE_REACHED_MINIMUM + - CHANGE_SUCCESS + description: Throughput change request state + ThroughputPatchStatus: + type: string + x-enum: + - PATCH_ACCEPTED + - PATCH_FAILED + - PATCH_REJECTED + description: Response status for throughput change requests UpsertDataStatus: type: string x-enum: @@ -1846,19 +2979,22 @@ components: - DELTA_SYNC - DIRECT_ACCESS description: >- - There are 2 types of Vector Search indexes: - `DELTA_SYNC`: An index - that automatically syncs - - with a source Delta Table, automatically and incrementally updating the - index as the underlying - - data in the Delta Table changes. - `DIRECT_ACCESS`: An index that - supports direct read and write + There are 2 types of AI Search indexes: - of vectors and metadata through our REST and SDK APIs. With this model, - the user manages index - updates. + - ``DELTA_SYNC``: An index that automatically syncs with a source Delta + Table, automatically and + incrementally updating the index as the underlying data in the Delta Table changes. + - ``DIRECT_ACCESS``: An index that supports direct read and write of + vectors and metadata + through our REST and SDK APIs. With this model, the user manages index updates. + VectorSearchEndpointPermissionLevel: + type: string + x-enum: + - CAN_CREATE + - CAN_MANAGE + - CAN_USE + description: Permission level x-stackQL-resources: endpoints: id: databricks_workspace.vectorsearch.endpoints @@ -1938,7 +3074,7 @@ components: name: vector_search_endpoints title: Vector Search Endpoints methods: - vector_search_endpoints_patch_endpoint: + update: config: requestBodyTranslate: algorithm: naive @@ -1948,14 +3084,89 @@ components: response: mediaType: application/json openAPIDocKey: '200' + patch_throughput: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1vector-search~1endpoints~1{endpoint_name}~1throughput/patch + response: + mediaType: application/json + openAPIDocKey: '200' sqlVerbs: select: [] insert: [] update: - $ref: >- - #/components/x-stackQL-resources/vector_search_endpoints/methods/vector_search_endpoints_patch_endpoint + #/components/x-stackQL-resources/vector_search_endpoints/methods/update + delete: [] + replace: [] + vector_search_endpoint_permission_levels: + id: >- + databricks_workspace.vectorsearch.vector_search_endpoint_permission_levels + name: vector_search_endpoint_permission_levels + title: Vector Search Endpoint Permission Levels + methods: + get: + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1vector-search-endpoints~1{endpoint_id}~1permissionLevels/get + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/vector_search_endpoint_permission_levels/methods/get + insert: [] + update: [] delete: [] replace: [] + vector_search_endpoint_permissions: + id: databricks_workspace.vectorsearch.vector_search_endpoint_permissions + name: vector_search_endpoint_permissions + title: Vector Search Endpoint Permissions + methods: + get: + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1vector-search-endpoints~1{endpoint_id}/get + response: + mediaType: application/json + openAPIDocKey: '200' + set: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1vector-search-endpoints~1{endpoint_id}/put + response: + mediaType: application/json + openAPIDocKey: '200' + update: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1permissions~1vector-search-endpoints~1{endpoint_id}/patch + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/vector_search_endpoint_permissions/methods/get + insert: [] + update: + - $ref: >- + #/components/x-stackQL-resources/vector_search_endpoint_permissions/methods/update + delete: [] + replace: + - $ref: >- + #/components/x-stackQL-resources/vector_search_endpoint_permissions/methods/set indexes: id: databricks_workspace.vectorsearch.indexes name: indexes @@ -2053,6 +3264,54 @@ components: delete: - $ref: '#/components/x-stackQL-resources/indexes/methods/delete' replace: [] + vector_search_indexes: + id: databricks_workspace.vectorsearch.vector_search_indexes + name: vector_search_indexes + title: Vector Search Indexes + methods: + vector_search_indexes_get_auto_eval_status: + operation: + $ref: '#/paths/~1api~12.0~1vector-search~1indexes~1{name}~1autoeval/get' + response: + mediaType: application/json + openAPIDocKey: '200' + run_auto_eval: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: '#/paths/~1api~12.0~1vector-search~1indexes~1{name}~1autoeval/post' + response: + mediaType: application/json + openAPIDocKey: '200' + run_reranker_finetuning: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1vector-search~1indexes~1{name}~1reranker-finetuning/post + response: + mediaType: application/json + openAPIDocKey: '200' + update_index_budget_policy: + config: + requestBodyTranslate: + algorithm: naive + operation: + $ref: >- + #/paths/~1api~12.0~1vector-search~1indexes~1{index_name}~1usage-policy/patch + response: + mediaType: application/json + openAPIDocKey: '200' + sqlVerbs: + select: + - $ref: >- + #/components/x-stackQL-resources/vector_search_indexes/methods/vector_search_indexes_get_auto_eval_status + insert: [] + update: [] + delete: [] + replace: [] x-stackQL-config: pagination: requestToken: diff --git a/providers/src/databricks_workspace/v00.00.00000/services/workspace.yaml b/providers/src/databricks_workspace/v00.00.00000/services/workspace.yaml index 3d75e16b..e87a2eee 100644 --- a/providers/src/databricks_workspace/v00.00.00000/services/workspace.yaml +++ b/providers/src/databricks_workspace/v00.00.00000/services/workspace.yaml @@ -5,8 +5,8 @@ info: OpenAPI specification for the Databricks workspace service (workspace-level APIs), generated from the Databricks Python SDK. version: 0.1.0 - x-stackql-sdk-version: 0.86.0 - x-stackql-date-generated: '2026-03-20' + x-stackql-sdk-version: 0.123.0 + x-stackql-date-generated: '2026-07-31' x-stackql-sdk-namespace: databricks.sdk.service.workspace servers: - url: https://{deployment_name}.cloud.databricks.com @@ -32,9 +32,11 @@ paths: :param git_provider: str - Git provider. This field is case-insensitive. The available Git providers are `gitHub`, - `bitbucketCloud`, `gitLab`, `azureDevOpsServices`, `gitHubEnterprise`, `bitbucketServer`, - `gitLabEnterpriseEdition` and `awsCodeCommit`. + Git provider. This field is case-insensitive. The available Git providers are ``gitHub``, + ``bitbucketCloud``, ``gitLab``, ``azureDevOpsServices`` (Azure DevOps Services, including Microsoft + Entra ID authentication), ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data Center), + ``gitLabEnterpriseEdition`` (GitLab Self-Managed), and ``awsCodeCommit`` (deprecated by AWS, not + accepting new customers). :param git_email: str (optional) The authenticating email associated with your Git provider user account. Used for authentication with the remote repository and also sets the author & committer identity for commits. Required for @@ -51,9 +53,8 @@ paths: the name of the git credential, used for identification and ease of lookup :param personal_access_token: str (optional) The personal access token used to authenticate to the corresponding Git provider. For certain - providers, support may exist for other types of scoped access tokens. [Learn more]. - - [Learn more]: https://docs.databricks.com/repos/get-access-tokens-from-git-provider.html + providers, support may exist for other types of scoped access tokens. `Learn more + `__. :param principal_id: int (optional) The ID of the service principal whose credentials will be modified. Only service principal managers can perform this action. @@ -69,10 +70,13 @@ paths: type: string description: >- Git provider. This field is case-insensitive. The available - Git providers are `gitHub`, `bitbucketCloud`, `gitLab`, - `azureDevOpsServices`, `gitHubEnterprise`, - `bitbucketServer`, `gitLabEnterpriseEdition` and - `awsCodeCommit`. + Git providers are ``gitHub``, ``bitbucketCloud``, + ``gitLab``, ``azureDevOpsServices`` (Azure DevOps Services, + including Microsoft Entra ID authentication), + ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data + Center), ``gitLabEnterpriseEdition`` (GitLab Self-Managed), + and ``awsCodeCommit`` (deprecated by AWS, not accepting new + customers). git_email: type: string description: >- @@ -104,9 +108,9 @@ paths: description: >- The personal access token used to authenticate to the corresponding Git provider. For certain providers, support - may exist for other types of scoped access tokens. [Learn - more]. [Learn more]: - https://docs.databricks.com/repos/get-access-tokens-from-git-provider.html + may exist for other types of scoped access tokens. `Learn + more + `__. principal_id: type: integer description: >- @@ -283,9 +287,11 @@ paths: :param credential_id: int The ID for the corresponding credential to access. :param git_provider: str - Git provider. This field is case-insensitive. The available Git providers are `gitHub`, - `bitbucketCloud`, `gitLab`, `azureDevOpsServices`, `gitHubEnterprise`, `bitbucketServer`, - `gitLabEnterpriseEdition` and `awsCodeCommit`. + Git provider. This field is case-insensitive. The available Git providers are ``gitHub``, + ``bitbucketCloud``, ``gitLab``, ``azureDevOpsServices`` (Azure DevOps Services, including Microsoft + Entra ID authentication), ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data Center), + ``gitLabEnterpriseEdition`` (GitLab Self-Managed), and ``awsCodeCommit`` (deprecated by AWS, not + accepting new customers). :param git_email: str (optional) The authenticating email associated with your Git provider user account. Used for authentication with the remote repository and also sets the author & committer identity for commits. Required for @@ -302,9 +308,8 @@ paths: the name of the git credential, used for identification and ease of lookup :param personal_access_token: str (optional) The personal access token used to authenticate to the corresponding Git provider. For certain - providers, support may exist for other types of scoped access tokens. [Learn more]. - - [Learn more]: https://docs.databricks.com/repos/get-access-tokens-from-git-provider.html + providers, support may exist for other types of scoped access tokens. `Learn more + `__. :param principal_id: int (optional) The ID of the service principal whose credentials will be modified. Only service principal managers can perform this action. @@ -325,10 +330,13 @@ paths: type: string description: >- Git provider. This field is case-insensitive. The available - Git providers are `gitHub`, `bitbucketCloud`, `gitLab`, - `azureDevOpsServices`, `gitHubEnterprise`, - `bitbucketServer`, `gitLabEnterpriseEdition` and - `awsCodeCommit`. + Git providers are ``gitHub``, ``bitbucketCloud``, + ``gitLab``, ``azureDevOpsServices`` (Azure DevOps Services, + including Microsoft Entra ID authentication), + ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data + Center), ``gitLabEnterpriseEdition`` (GitLab Self-Managed), + and ``awsCodeCommit`` (deprecated by AWS, not accepting new + customers). git_email: type: string description: >- @@ -360,9 +368,9 @@ paths: description: >- The personal access token used to authenticate to the corresponding Git provider. For certain providers, support - may exist for other types of scoped access tokens. [Learn - more]. [Learn more]: - https://docs.databricks.com/repos/get-access-tokens-from-git-provider.html + may exist for other types of scoped access tokens. `Learn + more + `__. principal_id: type: integer description: >- @@ -406,12 +414,17 @@ paths: :param url: str URL of the Git repository to be linked. :param provider: str - Git provider. This field is case-insensitive. The available Git providers are `gitHub`, - `bitbucketCloud`, `gitLab`, `azureDevOpsServices`, `gitHubEnterprise`, `bitbucketServer`, - `gitLabEnterpriseEdition` and `awsCodeCommit`. + Git provider. This field is case-insensitive. The available Git providers are ``gitHub``, + ``bitbucketCloud``, ``gitLab``, ``azureDevOpsServices`` (Azure DevOps Services, including Microsoft + Entra ID authentication), ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data Center), + ``gitLabEnterpriseEdition`` (GitLab Self-Managed), and ``awsCodeCommit`` (deprecated by AWS, not + accepting new customers). + :param git_credential_id: int (optional) + Git credential ID to use when cloning the repository. The Git credential must be configured for the + current user. :param path: str (optional) Desired path for the repo in the workspace. Almost any path in the workspace can be chosen. If repo - is created in `/Repos`, path must be in the format `/Repos/{folder}/{repo-name}`. + is created in ``/Repos``, path must be in the format ``/Repos/{folder}/{repo-name}``. :param sparse_checkout: :class:`SparseCheckout` (optional) If specified, the repo will be created with sparse checkout enabled. You cannot enable/disable sparse checkout after the repo is created. @@ -430,17 +443,25 @@ paths: type: string description: >- Git provider. This field is case-insensitive. The available - Git providers are `gitHub`, `bitbucketCloud`, `gitLab`, - `azureDevOpsServices`, `gitHubEnterprise`, - `bitbucketServer`, `gitLabEnterpriseEdition` and - `awsCodeCommit`. + Git providers are ``gitHub``, ``bitbucketCloud``, + ``gitLab``, ``azureDevOpsServices`` (Azure DevOps Services, + including Microsoft Entra ID authentication), + ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data + Center), ``gitLabEnterpriseEdition`` (GitLab Self-Managed), + and ``awsCodeCommit`` (deprecated by AWS, not accepting new + customers). + git_credential_id: + type: integer + description: >- + Git credential ID to use when cloning the repository. The + Git credential must be configured for the current user. path: type: string description: >- Desired path for the repo in the workspace. Almost any path in the workspace can be chosen. If repo is created in - `/Repos`, path must be in the format - `/Repos/{folder}/{repo-name}`. + ``/Repos``, path must be in the format + ``/Repos/{folder}/{repo-name}``. sparse_checkout: type: object properties: @@ -449,9 +470,10 @@ paths: items: type: string description: >- - List of sparse checkout cone patterns, see [cone mode - handling] for details. [cone mode handling]: - https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling + List of sparse checkout cone patterns, see `cone mode + handling + `__ + for details. description: >- If specified, the repo will be created with sparse checkout enabled. You cannot enable/disable sparse checkout after the @@ -482,24 +504,33 @@ paths: operationId: repos_list summary: >- Returns repos that the calling user has Manage permissions on. Use - `next_page_token` to iterate + ``next_page_token`` to iterate tags: - workspace - repos description: >- Returns repos that the calling user has Manage permissions on. Use - `next_page_token` to iterate + ``next_page_token`` to iterate through additional pages. + Deprecated: This operation does not return a complete list of the repos + in the workspace, because + + repos with the Git CLI enabled are not included in its results. Instead, + use the Repos and Workspace + + APIs to find repos and their associated metadata in the workspace. + + :param next_page_token: str (optional) Token used to get the next page of results. If not specified, returns the first page of results as well as a next page token if there are more results. :param path_prefix: str (optional) Filters repos that have paths starting with the given path prefix. If not provided or when provided - an effectively empty prefix (`/` or `/Workspace`) Git folders (repos) from `/Workspace/Repos` will - be served. + an effectively empty prefix (``/`` or ``/Workspace``) Git folders (repos) from ``/Workspace/Repos`` + will be served. :returns: Iterator over :class:`RepoInfo` parameters: @@ -519,9 +550,9 @@ paths: type: string description: >- Filters repos that have paths starting with the given path prefix. - If not provided or when provided an effectively empty prefix (`/` or - `/Workspace`) Git folders (repos) from `/Workspace/Repos` will be - served. + If not provided or when provided an effectively empty prefix (``/`` + or ``/Workspace``) Git folders (repos) from ``/Workspace/Repos`` + will be served. responses: '200': description: Success @@ -633,6 +664,20 @@ paths: ID of the Git folder (repo) object in the workspace. :param branch: str (optional) Branch that the local version of the repo is checked out to. + :param dangerously_force_discard_all: bool (optional) + WARNING: DESTRUCTIVE AND IRREVERSIBLE. If true, permanently deletes ALL uncommitted changes in the + Git folder — staged, unstaged, and untracked files — before updating. Lost data CANNOT be + recovered. + + NEVER use this on Git folders where users author or edit files. This flag is intended ONLY for + automated jobs that treat the Git folder as a read-only mirror of a remote branch and need to + force-sync it. If any user has uncommitted work in the Git folder, that work will be permanently + destroyed without warning. + + Local commits that have been made but not yet pushed to the remote are preserved. + :param git_credential_id: int (optional) + Git credential ID to use for this update operation. The Git credential must be configured for the + current user. :param sparse_checkout: :class:`SparseCheckoutUpdate` (optional) If specified, update the sparse checkout settings. The update will fail if sparse checkout is not enabled for the repo. @@ -656,6 +701,25 @@ paths: branch: type: string description: Branch that the local version of the repo is checked out to. + dangerously_force_discard_all: + type: boolean + description: >- + WARNING: DESTRUCTIVE AND IRREVERSIBLE. If true, permanently + deletes ALL uncommitted changes in the Git folder — staged, + unstaged, and untracked files — before updating. Lost data + CANNOT be recovered. NEVER use this on Git folders where + users author or edit files. This flag is intended ONLY for + automated jobs that treat the Git folder as a read-only + mirror of a remote branch and need to force-sync it. If any + user has uncommitted work in the Git folder, that work will + be permanently destroyed without warning. Local commits that + have been made but not yet pushed to the remote are + preserved. + git_credential_id: + type: integer + description: >- + Git credential ID to use for this update operation. The Git + credential must be configured for the current user. sparse_checkout: type: object properties: @@ -664,9 +728,10 @@ paths: items: type: string description: >- - List of sparse checkout cone patterns, see [cone mode - handling] for details. [cone mode handling]: - https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling + List of sparse checkout cone patterns, see `cone mode + handling + `__ + for details. description: >- If specified, update the sparse checkout settings. The update will fail if sparse checkout is not enabled for the @@ -952,8 +1017,11 @@ paths: { "scope": "my-simple-databricks-scope", "initial_manage_principal": "users" "scope_backend_type": - "databricks|azure_keyvault", # below is only required if scope type is - azure_keyvault + "databricks|azure_keyvault", + + + below is only required if scope type is azure_keyvault + "backend_azure_keyvault": { "resource_id": @@ -1637,13 +1705,13 @@ paths: permissions are ordered as follows: - * ``MANAGE`` - Allowed to change ACLs, and read and write to this secret - scope. * ``WRITE`` - Allowed + - ``MANAGE`` - Allowed to change ACLs, and read and write to this secret + scope. - to read and write to this secret scope. * ``READ`` - Allowed to read - this secret scope and list what + - ``WRITE`` - Allowed to read and write to this secret scope. - secrets are available. + - ``READ`` - Allowed to read this secret scope and list what secrets are + available. Note that in general, secret values can only be read from within a @@ -1849,21 +1917,20 @@ paths: operationId: workspace_delete summary: >- Deletes an object or a directory (and optionally recursively deletes all - objects in the directory). * + objects in the directory). tags: - workspace description: >- Deletes an object or a directory (and optionally recursively deletes all - objects in the directory). * - - If `path` does not exist, this call returns an error - `RESOURCE_DOES_NOT_EXIST`. * If `path` is a + objects in the directory). - non-empty directory and `recursive` is set to `false`, this call returns - an error - `DIRECTORY_NOT_EMPTY`. + - If ``path`` does not exist, this call returns an error + ``RESOURCE_DOES_NOT_EXIST``. + - If ``path`` is a non-empty directory and ``recursive`` is set to + ``false``, this call returns an + error ``DIRECTORY_NOT_EMPTY``. Object deletion cannot be undone and deleting a directory recursively is not atomic. @@ -1872,7 +1939,7 @@ paths: :param path: str The absolute path of the notebook or directory. :param recursive: bool (optional) - The flag that specifies whether to delete the object recursively. It is `false` by default. Please + The flag that specifies whether to delete the object recursively. It is ``false`` by default. Please note this deleting directory is not atomic. If it fails in the middle, some of objects under this directory may be deleted and cannot be undone. requestBody: @@ -1888,7 +1955,7 @@ paths: type: boolean description: >- The flag that specifies whether to delete the object - recursively. It is `false` by default. Please note this + recursively. It is ``false`` by default. Please note this deleting directory is not atomic. If it fails in the middle, some of objects under this directory may be deleted and cannot be undone. @@ -1919,30 +1986,38 @@ paths: Exports an object or the contents of an entire directory. - If `path` does not exist, this call returns an error - `RESOURCE_DOES_NOT_EXIST`. + If ``path`` does not exist, this call returns an error + ``RESOURCE_DOES_NOT_EXIST``. If the exported data would exceed size limit, this call returns - `MAX_NOTEBOOK_SIZE_EXCEEDED`. + ``MAX_NOTEBOOK_SIZE_EXCEEDED``. Currently, this API does not support exporting a library. :param path: str - The absolute path of the object or directory. Exporting a directory is only supported for the `DBC`, - `SOURCE`, and `AUTO` format. + The absolute path of the object or directory. Exporting a directory is only supported for the + ``DBC``, ``SOURCE``, and ``AUTO`` format. :param format: :class:`ExportFormat` (optional) - This specifies the format of the exported file. By default, this is `SOURCE`. + This specifies the format of the exported file. By default, this is ``SOURCE``. The value is case sensitive. - - `SOURCE`: The notebook is exported as source code. Directory exports will not include non-notebook - entries. - `HTML`: The notebook is exported as an HTML file. - `JUPYTER`: The notebook is exported - as a Jupyter/IPython Notebook file. - `DBC`: The notebook is exported in Databricks archive format. - Directory exports will not include non-notebook entries. - `R_MARKDOWN`: The notebook is exported to - R Markdown format. - `AUTO`: The object or directory is exported depending on the objects type. - Directory exports will include notebooks and workspace files. + - ``SOURCE``: The notebook is exported as source code. Directory exports will not include + non-notebook entries. + - ``HTML``: The notebook is exported as an HTML file. + - ``JUPYTER``: The notebook is exported as a Jupyter/IPython Notebook file. + - ``DBC``: The notebook is exported in Databricks archive format. Directory exports will not include + non-notebook entries. + - ``R_MARKDOWN``: The notebook is exported to R Markdown format. + - ``AUTO``: The object or directory is exported depending on the objects type. Directory exports + will include notebooks and workspace files. + :param outputs: :class:`ExportOutputs` (optional) + This specifies which cell outputs should be included in the export (if the export format allows it). + If not specified, the behavior is determined by the format. For JUPYTER format, the default is to + include all outputs. This is a public endpoint, but only ALL or NONE is documented publically, + DATABRICKS is internal only :returns: :class:`ExportResponse` parameters: @@ -1953,7 +2028,7 @@ paths: type: string description: >- The absolute path of the object or directory. Exporting a directory - is only supported for the `DBC`, `SOURCE`, and `AUTO` format. + is only supported for the ``DBC``, ``SOURCE``, and ``AUTO`` format. - name: format in: query required: false @@ -1961,16 +2036,27 @@ paths: $ref: '#/components/schemas/ExportFormat' description: >- This specifies the format of the exported file. By default, this is - `SOURCE`. The value is case sensitive. - `SOURCE`: The notebook is - exported as source code. Directory exports will not include - non-notebook entries. - `HTML`: The notebook is exported as an HTML - file. - `JUPYTER`: The notebook is exported as a Jupyter/IPython - Notebook file. - `DBC`: The notebook is exported in Databricks - archive format. Directory exports will not include non-notebook - entries. - `R_MARKDOWN`: The notebook is exported to R Markdown - format. - `AUTO`: The object or directory is exported depending on - the objects type. Directory exports will include notebooks and - workspace files. + ``SOURCE``. The value is case sensitive. - ``SOURCE``: The notebook + is exported as source code. Directory exports will not include + non-notebook entries. - ``HTML``: The notebook is exported as an + HTML file. - ``JUPYTER``: The notebook is exported as a + Jupyter/IPython Notebook file. - ``DBC``: The notebook is exported + in Databricks archive format. Directory exports will not include + non-notebook entries. - ``R_MARKDOWN``: The notebook is exported to + R Markdown format. - ``AUTO``: The object or directory is exported + depending on the objects type. Directory exports will include + notebooks and workspace files. + - name: outputs + in: query + required: false + schema: + $ref: '#/components/schemas/ExportOutputs' + description: >- + This specifies which cell outputs should be included in the export + (if the export format allows it). If not specified, the behavior is + determined by the format. For JUPYTER format, the default is to + include all outputs. This is a public endpoint, but only ALL or NONE + is documented publically, DATABRICKS is internal only responses: '200': description: Success @@ -2286,15 +2372,15 @@ paths: get: operationId: workspace_get_status summary: >- - Gets the status of an object or a directory. If `path` does not exist, + Gets the status of an object or a directory. If ``path`` does not exist, this call returns an error tags: - workspace description: >- - Gets the status of an object or a directory. If `path` does not exist, + Gets the status of an object or a directory. If ``path`` does not exist, this call returns an error - `RESOURCE_DOES_NOT_EXIST`. + ``RESOURCE_DOES_NOT_EXIST``. :param path: str @@ -2339,21 +2425,22 @@ paths: Imports a workspace object (for example, a notebook or file) or the contents of an entire directory. - If `path` already exists and `overwrite` is set to `false`, this call - returns an error + If ``path`` already exists and ``overwrite`` is set to ``false``, this + call returns an error - `RESOURCE_ALREADY_EXISTS`. To import a directory, you can use either the - `DBC` format or the `SOURCE` + ``RESOURCE_ALREADY_EXISTS``. To import a directory, you can use either + the ``DBC`` format or the - format with the `language` field unset. To import a single file as - `SOURCE`, you must set the + ``SOURCE`` format with the ``language`` field unset. To import a single + file as ``SOURCE``, you must - `language` field. Zip files within directories are not supported. + set the ``language`` field. Zip files within directories are not + supported. :param path: str - The absolute path of the object or directory. Importing a directory is only supported for the `DBC` - and `SOURCE` formats. + The absolute path of the object or directory. Importing a directory is only supported for the + ``DBC`` and ``SOURCE`` formats. :param content: str (optional) The base64-encoded content. This has a limit of 10 MB. @@ -2364,17 +2451,19 @@ paths: The value is case sensitive. - - `AUTO`: The item is imported depending on an analysis of the item's extension and the header - content provided in the request. If the item is imported as a notebook, then the item's extension is - automatically removed. - `SOURCE`: The notebook or directory is imported as source code. - `HTML`: - The notebook is imported as an HTML file. - `JUPYTER`: The notebook is imported as a Jupyter/IPython - Notebook file. - `DBC`: The notebook is imported in Databricks archive format. Required for - directories. - `R_MARKDOWN`: The notebook is imported from R Markdown format. + - ``AUTO``: The item is imported depending on an analysis of the item's extension and the header + content provided in the request. If the item is imported as a notebook, then the item's extension + is automatically removed. + - ``SOURCE``: The notebook or directory is imported as source code. + - ``HTML``: The notebook is imported as an HTML file. + - ``JUPYTER``: The notebook is imported as a Jupyter/IPython Notebook file. + - ``DBC``: The notebook is imported in Databricks archive format. Required for directories. + - ``R_MARKDOWN``: The notebook is imported from R Markdown format. :param language: :class:`Language` (optional) - The language of the object. This value is set only if the object type is `NOTEBOOK`. + The language of the object. This value is set only if the object type is ``NOTEBOOK``. :param overwrite: bool (optional) - The flag that specifies whether to overwrite existing object. It is `false` by default. For `DBC` - format, `overwrite` is not supported since it may contain a directory. + The flag that specifies whether to overwrite existing object. It is ``false`` by default. For + ``DBC`` format, ``overwrite`` is not supported since it may contain a directory. requestBody: content: application/json: @@ -2385,7 +2474,7 @@ paths: type: string description: >- The absolute path of the object or directory. Importing a - directory is only supported for the `DBC` and `SOURCE` + directory is only supported for the ``DBC`` and ``SOURCE`` formats. content: type: string @@ -2406,17 +2495,17 @@ paths: - SOURCE description: >- This specifies the format of the file to be imported. The - value is case sensitive. - `AUTO`: The item is imported + value is case sensitive. - ``AUTO``: The item is imported depending on an analysis of the item's extension and the header content provided in the request. If the item is imported as a notebook, then the item's extension is - automatically removed. - `SOURCE`: The notebook or directory - is imported as source code. - `HTML`: The notebook is - imported as an HTML file. - `JUPYTER`: The notebook is - imported as a Jupyter/IPython Notebook file. - `DBC`: The - notebook is imported in Databricks archive format. Required - for directories. - `R_MARKDOWN`: The notebook is imported - from R Markdown format. + automatically removed. - ``SOURCE``: The notebook or + directory is imported as source code. - ``HTML``: The + notebook is imported as an HTML file. - ``JUPYTER``: The + notebook is imported as a Jupyter/IPython Notebook file. - + ``DBC``: The notebook is imported in Databricks archive + format. Required for directories. - ``R_MARKDOWN``: The + notebook is imported from R Markdown format. language: type: string x-enum: @@ -2426,13 +2515,13 @@ paths: - SQL description: >- The language of the object. This value is set only if the - object type is `NOTEBOOK`. + object type is ``NOTEBOOK``. overwrite: type: boolean description: >- The flag that specifies whether to overwrite existing - object. It is `false` by default. For `DBC` format, - `overwrite` is not supported since it may contain a + object. It is ``false`` by default. For ``DBC`` format, + ``overwrite`` is not supported since it may contain a directory. required: - path @@ -2463,7 +2552,7 @@ paths: Lists the contents of a directory, or the object if it is not a directory. If the input path does not - exist, this call returns an error `RESOURCE_DOES_NOT_EXIST`. + exist, this call returns an error ``RESOURCE_DOES_NOT_EXIST``. :param path: str @@ -2519,7 +2608,7 @@ paths: an object (not a directory) at any prefix of the input path, this call returns an error - `RESOURCE_ALREADY_EXISTS`. + ``RESOURCE_ALREADY_EXISTS``. Note that if this operation fails it may have succeeded in creating some @@ -2595,7 +2684,7 @@ components: - dns_name description: >- The metadata of the Azure KeyVault for a secret scope of type - `AZURE_KEYVAULT` + ``AZURE_KEYVAULT`` CreateCredentialsResponse: type: object properties: @@ -2649,7 +2738,11 @@ components: description: Path of the Git folder (repo) in the workspace. provider: type: string - description: Git provider of the linked Git repository. + description: >- + Git provider of the linked Git repository, e.g. ``gitHub``, + ``azureDevOpsServices``, ``bitbucketServer`` (Bitbucket Data + Center), ``gitLabEnterpriseEdition`` (GitLab Self-Managed), or + ``awsCodeCommit`` (deprecated). sparse_checkout: $ref: '#/components/schemas/SparseCheckout' description: Sparse checkout settings for the Git folder (repo). @@ -2671,7 +2764,13 @@ components: https://docs.databricks.com/aws/en/repos/get-access-tokens-from-git-provider git_provider: type: string - description: The Git provider associated with the credential. + description: >- + The Git provider associated with the credential. One of ``gitHub``, + ``bitbucketCloud``, ``gitLab``, ``azureDevOpsServices`` (Azure + DevOps Services, including Microsoft Entra ID authentication), + ``gitHubEnterprise``, ``bitbucketServer`` (Bitbucket Data Center), + ``gitLabEnterpriseEdition`` (GitLab Self-Managed), or + ``awsCodeCommit`` (deprecated). git_username: type: string description: >- @@ -2702,6 +2801,22 @@ components: DeleteSecretResponse: type: object properties: {} + DirectoryInfo: + type: object + properties: + is_git_folder: + type: boolean + description: >- + Whether the directory is a Git folder, whose contents are + version-controlled by a remote Git repository. How a Git folder is + represented depends on whether it has Git CLI access: - A Git folder + with Git CLI access has an object type of ``DIRECTORY``, with this + field set to ``true``. - A standard Git folder, which does not have + Git CLI access, has an object type of ``REPO`` and does not include + this field. - A directory that is not Git-backed has this field set + to ``false``. Use this field together with ``object_type`` to + identify every Git folder in a workspace. + description: Additional metadata about a directory. ExportResponse: type: object properties: @@ -2714,7 +2829,7 @@ components: type: string description: The file type of the exported file. description: >- - The request field `direct_download` determines whether a JSON response + The request field ``direct_download`` determines whether a JSON response or binary contents are returned by this endpoint. GetCredentialsResponse: @@ -2763,6 +2878,12 @@ components: properties: branch: type: string + git_cli_enabled: + type: boolean + description: >- + Whether the Git CLI is enabled for this Git folder (repo). When + true, Git commands can be run directly against this Git folder using + the Git CLI. head_commit_id: type: string description: >- @@ -2776,7 +2897,11 @@ components: description: Path of the Git folder (repo) in the workspace. provider: type: string - description: Git provider of the linked Git repository. + description: >- + Git provider of the linked Git repository, e.g. ``gitHub``, + ``azureDevOpsServices``, ``bitbucketServer`` (Bitbucket Data + Center), ``gitLabEnterpriseEdition`` (GitLab Self-Managed), or + ``awsCodeCommit`` (deprecated). sparse_checkout: $ref: '#/components/schemas/SparseCheckout' description: Sparse checkout settings for the Git folder (repo). @@ -2855,11 +2980,17 @@ components: created_at: type: integer description: Only applicable to files. The creation UTC timestamp. + directory_info: + $ref: '#/components/schemas/DirectoryInfo' + description: >- + Additional metadata about the directory. Only set for objects of + type ``DIRECTORY``. language: $ref: '#/components/schemas/Language' description: >- The language of the object. This value is set only if the object - type is ``NOTEBOOK``. + type is ``NOTEBOOK``. For Jupyter (.ipynb) notebooks, this is always + ``PYTHON``. modified_at: type: integer description: Only applicable to files, the last modified UTC timestamp. @@ -2869,10 +3000,10 @@ components: object_type: $ref: '#/components/schemas/ObjectType' description: >- - The type of the object in workspace. - `NOTEBOOK`: document that + The type of the object in workspace. - ``NOTEBOOK``: document that contains runnable code, visualizations, and explanatory text. - - `DIRECTORY`: directory - `LIBRARY`: library - `FILE`: file - `REPO`: - repository - `DASHBOARD`: Lakeview dashboard + ``DIRECTORY``: directory - ``LIBRARY``: library - ``FILE``: file - + ``REPO``: repository - ``DASHBOARD``: Lakeview dashboard path: type: string description: The absolute path of the object. @@ -2936,7 +3067,11 @@ components: description: Root path of the git folder (repo) in the Workspace. provider: type: string - description: Git provider of the remote git repository, e.g. `gitHub`. + description: >- + Git provider of the remote git repository, e.g. ``gitHub``, + ``azureDevOpsServices``, ``bitbucketServer`` (Bitbucket Data + Center), ``gitLabEnterpriseEdition`` (GitLab Self-Managed), or + ``awsCodeCommit`` (deprecated). sparse_checkout: $ref: '#/components/schemas/SparseCheckout' description: Sparse checkout config for the git folder (repo). @@ -3011,9 +3146,9 @@ components: items: type: string description: >- - List of sparse checkout cone patterns, see [cone mode handling] for - details. [cone mode handling]: - https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling + List of sparse checkout cone patterns, see `cone mode handling + `__ + for details. description: Sparse checkout configuration, it contains options like cone patterns. SparseCheckoutUpdate: type: object @@ -3023,9 +3158,9 @@ components: items: type: string description: >- - List of sparse checkout cone patterns, see [cone mode handling] for - details. [cone mode handling]: - https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling + List of sparse checkout cone patterns, see `cone mode handling + `__ + for details. description: Sparse checkout configuration, it contains options like cone patterns. UpdateCredentialsResponse: type: object @@ -3112,6 +3247,48 @@ components: - R_MARKDOWN - SOURCE description: The format for workspace import and export. + ExportOutputs: + type: string + x-enum: + - ALL + - NONE + description: |- + Create a collection of name/value pairs. + + Example enumeration: + + >>> class Color(Enum): + ... RED = 1 + ... BLUE = 2 + ... GREEN = 3 + + Access them by: + + - attribute access: + + >>> Color.RED + + + - value lookup: + + >>> Color(1) + + + - name lookup: + + >>> Color['RED'] + + + Enumerations can be iterated over, and know how many members they have: + + >>> len(Color) + 3 + + >>> list(Color) + [, , ] + + Methods can be added to enumerations, and members can have their own + attributes -- see the documentation for details. ImportFormat: type: string x-enum: