Configure a Databricks Unity Catalog Integration

graph LR
A(Authenticate) --> B(Configure integration) --> C(Map users and create policies) --> D(Register metadata);
style A fill:#fff
style B fill:#2F6BA7,color:#fff
style C fill:#fff
style D fill:#fff
click A "../../getting-started/#authenticate-with-the-api" _blank
click C "../../getting-started/#protect-your-data" _blank
click D "../../getting-started/#register-metadata" _blank

Immuta’s integration with Unity Catalog allows you to manage multiple Databricks workspaces through Unity Catalog while protecting your data with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and enforce Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouses.

Use the /integrations endpoint to

• configure a Databricks Unity Catalog integration
• get a Databricks Unity Catalog integration
• get all integrations
• update a Databricks Unity Catalog integration
• delete a Databricks Unity Catalog integration

Requirements

• APPLICATION_ADMIN Immuta permission
• A service principal with the following privileges is required for Immuta to use to manage permissions on objects in the Unity Catalog metastore:
  • CREATE CATALOG on the Unity Catalog metastore to create an Immuta-owned catalog and tables
  • OWNERSHIP on the Immuta catalog you configure
  • USE CATALOG and USE SCHEMA on all catalogs and schemas with tables managed by Immuta
  • SELECT and MODIFY on all tables in the metastore managed my Immuta
• Authentication:
  • Access token authentication: If using this method, generate a personal access token for the service principal that Immuta will use to manage policies in Unity Catalog.
  • OAuth machine-to-machine (M2M) authentication: If using this method, follow Databricks documentation to create a client secret for the Immuta service principal. This service principal must have the metastore privileges listed above for the metastore associated with the Databricks workspace.

Prerequisite

Enable Databricks Unity Catalog on the Immuta app settings page:

1. Click the App Settings icon in the left sidebar.
2. Scroll to the Global Integrations Settings section and check the Enable Databricks Unity Catalog support in Immuta checkbox.

Configure the integration

You have two options for configuring your Databricks Unity Catalog integration:

• Automatic setup: When performing an automatic setup, the Databricks personal access token you configure below must be attached to an account with these Databricks permissions for the metastore associated with the specified Databricks workspace. Immuta creates the catalogs, schemas, tables, and functions using the integration's configured personal access token.

• Manual setup: Run the Immuta script in Databricks yourself to create the catalog. You can also modify the script to customize your storage location for tables, schemas, or catalogs. The user running the script needs to have the CREATE CATALOG permission on the workspace metastore. The Databricks personal access token you configure must be attached to an account with the Databricks permissions listed in the requirements section.

Automatic setup

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": true,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": true,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "oAuthM2M",
      "oAuthClientConfig": {
        "clientId": "123456abc",
        "clientSecret": "secret",
        "authorityUrl": "example.authority.com"
      },
      "catalog": "immuta"
    }
    }'

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

{
  "id": "123456789",
  "status": "creating",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Databricks Unity Catalog integration already exists on www.example-workspace.cloud.databricks.com (id = 123456789)"
}

Manual setup

To manually configure the integration, complete the following steps:

1. Generate the Immuta script and run it in your Databricks environment.
2. Configure the integration in Immuta.

Generate the script

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations/scripts/create' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations/scripts/create' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "oAuthM2M",
      "oAuthClientConfig": {
        "clientId": "123456abc",
        "clientSecret": "secret",
        "authorityUrl": "example.authority.com"
      },
      "catalog": "immuta"
    }
    }'

Response

The response returns the script for you to run in your environment.

Configure the integration in Immuta

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML. The payload you provide must match the payload sent when generating the script.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "oAuthM2M",
      "oAuthClientConfig": {
        "clientId": "123456abc",
        "clientSecret": "secret",
        "authorityUrl": "example.authority.com"
      },
      "catalog": "immuta"
    }
    }'

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

{
  "id": "123456789",
  "status": "creating",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Databricks Unity Catalog integration already exists on www.example-workspace.cloud.databricks.com (id = 123456789)"
}

Get an integration

curl -X 'GET' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

Response

The response returns a Databricks Unity Catalog integration configuration. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "enabled",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  },
  "type": "Databricks",
  "autoBootstrap": false,
  "config": {
    "port": 443,
    "workspaceUrl": "www.example-workspace.cloud.databricks.com",
    "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
    "authenticationType": "token",
    "token": "REDACTED",
    "audit": {
      "enabled": false
    },
    "catalog": "immuta"
  }
}

Get all integrations

curl -X 'GET' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

Response

The response returns the configuration for all integrations. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

[
  {
    "id": "1",
    "status": "enabled",
    "validationResults": {
      "status": "passed",
      "validationTests": [
      {
        "name": "Initial Validation: Basic Connection Test",
        "status": "passed"
      },
      {
        "name": "Initial Validation: Default Warehouse Access Test",
        "status": "passed",
        "result": []
      },
      {
        "name": "Initial Validation: Validate access to Privileged Role",
        "status": "passed",
        "result": []
      },
      {
        "name": "Validate Automatic: Database Does Not Exist",
        "status": "passed"
      },
      {
        "name": "Validate Automatic: Impersonation Role Does Not Exist",
        "status": "skipped"
      },
      {
        "name": "Validate Automatic Bootstrap User Grants",
        "status": "passed"
      }
    ] },
    "type": "Snowflake",
    "autoBootstrap": true,
    "config": {
      "host": "organization.us-east-1.snowflakecomputing.com",
      "warehouse": "SAMPLE_WAREHOUSE",
      "database": "SNOWFLAKE_SAMPLE_DATA",
      "port": 443,
      "audit": {
          "enabled": false
        },
      "workspaces": {
        "enabled": false
      },
      "impersonation": {
        "enabled": false
      },
      "lineage": {
        "enabled": false
      },
      "authenticationType": "userPassword",
      "username": "<REDACTED>",
      "password": "<REDACTED>",
      "role": "ACCOUNTADMIN"
    },
    {
      "id": "2",
      "status": "enabled",
      "type": "Databricks",
      "validationResults": {
        "status": "passed",
        "validationTests": [
        {
          "name": "Metastore validation",
          "status": "passed"
        },
        {
          "name": "Basic Connection Test",
          "result": [
          {
            "1": 1
          }
          ],
          "status": "passed"
        }
        ]
      },
      "autoBootstrap": true,
      "config": {
        "workspaceUrl": "www.example-workspace.cloud.databricks.com",
        "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
        "authenticationType": "token",
        "token": "REDACTED",
        "audit": {
          "enabled": false
        },
        "catalog": "immuta"
      }
    }
  }
]

Update an integration configuration

You have two options for updating your integration. Follow the steps that match your initial configuration of autoBootstrap:

• automatic update (autoBootstrap is true)
• manual update (autoBootstrap is false)

Automatic update

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'PUT' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": true,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "groupPattern": {
        "deny": "admins"
      },
      "catalog": "immuta"
    }
    }'

This example updates the name of the group that will be excluded from having Immuta policies applied. Updating this exemption group requires you to re-push all of your policies to your Databricks cluster.

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

{
  "id": "123456789",
  "status": "editing",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Unable to edit integration with ID 123456789 in current state editing."
}

Manual update

To manually update the integration, complete the following steps:

1. Generate the updated Immuta script and run it in your Databricks environment.
2. Update the integration in Immuta.

Generate the updated script

Copy the request example, and replace the values with your own as directed to generate the script. The example provided uses JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations/{id}/scripts/edit' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

Response

The response returns the script for you to run in your Databricks environment.

Update the integration in Immuta

Copy the request example, and replace the values with your own as directed to update the integration settings. The example provided uses JSON format, but the request also accepts YAML. The payload you provide must match the payload sent when generating the script.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'PUT' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

{
  "id": "123456789",
  "status": "editing",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Unable to edit integration with ID 123456789 in current state editing."
}

Delete an integration

curl -X 'DELETE' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

Response

The response returns the status of the Databricks Unity Catalog integration configuration that has been deleted. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "deleting",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}