Instruction file imported from databricks-solutions/partner-ai-dev-kit (.cursor/rules/databricks-isv-integration.mdc). Copyright stays with the author.

Databricks PWAF Integration – Cursor Rules

When writing or testing PWAF-compliant Databricks partner (ISV) integrations, follow these patterns and avoid these pitfalls. Reference: Partner Well-Architected Framework (PWAF).

Auth isolation (critical)

Read skills/connector-testing/env-isolation.md before testing. It covers the three most common test-environment pitfalls: DATABRICKS_AUTH_TYPE pollution (breaks Python SDK, Go SDK, Java SDK), ~/.databrickscfg DEFAULT profile leakage (triggers "more than one authorization method configured"), and the env -i isolation pattern with ready-to-use examples for PAT, M2M, and U2M tests.

• Never mix auth methods in one process. If both PAT and OAuth M2M env vars are set (e.g. DATABRICKS_TOKEN and DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET), the Databricks SDK reports: "more than one authorization method configured: oauth and pat".
• Run each auth path with a clean env. For scripts or test runners, use env -i plus only the variables required for that auth type (e.g. for PAT: only DATABRICKS_HOST and DATABRICKS_TOKEN; for M2M: only DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET). Do not source a full config that sets both PAT and M2M when running a single-auth script.
• DATABRICKS_AUTH_TYPE is reserved for SDK internal use. Never set it in the process env or test runner — the Python SDK, Go SDK, and Java SDK all read it as an internal auth type selector and will fail if it contains application-level values like "oauth_m2m" or "u2m_custom_oauth_app". Use APP_AUTH_TYPE as your application-level selector instead.
• Default credential resolution: When using OAuth M2M programmatically, pass auth_type="oauth-m2m" into Config(...) so the SDK does not try default credential resolution (which can trigger "cannot configure default credentials" when env is ambiguous or incomplete).

U2M (user-to-machine) flows

• M2M client_id ≠ U2M OAuth app. The client_id for OAuth M2M (service principal) is not valid for the browser (authorization code) flow. If you use that client_id with the authorize URL or SDK external-browser, Databricks returns: "OAuth application with client_id: '...' not available in Databricks account". For U2M you need either:
  • external-browser (rest_api_u2m_external_browser_example.py): Use the SDK with auth_type="external-browser" and do not set or pass DATABRICKS_CLIENT_ID (the SDK uses Databricks' built-in OAuth app). In code, temporarily unset DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET when calling the SDK for external-browser so the built-in app is used. SDK caches tokens in ~/.databricks/token-cache.json.
  • custom-oauth-app (rest_api_u2m_custom_oauth_app_example.py): A separate custom OAuth app (created in the account, Settings -> Developer / App connections) with redirect URI e.g. http://localhost:8080/callback. Uses DATABRICKS_U2M_CLIENT_ID and DATABRICKS_U2M_CLIENT_SECRET (not the M2M service principal vars).
  • token-env (rest_api_u2m_token_env_example.py): A pre-obtained access token (e.g. from a hosted callback or refresh) passed via DATABRICKS_ACCESS_TOKEN or DATABRICKS_TOKEN. No browser needed.
• Token usage: All API calls use the access_token. The refresh_token is only for obtaining a new access_token when it expires or in headless flows; do not send refresh_token on every request.

Python SDK

• Config import: Use from databricks.sdk.config import Config (not from databricks.sdk import Config).
• OAuth M2M: When building a client with service principal credentials, pass auth_type="oauth-m2m" to Config(...) to avoid default credential resolution errors.
• Telemetry: Use useragent.with_partner("<isv>") and useragent.with_product("<product>", "<version>") from from databricks.sdk import useragent. These are global registrations applied to all subsequent SDK requests. Call once before creating WorkspaceClient.

Java SDK (databricks-sdk-java)

• Package: com.databricks:databricks-sdk-java:0.54.0 (Maven). Use for workspace APIs, UC metadata, Jobs — no SQL warehouse needed.
• Auth: DatabricksConfig.setHost() + .setToken() (PAT) or .setClientId() + .setClientSecret() (M2M). SDK auto-detects auth method; auto-handles M2M token fetch + refresh.
• Telemetry: Use UserAgent.withProduct("<product>", "<version>") and UserAgent.withPartner("<isv>") from com.databricks.sdk.core.UserAgent. These are static/global registrations — call once before creating WorkspaceClient.
• DATABRICKS_AUTH_TYPE conflict: The Java SDK reads DATABRICKS_AUTH_TYPE from the env internally. Do not use this name for your own app-level auth selector; use APP_AUTH_TYPE or similar.
• Java 11+ required. No --add-opens flag needed (unlike JDBC driver on Java 17+).
• See skills/java-sdk/authentication.md for complete patterns and skills/java-sdk/SKILL.md for task-focused guidance.

Python SQL connector

• M2M credentials_provider: The SQL connector expects a callable that returns a callable that returns headers. Pass e.g. lambda: config.authenticate (no parentheses – return the method, do not call it), not config.authenticate directly, to avoid "'dict' object is not callable" or similar.
• OAuth M2M in all_auth scripts: When using Config for M2M in a multi-auth script, set auth_type="oauth-m2m" on Config so the SDK does not try default credentials.
• Deprecation: Prefer user_agent_entry= over _user_agent_entry= when the connector supports it.

Python SQLAlchemy (databricks-sqlalchemy)

• URL: databricks://token:<token>@<host>?http_path=...&catalog=...&schema=.... Use create_engine(url, connect_args={"user_agent_entry": "<isv>_<product>/<version>"}). Use user_agent_entry, not _user_agent_entry (deprecated).
• OAuth M2M: Config.authenticate() returns a headers dict, not an object with .token. Extract the token with headers.get("Authorization", "").replace("Bearer ", "").strip() and put it in the URL.
• U2M: For external-browser, unset DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET before calling the SDK so the built-in OAuth app is used. For custom-oauth-app, the app must be in the same account as the workspace; use DATABRICKS_U2M_CLIENT_ID/DATABRICKS_U2M_CLIENT_SECRET to avoid conflicts with M2M vars.

REST API

• Headers: Always set Authorization: Bearer <token> and User-Agent: <isv>_<product>/<version> on every request.
• Common validation tests: (1) Unity Catalog Tables API – e.g. GET table (no warehouse). (2) Statement Execution API – run SQL (requires warehouse_id). Use these two to validate PAT, M2M, and U2M flows.

Databricks Connect (databricks-connect)

• User-Agent: Use .userAgent("<isv>_<product>/<version>") on DatabricksSession.builder — this is the PWAF-recommended approach. Do not use product=/product_version= on Config (older, non-PWAF pattern).
• Session builder pattern: DatabricksSession.builder.sdkConfig(config).userAgent(USER_AGENT).getOrCreate().
• Scala: DatabricksSession.builder.userAgent("<isv>_<product>/<version>").getOrCreate().
• See skills/databricks-connect/authentication.md for complete patterns and skills/databricks-connect/SKILL.md for task-focused guidance.

Node.js SQL Driver (@databricks/sql)

• Package: @databricks/sql (npm). Version 1.5.0+ for M2M, 1.3.0+ for U2M.
• Host normalization: The host connect option must be bare hostname (no https://). Strip scheme: host.replace('https://', '').split('/')[0].
• User-Agent: Set userAgentEntry on every client.connect() call (format <isv>_<product>/<version>).
• PAT: { host, path, token, userAgentEntry }.
• OAuth M2M (driver-native, preferred): { host, path, authType: 'databricks-oauth', oauthClientId, oauthClientSecret, userAgentEntry }. The driver handles token fetch and auto-refresh internally. No need for custom token providers or axios.
• OAuth U2M (driver-native): { host, path, authType: 'databricks-oauth', userAgentEntry } — without oauthClientId/oauthClientSecret. The driver opens a browser, starts a local server on a dynamic port for the redirect, and handles PKCE automatically. Do not pass M2M service principal client_id for U2M.
• M2M vs U2M: Both use authType: 'databricks-oauth'. With oauthClientId + oauthClientSecret → M2M. Without → U2M (browser).
• DESCRIBE TABLE results: Returns objects { col_name, data_type, comment }. comment can be null; handle with row.comment != null ? row.comment : 'NULL'.
• See skills/nodejs-sql-driver/authentication.md for complete patterns and skills/nodejs-sql-driver/SKILL.md for task-focused guidance.

Go SDK (databricks-sdk-go)

• Package: github.com/databricks/databricks-sdk-go v0.107.0+. Use for workspace APIs, UC metadata, Jobs — no SQL warehouse needed.
• Auth: databricks.Config{Host, Token} (PAT) or databricks.Config{Host, ClientID, ClientSecret} (M2M). SDK auto-detects auth method; auto-handles M2M token fetch + refresh.
• Telemetry: Use useragent.WithPartner("<isv>") and useragent.WithProduct("<product>", "<version>") from github.com/databricks/databricks-sdk-go/useragent. These are package-level/global registrations — call once before creating WorkspaceClient.
• Validation: w.Tables.Get(ctx, catalog.GetTableRequest{FullName: "samples.nyctaxi.trips"}) — UC Tables API, no warehouse needed.
• DATABRICKS_AUTH_TYPE conflict: The Go SDK may read DATABRICKS_AUTH_TYPE from the env internally. Do not use this name for your own app-level auth selector; use APP_AUTH_TYPE or similar.
• See skills/go-sdk/authentication.md for complete patterns and skills/go-sdk/SKILL.md for task-focused guidance.

Databricks SQL Driver for Go (databricks-sql-go)

• Package: github.com/databricks/databricks-sql-go. Use dbsql.NewConnector + functional options. Requires SQL warehouse (DATABRICKS_HTTP_PATH).
• Host normalization: The WithServerHostname option must be bare hostname (no https://). Strip scheme in code.
• User-Agent: Use dbsql.WithUserAgentEntry("YourCompany_YourProduct/1.0.0") (per-connector).
• PAT: WithAccessToken(token).
• OAuth M2M: Use m2m.NewAuthenticator(clientId, clientSecret, host) + dbsql.WithAuthenticator(auth). Driver handles token fetch + refresh.
• OAuth U2M: Use u2m.NewAuthenticator(host, 120*time.Second) + dbsql.WithAuthenticator(auth). Opens browser; driver manages redirect on a dynamic localhost port (not configurable).
• DESCRIBE TABLE results: Scan with sql.NullString to handle nullable comment column.
• See skills/go-sql-driver/authentication.md for complete patterns and skills/go-sql-driver/SKILL.md for task-focused guidance.

Adding Databricks to an existing project

• No Databricks yet? Use skills/adding-databricks-connector/SKILL.md: choose stack (REST vs Python SDK/SQL vs JDBC vs Connect), where to put the integration (dedicated module, adapter to your connection abstraction), and minimal steps (dependency → config → connect → operations → validation).
• Then use connector-structure and the authentication.md for the chosen stack.

Connector structure

• User-Agent required, connector-level: Set User-Agent on every API/driver call. Format <isv>_<product>/<version>. It is coded in the connector (e.g. constant or build-time value), not a user-configurable option; the end user does not provide product or product_version.
• Recommend all three auth types: PAT, OAuth M2M, and OAuth U2M. Design config and connect() so the user chooses one per connection. Collect only the options for the selected auth type (see connector-structure skill for full “User input by auth type” tables).
• Single config: One connection = one auth type (host, auth_type, credentials for that type, product/product_version; optional warehouse_id for SQL). Do not mix PAT and M2M (or multiple auth methods) in one config.
• Single entry point: Expose connect(config) that returns a client (REST wrapper or WorkspaceClient). Inside, branch on auth_type and build the right auth; use auth_type="oauth-m2m" for M2M when building Config.
• Operations via client: All API/SDK calls go through that client so headers (Authorization, User-Agent) and token refresh (M2M) are in one place.
• Validation: Optionally run the two tests (UC table get, Statement Execution with warehouse_id) to verify credentials. See connector-structure skill for full pattern.

Testing

• Per-test clean env: Run each auth script with only the env vars it needs (e.g. env -i PATH=$PATH HOME=$HOME VAR1=... VAR2=... python script.py). This prevents "more than one authorization method configured" and ensures each mechanism is tested in isolation.
• U2M browser test: For U2M external-browser, ensure no DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET in the env (or that the script unsets them before calling the SDK) so the browser flow uses the built-in OAuth app. Pass DISPLAY (or equivalent) when running in a GUI environment so the browser can open.

Reference (skills/)

Authentication patterns (co-located with each skill): skills/rest-api/authentication.md · skills/python-sdk/authentication.md · skills/python-sql-connector/authentication.md · skills/python-sqlalchemy/authentication.md · skills/python-dbconnect/authentication.md · skills/databricks-connect/authentication.md · skills/java-jdbc/authentication.md · skills/java-sdk/authentication.md · skills/nodejs-sql-driver/authentication.md · skills/go-sdk/authentication.md · skills/go-sql-driver/authentication.md · skills/odbc/authentication.md

Testing support: skills/connector-testing/env-isolation.md · skills/connector-testing/integration-checklist.md · skills/telemetry-attribution/SKILL.md

Skills (task-based; read when relevant): skills/adding-databricks-connector/SKILL.md (add connector to existing project) · skills/connector-structure/SKILL.md (config, connect, operations) · skills/rest-api/SKILL.md · skills/python-sdk/SKILL.md · skills/python-sql-connector/SKILL.md · skills/python-sqlalchemy/SKILL.md · skills/python-dbconnect/SKILL.md · skills/databricks-connect/SKILL.md · skills/java-jdbc/SKILL.md · skills/java-sdk/SKILL.md (Java SDK) · skills/go-sdk/SKILL.md (Databricks SDK for Go) · skills/go-sql-driver/SKILL.md (Databricks SQL Driver for Go) · skills/nodejs-sql-driver/SKILL.md (Node.js SQL Driver) · skills/u2m/SKILL.md · skills/testing/SKILL.md · skills/connector-testing/SKILL.md · skills/build-report/SKILL.md