Index live · v1.3.0 · JUN 8 2026

Azure Databricks

A-rank

Run Spark, Delta Lake, and ML workloads on Azure Databricks: workspaces, clusters, Unity Catalog, and Lakehouse patterns. Requires the mcp_microsoftdocs MCP for live docs.

Claude CodeCodexCursorGemini CliGithub CopilotMulti Platform
596stars
Updated 7 days ago
2contributors

Install This Skill

npx skills add MicrosoftDocs/Agent-Skills

SKILL.md

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

  • Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
  • Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

CategoryLocationDescription
TroubleshootingL37-L140Diagnosing and fixing Databricks errors and failures across compute, Spark/SQL, connectors/Lakeflow, Model Serving, VS Code/CLI, and AI agents, plus tools and logs for debugging performance.
Best PracticesL141-L312Best-practice guidance for Databricks architecture, governance, performance, cost, streaming, Lakeflow, ML/LLM/RAG, BI, vector search, and operations across Azure Databricks workloads.
Decision MakingL313-L404Guides for choosing Azure Databricks options and planning migrations: tiers, compute, Unity Catalog, runtimes, ingestion, ML/LLM, Lakebase, budgets, and tooling/SDK/CLI decisions.
Architecture & Design PatternsL405-L445Architectural blueprints and design patterns for Databricks lakehouse, including DR, networking, storage, governance, RAG/agents, MLOps, streaming, data modeling, and performance/cost optimization.
Limits & Quotaslimits-quotas.mdLimits, quotas, and constraints for Azure Databricks compute, AI/BI, connectors, Lakeflow, Lakebase, SQL types, Model Serving/foundation models, tokens, and Unity Catalog resources.
Securitysecurity.mdIdentity, access control, encryption, networking, compliance, and governance for Azure Databricks, Unity Catalog, Delta Sharing, Lakeflow, Lakebase, apps, agents, and external integrations.
Configurationconfiguration.mdConfiguring and administering Azure Databricks: accounts, workspaces, security, networking, compute, jobs, Unity Catalog, Lakeflow, ML/serving, Marketplace, SQL, bundles, apps, and connectors.
Integrations & Coding Patternsintegrations.mdIntegrating Databricks with external systems, SDKs, CLIs, agents, and data sources, plus detailed Spark/SQL/PySpark APIs, UDFs, and patterns for building pipelines, AI/ML, and streaming apps.
Deploymentdeployment.mdDeploying and operating Azure Databricks: workspace setup, CI/CD, IaC, Unity Catalog migration, model/agent serving, feature endpoints, serverless, and release/region considerations

Troubleshooting

TopicURL
Troubleshoot Azure Databricks compute startup issueshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codeshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UIhttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot Apache Kafka usage on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/connect/streaming/kafka/faq
Diagnose and fix common Delta Sharing errorshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Troubleshoot common Databricks CLI issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Diagnose and fix Databricks Connect Python issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Diagnose and fix Databricks Connect Scala issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extensionhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot Databricks VS Code extension errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Resolve ARITHMETIC_OVERFLOW errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Handle CAST_INVALID_INPUT errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in GA4 connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in SQL Server connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Understand DELTA_ICEBERG_COMPAT_V1_VIOLATION errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Handle DIVIDE_BY_ZERO errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Handle Azure Databricks named error conditionshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Fix EWKB_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Fix EWKT_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Resolve GEOJSON_PARSE_ERROR in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Address GROUP_BY_AGGREGATE errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Interpret and resolve H3_INVALID_GRID_DISTANCE_VALUE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors and tier requirementshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Fix INSUFFICIENT_TABLE_PROPERTY errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Troubleshoot INVALID_ARRAY_INDEX errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Troubleshoot INVALID_ARRAY_INDEX_IN_ELEMENT_AT in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks querieshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Diagnose ROW_COLUMN_ACCESS errors for filters and maskshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Interpret Azure Databricks SQLSTATE error codeshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Fix TABLE_OR_VIEW_NOT_FOUND errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Resolve UNRESOLVED_ROUTINE function resolution errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Understand UNSUPPORTED_TABLE_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Understand UNSUPPORTED_VIEW_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot MLflow 2 Agent Evaluation issueshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/troubleshooting
Troubleshoot and debug Databricks AI agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/debug-agent
Troubleshoot common Genie Space issueshttps://learn.microsoft.com/en-us/azure/databricks/genie/troubleshooting
Troubleshoot common Databricks Auto Loader issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/faq
Resolve common Confluence connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-faq
Troubleshoot authentication and rate limit errors for Confluencehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 Lakeflow ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Resolve common issues with Lakeflow managed connectorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/faq
Troubleshoot Google Ads connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics raw data ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot common HubSpot connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Troubleshoot Jira Lakeflow ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Meta Ads ingestion connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Diagnose and fix MySQL Lakeflow Connect ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Troubleshoot common Outlook connector ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-troubleshoot
Troubleshoot PostgreSQL Lakeflow Connect ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Lakeflow Connect query-based connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Salesforce Lakeflow ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Diagnose and fix Databricks ServiceNow connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Diagnose and fix Lakeflow SharePoint connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Troubleshoot Databricks Smartsheet connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/smartsheet-troubleshoot
Answer common SQL Server Lakeflow Connect connector questionshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-faq
Resolve SQL Server Lakeflow Connect ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot TikTok Ads connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Fix UNITY_CATALOG_INITIALIZATION_FAILED in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/uc-initialization-troubleshoot
Troubleshoot Workday HCM connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Diagnose and fix Databricks Workday connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Diagnose and fix Zendesk Support connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Handle Zerobus Ingest errors and retrieshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Inspect logs for Databricks init script executionhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Test and validate Databricks ODBC driver connectionshttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Configure and troubleshoot Lakeflow Jobs with many taskshttps://learn.microsoft.com/en-us/azure/databricks/jobs/large-jobs
Troubleshoot and repair Azure Databricks Lakeflow job failureshttps://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Monitor and troubleshoot Lakeflow Spark pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/observability
Use pipeline query history for debugging and tuninghttps://learn.microsoft.com/en-us/azure/databricks/ldp/query-history
Recover Lakeflow pipelines from streaming checkpoint failureshttps://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
User guides, migration, and troubleshooting for AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/guides
Troubleshoot Databricks Feature Store issueshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug common Databricks Model Serving endpoint issueshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving issues with Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Diagnose and resolve Databricks Model Serving timeoutshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-timeouts
Debug Python code in Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger
Troubleshoot failing Spark jobs and executors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Use Databricks Spark jobs timeline for debugginghttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/jobs-timeline
Diagnose long-running Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Investigate high I/O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-io
Debug slow low-I/O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Identify expensive reads in Spark DAG on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Diagnose gaps between Spark jobs in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Diagnose and fix Spark memory issues on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Azure Databricks Partner Connect issueshttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot Databricks Git folder sync errorshttps://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Fetch cursor rows and handle SQLSTATE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/fetch-stmt
Use GET DIAGNOSTICS for SQL error handling in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/get-diagnostics-stmt
Open cursors and handle errors with OPEN in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/open-stmt
Validate UTF-8 strings and handle INVALID_UTF8_STRINGhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Interpret Databricks SQL query performance insightshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/performance-insights
Use Databricks SQL query history to debug performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Analyze Databricks SQL query profiles to find bottleneckshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile
Troubleshoot and configure Databricks SQL scheduled querieshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/schedule-query
Inspect Structured Streaming state data for monitoring and debugginghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/read-state

Best Practices

TopicURL
Apply Databricks usage tags for accurate cost attributionhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags
Use default Databricks policy families to enforce compute best practiceshttps://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply Azure Databricks identity configuration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Configure default deletion vectors for Databricks Delta tableshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace-settings/deletion-vectors
Apply best practices for Azure Databricks serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Migrate Databricks library installs from init scriptshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply compute policy best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hintshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Use advanced techniques in Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/advanced-techniques
Apply Azure Databricks administration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Prepare and model data for high-performance BI on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI servinghttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Best practices for Power BI dashboards on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply Databricks compute configuration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Apply serverless compute best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Tune Databricks SQL warehouses for BI workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Use system table queries to monitor SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/monitor/queries
Control large interactive queries with Query Watchdoghttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Apply observability best practices for Databricks jobs and pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Best practices for designing Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/best-practices
Optimize performance of Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/performance
Apply Unity Catalog data governance best practiceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Apply row filters and column masks in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/filters-and-masks/
Work with legacy Hive metastore database objectshttps://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Follow DBFS root storage recommendations in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Migrate from DBFS mounts to Unity Catalog external locationshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts
Apply DBFS and Unity Catalog usage best practiceshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Optimize Delta Sharing egress costs for providershttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress
Apply Delta Lake best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Optimize tables with liquid clustering instead of partitioninghttps://learn.microsoft.com/en-us/azure/databricks/delta/clustering
Tune Azure Databricks data skipping with stats and Z-orderhttps://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping
Use deletion vectors to optimize Delta table updateshttps://learn.microsoft.com/en-us/azure/databricks/delta/deletion-vectors
Drop or replace Delta and Unity Catalog tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/drop-table
Optimize Delta table file layout on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/optimize
Handle Delta Lake limitations on Amazon S3https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Choose selective overwrite options in Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Control Delta table data file size on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
Vacuum Delta tables and manage retention safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/vacuum
Optimize VARIANT data performance with shredding on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/variant-shredding
Apply MLOps Stack best practices with bundleshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/mlops-stacks
Apply CI/CD best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
View Databricks cluster policy families via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/policy-families-commands
Apply security and performance best practices for Databricks appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Test Databricks Connect for Python code with pytesthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions in Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Choose between Databricks volumes and workspace fileshttps://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Apply best practices for MLflow 2 evaluation setshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/evaluation-set
Use Databricks review app for human GenAI evaluationshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/review-app
Load test Databricks Apps agents to determine sustainable QPShttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/load-test-agent-app
Follow an end-to-end Databricks agents development workflowhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agents-dev-workflow
Measure RAG performance with Databricks metricshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-assess-performance
Evaluate and monitor RAG apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Optimize Databricks RAG application qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-overview
Improve Databricks RAG chain qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain
Write effective custom instructions for Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/genie-code/instructions
Create and optimize Genie Code agent skillshttps://learn.microsoft.com/en-us/azure/databricks/genie-code/skills
Apply practical tips to improve Genie Code responseshttps://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Curate effective Genie Spaces for accurate answershttps://learn.microsoft.com/en-us/azure/databricks/genie/best-practices
Test, refine, and monitor Genie Space responseshttps://learn.microsoft.com/en-us/azure/databricks/genie/monitor
Tune Genie Space quality with SQL and knowledge storeshttps://learn.microsoft.com/en-us/azure/databricks/genie/tune-quality
Migrate existing Auto Loader streams to file eventshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/migrating-to-file-events
Configure Auto Loader for production workloadshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Apply common COPY INTO data loading patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Apply common patterns for Lakeflow ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Perform full refreshes of Lakeflow target tableshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/full-refresh
Query system.billing.usage to monitor Lakeflow costshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monitor-costs
Maintain Lakeflow managed ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Maintain and operate PostgreSQL ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Enable incremental ingestion for Salesforce formula fieldshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
Use Databricks init scripts for cluster customizationhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Configure and optimize compute for Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/compute
Build metadata-driven For each jobs with control tableshttps://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/foreach-sql-lookup-tutorial
Apply best practices for configuring classic Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Reduce Databricks lakehouse costs with optimization practiceshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Implement data and AI governance on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Design observability and monitoring strategy for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Apply interoperability and usability best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/best-practices
Apply operational excellence practices in Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
Optimize performance efficiency in Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
Apply reliability best practices on Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/best-practices
Implement security and compliance best practices in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/security-compliance-and-privacy/best-practices
Optimize Lakeflow pipelines with enhanced autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Apply best practices for Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices
Use advanced AUTO CDC features and monitor processing metricshttps://learn.microsoft.com/en-us/azure/databricks/ldp/cdc-advanced
Develop and test Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/develop
Manage Python dependencies in Lakeflow pipelines safelyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/external-dependencies
Implement advanced expectation patterns at scalehttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Apply data quality expectations in Databricks pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectations
Reduce high initialization times in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Backfill historical data with Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/flows-backfill
Run full refresh operations for Databricks streaming tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/full-refresh-st
Optimize stateful stream processing with watermarkshttps://learn.microsoft.com/en-us/azure/databricks/ldp/stateful-processing
Design CDC and snapshot patterns in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ldp/what-is-change-data-capture
Restart the Python process to refresh Databricks librarieshttps://learn.microsoft.com/en-us/azure/databricks/libraries/restart-python-process
Apply data loading best practices on Databricks AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/dataloading
Apply Hyperopt best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/hyperopt-best-practices
Improve Databricks AutoML forecasting with covariateshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl/automl-covariate-forecast
Implement point-in-time correct feature joins for time series MLhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Benchmark Databricks LLM endpoints for latency and TPShttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/prov-throughput-run-benchmark
Apply LLMOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/llmops
Validate Databricks models before serving deploymenthttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-pre-deployment-validation
Monitor Databricks model quality and endpoint healthhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/monitor-diagnose-endpoints
Optimize Databricks Model Serving endpoints for productionhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/production-optimization
Plan and execute load testing for Databricks serving endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/what-is-load-test
Tune and scale Ray clusters on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/scale-ray
Follow deep learning best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
Fine-tune Hugging Face models on a single GPU in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/fine-tune-model
Prepare datasets for Hugging Face fine-tuning on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/load-data
Adapt Apache Spark workloads for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/spark
Align MLflow LLM judges with human evaluatorshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/align-judges
Evaluate and compare MLflow prompt versions effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-prompts
Use manual MLflow tracing for production GenAI appshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/app-instrumentation/manual-tracing/
Analyze GenAI traces for errors and performancehttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/observe-with-traces/analyze-traces
Run Databricks notebooks safely and efficientlyhttps://learn.microsoft.com/en-us/azure/databricks/notebooks/run-notebook
Test and schedule Databricks notebook codehttps://learn.microsoft.com/en-us/azure/databricks/notebooks/test-notebooks
Apply performance optimization recommendations on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/
Use adaptive query execution on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/aqe
Leverage cost-based optimizer in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo
Improve read performance with Databricks disk cachehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache
Improve Delta query performance with dynamic file pruning on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/dynamic-file-pruning
Accelerate data access with predictive I/Ohttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-io
Use predictive optimization for Unity Catalog tableshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization
Tune Azure Databricks range join optimizationhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/range-join
Diagnose Databricks Spark cost and performance in UIhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/
Debug skew and spill in Databricks Spark stageshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Handle Databricks spot instance losses effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Resolve long Spark stages with a single taskhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/one-spark-task
Optimize many small Spark jobs on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/small-spark-jobs
Mitigate overloaded Spark driver on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-driver-overloaded
Detect unnecessary data rewriting in Databricks Spark writeshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-rewriting-data
Best practices for setting up Databricks Partner Connecthttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/best-practice
Optimize joins with broadcast hints in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/functions/broadcast
Configure networking for Databricks Lakehouse Federation data sourceshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/networking
Optimize performance of Databricks Lakehouse Federation querieshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations
Transform complex and nested data types in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/complex-types
Use higher-order functions on arrays in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/higher-order-functions
Differences between VARIANT and JSON strings in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/variant-json-diff
Work with OBJECT type and VARIANT schemas in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/object-type
Use TIMESTAMP_NTZ type and Delta support in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/timestamp-ntz-type
Use VARIANT type and Iceberg compatibility in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type
Collect table statistics with ANALYZE TABLE for optimizationhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-analyze-compute-statistics
Benchmark Databricks SQL warehouses with the TPC-DS datasethttps://learn.microsoft.com/en-us/azure/databricks/sql/tpcds-eval
Author effective SQL patterns for Databricks alertshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/query-patterns
Optimize Databricks SQL queries using primary key constraintshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-optimization-constraints
Use Structured Streaming checkpoints safely on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/checkpoints
Configure Databricks Structured Streaming for productionhttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/production
Optimize and monitor Databricks real-time streaming performancehttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/real-time/performance
Optimize stateless Structured Streaming queries on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stateless-streaming
Apply watermarks for stateful streaming on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/watermarks
Analyze Databricks table size and optimize storage costshttps://learn.microsoft.com/en-us/azure/databricks/tables/size
Design data models optimized for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/transform/data-modeling
Optimize join performance for Azure Databricks workloadshttps://learn.microsoft.com/en-us/azure/databricks/transform/optimize-joins
Clean and validate data with Databricks batch and streaminghttps://learn.microsoft.com/en-us/azure/databricks/transform/validate
Optimize Unity Catalog batch Python UDF performancehttps://learn.microsoft.com/en-us/azure/databricks/udf/python-batch-udf
Tune Azure Databricks vector search performance at scalehttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-best-practices
Optimize and control Azure Databricks vector search costshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-cost-management
Design and run load tests for Vector Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-endpoint-load-test
Improve retrieval quality for Databricks vector searchhttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-retrieval-quality
Identify and clean up unused Databricks Vector Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-unused-endpoints
Download internet data into Azure Databricks volumeshttps://learn.microsoft.com/en-us/azure/databricks/volumes/download-internet-files

Decision Making

TopicURL
Manage and change Azure Databricks subscription tierhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/account
Create and manage Databricks budgets to track usagehttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/budgets
Plan migration from Standard to Premium Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Migrate from SCIM to automatic identity managementhttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/automatic-identity-management/migrate-to-aim
Decide when and how to use serverless Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Decide and migrate from dbx to Databricks bundleshttps://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate
Migrate optimized LLM endpoints to provisioned throughputhttps://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/migrate-provisioned-throughput
Decide when to use Databricks Light runtimehttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime/light
Plan migration of Databricks workloads to Spark 3.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/
Choose and manage the default Unity Catalog cataloghttps://learn.microsoft.com/en-us/azure/databricks/catalogs/default
Choose appropriate Azure Databricks compute typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/choose-compute
Select compatible flexible node types for Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-type-instances
Decide when and how to use GPU Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/gpu
Decide when and how to use Azure Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-index
Plan migration from classic to serverless Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/migration
Choose and manage Azure Databricks SQL warehouse sizing and scalinghttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-behavior
Choose between Databricks SQL warehouse typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-types
Choose between ABAC and table-level filters in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/abac-vs-rls-cm
Choose between managed and external Unity Catalog assetshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/managed-versus-external
Plan and execute upgrade of Databricks workspaces to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/
Prepare and migrate to Unity Catalog–only Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/uc-only-migration
Choose Delta Lake protocol versions and feature setshttps://learn.microsoft.com/en-us/azure/databricks/delta/feature-compatibility
Choose local development tools for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/
Migrate from legacy to new Databricks CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/migrate
Manage Databricks account budget policies via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budget-policy-commands
Configure Databricks account budgets using CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budgets-commands
Manage Databricks account usage dashboards via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-usage-dashboards-commands
Plan migration from legacy Databricks Connect runtimeshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect-legacy
Migrate from older to new Databricks Connect for Pythonhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/migrate
Migrate from legacy to new Scala Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/migrate
Choose and use Databricks SDKs for automationhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/sdks
Decide between CDKTF and Databricks Terraform providerhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/cdktf
Decide when to migrate agents to Databricks Appshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/migrate-agent-to-apps
Select Azure Databricks generative AI capabilities for your workflowhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/gen-ai-capabilities
Choose between Databricks Free Edition and free trialhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/free-trial-vs-free-edition
Choose incremental ingestion options from cloud object storagehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/
Select Auto Loader file detection mode for your workloadhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/file-detection-modes
Plan migration of existing data to Delta Lake on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/data-migration/
Plan and choose a MySQL ingestion workflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql
Plan and choose a PostgreSQL ingestion workflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql
Select and plan a SQL Server ingestion workflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-overview
Choose and start with Databricks ODBC and JDBC drivershttps://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc-odbc-bi
Migrate from Simba Spark ODBC to Databricks ODBChttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/migration
Plan and manage production workloads with Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/
Migrate from Spark Submit tasks in Databricks jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/spark-submit
Plan production Azure Databricks lakehouse deploymentshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/
Design compute and workspace configuration for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/compute
Choose a programming language for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/languages/overview
Assess environment version compatibility for Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/environment-version-compatibility
Choose triggered vs continuous mode for pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/pipeline-mode
Migrate Databricks online tables to Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/migrate-from-online-tables
Choose and use Databricks Online Feature Storeshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/online-feature-store
Upgrade workspace feature tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/uc/upgrade-feature-table-to-uc
Migrate MLflow model versions to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-models
Decide and migrate to Unity Catalog model managementhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-to-uc
Upgrade Databricks ML workflows to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/upgrade-workflows
Choose Databricks options for batch model inferencehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-inference/
Migrate from legacy MLflow Model Serving to Databricks Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/migrate-model-serving
Choose between Spark and Ray on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/spark-ray-overview
Plan migration of data applications to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/
Scope and plan ETL pipeline migration to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/etl
Choose a migration path from Parquet to Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/migration/parquet-to-delta-lake
Plan migration from data warehouse to Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/migration/warehouse-to-lakehouse
Decide and migrate from Agent Evaluation to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration
Quick reference for migrating to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration-reference
Choose between open source and managed MLflow on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/oss-managed-diff
Choose compute resources for Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-compute
Choose Lakebase backup and restore methodshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/backup-methods
Choose between Databricks Apps and external Lakebase appshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/build-applications
Decide between Lakebase Provisioned and Autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/oltp/upgrade-to-autoscaling
Configure incremental refresh for Databricks materialized viewshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/incremental-refresh
Choose pandas options on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pandas/
Choose between Azure Databricks and Fabric integrationshttps://learn.microsoft.com/en-us/azure/databricks/partners/bi/fabric
Use Hive metastore federation in Unity Catalog migrationshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/hms-federation-concepts
Migrate legacy Databricks query federation to Lakehouse Federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/migrate
Plan and execute migration to Databricks Runtime 11.xhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/11.x-migration
Migrate workloads to Databricks Runtime 12.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/12.x-migration
Migrate workloads to Databricks Runtime 13.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/13.x-migration
Migrate workloads to Databricks Runtime 14.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/14.x-migration
Plan around Databricks Runtime and feature lifecycleshttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/databricks-runtime-ver
Understand serverless DBU billing by Azure Databricks SKUhttps://learn.microsoft.com/en-us/azure/databricks/resources/pricing
Plan and manage Azure Databricks serverless networking costshttps://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/cost-management
Decide between Spark Connect and Spark Classic on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/spark/connect-vs-classic
Choose between SparkR and sparklyr on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sparkr/sparkr-vs-sparklyr
Use SYNC to upgrade Hive tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-sync
Choose and size SQL warehouses for alertshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/compute
Choose Structured Streaming output modes on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/output-mode
Choose and implement Databricks transaction modeshttps://learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes

Architecture & Design Patterns

TopicURL
Plan disaster recovery architecture for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/admin/disaster-recovery
Design and use materialization for Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/materialization
Implement fan-in and fan-out in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/fan-in-fan-out
Choose patterns for external access to Databricks datahttps://learn.microsoft.com/en-us/azure/databricks/external-access/
Build an IDP pipeline with Databricks AI Functionshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/idp-pipeline-tutorial
Design intelligent document processing pipelines on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/intelligent-document-processing
Design multi-agent orchestrator apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-apps
Apply agent system design patterns on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agent-system-design-patterns
Design measurement infrastructure for RAG quality on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-enable-measurement
Design and tune Databricks RAG inference chainshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Use Agent mode in Genie Spaces for complex analysishttps://learn.microsoft.com/en-us/azure/databricks/genie/agent-mode
Design cost optimization architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/
Apply data and AI governance architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/
Design Delta Lake and medallion data architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/delta-lake
Design high availability and disaster recovery for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/ha-dr
Design Azure Databricks network and connectivity architecturehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/network
Design storage architecture for Azure Databricks and Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/storage
Design Azure Databricks workspace architecture strategyhttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/workspace-strategy
Design interoperability and usability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/
Design operational excellence architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/
Design performance efficiency architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/
Apply Azure Databricks lakehouse reference architectureshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reference
Design reliability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/
Apply medallion lakehouse architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion
Choose Databricks ML model deployment patternshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns
Implement MLOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/mlops-workflow
Design data serving patterns for Databricks AIhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/serve-data-ai
Choose and train deep learning recommender models on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-recommender-models
Back Databricks Online Feature Stores with Lakebase Autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/feature-store
Sync Lakebase Postgres tables to Unity Catalog with CDChttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/lakehouse-sync
Use Lakebase for durable AI agent state and memoryhttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/state-management
Serve lakehouse data via Lakebase synced tableshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/sync-tables
Set up Databricks Serverless Private Git with Private Linkhttps://learn.microsoft.com/en-us/azure/databricks/repos/serverless-private-git
Choose patterns for modeling semi-structured data on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/
Choose async checkpointing for Databricks stateful querieshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing
Use async progress tracking in Databricks streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-progress-checking
Decide when and how to partition Delta tableshttps://learn.microsoft.com/en-us/azure/databricks/tables/partitions
Synced from MicrosoftDocs/Agent-Skills@1888130fetched May 24, 2026

Quick Stats

Source repo · this path
Stars596
Forks67
Last commit2026-06-02
Contributors2
LicenseCC-BY-4.0
CategoryData Science
View on GitHub

Tags

azuredatabrickssparkdelta-lakelakehouse