Big Data News Hubb
Advertisement
  • Home
  • Big Data
  • News
  • Contact us
No Result
View All Result
  • Home
  • Big Data
  • News
  • Contact us
No Result
View All Result
Big Data News Hubb
No Result
View All Result
Home Big Data

Orchestrate Production dbt Projects on the Lakehouse With Databricks Workflows

admin by admin
February 15, 2023
in Big Data


We are pleased to announce the General Availability (GA) of support for orchestrating dbt projects in Databricks Workflows. Since the start of Public Preview, we have hundreds of customers leverage this integration with dbt to collaboratively transform, test, and document data in Databricks SQL warehouses.

With dbt support in Workflows, your dbt project is retrieved from a Git repository, and a single-node cluster is launched with dbt-core and project dependencies on it. The SQL generated by dbt is run on a serverless SQL warehouse, providing easy debugging and great performance. There are also robust and operational capabilities, such as the ability to repair failed runs and send alerts via Slack or a webhook destination when a dbt task fails, not to mention the ability to manage such jobs and retrieve dbt artifacts such as logs through the Jobs API.

With GA, we have extended support to SQL Pro Warehouses in addition to existing support for serverless SQL Warehouses. Moreover, we are happy to announce support for Databricks on Google Cloud Platform (GCP). Lineage from transforms specified in dbt projects is also automatically captured in Unity Catalog. Finally, even more dbt community packages such as dbt-artifacts now work with Databricks.

To get started with dbt on Databricks, simply run “pip install dbt-databricks.” This installs the open source dbt-databricks package built together with dbt Labs and other contributors. You can follow our detailed guide to get started with an example project. Once you commit your source code to a git repository, you can use Databricks Workflows to execute your dbt models in production (see our docs for (AWS | Azure | GCP).



Source link

Previous Post

How Strategic Blue uses Amazon QuickSight and AWS Cost and Usage Reports to help their customers save millions

Next Post

Video Highlights: Attention Is All You Need – Paper Explained

Next Post

Video Highlights: Attention Is All You Need - Paper Explained

Recommended

Solving for Data Drift from Class Imbalance with Model Monitoring

December 4, 2022

To Improve Enterprise Visibility, Shine a Light on Dark Data and Shadow IT

October 3, 2022

Upgrade Your Objects in Hive Metastore to Unity Catalog

December 3, 2022

Don't miss it

News

Bill Gates Says the Age of AI Has Begun, Bringing Opportunity and Responsibility

March 25, 2023
Big Data

Techniques for training large neural networks

March 25, 2023
Big Data

O’Reilly 2023 Tech Trends Report Reveals Growing Interest in Artificial Intelligence Topics, Driven by Generative AI Advancement

March 24, 2023
Big Data

Democratizing the magic of ChatGPT with open models

March 24, 2023
News

Introducing native support for Apache Hudi, Delta Lake, and Apache Iceberg on AWS Glue for Apache Spark, Part 2: AWS Glue Studio Visual Editor

March 24, 2023
News

ChatGPT Puts AI At Inflection Point, Nvidia CEO Huang Says

March 24, 2023

big-data-footer-white

© 2022 Big Data News Hubb All rights reserved.

Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Privacy Policy and Terms & Conditions.

Navigate Site

  • Home
  • Big Data
  • News
  • Contact us

Newsletter Sign Up

No Result
View All Result
  • Home
  • Big Data
  • News
  • Contact us

© 2022 Big Data News Hubb All rights reserved.