Databricks said Meta Platforms (META) had joined as an investor in a $10 billion funding round intended to fuel the data analytics startup's expansion plans and new product development.

Meta backs data analytics firm Databricks as AI boom attracts investors

(Bloomberg) -- Software maker Databricks Inc. has clinched more than $5 billion of financing from lenders including Blackstone Inc., Apollo Global Management Inc. and Blue Owl Capital Inc. in its largest debt raise to date, according to people with knowledge of the matter.Most Read from ...

Databricks Inks $5 Billion of Private Credit, Bank Funding

Software maker Databricks Inc. has clinched more than $5 billion of financing from lenders including Blackstone Inc., Apollo Global Management Inc. and Blue Owl Capital Inc. in its largest debt raise to date, according to people with knowledge of the matter.

Databricks Inks $5 Billion Financing From Private Credit, Banks

Young startup Databricks, which specializes in scaling and building artificial intelligence, said Tuesday it raised $10 billion, bringing the company's valuation to $62 billion.

AI startup Databricks raises $10 bn as value soars | National | theweeklyjournal.com

Databricks Inc., the fast-growing analytics software firm, is seeking about $2.5 billion of debt from private credit lenders to help it offset tax burdens associated with stock sales from staffers, according to people with knowledge of the matter.

Tech Start-Up Databricks Seeks $2.5 Billion Private Debt Financing - Bloomberg

The AI Times is a weekly newsletter covering the biggest AI, machine learning, big data, and automation news from around the globe.

A|I: The AI Times – Google & Nvidia are betting on this Canadian startup to boost ML model performance

In addition, the release will make data more accessible and governed through automatic data audit logging across data sources and will support 14 new data sources including SAP-HANA and Databricks. “Today, everyone can do AI proofs of concept — but ...

Domino Fall 2023 Release Expands Platform to Fast-Track and Future-Proof All AI

Expanded Data Connectivity, New Data Source Audit and Governance Capabilities Help Teams Overcome Barriers to Deploying All AI with Confidence SAN FRANCISCO,  /PRNewswire/ -- Domino Data Lab, provider of the leading Enterprise AI platform trusted by over 20% of the Fortune 100,

Domino Fall 2023 Release Expands Platform to Fast-Track All Enterprise AI, Including GenAI, Responsibly

Machine learning operations startup Domino Data Lab Inc. today announced a major update to its artificial intelligence development platform that it says will make it easier for companies to build generative AI models and other applications.

Domino Data bids to fast-track generative AI model development with project templates

SAN FRANCISCO, CALIFORNIA, UNITED STATES, October 31, 2023 /EINPresswire / -- In a strategic move that promises to redefine the landscape of B2B data solutions, FlashIntel is proud to announce its official partnership with Databricks , marking its esteemed ...

Flashintel Joins Databricks Marketplace: A Partnership Poised To Revolutionize B2B Data Solutions

Databricks

databricks

Building Data Intelligence Brick by Brick: From Databricks' Playbook

Die bou van data-intelligensie Baksteen vir baksteen: Uit Databricks se Playbook

Стварэнне інтэлектуальных дадзеных па цаглінках: з Playbook Databricks

Construyendo inteligencia de datos ladrillo a ladrillo: del manual de Databricks

בניית מודיעין נתונים לבנה אחר לבנה: מתוך ספר ההצגות של Databricks

データインテリジェンスを一歩ずつ構築する: Databricks のプレイブックから

Building Data Intelligence Brick ໂດຍ Brick: ຈາກ Databricks' Playbook

बिल्डिङ डाटा इंटेलिजेंस ब्रिक बाइ ब्रिक: डाटाब्रिक्स प्लेबुकबाट

Willayta Yachayta Ladrillomanta Ladrilloman ruway: Databrickspa Pukllay Libronmanta

Construirea inteligenței datelor cărămidă cu cărămidă: Din manualul de joc al Databricks

Ndërtimi i inteligjencës së të dhënave Brick by Brick: From Databricks' Playbook

Изградња Интелигенције података Брицк би Брицк: Фром Датабрицкс' Плаибоок

கட்டிட தரவு நுண்ணறிவு செங்கல் மூலம் செங்கல்: டேட்டாபிரிக்ஸ் பிளேபுக்கிலிருந்து

Let's Build an MLOps Pipeline With Databricks and Spark - Part 2

At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers. Introduction My name is Aniruth. I work on the storage team at Databricks, where we work on enabling customers to save large amounts of data in an open, scalable format with our Data Intelligence Platform. Specifically, I work on our interoperability efforts with Delta Lake and Apache Iceberg. HackerNoon: What’s the main problem your AI product is designed to solve, and what is effective about your approach to solving it? Aniruth: Databricks unifies data and AI to give customers actionable intelligence—what we call data intelligence. This includes ingesting large amounts of data, ETL, large-scale storage, business intelligence queries, and AI workloads. The techniques used in machine learning in the past decade have been around since the 1980s; the rise of big data made it feasible to run algorithms at scale. Techniques like few shot prompting or RAG rely on high-quality data. Models that have better data win against those with better architectures. Databricks has put significant investment into leading efforts in the data space, pioneering the lakehouse architecture with open data formats and open governance, where customers are able to get the best insights with the best performance from data lakes. What criteria did you use to select the specific AI models for this product, and how does your company make decisions like this? We use AI models in a number of ways in the product - such as Llama 3 for the AI Assistant. We believe in an open data and AI ecosystem and encourage our customers to use any model of their choice. We help make sure customers have end-to-end governance across the AI lifecycle regardless of the models they use, so they can focus on making their models purpose-built for their use cases. How do you ensure your product delivers accurate and unbiased results to users? We have spent significant effort and investment in prioritizing accuracy and unbiased answers for AI usage within our products, and continue to conduct frequent testing. What is your day in the life like as a product manager? The data and AI space is rapidly evolving, so it’s very important to keep up to date. My day can include talking to customers, market analysis, putting together a product requirements document, or preparing marketing materials. My favorite part is getting to make diagrams illustrating how things will work, as it’s pretty fun to transform an idea into a visual. What’s the next big breakthrough in AI that everyone should be watching for? There’s a lot of big breakthroughs coming soon. One that I’m particularly interested in is the hyperpersonalization of content. For the past decade, ads have been fine-tuned to the specific watcher. Some elements of content have been tuned, such as what thumbnail Netflix shows a user, but the actual content (the video itself) has not been. It’ll be interesting to see how directors/producers balance telling the story they want to and the user’s interests. What’s been the biggest challenge you’ve faced in bringing this AI product to market, from concept to launch? I work on large-scale data storage, which can be very confusing to understand. We have various AI optimizations on data, but there’s often questions about when these optimizations are run, how they work, what they don’t cover, etc. With these kinds of questions, it is important to ensure we have clear, consistent messaging about what we are building and why. I’ve found that explaining the cause of limitations resonates very well with customers. How do you envision AI evolving to better understand and respond to human emotions, and what challenges or opportunities do you see in that area? Multimodal models are going to get significantly better in the coming years, which will change our primary mode of interaction with AI. Figuring out human emotion is significantly easier from visual or audio information compared to text. I think there's an opportunity to create more natural interactions in a wider array of scenarios. What performance metrics or KPIs do you track to gauge the success of this AI product? We typically want to see good feedback and usage. I talk to customers pretty often to get a sense of how and why they think about our products, which is key to explain why we see certain trends in metrics. How do you approach designing a product experience that’s not only functional but also engaging and memorable for users? Large-scale data products are notoriously challenging to use. Simple examples are easy to set up, but production workloads typically involve confusing configuration and code. It’s been a high priority for me to build out functionality that customers need, while making the product very simple to use. The future is where any business gets insights from their data easily. In the current world, data-driven business insights are typically restricted to the biggest companies - but even they would prefer a simpler experience. In the long-term, how do you think AI will help the individual reach one’s full potential? For the individual, I’m very excited about AI integrations into hardware. Up to now, we’ve largely seen AI in software applications like websites. There’s many larger applications of building devices that utilize AI, and we’re already starting to see some of those implications in cars and phones. Where do you see the product evolving in the next few years, and what features are you most excited to add? Databricks is on a path towards becoming increasingly simple and more powerful at the same time. There are a lot of efforts we are working on across the board, from making big-scale data and compute easy to work with to improving performance on queries and workflows. Personally, I think we have some exciting features coming soon throughout the product that make workflows way easier with AI. Examples include AI-generated comments on data, AI code suggestions in the notebook editors, and AI interfaces to chat with data (for example, Databricks AI/BI Genie). What’s your perspective on AI’s impact on jobs, and how do you address those concerns in your product strategy? There are concerns over whether AI would reduce the number of jobs. Our products are designed to increase valuable insights, which often come in conjunction with users. For example, with AI/BI Genie, users can create interfaces on their own data. This is a magical experience, where users can ask questions and get answers specific to them. In fact, users can even check the SQL being used to confirm it’s what they’re looking for. This is collaborative with analysts, reducing the time it takes them to go from idea to insight. What user feedback surprised you and led to changes in your roadmap, strategy or product experience? One major surprise for me was the complexity that some of the larger companies have. This introduces requirements into the product that I wouldn’t have considered on my own. A common example is thinking of migration strategies when introducing a new product. Typically, large companies will have either put together existing technologies (commonly open-source software) or have built custom software to solve the problem that our new product offering tackles. It usually takes a bit of time to understand why and how these are put together to ensure we have a solution that covers all possibilities. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Interested in reading the content from all of our writing prompts? Click HERE. At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers. At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers. Introduction My name is Aniruth . I work on the storage team at Databricks , where we work on enabling customers to save large amounts of data in an open, scalable format with our Data Intelligence Platform. Specifically, I work on our interoperability efforts with Delta Lake and Apache Iceberg. Aniruth Databricks HackerNoon: What’s the main problem your AI product is designed to solve, and what is effective about your approach to solving it? HackerNoon: Aniruth : Databricks unifies data and AI to give customers actionable intelligence—what we call data intelligence. This includes ingesting large amounts of data, ETL, large-scale storage, business intelligence queries, and AI workloads. The techniques used in machine learning in the past decade have been around since the 1980s; the rise of big data made it feasible to run algorithms at scale. Aniruth Aniruth Techniques like few shot prompting or RAG rely on high-quality data. Models that have better data win against those with better architectures. Databricks has put significant investment into leading efforts in the data space, pioneering the lakehouse architecture with open data formats and open governance, where customers are able to get the best insights with the best performance from data lakes. What criteria did you use to select the specific AI models for this product, and how does your company make decisions like this? We use AI models in a number of ways in the product - such as Llama 3 for the AI Assistant. We believe in an open data and AI ecosystem and encourage our customers to use any model of their choice. We help make sure customers have end-to-end governance across the AI lifecycle regardless of the models they use, so they can focus on making their models purpose-built for their use cases. How do you ensure your product delivers accurate and unbiased results to users? We have spent significant effort and investment in prioritizing accuracy and unbiased answers for AI usage within our products, and continue to conduct frequent testing. What is your day in the life like as a product manager? The data and AI space is rapidly evolving, so it’s very important to keep up to date. My day can include talking to customers, market analysis, putting together a product requirements document, or preparing marketing materials. My favorite part is getting to make diagrams illustrating how things will work, as it’s pretty fun to transform an idea into a visual. What’s the next big breakthrough in AI that everyone should be watching for? There’s a lot of big breakthroughs coming soon. One that I’m particularly interested in is the hyperpersonalization of content. For the past decade, ads have been fine-tuned to the specific watcher. Some elements of content have been tuned, such as what thumbnail Netflix shows a user, but the actual content (the video itself) has not been. It’ll be interesting to see how directors/producers balance telling the story they want to and the user’s interests. What’s been the biggest challenge you’ve faced in bringing this AI product to market, from concept to launch? I work on large-scale data storage, which can be very confusing to understand. We have various AI optimizations on data, but there’s often questions about when these optimizations are run, how they work, what they don’t cover, etc. With these kinds of questions, it is important to ensure we have clear, consistent messaging about what we are building and why. I’ve found that explaining the cause of limitations resonates very well with customers. How do you envision AI evolving to better understand and respond to human emotions, and what challenges or opportunities do you see in that area? Multimodal models are going to get significantly better in the coming years, which will change our primary mode of interaction with AI. Figuring out human emotion is significantly easier from visual or audio information compared to text. I think there's an opportunity to create more natural interactions in a wider array of scenarios. What performance metrics or KPIs do you track to gauge the success of this AI product? We typically want to see good feedback and usage. I talk to customers pretty often to get a sense of how and why they think about our products, which is key to explain why we see certain trends in metrics. How do you approach designing a product experience that’s not only functional but also engaging and memorable for users? Large-scale data products are notoriously challenging to use. Simple examples are easy to set up, but production workloads typically involve confusing configuration and code. It’s been a high priority for me to build out functionality that customers need, while making the product very simple to use. The future is where any business gets insights from their data easily. In the current world, data-driven business insights are typically restricted to the biggest companies - but even they would prefer a simpler experience. In the long-term, how do you think AI will help the individual reach one’s full potential? For the individual, I’m very excited about AI integrations into hardware. Up to now, we’ve largely seen AI in software applications like websites. There’s many larger applications of building devices that utilize AI, and we’re already starting to see some of those implications in cars and phones. Where do you see the product evolving in the next few years, and what features are you most excited to add? Databricks is on a path towards becoming increasingly simple and more powerful at the same time. There are a lot of efforts we are working on across the board, from making big-scale data and compute easy to work with to improving performance on queries and workflows. Personally, I think we have some exciting features coming soon throughout the product that make workflows way easier with AI. Examples include AI-generated comments on data, AI code suggestions in the notebook editors, and AI interfaces to chat with data (for example, Databricks AI/BI Genie). What’s your perspective on AI’s impact on jobs, and how do you address those concerns in your product strategy? There are concerns over whether AI would reduce the number of jobs. Our products are designed to increase valuable insights, which often come in conjunction with users. For example, with AI/BI Genie, users can create interfaces on their own data. This is a magical experience, where users can ask questions and get answers specific to them. In fact, users can even check the SQL being used to confirm it’s what they’re looking for. This is collaborative with analysts, reducing the time it takes them to go from idea to insight. What user feedback surprised you and led to changes in your roadmap, strategy or product experience? One major surprise for me was the complexity that some of the larger companies have. This introduces requirements into the product that I wouldn’t have considered on my own. A common example is thinking of migration strategies when introducing a new product. Typically, large companies will have either put together existing technologies (commonly open-source software) or have built custom software to solve the problem that our new product offering tackles. It usually takes a bit of time to understand why and how these are put together to ensure we have a solution that covers all possibilities. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Interested in reading the content from all of our writing prompts? Click HERE. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Interested in reading the content from all of our writing prompts? Click HERE. The link for the template is HERE. The link for the template is HERE. HERE. HERE.

"The Future is Where Any Business Gets Insights From Their Data Easily" says Aniruth from Databricks

Databricks-dən Aniruth deyir: "Gələcək hər hansı bir biznesin məlumatlarından asanlıqla məlumat əldə etdiyi yerdir"

"El futuro es donde cualquier empresa puede obtener información de sus datos fácilmente", afirma Aniruth de Databricks

“Masa Depan adalah Tempat Setiap Bisnis Mendapatkan Wawasan Dari Data Mereka Dengan Mudah” kata Aniruth dari Databricks

"Il futuro è dove qualsiasi azienda ottiene facilmente informazioni dai propri dati", afferma Aniruth di Databricks

「未来は、どんな企業でもデータから簡単に洞察を得られる時代です」とDatabricksのAniruth氏は語る

„Budúcnosť je tam, kde každá firma ľahko získa informácie zo svojich údajov,“ hovorí Aniruth z Databricks

"Mustaqbalku waa halka uu ganacsi kastaa si sahal ah uga helo xogta xogtooda" ayuu yidhi Aniruth oo ka socda Databricks

"Framtiden är där alla företag enkelt får insikter från sina data" säger Aniruth från Databricks

Анирут аз Databricks мегӯяд: "Оянда он ҷоест, ки ҳама гуна тиҷорат аз маълумоти худ ба осонӣ фаҳмиш мегирад"

"Kelajak - bu har qanday biznes o'z ma'lumotlaridan osongina ma'lumot oladigan joy", deydi Databricks'dan Aniruth

"Ikamva kulapho naliphi na ishishini lifumana ulwazi kwidatha yalo ngokulula" utshilo uAniruth ovela kwiDatabricks.

"Ikusasa Yilapho Noma Yiliphi Ibhizinisi Lithola Ukuqonda Emininingwaneni Yalo Kalula" kusho u-Aniruth ovela ku-Databricks

Let's Build an MLOps Pipeline With Databricks and Spark - Part 1

Building CI Pipeline with Databricks Asset Bundle and GitLab

ডেটাব্রিক্স অ্যাসেট বান্ডেল এবং গিটল্যাব সহ সিআই পাইপলাইন তৈরি করা

Erstellen einer CI-Pipeline mit Databricks Asset Bundle und GitLab

Creación de canalización de CI con Databricks Asset Bundle y GitLab

Création d'un pipeline CI avec Databricks Asset Bundle et GitLab

डेटाब्रिक्स एसेट बंडल और गिटलैब के साथ CI पाइपलाइन का निर्माण

Databricks Asset Bundle と GitLab を使用した CI パイプラインの構築

Databricks Asset Bundle 및 GitLab을 사용하여 CI 파이프라인 구축

Construindo Pipeline de CI com Databricks Asset Bundle e GitLab

Создание конвейера CI с помощью Databricks Asset Bundle и GitLab

Databricks Asset Bundle ve GitLab ile CI Pipeline Oluşturma

Xây dựng quy trình CI với gói tài sản Databricks và GitLab

使用 Databricks Asset Bundle 和 GitLab 构建 CI 管道

If you are using Databricks serving endpoint, and you wish to export metrics to Datadog, you can face with some challenges in Datadog documentation. This article will help you to solve them, at least it will help you to make a PoC. Here are list of links which you can use: https://docs.databricks.com/en/machine-learning/model-serving/metrics-export-serving-endpoint.html https://docs.datadoghq.com/agent/basic_agent_usage/ubuntu/?tab=agentv6v7 Databricks serving endpoint is a service which allows you to deploy your model and serve it. It is a part of Databricks MLflow. You can find more information about it here: https://docs.databricks.com/applications/mlflow/model-serving.html Once you created an endpoint, out of the box you can see some metrics in Databricks UI. Also list of metrics can be received via REST API: curl --request GET "https://${DATABRICKS_HOST}/api/2.0/serving-endpoints/{endpoint_name}/metrics" --header "Authorization: Bearer ${DATABRICKS_TOKEN}" But if you want to export them to Datadog, you need to install and configure Datadog agent. Pre-requisites Created Databricks serving endpoint Bearer PAT token behalf of Databricks user/service principal, which has `CAN_QUERY` permission on the endpoint Datadog API key Enabled OpenMetrics integration in your Datadog account, link: https://app.datadoghq.com/integrations Install Datadog agent Datadog offers several ways to install agent, they are listed here: , however looks like some of them are not working, or at least I was not able to make them work. That is why I decided to use the most simple way — install agent on Docker. https://app.datadoghq.com/account/settings/agent/latest?platform=overview Before agent installation, you need to create a file `datadog.yaml`, which should be located ` `, with the following content: /etc/datadog-agent/datadog.yaml api_key: "__DD_API_KEY__"
site: "datadoghq.com"
hostname: "__YOUR_HOST_NAME__" Then you need to create `metric-config.yaml` file, which should be located ` `, with the following content: /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default instances:
 - openmetrics_endpoint: https://__DATABRICKS_HOST__/api/2.0/serving-endpoints/__ENDPOINT_NAME__/metrics

   metrics:
     - cpu_usage_percentage:
         name: cpu_usage_percentage
         type: gauge
     - mem_usage_percentage:
        name: mem_usage_percentage
        type: gauge
     - provisioned_concurrent_requests_total:
        name: provisioned_concurrent_requests_total
        type: gauge
     - request_4xx_count_total:
        name: request_4xx_count_total
        type: gauge
     - request_5xx_count_total:
        name: request_5xx_count_total
        type: gauge
     - request_count_total:
        name: request_count_total
        type: gauge
     - request_latency_ms:
        name: request_latency_ms
        type: histogram

   tag_by_endpoint: false

   send_distribution_buckets: true

   headers:
     Authorization: Bearer __DATABRICKS_TOKEN__
     Content-Type: application/openmetrics-text The full possible options for ` ` can be found in template file which is located in a container under ` ` conf.yaml.default /opt/datadog-agent/agent/conf.d/openmetrics.d/ I’ve created the next , because I have some difficulties with that one which is provided in Datadog documentation: Dockerfile FROM ubuntu:20.04

# Set environment variables
ENV DD_API_KEY=[SPECIFY_DD_API_KEY]
ENV DD_SITE=datadoghq.com
ENV DD_HOSTNAME=datadog-agent-ubuntu
ENV DATABRICKS_HOST=[SPECIFY_DATABRICKS_HOST]
ENV DATABRICKS_TOKEN=[SPECIFY_DATABRICKS_TOKEN]
ENV ENDPOINT_NAME=[SPECIFY_ENDPOINT_NAME]

# Install curl
RUN apt-get update && apt-get install -y curl

# Copy the datadog.yaml file into the Docker image
COPY datadog.yaml /etc/datadog-agent/datadog.yaml
RUN sed -i -e "s@__DD_API_KEY__@$DD_API_KEY@g" /etc/datadog-agent/datadog.yaml

# Install the Datadog agent
RUN bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

# Copy the metrics-config.yaml file into the Docker image
COPY metrics-config.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN sed -i -e "s@__DATABRICKS_HOST__@$DATABRICKS_HOST@g; s@__ENDPOINT_NAME__@$ENDPOINT_NAME@g; s@__DATABRICKS_TOKEN__@$DATABRICKS_TOKEN@g; " /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN service datadog-agent restart

# Run a process that doesn't exit
CMD ["tail", "-f", "/dev/null"] Usage Run serving endpoint Define environment variables in `Dockerfile’ Build docker image Run docker container # Step 1: Build the Docker image
# Navigate to the directory containing the Dockerfile
cd /path/to/directory/with/dockerfile

# Build the Docker image
# Replace 'my-image' with the name you want to give to your Docker image
docker build -t datadoc-agent-image .

# Step 2: Run a container from the built image
# Replace 'my-container' with the name you want to give to your Docker container
docker run --name datadog-agent -d datadoc-agent-image

# Step 3: Connect to a container and restart agent
docker exec -it datadog-agent  bash
service datadog-agent restart Some usefully commands for DataDog agent: datadog-agent status
datadog-agent health

service datadog-agent start
service datadog-agent restart Also appears . here

How to Export Metrics from Databricks Serving Endpoint to Datadog

Introduction Databricks is emerging as one of the main players in the MLOps and DevOps world. In the last month, I experienced this as part of my project at Areto and decided to write a series of hands-on blog posts on how to implement a MLOps Pipeline with Databricks. This is an ongoing learning process for me as well. I write each blog post after I experiment with different features and tools. So, I like to see this blog series as collective learning. Any comments and suggestions are welcome. In this blog post, we'll explore how to build a pipeline using the Databricks DBX tool and GitLab. This guide is designed for data engineers and scientists who are looking to streamline their data processing workflows. We'll cover key concepts, practical applications, and step-by-step instructions to help you integrate these tools effectively into your data projects. Continuous Integration (CI) CI/CD (Continuous Integration/Continuous Delivery): Continuous Integration and Continuous Delivery/Deployment (CI/CD) have become fundamental in modern software development practices. At its core, CI/CD is a method that emphasizes short, frequent updates to software through automated pipelines. While traditionally associated with software development, its principles are increasingly being adopted in data engineering and data science. The essence of CI/CD lies in its ability to automate the stages of software development, particularly in building, testing, and deploying code.  Continuous Integration begins with regular code commits to a shared repository branch, ensuring collaborative development without version conflicts. Each commit undergoes an automated build and test process, validating changes to enhance the quality and reliability of the final product. Databricks: Databricks has emerged as a premier cloud-based platform, uniting the realms of data engineering, machine learning, and analytics. Databricks excels in handling large-scale data processing and complex analytical tasks, but to leverage its full potential, teams need to navigate its multifaceted environment effectively. Integrating CI/CD processes within Databricks environments can streamline workflows, ensuring consistent, reproducible, and scalable data operations. This integration is crucial for teams aiming to maintain agility and efficiency in their data projects, enabling them to deliver reliable, high-quality data solutions consistently. Databricks Workflows: Databricks workflows are particularly significant in the context of CI/CD, as they offer a robust platform for automating and orchestrating data pipelines crucial to continuous integration and deployment. These workflows allow for the seamless scheduling and execution of tasks, which is essential for maintaining the frequent, automated update cycles characteristic of CI/CD. In a CI/CD pipeline, Databricks workflows can be used to automatically process and test large datasets, ensuring that data transformations and analyses are consistently accurate and up-to-date. The diversity of tasks that can be executed within Databricks workflows, such as Notebooks for exploratory data analysis, Python scripts for structured data processing, and Python wheels for custom package deployment, aligns well with the varied needs of CI/CD pipelines. By integrating these tasks into CI/CD workflows, data teams can ensure that every aspect of their data processing and analysis is continuously tested and integrated into the larger data strategy. This integration is key for developing resilient, scalable, and efficient data operations, enabling teams to deliver high-quality, reliable data products rapidly. Databricks CLI Extension (DBX): The Databricks CLI Extension, or DBX, is a pivotal tool in integrating Databricks with CI/CD pipelines, enhancing the automation and management of data workflows. The ability to programmatically control and manipulate data processes is crucial to implementing a CI/CD workflow, and DBX fills this role effectively. It provides a command-line interface for interacting with various Databricks components, such as workspaces, workflows, and clusters, facilitating seamless integration into automated pipelines. Hands-on In our hands-on example, we will create a (very) minimal project that uses dbx to deploy and run a Databricks workflow to manipulate, analyze, and test some data. This practical exercise showcases how Databricks can be used for real-world data analysis tasks. Our CI pipeline will play a crucial role in this process, as it will automate the deployment of our code as a Databricks workflow and validate its output. This validation step is vital to ensure the accuracy and reliability of our analysis. By the end of this exercise, you'll have a clear understanding of how to set up and run a Databricks workflow within a CI pipeline and how such a setup can be beneficial in analyzing and deriving insights from large datasets. This example will not only demonstrate the technical application of Databricks and CI principles but also offer a glimpse into the practical benefits of automated data analysis in a business context. Development pattern using Databricks GUI There are various development patterns to implement our CI pipeline. The common development workflows with dbx are to develop, test, and debug your code on your local environments and then use dbx to batch-run the local code on a target cluster. By integrating a remote repository with Databricks, we use a CI/CD platform such as , , or to automate running our remote repo’s code on our clusters. GitHub Actions Azure DevOps GitLab In this tutorial, for the sake of simplicity, we use Databricks GUI to develop and test our code. We follow these steps: Create a remote repository and clone it into our Databricks workspace. We use here. Gitlab Develop the program logic and test it inside the Databricks GUI. This includes Python scripts to build a Python Wheel package, scripts to test the data quality using Pytest, and a notebook to run the Pytest. Push the code to GitLab. The git push will trigger a Gitlab Runner to build, deploy and launch our workflows on Databricks using dbx Git integration As the first step, we . Next, we create a remote repository and . To allow our Gitlab runner to communicate with Databricks API through dbx, we should add two environment variables, and to our CI/CD pipeline configurations. configure Git credentials & connect a remote repo to Databricks clone it to our Databricks repo DATABRICKS_HOST DATABRICKS_TOKEN to generate a Databricks token, in your Databricks, go to User Settings →  Developer → Access tokens → manage → Generate new token. The Databricks host is the URL when you login into your Databricks workspace. It would be something like . The last part of the URL is your workspace id, and you should ignore that. https://dbc-dc87ke16-h5h2.cloud.databricks.com/ Finally we add the token and host to our Gitlab CI/CD setting. For that open your repo in Gitlab, go to Settings→ CI/CD → Variables → Add variables The project skeleton the project is structured into several folders and key files, each serving a specific purpose: .dbx folder: : defining the configuration of your DBX project. It contains environment settings, dependencies, and other project-specific parameters. project.json conf folder: : outlining the deployment configurations and environment settings. It defines workflows and their respective parameters and cluster configurations. deployment.yml : our wheel package. This folder includes: my_package folder A subfolder containing the main ETL task script . The ETL task load our dataset and create two new tables. tasks sample_etl_job.py The file. it includes common utilities that provide access to components such as common.py SparkSession Contains two Jupyter notebooks: notebooks folder: : plot the distribution of different features in our dataset explorative_analysis : used to execute for unit testing run_unit_test pytest This folder is dedicated to testing: tests folder: : includes pytest fixtures conftest.py : contains unit tests to validate the data structure in our tables. test_data.py In addition to these folders, there are two important files in the root directory: : the configuration file for GitLab's Continuous Integration (CI) service, defining the instructions and commands the CI pipeline should execute. .gitlab-ci.yml : for building my Python wheel package. It defines the package's metadata, dependencies, and build instructions. setup.py dbx-tutorial/
├─ .dbx/
│  ├─ project.json
├─ conf/
│  ├─ deployment.yml
├─ my_package/
│  ├─ tasks/
│  │  ├─ __init__.py
│  │  ├─ sample_etl_job.py
│  ├─ __init__.py
│  ├─ common.py
├─ tests/
│  ├─ conftest.py
│  ├─ test_data.py
├─ notebooks/
│  ├─ explorative_analysis
│  ├─ run_unit_test
├─ .gitlab-ci.yml
├─ setup.py Deployment configuration file is crucial for defining your DBX project's configuration. This file could be generated automatically when you run . Check the dbx website for more details about the . project.json dbx --init Profile file reference The “ ” option is used for local development. If you run the command inside the CI tool, you need to specify the and environment variables, and they will the profile variable. Since we don’t want to use the local environment and develop and run our code on Databricks UI, we don’t need to specify this option here. Instead, the dbx will pick them up from the CI/CD setting that we set earlier. profile dbx databricks host databricks token overwrite Your files that dbx will automatically upload to the system will be stored in They will be stored in .  You can read more about this in ml experiment. artifact_location dbx documentation {
    "environments": { 
        "default": {
            "profile": "dbx-tutorial",
            "storage_type": "mlflow",
            "properties": {
                "workspace_directory": "/Shared/dbx/dbx-tutorial", 
                "artifact_location": "dbfs:/Shared/dbx/projects/dbx-tutorial"
            }
        }
    },
    "inplace_jinja_support": false,
    "failsafe_cluster_reuse_with_assets": false,
    "context_based_upload_for_execute": false
} outlines the deployment configurations and environment settings. Here, we use and format to define the workflows. Each workflow defines an object inside the Databricks workspace: or deployment.yml 2.1 API wheel_task job delta live tables Pipeline. custom:
  existing_cluster_id: &existing_cluster_id
    existing_cluster_id: "1064-xxxxxx-xxxxxxxxx"

environments:
  default: #this should be same envoriment name as you defined in the project.json
    workflows:
      - name: "etl_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            python_wheel_task:
              package_name: "my_package"
              entry_point: "etl_job" # take a look at the setup.py entry_points section for details on how to define an entrypoint
          - task_key: "eda"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/explorative_analysis"
              source: WORKSPACE
            depends_on:
              - task_key: "main"    
      - name: "test_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/run_unit_test"
              source: WORKSPACE
            libraries:
            - pypi:
                package: pytest Here, we define two workflows: : consists of two tasks/jobs. "etl_job" The first task ( ) is of type . It used to run ETL task using the entry point that we define in the file. main python_wheel_task setup.py The second task ( ) is of type . It is used to run the notebook after the main tasks run successfully. Notice the use of property. etl notebook_task explorative_analysis depends_on : consists of only one job of type . It is used to run the notebook that is "test_job" notebook_task responsible for running the pytest. : in the above example, we run the workflow on an existing cluster. You can also create and run a new cluster for every deployment and launch of your work. For that, you should change the code as follows: Note custom:
  basic-cluster-props: &basic-cluster-props
    spark_version: "11.3.x-cpu-ml-scala2.12"

  basic-static-cluster: &basic-static-cluster
    new_cluster:
      <<: *basic-cluster-props
      num_workers: 1
      node_type_id: "Standard_E8_v3"

environments:
  default: 
    workflows:
      - name: "et_job"
        tasks:
          - task_key: "main"
            <<: *basic-static-cluster
						.... CI pipeline configuration Your GitLab CI pipeline is structured to automate the testing and deployment processes of your Databricks project. It consists of two main stages: and . In the stage, the runs the unit tests and deploys a separate workflow for testing. The stage, activated upon successful completion of the test stage, handles the deployment of your main ETL workflow. In general, the pipeline follows these steps: test deployment test unit-test-job deploy Build the project Push the build artifacts to the Databricks workspace Install the wheel package on your cluster Create the jobs on Databricks Workflows Run the jobs image: python:3.9

stages:          # List of stages for jobs, and their order of execution
  - test
  - deploy

unit-test-job:   # This job runs in the test stage.
  stage: test    # It only starts when the job in the build stage completes successfully.
  script:
    - echo "Running unit tests... This will take about 60 seconds."
    - echo "Code coverage is 90%"
    - pip install -e ".[local]"
    - dbx deploy --deployment-file conf/deployment.yml test_job --assets-only
    - dbx launch test_job --from-assets --trace

deploy-job:      # This job runs in the deploy stage.
  stage: deploy  # It only runs when *both* jobs in the test stage complete successfully.
  script:
    - echo "Deploying application..."
    - echo "Install dependencies"
    - pip install -e ".[local]"
    - echo "Deploying Job"
    - dbx deploy --deployment-file conf/deployment.yml etl_job
    - dbx launch etl_job --trace
    - echo "Application successfully deployed."
    - echo "remove all workflows." For workflow deployment and lunch, we use two commands: : to deploy the workflow definitions to the given environment. dbx deploy : to launch the given workload by its name in a given environment. dbx lunch In the we use the flag to avoid creating a job definition on our Databricks workflows during deployment. For the lunch command, we should use the flag. If this flag is provided, the launch command will search for the latest deployment that was assets-only. Lunching this, we can not see anything in the Workflows UI, but it will be a non-titled job  running by submit API. You can see this on the Job Runs page **** of your Databricks workspace. Check out the dbx documentation to read about and unit-test-job --assets-only --from-assets assest-based workflows other commands If you check the “Experiments” page of your Databricks workspace, you will find the build artifacts of each run of your CI pipeline. Databricks Asset Bundles currently, Databricks recommends using Asset Bundles for CI/CD and offers a migration guide from dbx to bundles.  If you understand the concepts in this post, it would be easy to switch to bundles. In the next post, I am going to explain how we can convert this project to an asset bundle project. Resources for this tutorial. Make sure you update the file with the correct and Repository conf/deployment.yml cluster_id DB username dbx documentation Databricks Best Practices: Development loop and CI/CD on Databricks with dbx Databricks Jobs API CI/CD techniques with Git and Databricks Repos What is CI/CD on Azure Databricks?

Building a CI Pipeline with Databricks dbx Tool and GitLab

In this post, we will look at creating an Azure data factory with a pipeline that loads Office 365 event data incrementally based on change data capture (CDC) information in the source of Change Data Feed(CDF) of a Delta lake table to an AWS S3 bucket. What we’ll cover: Create an ADF Pipeline that loads Calendar events from Offfice365 to a Blob container. Run a Databricks Notebook with the activity in the ADF pipeline, transform extracted Calendar event and merge to a Delta lake table. Load Change Data Feed on the Delta lake table to an AWS S3 bucket. The following diagram shows the architecture of the above steps: Prerequisites To build the pipeline, there are a few prerequisites that need to be met: Setup an Azure account. Register an AD application with Microsoft Graph permission. . Configure Microsoft Graph data connect consent request approver group t. Create an Azure Storage Accoun Create an AWS S3 bucket. Before going any further, we will need to create an Azure Storage account and grant the Azure AD application we registered the proper access to the container of the storage account. Check out the step-by-step Azure for Authorizing access to blobs using Azure Active Directory. guide 1. Create an Azure Data Factory Pipeline In this step, we will create an ADF Pipeline that loads Office365 Calendar Views data to an Azure Blob container. I will assume you already have an Azure Subscription and Resource Group setup. To create an Azure data factory, run the following command: az datafactory create --resource- --factory-name ADFDemo group RESOURCE_GROUP 1.1 Create linked services and datasets To connect to Office 365, we need to use the following command to create a linked service for Office 365 connection: az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name Offic365LinkedService - - properties @offic365LinkedService . json Replace APP_CLIENT_ID , TENANT_ID and APP_CLIENT_SECRET with corresponding values from your Active Directory tenant and App. Next, create a Calendar dataset named calendarDataset to reference the Calendar view data from Office 365. az datafactory dataset create - - resource - group RESOURCE_GROUP - - dataset - name calendarDataset - - factory - name ADFDemo - - properties @calendarDataset . json Now, let’s create another Linked Services for Blob Storage Account to store calendar data extracted from Office 365. az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name AzureStorageLinkedService - - properties @azureStorageLinkedService . json ECRET Replace YOUR_CONTAINER , APP_CLIENT_S , TENANT_ID and APP_CLIENT_SECRET with corresponding values from your Active Directory tenant, App, and storage account. Next, create a binary dataset named . outputDataset az datafactory dataset create - - resource - group RESOURCE_GROUP - - dataset - name outputDataset - - factory - name ADFDemo - - properties @outputDataset . json Note that @dataset.outputPath in the folderPath section is a global variable that we will set in the next step. 1.2 Create an ADF pipeline In this section, we will use the Azure CLI command to create a simple ADF pipeline with three activities: to set a global variable . SetVariableActivity outputPath named to load data from Office 365 to Azure blob storage. CopyActivity ExtractCalendarView to throw an error when the pipeline fails. FailActivity Enter the following commands to create ADF Pipeline with activities: az datafactory pipeline create --resource-group RESOURCE_GROUP --factory-name ADFDemo --name DemoPipeline  --pipeline @demoPipeline.json Once the pipeline is created, Navigate to the Azure portal and launch the Azure Data Factory studio. We can view the pipeline activities we have created from the pipeline designer. 2. Create Notebook in Azure Databricks In this section, we will create a notebook in Azure Databricks. You can refer to this to create a notebook in the Azure Databricks workspace. tutorial Before starting the notebook, we have to set up a connection from Azure Databricks to Azure Storage Account using a SAS key. Check out this for creating a Databricks secret scope. tutorial We are going to perform the following steps in the notebook: Mount a Blob storage container. Transform calendar view data. Create an SCD Type 1 delta table with Change Data Feed enabled. Merge input data frame to the delta table. Configure AWS S3 connection, and load change feed data to s3 bucket. The notebook would look similar to this: Now we have our notebook created, Let’s move back to step 1.1 to create linked services for Azure Databricks connection. az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name databricksLinkedService - - properties @databricksLinkedService . json Next, delete the pipeline that we created in step 1.2, recreate it to execute the Databricks notebook against the Databricks jobs cluster. Delete the existing pipeline we have created: az datafactory pipeline delete - - factory - name - - name ADFDemo - - resource - group RESOURCE_GROUP then recreate the pipeline using the following command: az datafactory pipeline create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - name DemoPipeline - - pipeline @demoPipeline . json Now we should have our pipeline that looks like the following diagram. Let’s publish entities (linked services and pipeline) to the Azure Data Factory service using ADF studio. and trigger the pipeline. We can log on to the Azure Databricks workspace, go to Clusters and see the Job status as . pending execution, running, or terminated After the pipeline run is completed, log on to the AWS S3 console, we should see the CSV file in the following format is loaded to the destination s3 bucket. Conclusion In this article, I introduced the approach to load data incrementally from Office365 to an AWS S3 bucket using ADF and Azure Databricks. I hope you found this article useful. You can find the ARM template and notebook in my . GitHub repository

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Introduction The Data space is almost as volatile as the AI space this year, with many players consolidating. In the summer of 2022, I wrote about The Data Table Format Wars, and the ecosystem has only gotten more interesting since then. I won’t get deep into the weeds on Data Lakes vs. Data Warehouse, etc., but suffice it to say, we are seeing everything coalesce around decoupling data storage from computing. Table formats are a way to standardize lightly opinionated storage. As I opined a couple of years ago, Apache Iceberg seems to be where they are settling. These formats tend to use Apache Parquet on object storage and provide a method for tracking new data and data state changes. In 2023, Databricks announced the open-sourcing of their Unity Catalog, and Onehouse announced OneTable (now X-Table). These projects are meant to give you a single interface to the three popular table formats: Delta, Hudi, and Iceberg. Then, June 2024 happened… What Happened? Snowflake was having its annual Snowflake Summit conference, and on June 3, 2024, they announced Apache Polaris (incubating), a vendor-neutral, open catalog for Apache Iceberg. The next day, and a week before the Databricks conference, Databricks announced their acquisition of Tabular, a company founded by the creators of Iceberg. This still seems weird to everyone from the outside because Iceberg is a spec; it isn’t a product, and Tabular isn’t even the main contributor to Iceberg. In my opinion, Databricks will somehow roll their Deltalake into Iceberg. Behind the scenes, Snowflake had been trying to acquire Tabular. Something happened at the 11th hour, and Databricks came in and laid down a ton more money, and the Snowflake bid was discarded. Snowflake was already two years into their commitment to supporting Iceberg as external storage, which made a lot of sense for them regardless. Snowflake has a reputation concerning the release speed after an announcement, so there was some skepticism, and some even wondered if any code had been written. It turns out Snowflake had been working on this for a while, and it was made available in very short order, with vendors like Dremio, Starburst, CelerData, and Upsolver showing solutions using it in less than two months: a delightful surprise, and a positive sign of things to come. All of that is a long way to set up Apache Polaris, and hopefully, it will give you a sense of the importance that a good catalog plays in all of this. How will any of your compute tools work if you don’t have a unified system to describe all those parquet files in your data lake? Polaris Overview Apache Polaris is an open-source catalog implementation for Apache Iceberg tables. Its main features include: REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines.
Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog.
Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines.
Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables.
Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage.
Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution.
Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality.
Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs.
Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place.
Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. And a lovely little graphic from the docs to put a picture on everything. Borrowing from the Polaris docs for a picture once again, let’s dive into what is happening here in this entity hierarchy graphic: We can have multiple catalogs in a single Polaris install; in this diagram, it is only Catalog1 that has any namespaces or tables defined. A single catalog can be either internal or external. The internal tables allow for read/write functionality, meaning you have full control. The external tables are currently read-only and are intended to mirror data that is in some external catalog, such as a Glue catalog, which Polaris will see as the source of truth once connected. The catalog organizes your iceberg tables; you’ll want to configure it with your storage configuration for the applicable cloud system that you are on. This forms the first architectural layer in the Iceberg table spec. A namespace can have an arbitrary level of nesting and is used to logically group Iceberg tables within a catalog. An Iceberg table will always belong to a namespace. With this configuration in place, you can then use the Polaris catalog to describe your Iceberg tables in such a way as to make them easily accessible and manageable using a large variety of tools. That’s a quick overview of what is happening and what it looks like. The implementation inside of Snowflake provides a really handy UI to manage the implementation; other vendors have their own, and you can just do it through Spark if you like. Summary I continue to believe that Apache Iceberg is going to “win” the table format wars. Apache Polaris focusing on Iceberg is not a bad thing. I want to dig into how RBAC has been implemented in Polaris another time and see if it is as robust as what the Tabular product implemented, which I was a big fan of. I believe Polaris is a big win for the Iceberg community and will help accelerate adoption even further. I’ve seen talk about the Dremio Nessie project getting rolled in, and I’m curious to see what that will look like. I haven’t looked at Nessie in a couple of years, and I thought their git-like semantics mainly had been covered by the Iceberg tagging and branching, which I wrote about last year—yet another story for another day. I’m a big fan of what is happening here. If you are in this space, this is the right train to jump on. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB?
What the Heck Is Malloy?
What the Heck is PRQL?
What the Heck is GlareDB?
What the Heck is SeaTunnel?
What the Heck is LanceDB?
What the heck is SDF?
What the Heck is Paimon?
What the Heck is Proton?
What the Heck is PuppyGraph?
What the Heck is GPTScript?
What the Heck is WarpStream? Introduction Introduction The Data space is almost as volatile as the AI space this year, with many players consolidating. In the summer of 2022, I wrote about The Data Table Format Wars , and the ecosystem has only gotten more interesting since then. I won’t get deep into the weeds on Data Lakes vs. Data Warehouse, etc., but suffice it to say, we are seeing everything coalesce around decoupling data storage from computing. The Data Table Format Wars Table formats are a way to standardize lightly opinionated storage. As I opined a couple of years ago, Apache Iceberg seems to be where they are settling. These formats tend to use Apache Parquet on object storage and provide a method for tracking new data and data state changes. In 2023, Databricks announced the open-sourcing of their Unity Catalog, and Onehouse announced OneTable (now X-Table). These projects are meant to give you a single interface to the three popular table formats: Delta, Hudi, and Iceberg. Then, June 2024 happened… announced announced What Happened? Snowflake was having its annual Snowflake Summit conference, and on June 3, 2024, they announced Apache Polaris (incubating), a vendor-neutral, open catalog for Apache Iceberg. The next day, and a week before the Databricks conference, Databricks announced their acquisition of Tabular, a company founded by the creators of Iceberg. This still seems weird to everyone from the outside because Iceberg is a spec; it isn’t a product, and Tabular isn’t even the main contributor to Iceberg. In my opinion, Databricks will somehow roll their Deltalake into Iceberg. announced announced Behind the scenes, Snowflake had been trying to acquire Tabular. Something happened at the 11th hour, and Databricks came in and laid down a ton more money, and the Snowflake bid was discarded. Snowflake was already two years into their commitment to supporting Iceberg as external storage, which made a lot of sense for them regardless. Snowflake has a reputation concerning the release speed after an announcement, so there was some skepticism, and some even wondered if any code had been written. It turns out Snowflake had been working on this for a while, and it was made available in very short order, with vendors like Dremio, Starburst, CelerData, and Upsolver showing solutions using it in less than two months: a delightful surprise, and a positive sign of things to come. All of that is a long way to set up Apache Polaris , and hopefully, it will give you a sense of the importance that a good catalog plays in all of this. How will any of your compute tools work if you don’t have a unified system to describe all those parquet files in your data lake? Apache Polaris Polaris Overview Apache Polaris is an open-source catalog implementation for Apache Iceberg tables. Its main features include: REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines. Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog. Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines. Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables. Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage. Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution. Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality. Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs. Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place. Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines. Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog. Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines. Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables. Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage. Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution. Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality. Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs. Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place. Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. And a lovely little graphic from the docs to put a picture on everything. Borrowing from the Polaris docs for a picture once again, let’s dive into what is happening here in this entity hierarchy graphic: We can have multiple catalogs in a single Polaris install; in this diagram, it is only Catalog1 that has any namespaces or tables defined. A single catalog can be either internal or external. The internal tables allow for read/write functionality, meaning you have full control. The external tables are currently read-only and are intended to mirror data that is in some external catalog, such as a Glue catalog, which Polaris will see as the source of truth once connected. The catalog organizes your iceberg tables; you’ll want to configure it with your storage configuration for the applicable cloud system that you are on. This forms the first architectural layer in the Iceberg table spec. A namespace can have an arbitrary level of nesting and is used to logically group Iceberg tables within a catalog. An Iceberg table will always belong to a namespace. With this configuration in place, you can then use the Polaris catalog to describe your Iceberg tables in such a way as to make them easily accessible and manageable using a large variety of tools. That’s a quick overview of what is happening and what it looks like. The implementation inside of Snowflake provides a really handy UI to manage the implementation; other vendors have their own, and you can just do it through Spark if you like. Summary I continue to believe that Apache Iceberg is going to “win” the table format wars. Apache Polaris focusing on Iceberg is not a bad thing. I want to dig into how RBAC has been implemented in Polaris another time and see if it is as robust as what the Tabular product implemented, which I was a big fan of. I believe Polaris is a big win for the Iceberg community and will help accelerate adoption even further. I’ve seen talk about the Dremio Nessie project getting rolled in, and I’m curious to see what that will look like. I haven’t looked at Nessie in a couple of years, and I thought their git-like semantics mainly had been covered by the Iceberg tagging and branching, which I wrote about last year—yet another story for another day. wrote about I’m a big fan of what is happening here. If you are in this space, this is the right train to jump on. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is WarpStream? What The Heck Is DuckDB? What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is LanceDB? What the Heck is LanceDB? What the heck is SDF? What the heck is SDF? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Proton? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is GPTScript? What the Heck is WarpStream? What the Heck is WarpStream?

What The Heck is Apache Polaris?

This summer both Databricks and Apache Iceberg rolled out enhancements to their open table formats. Databricks announced can read and write data to all of the most popular open table formats: Delta Table, Iceberg and Apache Hudi. Delta Universal Format (UniForm) makes it so that open table formats become interoperable, avoiding the need for creating and storing extra copies of data in this or that format. Data teams using existing query engines like , , and others to query over Iceberg or Hudi files can read Delta tables directly without conversion. Delta Lake 3.0 DuckDB Dremio Around the same time, Iceberg announced a slew of new support for query engines and platforms including , , and StarRocks. With these announcements from Databricks and Iceberg, interoperability joined hand-in-hand with data portability. Open table formats by design promote the concept that you should be able to access, control, share, and operate on your data with whatever tool you want, wherever you want to, whether that is in the public clouds, in your private clouds, on-edge, or on bare-metal. Snowflake AWS Athena Apache Doris Understanding Open Table Formats Let’s put these announcements in context. Open table formats allow data lakes to attain performance and compliance standards that in the past could only be achieved by traditional data warehouses or databases, all the while preserving the flexibility of a data lake environment. There are three major open table formats: was originally designed by Netflix specifically for handling substantial data volumes within data lakes. This open table format boasts distinctive features such as time travel, dynamic schema evolution, and partition evolution. These capabilities make it revolutionary, enabling concurrent and secure operations by query engines on the same dataset. Iceberg is an open-source storage framework in the Lakehouse architecture that empowers data lakes on object storage like MinIO. It ensures ACID transactions, scalable metadata handling, and unified processing for Apache Spark, offering reliability and scalability. Delta Lake can handle the performance and correctness challenges of complex Spark workloads, especially under heavy concurrency, with non-atomic updates and metadata operations causing significant bottlenecks. Delta Lake is rooted in the Hadoop ecosystem, and Hudi's primary purpose is to decrease latency during the ingestion of streaming data, offering features such as tables, transactions, upserts/deletes, advanced indexes, and compatibility with various storage implementations, including cloud-native object storage like MinIO. Hudi Much has been written about choosing between different formats, with some asserting up to among the three primary Open Table formats. This blending of distinctions makes sense given the environment of interoperability in which these open table formats were created and continue to thrive. The creators of these formats prioritized capability over traditional notions of vendor lock-in and operational control. 80% functional equivalence Open Table Formats as Part of the Modern Data Stack Even before these recent announcements, open table formats had already become integral to modern data lake design. And reciprocally, data lakes have been integral to the modern data stack. A recent by found that 70% of respondents said that more than half of their analytics is or would be in a data lake within three years. This pervasive adoption signifies a paradigm shift in how organizations structure and manage their data, placing a strong emphasis on interoperability, flexibility and performance. survey Dremio It’s no surprise really that cloud-native data lakes and their components and technologies like open table formats have become center stage in the modern data stack. This stands in stark contrast to traditional, monolithic legacy hardware and software sold wholesale to organizations hoping to slap the phrase ‘cloud technology’ onto their aging systems. Becoming cloud-native is more than adding an API – the modern data stack is a modular and specialized ensemble of tools tailored for various data handling facets. It is built for adaptability, born in the  cloud and held to high-performance standards. Features that make the modern data stack a  compelling choice for organizations. The stack's modularity provides a range of options, allowing organizations to craft a bespoke data infrastructure that aligns with their specific needs, fostering agility in the continually evolving data landscape. Despite this continuously evolving range of options, there are defining characteristics that weave through the components of the stack: The modern data stack is designed to seamlessly scale across diverse cloud environments, ensuring compatibility with multiple clouds to prevent vendor lock-in. Cloud-Native: Engineered for efficiency, the stack incorporates components that take a software-first approach and design for performance. Optimized Performance: The stack establishes a standardized communication framework between its components. This promotes interoperability and supports the creation of microservices. RESTful API compatibility: The stack enables independent scaling of computational resources and storage capacity. This approach optimizes cost efficiency and enhances overall performance by allowing each aspect to scale according to specific needs. Disaggregated Storage and Compute: Beyond supporting open table formats, the modern data stack embraces openness in the form of open-source solutions. This commitment eliminates proprietary silos and mitigates vendor lock-in, fostering collaboration, innovation, and improved data accessibility. The dedication to openness reinforces the stack's adaptability across various platforms and tools, ensuring inclusivity. Commitment to Openness: Data Portability and Interoperability as a Business Standard Truly embracing data portability and interoperability means being able to create and access data wherever it is. This approach facilitates flexibility, allowing organizations to harness the capabilities of diverse tools without being constrained by either vendor lock-in or data silos. The goal is to enable universal access to data, promoting a more agile and adaptable data ecosystem within organizations. Understanding that the cloud as an operating model is built on principles of cloud-native technology rather than a specific location is critical to achieving data portability. Some organizations in this endeavor and attempt to buy their way into the cloud at a tremendous cost. The reality is that while cloud adoption presents an opportunity for the average company to , the real impact and true cost savings comes from embracing the cloud operating model on private infrastructure. struggle increase profitability by 20 to 30 percent Many established organizations are actively adopting this philosophy, choosing to repatriate workloads from the cloud and achieving substantial cost savings, with companies like , 37Signals, and a major enterprise security firm from cloud exits. The cloud operating model allows for the coexistence of seemingly contradictory ideas: companies can benefit from migrating to the cloud repatriating workloads. The key determinant is the adoption of the cloud operating model, fundamentally transforming how organizations approach infrastructure, development and technical efficiency. This model optimizes for flexibility, efficiency and long-term success – whether in the public cloud or beyond – and dovetails precisely with the concept of the modern data stack, enabling data portability and interoperability with open table formats. X.com saving an average 60% and Conclusion Recent strides in open table formats by Databricks,  Apache Iceberg and Hudi signify a pivotal moment in data management. Delta Lake 3.0's universal compatibility and expanded support for Apache Iceberg showcase a commitment by both data infrastructure companies and on the ground implementers to seamless data portability and interoperability. These developments align with the inherent modularity of the modern data stack, where open table formats play a central role in achieving performance and compliance standards. This shift is not isolated but intersects with the cloud operating model. Beyond the allure of public clouds, real impact and cost savings emerge by embracing the cloud operating model on private infrastructure. The confluence of open table formats, the modern data stack, and the cloud operating model signifies a transformative era in data management. This approach ensures adaptability across various environments, whether public or private, on-prem on edge. For those navigating data lake architecture complexities, our team at MinIO is ready to assist. Join us at hello@minio.io or on our channel for collaborative discussions as you embark on your data journey. Slack

What the Interoperability Trend in Open Table Formats Signify for Enterprise Data Architectures

এন্টারপ্রাইজ ডেটা আর্কিটেকচারের জন্য ওপেন টেবিল ফরম্যাটে ইন্টারঅপারেবিলিটি ট্রেন্ড কী বোঝায়

Was der Interoperabilitätstrend bei offenen Tabellenformaten für Unternehmensdatenarchitekturen bedeutet

Qué significa la tendencia de interoperabilidad en los formatos de tablas abiertas para las arquitecturas de datos empresariales

Ce que signifie la tendance à l'interopérabilité dans les formats de tables ouvertes pour les architectures de données d'entreprise

ओपन टेबल फॉर्मेट में इंटरऑपरेबिलिटी ट्रेंड एंटरप्राइज डेटा आर्किटेक्चर के लिए क्या दर्शाता है

オープン テーブル フォーマットの相互運用性トレンドがエンタープライズ データ アーキテクチャに何を意味するか

오픈 테이블 형식의 상호 운용성 추세가 엔터프라이즈 데이터 아키텍처에 미치는 영향

O que a tendência de interoperabilidade em formatos de tabelas abertas significa para arquiteturas de dados empresariais

Что означает тенденция совместимости в форматах открытых таблиц для архитектур корпоративных данных

Açık Tablo Formatlarındaki Birlikte Çalışabilirlik Trendi Kurumsal Veri Mimarileri İçin Ne Anlam Taşıyor?

Xu hướng tương tác trong các định dạng bảng mở có ý nghĩa gì đối với Kiến trúc dữ liệu doanh nghiệp

开放表格式的互操作性趋势对企业数据架构意味着什么

Stories, News & Mentions about DATABRICKS

Databricks’s Stories on HackerNoon

Databricks's News & Mentions