Databricks Fundamentals

-

+ Databricks is an integrated platform for data and AI that enables large-scale development, deployment, and management of data, analytics, and AI-driven solutions.

+ Founded in 2013 at University of California, Berkeley

+ Mission: democratize data and AI

+ 2020- Introduced Lakehouse

+ 2023- Rise of GenAI (Data +AI) 


Some Basic Terms:

Data Warehouse: Storage of Structured data for BI & reporting

Data Lakes: Storage of structured, semi & unstructured data used for almost all type of data suitable                         for DS,ML

we can't use both by connecting each other i.e; as both storage systems with same data to be ingested. People are copying the data to use for each application as for each  system the data governance and security rules are different

Data Lakehouse: (Data Warehouse + Data Lake) one platform to unify all data, analytics and AI workloads. one security and governance approach. created over the data lake

Data Intelligence: Data Lakehouse + Gen AI

implementing AI (Gen AI & Traditional AI) to understand the organizational data usage


Databricks Architecture:

                                            Unified Architecture 

data sources->(Ingest->Transform->Query->Visualize->Serve)->External Apps

                                                Governance and Security

Compute: selection of computing resources in databricks

Serverless- scale dynamically based on the load and efficient in time sensitive tasks, default compute in the databricks. removes the complexity of managing underlying computing resources


Data Governance: managing the data in an organization across it's life long

Key Elements- Data cataloging, Data classification, Auditing data entitlements and access, Data discovery, Data sharing & collaboration, Data lineage, Data security, Data quality.

Unity Catalog: Unified governance model for all the data & AI  assets within the databricks data intelligence programme


Data Ingestion: Lakeflow,ETL,Streaming

Lakeflow- connect - easily connect key data sources (No code, low maintenance design, auto scale)

DLT- Reliable data pipelines made easy (declarative pipelines with automatic incrementalization, batching and streaming

Jobs- unified orchestration for analytics and AI Automate data and AI workloads with real time triggers and more

Data Ingestion Techniques from cloud storage: converts raw file format to delta table format

CREATE TABLE AS (CTAS)- Batch Ingestion

+ read_files() - function reads files under provided location and returns the data in tabular form

+ can detect the file format automatically and infer a unified schema across all the files


COPY INTO - Incremental batch Ingestion

+ FROM specifies the cloud storage location continuously adding files

+ FORMAT_OPTIONS() - control how the source files are parsed and interpreted

+ COPY_OPTIONS() - controls th ebehaviour of COPY INTO itself, such as schema evolution and idempotency


AUTO LOADER - Incremental batch/Stream Ingestion

+ support for both python and SQL(automatically leverges DLT)



Data warehousing: Databricks SQL- collection of tools for BI and analytics

AI/BI: analyze and visualize data, Power BI and Tableau integration

Genie- users can directly interact with data by asking questions which are addressed directly. 

Mosaic AI:  Gen AI and DS/ML 

Model Serving- scalable deployment for traditional and LLM ML models- unified interface to manage all deployed models

Delta Sharing: unified platform sharing app to share data across platforms

Databricks Market Place- Databricks own place for all DS/ML data products based on Delta Sharing


Lakeflow: consists of Connect, declarative Pipeline, Jobs

Connect: getting data from all sources to the data intelligence platform

Connectors- Ingesting data into lakehouse by

+ Upload Files

+ Standard Connectors- cloud object storage, kafka 

+ Managed Connectors- SaaS applications, Databases - leverage efficient incremental reads and writes 


Ingestion Methods:

+ Batch- load data as batches of rows into databricks often based on a schedule. 1. 3 rows 2. 3+3 rows 3. 3+3+3 rows

+ Incremental- only new data is ingested, faster ingestion and better resource efficiency

+ Streaming- Continuously load data- suitable for Micro-batch processes


Delta Lake: open source protocol for reading and writing files to cloud storage. In delta lake we create delta tables consists of parquet files and logs with metadata

Medallion Architecture (Multi Hop)- Ingestion -> Data Processing and Transformation -> Consumers




Comments

Post a Comment

Popular posts from this blog

Understanding Titanic Survival Predictions with Decision Trees

-
Image
  Decision Tree: Supervised learning Algorithm Can do both Regression and Classification capabilities Inverted tree like structure with bunch of if-else statements The Titanic dataset is one of the most famous case studies in data science. It captures passenger details such as age, sex, class, fare, and family aboard, along with whether they survived or not. In this post, we’ll walk through how a decision tree model uses these features to predict survival, and what the visualization reveals. How the Decision Tree Works Note: These features are based on the importance table at the right side Root node: “Was the passenger male?” If female , survival chances are much higher. The tree then considers passenger class and fare to refine predictions. If male , survival chances drop significantly, but age and family aboard (siblings/spouses) still play a role. Each node in the tree shows: Gini impurity: How mixed the group is (lower / purer). Samples: Number of passengers in that group. ...
Read more

From Raw Logistics Data to Business Insights: An Excel Analytics Project (RFM Analysis)

-
Image
Analyzed logistics shipment data in Excel to evaluate delivery performance, cost, transit time, distance trends. Extended the analysis with RFM-based carrier segmentation to understand carrier utilization and spending patterns, supporting more data-driven operational decisions. My objective was to explore two practical business questions: How are delivery delays and shipment outcomes behaving across time and distance? How can carrier performance be evaluated using cost, transit time, shipment activity, and delivery performance? Starting with data quality I first reviewed the dataset for completeness and consistency. The initial cleaning process included identifying missing delivery dates and transportation costs, checking transit-day inconsistencies, and investigating numerical outliers. One thing I wanted to avoid was treating every unusual value as bad data. After null-value treatment, the working dataset contained 1,928 shipment records , while the dataset used after the selected ...
Read more

Unable to access the PaloAlto VM image via HTTPS

-
Image
Recently, I encountered an Issue of "Unable to access the PaloAlto VM image via HTTPS". Here's how I applied troubleshooting steps to resolve it: Configuration Changes done before: Configured static management IP & default gateway and rebooted the PA VM, then faced the issue. Observed: Unable to access the PaloAlto VM image via HTTPS. Unable to ping to the default Gateway from the PA and viceversa. Error- getting Destination Host Unreachable in both cases. Analyzed: + Checked the Management IP address and netmask, both are same as bridged network (physical) + Checked the default route "show routing route" - no route configured + While pinging from the base machine to PA and vice versa getting error as "Destination is Unreachable". Hypothesized: + As per the Error message, primarily issue is Network related, that the PA VM is unable to reach physical NIC IP. Tested: + Tried with changing the management IP address within the same subnet (as physic...
Read more