Home

sreeram - Senior Databricks Data Engineer

[email protected]

Location: Austin, Texas, USA

Relocation: Any state

Visa: H1B

Resume file: Sreeram_Databricks_Resume_1791398933692.docx Please check the file(s) for viruses. Files are checked manually and then made available for download.

Sreeram Kandula Email: [email protected]| Mobile: +1 972 678 0442 PROFESSIONAL SUMMARY: Seasoned Data Engineer with 15+ years of experience in building and delivering cloud-native data platforms, Lakehouse architectures, and enterprise analytics solutions. Expert in Databricks, PySpark, Delta Lake, Azure Data Factory, ADLS Gen2, Snowflake, Spark SQL, and distributed data processing. Experienced in designing Medallion Architecture (Bronze, Silver, Gold) and scalable ETL/ELT frameworks supporting batch and streaming workloads. Proven expertise optimizing Spark workloads, Delta Lake performance, and cloud resource utilization to improve scalability and reduce cost. Strong background in Data Governance, Unity Catalog, RBAC, Data Lineage, Security, and Enterprise Data Management. Enhanced data quality and consistency by applying schema validation, data cleansing, and enrichment techniques across different layers. Hands-on experience building AI-enabled data solutions leveraging LLMs, Vector Search, RAG architectures, and intelligent analytics platforms. Skilled in Azure DevOps, CI/CD automation, production support, incident management, and SLA-driven data operations. Experienced in architecting enterprise data platforms integrating structured, semi-structured, and unstructured data sources across Azure and AWS ecosystems. Experience with Power BI integration for real-time business intelligence dashboards and reporting. Proven track record leading cross-functional teams, mentoring engineers, and delivering high-impact data solutions aligned with business objectives. TECHNICAL SKILLS: Cloud Platforms: Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure Blob Storage, Azure Key Vault, Azure Data Explorer, Snowflake, AWS S3, AWS Glue, AWS Lambda Data Engineering: PySpark, Spark SQL, Delta Lake, Auto Loader, LakeFlow, Delta Live Tables (DLT), Kafka, Airflow, Python Storage & Data Lakes: ADLS Gen2, Azure Blob Storage, Delta Lake, Amazon S3 Integration & Orchestration: ADF, Informatica PowerCenter, IICS, ODI, REST APIs, Logic Apps Databases: Azure SQL Database, Azure Synapse, SQL Server, Oracle, T-SQL, PL/SQL Governance & Security: Unity Catalog, RBAC, Data Lineage, Data Quality Frameworks, Schema Enforcement DevOps & Automation: Azure DevOps, Git, CI/CD, Shell Scripting, Jira, ServiceNow, Flex Deploy BI & Analytics: Power BI, Tableau, OBIEE, OBIA, BI Publisher AI & Advanced Analytics: Azure OpenAI, RAG, Vector Search, Knowledge Graphs, AI Agents, LLM-powered Analytics CERTIFICATIONS: Databricks Certified Data Engineer Professional. Snowflake SnowPro Certified. Microsoft Certified Power BI Data Analyst Associate. Oracle Business Intelligence Foundation Suite 11g Certified. EDUCATION: B.Tech. (2005) in CSE from Acharya Nagarjuna University. WORK EXPERIENCE: Senior Databricks Data Engineer Selective Insurance, (Remote) June 2026 - Till Date Responsibilities: Architected and led Auto Fleet insurance claims lakehouse platform on Azure Databricks for premium pricing and fleet analytics. Designed and implemented metadata-driven ingestion frameworks integrating Pega365 APIs, flat files, and on-premises applications using Informatica IICS, LakeFlow Connect, ADLS Gen2, and Databricks, reducing onboarding effort for new data sources. Architected Medallion (Bronze, Silver, Gold) Lakehouse architecture using Delta Lake to support scalable, governed, and high-performance analytical workloads. Developed Auto Loader ingestion pipelines for claims and policy datasets. Designed Spark Declarative Pipelines and SCD Type 2 dimensions. Implemented data quality expectations, null validations, deduplication, and schema enforcement. Leveraged Photon engine and Delta optimization strategies to reduce processing times by 40% and improve workload efficiency. Processed multi-terabyte datasets containing millions of claim records. Implemented Azure DevOps CI/CD deployment pipelines. Partnered with Data Governance and Architecture teams to define governance standards, data quality rules, and compliance controls. Environment: Azure Databricks, Azure Data Lake Storage Gen2 (ADLS), Azure Blob Storage, Apache Spark, PySpark, Python, Spark SQL, LakeFlow Connect, Azure DevOps, Delta Lake, Medallion Architecture, SQL, Power BI, Active Directory, Informatica, IICS. Senior Data Engineer Signet, Dallas TX April 2024 - June 2026 Responsibilities: Designed and developed scalable Databricks data pipelines supporting low-latency ingestion, incremental processing, and high-volume analytical workloads. Architected an AI-powered self-service analytics platform leveraging Databricks LLM capabilities, enabling business users to generate SQL from natural language queries and retrieve governed insights from enterprise datasets. Integrated structured and unstructured data from APIs, JSON payloads, and third-party platforms into governed Delta Lake tables using Azure Data Factory, Databricks Auto Loader, and REST-based ingestion frameworks. Implemented Unity Catalog governance, RBAC, and lineage capabilities across enterprise Delta Lake environments. Developed CDC pipelines using Delta Lake MERGE strategies to support near real-time synchronization of source system changes. Led the architecture and optimization of enterprise-scale ETL/ELT solutions supporting both batch and streaming workloads using Azure Data Factory, Azure Databricks, and Apache Spark. Designed Medallion architecture (Bronze, Silver, Gold layers) to modularize raw ingestion, curated transformations, and business-level aggregations. Design and develop complex Azure Data Factory (ADF) pipelines for batch and near-real-time data ingestion from SQL Server, Oracle, Salesforce, and Azure Blob Storage. Architect and implement end-to-end data lakehouse solutions on Azure using ADF, Azure Databricks, Delta Lake, Unity Catalog, and Azure Data Lake Storage Gen2 along with governance standards. Implement data transformations using PySpark in Azure Databricks for deduplication, cleansing, joins, and aggregations across structured and semi-structured sources (CSV, JSON, Parquet). Resolve data skew, shuffle spills, and job failures by analyzing DAGs, Spark UI, and cluster metrics. Implement data quality checks and validation frameworks for datasets processed within Databricks. Design and develop scalable notebooks using PySpark, Delta Lake, and Unity Catalog to enable efficient data processing, ensure data quality, and enforce robust data governance. Collaborated with enterprise architects and data stewards to define governance standards, metadata management strategies, and data security frameworks. Leveraged Databricks Photon engine and Delta optimization techniques to improve query performance and reduce compute costs for large-scale analytical workloads. Implemented LakeFlow Connect pipelines to streamline ingestion of API, file-based, and enterprise source system data into the Databricks Lakehouse. Monitor and optimize data pipelines for performance and reliability, identifying and addressing any bottlenecks and inefficiencies. Partnered with business stakeholders to design data models, KPIs, and interactive dashboards in Power BI, improving decision-making speed by 20% and delivering actionable insights to leadership. Mentored junior data engineers and provided technical leadership on Databricks development standards, performance tuning, and production support activities. Implemented CI/CD pipelines in collaboration with DevOps, ensuring faster deployments and accelerating delivery. Environment: Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), Azure Blob Storage, Apache Spark, PySpark, Python, Spark SQL, Azure DevOps, SQL, Power BI, Active Directory, REST APIs. Senior Data Engineer Pacific Gas and Electric, Oakland CA January 2024 - March 2024 Responsibilities: Designed and implemented enterprise-scale ETL/ELT pipelines using Azure Databricks and Azure Data Factory, enabling reliable ingestion, transformation, and delivery of analytics-ready datasets. Implemented Delta Lake optimization strategies including partitioning, schema enforcement, compaction, and ACID-compliant transactions, significantly improving data reliability and query performance. Developed scalable batch and near real-time processing pipelines using Databricks, PySpark, Spark SQL, and Python to support enterprise reporting and advanced analytics workloads. Implemented data validation and quality checks to ensure accuracy, completeness, and consistency across downstream analytical datasets. Monitored, troubleshot, and resolved Databricks performance and stability issues, optimizing Spark configurations and pipeline execution. Collaborated closely with BI developers and analysts to design analytics-optimized data models, ensuring seamless consumption by Power BI dashboards. Environment: Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), PySpark, Python, Spark SQL, Informatica, Power BI. Senior Data Engineer Viavi, San Jose CA January 2018 - December 2023 Responsibilities: Designed and developed data pipelines to deliver QoQ and YoY aggregates, including bookings, billings, and backlog, powering Finance leadership dashboards for the VP and CFO. Integrated data from various sources, such as production lines, supply chain systems, order management system to cloud data warehouse. Designed and developed scalable ETL/ELT pipelines using Azure Data Factory and Databricks to support enterprise analytics and reporting workloads. Developed and optimized PySpark transformation frameworks in Databricks for processing large-scale datasets and improving workload performance. Designed Bronze, Silver, and Gold data layers to standardize ingestion, transformation, and business consumption patterns across the platform. Designed and deployed SCD Type 2 logic and incremental load strategies using PySpark and SQL. Designed and developed parameterized ADF pipelines using lookup, foreach, filter, and copy activities to orchestrate dynamic ETL processes across multiple environments. Implemented automated data quality and validation frameworks to ensure data accuracy, completeness, and consistency across downstream reporting systems. Integrated Azure Key Vault for managing secrets and credentials securely within pipelines and notebooks. Led migration of enterprise data assets from on-premises platforms to Azure Data Lake Storage (ADLS Gen2) using Azure Data Factory and Databricks, enabling scalable cloud-based analytics and reporting. Applied Delta caching, Spark caching, and query optimization techniques to improve workload performance and reduce Databricks compute costs. Designed Power BI semantic models and executive dashboards delivering actionable insights into financial KPIs, improving reporting efficiency and business visibility. Leveraged CI/CD automation using Azure DevOps pipelines for deploying ADF and Databricks artifacts across multiple environments. Design and develop new subject areas in OBIEE RPD and create new dashboards and reports in OBIEE Analytics. Implemented enterprise-level object and data-level security in OBIEE by defining roles, privileges, and row-level filters to ensure compliance with organizational governance. Developed complex Informatica PowerCenter mappings utilizing Lookup, Joiner, Router, and Expression transformations to support enterprise data warehouse requirements. Collaborated with business stakeholders, architects, and analytics teams to deliver scalable data warehousing and reporting solutions aligned with organizational objectives. Environment: Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), Azure Blob Storage, Delta Lake, Medallion Architecture, Apache Spark, PySpark, Python, Spark SQL, SQL, Azure Key Vault, Azure DevOps, Power BI. Lead Consultant HCL America Inc, Hyderabad, India / Sunnyvale CA March 2010 - December 2017 Clients: CSL Behring, Land O Lakes, Gilead Sciences, Marsh McLennan, Veritas Responsibilities: Delivered end-to-end BI and data warehousing solutions across Pharma, Insurance, Retail, and Manufacturing domains, supporting diverse business and analytical use cases. Designed and developed interactive Tableau dashboards by creating advanced calculated fields, parameters, and KPIs, enabling actionable insights for business stakeholders. Built Tableau workbooks using multiple heterogeneous data sources, leveraging data blending techniques to ensure unified and accurate reporting. Designed and developed OBIEE dashboards and reports with multiple analytical views, aligned to business and executive reporting needs. Created and enhanced OBIEE subject areas, dashboards, and repository objects to support scalable enterprise reporting. Provided design and development solutions for new analytical requirements, translating business needs into technical BI and ETL implementations. Implemented OBIEE security models by configuring Application Roles and Groups, ensuring secure, role-based access to dashboards and reports. Contributed to data warehouse design and OBIEE reporting architecture to support enterprise-wide analytics initiatives. Worked extensively on OBIA modules for multiple US-based clients, supporting packaged analytics implementations. Developed complex Informatica PowerCenter mappings to extract, transform, and load data efficiently, meeting stringent business and performance requirements. Designed Informatica mappings using transformations such as Joiner, Router, Expression, Lookup, and Source Qualifier to handle complex business logic. Creating database objects like Tables, Stored Procedures, Views, Indexes, Functions. Built ETL workflows using ODI Designer, creating Mappings, Procedures, Packages, and Scenarios to load data into target systems. Scheduled and monitored ETL jobs using DAC, AutoSys, and Control-M, ensuring reliable and timely data availability. Integrated data from enterprise source systems including Oracle E-Business Suite, JD Edwards, and PeopleSoft. Environment: OBIEE, OBIA, Informatica, ODI, Tableau, Oracle, SQL Server, Oracle EBS, JD Edwards, PeopleSoft. Project Engineer Wipro Technologies Ltd, Hyderabad, India June 2006 - March 2010 Responsibilities: Collaborated with business stakeholders to gather, analyze, and translate reporting and analytics requirements into technical solutions. Customized OBIEE RPD layers - Physical, Business Model & Mapping (BMM), and Presentation - to support evolving business requirements. Configured OBIEE security using the Administration Tool by managing groups, access privileges, and query permissions, and enforced role-based access in Answers. Created and maintained session variables, dynamic variables, and presentation variables to drive report filtering, data security, and dynamic calculations. Designed Dimensions, Hierarchies, and Measures within the repository to enable accurate and performant analytical reporting. Configured and scheduled iBots (Agents) to deliver analytics content, weekly reports, and automated alert notifications to business users. Developed interactive OBIEE dashboards and reports with multiple views, tailored to operational and management reporting needs. Environment: OBIEE, Informatica, Oracle. Keywords: continuous integration continuous deployment artificial intelligence user interface business intelligence sthree procedural language California Texas