Home

SivaKumar - Data Engineer | Azure & Databricks

[email protected]

Location: Frisco, Texas, USA

Relocation: Yes

Visa: H1B

Resume file: ShivaGandham_Data Engineer_1790203639375.docx Please check the file(s) for viruses. Files are checked manually and then made available for download.

Principal Data Engineer with deep expertise in building scalable data pipelines and modernizing legacy ETL platforms using Azure Data Factory, Azure Databricks, ADLS Gen2, Synapse Analytics, Delta Lake, and IDMC. Proven experience in leading large-scale cloud migrations, designing Medallion Architecture, implementing enterprise data integrations across Salesforce and SAP S/4HANA, and optimizing Azure data pipelines for performance, scalability, and reliability. PROFILE Senior Data Engineer specializing in Microsoft Azure and cloud data engineering, with proven expertise in designing, developing, and optimizing enterprise-scale ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, ADLS Gen2, Delta Lake, and Python/PySpark. Architected and implemented Azure data solutions leveraging ADLS Gen2, Azure Data Factory, Databricks, Synapse Analytics, Delta Lake, and Microsoft Fabric to build scalable, secure, and high-performance cloud data platforms. Skilled in Snowflake performance tuning, clustering, warehouse optimization, and secure data governance implementations. Designed and implemented Medallion Architecture (Bronze, Silver, Gold) on ADLS Gen2 using Azure Databricks, Apache Spark, PySpark, and Delta Lake, enabling reliable data processing, data quality, and analytics-ready datasets. Designed and implemented data integration between Salesforce and SAP S/4HANA using Azure Data Factory, developing scalable ingestion pipelines to extract, transform, and load enterprise data into downstream data platforms. Led cloud migration and modernization initiatives from legacy ETL platforms and on-premises data warehouses to Azure, leveraging ADF, Databricks, ADLS Gen2, Synapse Analytics, and modern ELT frameworks to improve scalability, performance, and operational efficiency. Enabled near real-time and batch analytics by integrating data from SAP, mainframe, APIs, databases, and enterprise applications into Azure Data Lake using Azure Data Factory, Databricks, and Delta Lake. Hands-on experience with Salesforce, Snowflake, and Looker, including designing and implementing integrations between Salesforce and Informatica, Salesforce and Looker, and Salesforce and Snowflake, along with Git-based workflows and automated CI/CD processes for Azure data platform deployments. Experience in AMS services, performing multiple roles including Data Engineer, Azure Developer, Administrator, and Production Support. Versatile Data Engineer with extensive experience working across legacy and modern ETL tools, relational databases, cloud data platforms, data lakes, and enterprise data warehouses. Experience working in Agile methodology, participating in sprint planning, daily stand-ups, backlog refinement, code reviews, testing, and continuous delivery activities. TECHNICAL SKILLS Cloud Data Platforms Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, Azure Data Lake Gen2, Azure Functions, Salesforce, Microsoft Fabric, AWS S3 (storage tiering), AWS Glue, AWS EMR, AWS Redshift, Snowflake Cloud Data Platform ETL & Orchestration Azure Data Factory Pipelines, Snowflake Tasks & Streams, IDMC (CDI), Informatica PowerCenter 10.X, HVR, Matillion ETL, Informatica Power Exchange for Oracle CDC Languages SQL, Python, PySpark, Jinja Data Quality & Governance Informatica Data Quality 9.6.1, Informatica Metadata Manager, Microsoft Purview Reporting/Visualization Looker and Qlik Sense. Operating Systems Windows, UNIX Development Tools ERWIN, ER Studio (Data Modeling Tools), Visual Studio Others/Databases Teradata, Oracle, SQL Server and HP Vertica Database Domain Skills Banking, Insurance, Healthcare CERTIFICATIONS Microsoft Certified: Azure Data Engineer Associate (DP-203) Microsoft Certified: Azure Developer Associate (AZ-204) Databricks Certified Data Engineer Associate Informatica PowerCenter Data Integration 9.x: Developer Specialist: Issued in Feb 2015 EDUCATION Bachelor of Engineering in Computer Science | VIT University, Vellore | 8.1/10CGPA PROFESSIONAL EXPERIENCE ModelN | Principal Data Engineer | Dallas, TX | Jan 2013 Present Clients: Pfizer, Novartis, GSK, BMS, Janssen, Sanofi, AbbVie, AstraZeneca, and Leo-Pharma. Project: Global Price Management, a cloud-based solution enabling organizations to cross-reference and manage drug pricing across countries for market access and pricing teams. Environment: Databricks, PySpark, Spark SQL, Informatica PowerCenter, IDMC, Salesforce, Looker, LookML, dbt, Airflow, Matillion ETL, Snowflake, Fivetran, Azure Data Factory (ADF), ADLS Gen2, Delta Lake, Python, Terraform, Jenkins, GitLab CI/CD, GitHub, Azure DevOps, Hive, Scala, Spark, Azure Blob Storage, Amazon S3, SQL, Power BI, SAP S/4HANA. Designed and built scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, and Delta Lake on ADLS Gen2, modernizing legacy ETL and SQL workloads into optimized Databricks SQL solutions for batch and near-real-time analytics. Developed and maintained debt models following ELT best practices, transforming raw data into analytics-ready datasets and orchestrating workflows using Apache Airflow, Azure Data Factory (ADF), Prefect, and dbt Cloud. Designed scalable ELT pipelines using Matillion ETL, leveraging push-down processing to deliver high-performance, cost-effective data integration across Snowflake and Amazon Redshift. Designed and delivered enterprise data integration and analytics solutions using Azure Data Factory, Informatica IDMC, Salesforce, and Looker, building scalable data pipelines and trusted data models to support sales, customer, inventory, and operational reporting for Global Price Management clients. Designed and developed scalable ELT pipelines using Matillion ETL, integrating data from multiple sources into cloud data warehouses such as Snowflake/Redshift for analytics. Designed and implemented Salesforce and SAP S/4HANA data integrations using Azure Data Factory and Informatica IDMC, developing scalable ingestion pipelines to extract, transform, and load data from Salesforce, SAP, databases, and APIs into Azure Data Lake and downstream platforms. Implemented Snowflake-based ELT frameworks for insurance reporting and downstream analytics. Developed and maintained Informatica workflows and mappings for enterprise data integration, including source-to-target mapping, transformations, and data quality validation across Salesforce and Azure data platforms. Designed and developed governed Looker analytics solutions using LookML, including Views, Explores, Dimensions, Measures, derived tables, and reusable semantic models for scalable, self-service enterprise reporting. Designed and developed ANT-based automation scripts to load and migrate data from CSV files into Salesforce, including data preparation, transformation, validation, and automated deployment processes. Built and maintained reusable, parameterized Matillion jobs using custom SQL and Python, integrating enterprise applications with APIs and cloud storage platforms, including Azure Blob Storage and Amazon S3. Optimized Snowflake queries and warehouse performance for high-volume insurance analytics workloads. Developed and maintained reusable Terraform modules for Infrastructure as Code (IaC), enabling standardized, scalable, and repeatable cloud infrastructure deployments on Microsoft Azure. Implemented CI/CD pipelines using Jenkins and GitLab CI/CD for Azure and Informatica data pipelines, automating build, test, and deployment processes while enforcing Git-based code reviews, branching strategies, and release management through GitHub and GitLab. Implemented automated, scheduled data integration workflows incorporating data validation, error handling, monitoring, dependency management, and performance optimization; performed unit and integration testing to ensure fault-tolerant, restartable pipeline reliability. Provided production support and root-cause analysis for data pipeline incidents as part of AMS services, monitoring data loads, resolving data quality issues, and ensuring adherence to enterprise SLAs. Collaborated with data analysts, solution architects, engineers, and business stakeholders across Agile sprints, including planning, backlog refinement, stand-ups, and retrospectives, to translate business requirements into scalable, enterprise-ready data solutions. MindTree Inc | Informatica Designer/Administrator | Bangalore, India | Nov 2011 Jan 2013 Client: American Express (AMEX), a global financial services company providing payment, credit, travel, and financial products and services. Environment: Informatica PowerCenter 10.1.1, Informatica Data Quality, PowerExchange CDC for Oracle, Metadata Manager. Worked extensively with Informatica PowerCenter, PowerExchange CDC for Oracle, and Informatica Metadata Manager to develop, maintain, and support enterprise data integration solutions for American Express (AMEX). Served as Informatica Administrator, managing the Informatica platform, environments, deployments, code migrations, repository objects, workflows, and production support activities. Analyzed existing client systems and business requirements to propose ETL and data integration architecture and design solutions, including technical specifications and implementation documentation. Built Snowflake data models supporting insurance claims adjudication and policy analytics reporting. Developed, supported, and delivered high-quality, defect-free ETL solutions, ensuring adherence to technical requirements, development standards, and project timelines. Improved Snowflake query performance for insurance analytics using clustering, partition pruning, and warehouse tuning. Provided technical support to development teams by analyzing and resolving application and ETL issues, troubleshooting production defects, and monitoring performance after implementation. Implemented secure data governance and compliance controls for sensitive insurance data in Snowflake. Improved Snowflake query performance for insurance analytics using clustering, partition pruning, and warehouse tuning. Performed Informatica workflow and session troubleshooting and performance tuning, optimizing mappings, transformations, SQL queries, and workflows to improve ETL performance and reliability. Created capture registrations in Informatica PowerExchange CDC and monitored CDC processes for Oracle, ensuring reliable change-data capture and downstream data synchronization. Prepared metrics and status reports for senior management on a daily and biweekly basis, while collaborating with the team to establish effective development and deployment practices. Mentored and trained team members on Informatica PowerCenter, ETL development, troubleshooting, performance tuning, and best practices. Hewlett-Packard | ETL Developer and Designer | Bangalore, India | Jul 2009 Nov 2011 Clients: HDFC, Bank of Baroda (BOB), and Amgen. Environment: Informatica PowerCenter 8.6.1/9.1/9.6, Informatica Data Quality, PowerExchange CDC for Oracle, Metadata Manager, TOAD, ERWIN, Oracle, Teradata. Delivered end-to-end ETL solutions for clients including HDFC, Bank of Baroda, and Amgen, working extensively with Informatica PowerCenter, Informatica Data Quality (IDQ), PowerExchange CDC for Oracle, and Informatica Metadata Manager to support enterprise data integration and data quality initiatives. Designed Snowflake schema models using Star and Snowflake schema for optimized data warehousing and reporting. Worked as an Informatica consultant, installing and configuring Informatica PowerCenter, Data Quality, PowerExchange CDC for Oracle, and Metadata Manager. Designed and developed Informatica PowerCenter mappings, workflows, and ETL processes, along with Informatica Data Quality mappings and PowerExchange CDC Navigator capture registrations, to support data integration, cleansing, validation, and quality requirements. Analyzed existing ETL code and business requirements, prepared technical specifications and design documents, and developed and tested enhancements and new functionality against business requirements. Performed unit testing, integration testing, and data validation to ensure ETL solutions met functional and technical requirements and delivered reliable data. Analyzed and resolved technical issues raised by development teams, performing root-cause analysis and post-implementation performance monitoring to ensure application and ETL stability. Performed troubleshooting and performance tuning of Informatica PowerCenter workflows, mappings, sessions, and underlying ETL processes to improve performance and reliability. Collaborated with team members to share Informatica development best practices, troubleshoot complex ETL issues, and continuously improve development and support processes; recognized by senior project stakeholders for consistent delivery and technical contributions. Keywords: continuous integration continuous deployment business intelligence sthree hewlett packard Arizona Texas