Home

Anusha kaithi - Data Engineer / Snowflake / Data Bricks / ADF /

[email protected]

Location: Kent, Ohio, USA

Relocation: Yes / Anywhere in USA

Visa: USC / Citizen

Resume file: AnushaKa-DataEngineer Updated _1790017597579.docx Please check the file(s) for viruses. Files are checked manually and then made available for download.

Anusha Kaithi Sr. Data Engineer [email protected] | +1 (330) 577-8643| LinkedIn Data Engineering professional with 8 years of experience building scalable cloud and big data solutions using GCP, AWS, Azure, and Hadoop. Expert in Python, Scala, and SQL, specializing in ETL development, data pipelines, and modern data architecture for analytics and data-driven insights. Data Engineer | GCP| AWS | Big Data Analytics | Python| ETL |Spark |Azure| SQL| Tableau PROFESSIONAL SUMMARY: Extensive experience building ETL/ELT pipelines using Apache Spark, Kafka, Airflow, Databricks, Delta Lake, AWS Glue, and Azure Data Factory supporting both batch and real-time data processing. Strong expertise with GCP services including BigQuery, Pub/Sub, Cloud Composer, Dataflow, Dataproc, and Cloud Storage for serverless data ingestion, orchestration, and large-scale analytics. Hands-on experience with Azure services including API Management (APIM), Azure Container Registry (ACR), Event Hub, Azure Data Lake, and Synapse for building cloud-native data integration solutions. Proficient in Python, Scala, Java, and SQL for developing distributed data processing frameworks, automation scripts, and scalable data pipelines. Deep expertise in Big Data ecosystems including Hadoop (HDFS, Hive, Pig, Sqoop, Oozie, YARN), Spark Core, Spark SQL, and Spark Streaming for large-scale distributed processing. Strong experience with data warehousing and analytics platforms including Snowflake, BigQuery, Redshift, and Teradata, implementing dimensional modeling, Star/Snowflake schemas, and OLTP/OLAP systems. Experienced in building real-time streaming architectures using Kafka, Pub/Sub, Event Hub, and Spark Streaming to process high-velocity data streams. Hands-on experience implementing DevOps and infrastructure automation using Terraform, CloudFormation, Jenkins, Git, and CI/CD pipelines for cloud deployments. Familiar with Generative AI (GenAI) integration for data analytics, and experienced working within SDLC frameworks including Agile/Scrum and Waterfall methodologies to deliver enterprise data solutions. Proficient in GCP BigQuery for query optimization, partitioning, clustering, and data modeling. Experienced in orchestrating complex workflows using Apache Airflow and developing robust data pipelines using Python. Adept in Bash/Shell scripting for automation and Git for version control and collaboration. Strong focus on data quality, including validation, testing, and monitoring frameworks. Hands-on experience with Dataflow and Apache Beam for both batch and real-time streaming data processing. TECHNICAL SKILLS: Cloud Environments AWS, Azure, GCP Big Data & Distributed Processing Apache Spark, PySpark, Scala, Spark SQL, Spark Streaming, Hadoop, HDFS, MapReduce, Hive, Pig, Sqoop, Oozie, YARN, Apache Beam, Databricks Data Warehousing & Analytics Platforms Snowflake, Google BigQuery, Amazon Redshift, Teradata Programming Languages Python, PySpark, Scala, Java, SQL, PL/SQL, T-SQL, Unix Shell Scripting Streaming & Messaging Apache Kafka, Spark Streaming, Google Pub/Sub, AWS Kinesis, Amazon SNS, Amazon SQS, Azure Event Hub Databases & NoSQL Databases PostgreSQL, MySQL, SQL Server, Oracle, DB2, MongoDB, Cassandra, HBase, DynamoDB, Redis, Neo4j Data Integration & ETL Tools Apache Airflow, DBT, Informatica IICS, Talend, Matillion, Prefect, Apache NiFi Machine Learning & AI TensorFlow, PyTorch, Spark MLlib, Databricks ML Runtime, AWS SageMaker, Google Vertex AI, AWS Bedrock DevOps & Infrastructure Docker, Kubernetes (GKE, EKS, AKS), Terraform, Ansible, GitHub Actions Visualization & Reporting Power BI, Tableau, Looker, Amazon QuickSight Methodologies Agile, Scrum, Waterfall, SDLC PROFESSIONAL EXPERIENCE: Molina Healthcare, Long Beach, CA Sep 2024 Present Role: Senior Data Engineer Responsibilities: Designed and implemented scalable ETL/ELT pipelines using Python, PySpark, SQL, Apache Airflow DAGs, DBT, Talend, Informatica IICS, and Matillion, integrating healthcare data from EMR/EHR systems (Cerner, NextGen) and relational databases including SQL Server, PostgreSQL, MySQL, and Teradata, transforming HL7/FHIR datasets and loading them into Snowflake, AWS Redshift, and Google BigQuery for enterprise analytics. Built distributed data processing solutions using Apache Spark, Databricks, Scala, Apache Beam, and the Hadoop ecosystem (Hive, HDFS, MapReduce, YARN, Oozie) to process large-scale structured and semi-structured healthcare datasets across cloud platforms. Developed real-time streaming and event-driven data pipelines using Apache Kafka, Spark Streaming, Google Cloud Pub/Sub, Redis, AWS Lambda, SNS, SQS, and Google Cloud Functions, enabling low-latency ingestion and processing of healthcare transactions and operational data. Architected and optimized cloud-native data platforms and data lakes leveraging AWS (S3, Redshift, EMR, Glue, Athena, EC2, CloudWatch, IAM) and GCP (BigQuery, Dataproc, Dataflow, Pub/Sub, Cloud Storage, GKE) for scalable analytics and enterprise reporting. Built and operationalized AI/ML and GenAI pipelines using TensorFlow, PyTorch, Spark MLlib, Databricks ML Runtime, AWS SageMaker, GCP Vertex AI, and AWS Bedrock with RAG architectures for predictive analytics, fraud detection, and healthcare insights. Implemented containerized microservices and scalable data platforms using Docker and Kubernetes (GKE/EKS), with automated CI/CD pipelines using Jenkins, Git, and GitHub, and managed infrastructure provisioning using Terraform. Designed and optimized data warehouse schemas and query performance across Snowflake, BigQuery, and Redshift, implementing partitioning, clustering, indexing, and star/snowflake models, and developed Snowflake stored procedures and UDFs (Python/JavaScript) for automated transformations. Delivered enterprise analytics through Power BI dashboards and BI reporting, ensuring HIPAA-compliant data governance, implementing IAM-based security, encryption, and auditing, while collaborating in Agile/Scrum environments using GitHub, Jira, and Confluence. Environment: Python, PySpark, Scala, SQL, Apache Spark, Apache Beam, Hadoop (HDFS, Hive, MapReduce, YARN, Oozie), Apache Airflow, DBT, Talend, Informatica IICS, Matillion, Apache Kafka, Spark Streaming, Google Cloud Pub/Sub, Redis, Snowflake, PostgreSQL, MySQL, SQL Server, Teradata, MongoDB, Cassandra, Neo4j, TensorFlow, PyTorch, Spark MLlib, Databricks ML Runtime, AWS SageMaker, GCP Vertex AI, AWS Bedrock. Barclays, New York, NY May 2022 - Aug 2024 Role: Senior Data Engineer Responsibilities: Architected and implemented scalable batch and real-time data pipelines on GCP using Dataflow, Dataproc, BigQuery, Pub/Sub, Cloud Storage, Compute Engine, and Cloud Functions, processing multi-terabyte financial datasets for enterprise analytics and reporting. Built high-performance distributed data processing frameworks using Apache Spark, PySpark, Scala, Hive, Pig, Hadoop, and MapReduce (Java/Python) across Dataproc and Databricks, optimizing transformations using partitioning, caching, and query execution tuning. Developed streaming data architectures leveraging Apache Kafka, Spark Streaming, and Azure Event Hub, enabling event-driven processing, fraud detection systems, and anomaly detection pipelines for high-volume financial transactions. Designed and orchestrated enterprise ETL/ELT pipelines using Apache Airflow, Oozie, DBT, Informatica, Azure Data Factory (ADF), and Microsoft Fabric pipelines, automating ingestion, transformation, workflow scheduling, and dependency management across hybrid cloud platforms. Built and optimized data lakehouse and data warehouse platforms using BigQuery, Snowflake, Teradata, Databricks, Google Cloud Storage, Azure Data Lake Storage, and Microsoft Fabric Lakehouse, implementing Medallion Architecture, clustering, partitioning, and materialized views for high-performance analytics. Developed and deployed AI/ML and advanced analytics pipelines using Python, PySpark MLlib, Spark, Databricks, Microsoft Fabric AI capabilities, and Palantir Foundry, enabling predictive analytics, recommendation systems, and anomaly detection models for financial insights. Implemented secure API-driven data services using REST, GraphQL, and Azure API Management (APIM) while integrating Palantir Foundry with enterprise data platforms for governed analytics, operational intelligence, and cross-platform data orchestration. Automated cloud infrastructure, containerization, and CI/CD pipelines using Terraform, Ansible, Jenkins, GitHub Actions, and Git, deploying containerized workloads via Docker, Kubernetes (GKE/AKS), and Azure Container Registry (ACR) with monitoring and observability through Looker, Palantir Foundry, Datadog, Fluentd, and Graylog. Environment: GCP (Dataflow, Dataproc, BigQuery, Pub/Sub, GCS, Compute Engine, Bigtable, Cloud Run), Snowflake, Apache Spark, PySpark, Scala, Kafka, Hive, Pig, Hadoop, DBT, Airflow, Oozie, Informatica, Python, Java, SQL, Terraform, Ansible, Jenkins, GitHub Actions, Git, Docker, Kubernetes (GKE, AKS), Azure Data Factory (ADF), Azure Event Hub, Azure API Management (APIM), Azure Container Registry (ACR), Azure Data Lake Storage, Azure Synapse. Belk, Charlotte, NC Sep 2020 - Apr 2022 Role: Senior Data Engineer Responsibilities: Designed and developed scalable ETL/ELT pipelines using AWS Glue, Matillion, Apache Airflow, Apache Oozie, and Prefect, ingesting multi-terabyte data from S3, Redshift, RDS, MongoDB, Oracle, SQL Server, DB2, and on-prem systems into Snowflake, Redshift, and AWS Data Lake environments. Built distributed data processing solutions using Apache Spark (PySpark, Scala), Spark SQL, DataFrames, RDDs, Hive, MapReduce, and Hadoop YARN on AWS EMR, Databricks, and HDInsight, optimizing performance through partitioning, caching, and columnar formats (Parquet, ORC, Avro). Implemented centralized metadata management and data discovery using AWS Glue Crawlers and Glue Data Catalog, enabling querying through Amazon Athena and analytics visualization with QuickSight, Tableau, Power BI, Streamlit, Plotly, and Matplotlib. Developed real-time and streaming data pipelines using Amazon Kinesis Data Streams, Kinesis Firehose, Kinesis Analytics, Spark Streaming, Flume, and Kafka integrations, processing event data and storing results in S3, DynamoDB, Redshift, and HDFS. Architected serverless and event-driven workflows using AWS Lambda and Step Functions, implementing Infrastructure-as-Code with Terraform, AWS CloudFormation, and Packer, and automating deployments via Azure DevOps, GitHub, and Stash CI/CD pipelines. Built and optimized enterprise data lake architectures on Amazon S3, managing ingestion, partitioning, archival of Redshift data to S3, lifecycle policies, and secure access using AWS IAM, VPC, and CloudWatch monitoring. Containerized and deployed data workloads using Docker, Kubernetes, and OpenShift, integrating automation scripts in Python and Scala (boto3, pandas, pyodbc) and enabling scalable data services across hybrid cloud infrastructure. Implemented monitoring, logging, and observability frameworks using Prometheus, Grafana, ELK Stack, AWS CloudWatch, and Elasticsearch, while supporting hybrid cloud integrations with Azure (AAD, API Management, VMs, VNet, Azure Functions, Logic Apps, Event Grid, Blob Storage) and enabling migrations via AWS DMS under Agile/JIRA-based SDLC practices. Environment: AWS (S3, Redshift, RDS, Glue Studio, Glue DataBrew, Glue Crawlers, Glue Data Catalog, Lambda, Step Functions, EMR, Athena, CloudFormation, DMS, IAM, VPC, CloudWatch), Kinesis Data Streams, MapReduce, Spark, PySpark, Scala, Kafka, MongoDB, DB2, Python (boto3, pandas, pyodbc), Parquet, Docker, Kubernetes, OpenShift, Terraform, Tableau, Power BI, Azure DevOps, AWS Data Lake, HDFS, DynamoDB. Progressive Insurance, Mayfield, OH Oct 2018 Aug 2020 Role: Big Data Engineer Responsibilities: Built scalable Big Data pipelines on AWS EMR and EC2 using the Hadoop ecosystem (HDFS, YARN, MapReduce, Hive, Spark) to process large-scale batch datasets and support distributed data processing. Developed distributed ETL and data processing applications using PySpark, Scala, Spark SQL, RDDs, and DataFrames, integrating data from HDFS, AWS S3, Oracle, SQL Server, DB2, and MongoDB for large-scale transformations and analytics. Implemented real-time streaming data pipelines using Apache Kafka, AWS Kinesis, Flume, and Spark Streaming, enabling continuous ingestion and processing of high-volume streaming datasets. Designed and optimized analytical data models using Snowflake, Redshift, Hive, and Elasticsearch, implementing star/snowflake schemas, partitioning, clustering, materialized views, and query optimization techniques for high-performance analytics. Built and orchestrated ETL/ELT workflows using AWS Glue, Apache Airflow, Luigi, and Oozie, performing large-scale ingestion using Sqoop, AWS DMS, Glue Crawlers, and Python/Shell scripts, and processing datasets in JSON and XML formats. Developed serverless and containerized data processing solutions using AWS Lambda, Docker, and Kubernetes, provisioning infrastructure using Terraform and CloudFormation to support scalable distributed environments. Enabled data analytics and reporting using AWS Athena and QuickSight, while performing cluster monitoring, troubleshooting, and performance tuning using CloudWatch and Spark UI to ensure optimal system performance. Environment: AWS (EMR, EC2, S3, Glue, Lambda, Redshift, Athena, Kinesis, CloudWatch, RDS, DynamoDB, DMS), Hadoop (HDFS, YARN, MapReduce), Apache Spark (RDD, DataFrame, Spark SQL, Spark Streaming), Hive, Kafka, Oozie, Sqoop, AWS Neptune (Cypher), MongoDB, Python, PySpark, Terraform, CloudFormation. Keywords: continuous integration continuous deployment artificial intelligence machine learning user interface business intelligence sthree procedural language California New York North Carolina Ohio