Senior Data Engineer; Must Have: Strong hands-on AWS, Snowflake, SQL and dbt experience. Nice to have – Python, PySpark, Airflow, Informatica
Requirements and Responsibilities
What You’ll Do
- Design, develop, and maintain scalable data pipelines and data transformation frameworks using AWS, Snowflake, and dbt.
- Develop robust ETL/ELT pipelines to ingest, transform, and integrate data from multiple source systems into Snowflake.
- Build and maintain data models in Snowflake supporting analytics, reporting, operational applications, and downstream data consumers.
- Develop reusable dbt models, macros, tests, snapshots, and incremental pipelines following established data engineering standards.
- Write complex and optimized SQL for data transformation, aggregation, validation, and analytical workloads.
- Design and implement scalable data ingestion solutions using AWS services such as S3, Glue, Lambda, and related data services.
- Develop data pipelines for both structured and semi-structured data and work with formats such as JSON, CSV, Parquet, and Avro.
- Implement incremental loading, CDC, partitioning, clustering, and other optimization techniques to efficiently process large datasets.
- Design and maintain appropriate data warehouse schemas and dimensional models, including fact tables, dimension tables, star schemas, and data marts.
- Implement data quality and validation frameworks using dbt tests, SQL-based validation, reconciliation, and data profiling.
- Monitor data pipelines and proactively identify failures, performance issues, data quality problems, and processing bottlenecks.
- Troubleshoot pipeline failures and data discrepancies and perform detailed root-cause analysis.
- Optimize Snowflake workloads through appropriate warehouse sizing, query optimization, clustering, partition pruning, and workload management.
- Implement appropriate logging, monitoring, error handling, alerting, and recovery mechanisms for production data pipelines.
- Develop automated data pipelines and workflows using orchestration tools such as AWS Glue, Airflow, or similar technologies.
- Implement version control, CI/CD, and deployment processes for dbt models, SQL code, and data pipelines.
- Work closely with Data Architects, Business Analysts, ETL Developers, SQL Developers, QA Engineers, and application teams to understand requirements and deliver scalable data solutions.
- Participate in data architecture and technical design discussions and contribute to data platform modernization initiatives.
- Support migration of legacy ETL processes to cloud-based AWS/Snowflake ELT architectures.
- Develop and maintain technical documentation covering data pipelines, data models, source-to-target mappings, dependencies, and operational procedures.
- Conduct code reviews and ensure adherence to development, security, performance, and data engineering standards.
- Support production releases, incident management, troubleshooting, and post-production validation.
- Work in an Agile/Scrum environment and actively participate in sprint planning, daily stand-ups, reviews, retrospectives, and technical discussions.
- Mentor junior data engineers and contribute to knowledge sharing and continuous improvement.
What You’ll Bring
- Bachelor’s degree or higher in Computer Science, Information Technology, Engineering, or a related field.
- 5+ years of relevant experience in data engineering, data integration, data warehousing, or cloud data platforms.
- Strong hands-on experience with Snowflake as an enterprise cloud data warehouse.
- Strong hands-on experience with AWS data services, particularly:
- Amazon S3
- AWS Glue
- AWS Lambda
- IAM and security concepts
- Other AWS data/analytics services as applicable
- Strong hands-on experience with dbt (data build tool), including:
- dbt models
- Incremental models
- Macros
- Sources
- Seeds
- Snapshots
- Tests
- Documentation
- Model dependencies
- Advanced proficiency in SQL, including complex joins, CTEs, subqueries, window functions, aggregations, and performance optimization.
- Strong understanding of ETL/ELT architecture and data pipeline development.
- Strong understanding of data warehousing and dimensional modeling concepts.
- Experience designing and implementing fact and dimension models, star schemas, snowflake schemas, and data marts.
- Experience working with large-volume datasets and optimizing data processing workloads.
- Experience with structured and semi-structured data such as JSON, CSV, Parquet, and Avro.
- Strong understanding of data quality, validation, reconciliation, and data governance practices.
- Experience implementing incremental loads and CDC processes.
- Experience with Git and CI/CD practices for data engineering projects.
- Experience with pipeline orchestration and scheduling tools such as Airflow, AWS Glue Workflows, or similar technologies.
- Strong troubleshooting and root-cause-analysis skills.
- Experience working in Agile/Scrum environments.
- Banking or financial-services domain experience is preferred.
- Experience with Python for data engineering, automation, or scripting is a plus.
- Experience with PySpark / Apache Spark is a plus.
- Experience with Informatica or other enterprise ETL tools is a plus, particularly for legacy ETL modernization.
- Experience with data migration from traditional data warehouses to AWS/Snowflake is a plus.
- Knowledge of Snowflake security, RBAC, data sharing, streams/tasks, and stored procedures is a plus.
- Strong analytical and problem-solving skills.
- Ability to work independently and collaborate effectively with cross-functional teams.
- Strong communication skills in English, both written and verbal.
Recommended External Screening Criteria
For Senior Data Engineer – AWS/Snowflake/dbt, I would keep the following as the core mandatory screening criteria:
Must Have
- 5+ years of Data Engineering / Data Warehousing experience
- Strong hands-on Snowflake
- Strong hands-on SQL
- Hands-on dbt development
- AWS data services experience, particularly S3 and Glue
- ETL/ELT pipeline development
- Data Warehousing & Dimensional Modeling
- Experience with large-volume data
- Data quality and validation
- Git / CI/CD experience
Preferred
- Python
- PySpark / Spark
- Airflow
- Snowflake Streams & Tasks
- Snowflake performance tuning
- AWS Lambda
- Informatica
- CDC
- Legacy ETL modernization
- Banking / Financial Services experience