IRIS direct end client which is one of the leading American provider of human resources management software and services is urgently looking to hire AWS Data Engineer Parsippany, NJ 07054 (Hybrid - 3 days per week). This is a Contract to hire Role opportunity.
AWS Data Engineer
Parsippany, NJ 07054 (Hybrid - 3 days per week)
Nature of Contract – Contract to hire Opportunity
Focus is : Databricks Products, AWS (Glue/Athena/kinesis/aurora etc.), Python or PySpark Development experience, building Data pipelines.
We are seeking a highly skilled AWS Data Engineer with expertise in building Data pipelines and processing solutions on the AWS platform, focusing on Databricks, Delta Lake, and Python development. The ideal candidate will have a strong background in relational and non-relational data architecture, data classification, and working collaboratively with Data Scientists and Architects.
Key Responsibilities:
- Design, develop, and optimize Data pipelines and processing solutions on the AWS platform, emphasizing Databricks, Delta Lake, and Python.
- Collaborate with Data Scientists, Architects, and business stakeholders to gather and clarify requirements for Data Lake components.
- Utilize AWS services such as CloudFormation, Lambda, API Gateway, S3, DynamoDB, RDS, Glue, Athena, and QuickSight for data management and analysis.
- Develop metadata-driven Data Lake solutions for ingesting historic and real-time structured/semi-structured/unstructured data.
- Implement ETL/ELT pipelines and ensure data integrity, quality, and security throughout the process.
- Manage changes in enterprise-grade operational Data Lake environments and ensure scalability and efficiency.
- Utilize tools like Gliffy for creating diagrams representing Business Processes, ERD, and AWS Architecture.
- Develop solutions for data Ingestion, Transformation, Cataloging, In-Place Querying, Storage, and Security following AWS best practices.
- Experience in Relational/NOSQL database integration into an Enterprise Data Lake environment.
- Set up AWS CloudWatch, CloudTrail for monitoring, and optimizing Data Lake environments.
- Write SQL queries using AWS Athena and map various relational DB queries to Athena.
- Proficient in Python, PySpark, and AWS Glue ETL jobs for data processing and transformations.
- Follow and enforce strict standards for code quality, automated testing, infrastructure-as-code, and code maintainability.
- Lead the development of user stories and translate requirements into Problem Statements in an agile collaborative environment.
Qualifications:
- 10+ years of experience in data management with a focus on AWS Data solutions.
- Strong expertise in relational and non-relational data architecture and classification.
- Hands-on experience with Databricks, Delta Lake, AWS services, Python, or PySpark development.
- Experience in designing and implementing ETL/ELT pipelines for Data processing.
- Knowledge of AWS EMR and its application in data processing and analytics.
- Proficiency in SQL queries using AWS Athena and mapping relational DB queries.
- Experience in de-normalizing/flattening data structures in Parquet/ORC formats.
- Good understanding of AWS IAM policies for implementing Data Lake security best practices.
- Robust debugging skills and knowledge of automated testing platforms and unit tests.
Looking forward to hear from you...!!!
Thanks and Regards,
Bharat Sharma
Sr. Technical Recruiter
IRIS Software, Inc.