Data Engineer - Cloud Architect

Data Engineer Role

As a Data Engineer at Crypto, you will be a key member of our data engineering team, responsible for designing, building, and maintaining our robust data infrastructure. You will collaborate with data analysts, scientists, and business stakeholders to develop and implement data solutions that drive informed decision-making. By adhering to industry best practices, you will ensure the efficient and cost-effective operation of our big data platform, empowering data-driven insights and innovation across the organization.

Responsibilities
• Design, develop, and maintain scalable data pipelines and ETL processes to efficiently extract, transform, and load high volumes of data from various sources into our data lake.
• Develop and implement data quality checks and monitoring processes to ensure data integrity and accuracy.
• Leverage AWS services (EKS, EMR, S3, Glue, Lambda) to build and maintain cloud-based data infrastructure.
• Implement CI/CD pipelines for data engineering processes.
• Utilize Infrastructure as Code (IaC) tools like CloudFormation and Terraform to manage AWS resources.
• Stay up-to-date with the latest data engineering technologies and trends.

Qualifications
• Bachelor's degree in Computer Science, Engineering, or a related field.
• Minimum of 3 years of experience as a Data Engineer.
• Strong proficiency in SQL, Python, and Spark.
• Familiarity with relational databases like PostgreSQL.
• Hands-on experience with AWS cloud services (EKS, EMR, S3, Athena, Lambda, etc).
• Experience with Kafka and real-time data processing.
• Experience with Tableau and knowledge of setup and maintenance.
• Experience with data lake and lake house architecture.
• Experience with Databricks and Delta Lake.
• Proficiency in Apache Airflow for workflow orchestration.
• Proficiency in Kubernetes and Docker for containerization and orchestration.
• Experience with CI/CD pipelines and tools.
• Experience with Infrastructure as Code (IaC) tools like CloudFormation and Terraform.
• Excellent problem-solving and analytical skills.
• Strong attention to detail and ability to work independently.
• Excellent communication and collaboration skills