Data Engineer

For one of my clients, a leading tendering and bidding platform, I am seeking a skilled Data Engineer with expertise in Python, SQL, Spark, and AWS technologies (S3, Glue, SQS), among others.

The current data landscape includes multiple formats and models, resulting in inconsistencies. Your primary responsibilities will include building and improving the existing data platform, as well as performing comprehensive data mapping to transform data from one format to another. The goal is to create a simple and understandable data mapping process that can be easily used across the organization.

Key Responsibilities:

• Develop, maintain, and optimize scalable data pipelines and architectures for batch and real-time data processing.
• Perform complex data mapping to transform and harmonize data from multiple sources and formats into a unified model.
• Collaborate with cross-functional teams to understand business needs and ensure data quality and consistency across the platform.
• Ensure the data mapping and transformation processes are easy to comprehend and usable by stakeholders across the organization.
• Maintain and improve the current data platform to ensure its robustness, scalability, and performance.
• Optimize workflows and data architectures to enhance overall system efficiency.

Requirements:

• Bachelor's degree in Computer Science, Information Systems, Data Engineering, or related fields.
• 3+ years of experience in data engineering, building and maintaining data pipelines and infrastructures.
• Strong proficiency in Python, SQL, and Spark.
• Hands-on experience with AWS services, including but not limited to S3, Glue, and SQS.
• Experience with ETL processes and familiarity with data transformation and mapping.
• Solid understanding of data modeling concepts and techniques.
• Ability to write clean, efficient, and well-documented code.
• Familiarity with CI/CD pipelines and version control.
• Experience working with large datasets and optimizing complex workflows.

Darwin Recruitment is acting as an Employment Business in relation to this vacancy.