As a Core Data Engineer, you will play a pivotal role in designing, building, and optimising data pipelines, architectures, and workflows for a diverse client base. You’ll work within dynamic, multi-disciplinary project teams to implement scalable data solutions while adhering to best practices and emerging trends in data engineering.
Key Responsibilities: * Data Pipeline Development: Design and implement robust, scalable, and efficient data pipelines to collect, transform, and integrate data from various sources. * Data Architecture: Develop and optimize data architectures, including data warehouses, data lakes, and other storage solutions. * ETL/ELT Processes: Create and maintain reliable ETL/ELT workflows that ensure data quality and accessibility. * Collaboration: Work closely with data scientists, analysts, and client stakeholders to understand requirements and deliver impactful solutions. * Performance Optimisation: Monitor and enhance the performance of data systems to ensure minimal downtime and fast query responses. * Automation and Tools: Identify opportunities for automation and recommend tools to improve data engineering workflows. * Documentation: Maintain detailed technical documentation for all solutions and processes.
Technical Skills: * Programming: Proficiency in Python, Java, Scala, or similar languages for data processing. * Big Data Technologies: Hands-on experience with big data tools e.g., Databricks, Apache Spark, Hadoop. * Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. * Database Management: Expertise in relational databases (e.g. postgres, sql server) * Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. * Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. * Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment.
Experience * 2+ years of experience in data engineering or a similar role. * Proven track record of delivering data solutions in a consulting or client-facing environment is a plus. * Experience with Agile or Scrum methodologies is beneficial.
Personal Qualities * Strong problem-solving and analytical thinking abilities. * Excellent communication skills to explain complex technical concepts to non-technical stakeholders. * A team-oriented mindset with a focus on collaboration and adaptability
