A Data Engineer is needed to design, implement, and maintain scalable data pipelines and analytics solutions on Databricks. Responsibilities include building data flows, optimizing performance, and collaborating with stakeholders to deliver data-driven insights. The role involves working with Databricks (Delta Lake, Spark), cloud platforms (Azure), data modeling, ETL/ELT processes, and CI/CD pipelines.
We are seeking a hands-on Data Engineer with strong experience in building scalable data pipelines and analytics solutions on Databricks . You will design, implement, and maintain end-to-end data flows, optimize performance, and collaborate with data scientists, analytics, and business stakeholders to turn raw data into trusted insights.
Design, develop, test, and maintain robust data pipelines and ETL/ELT processes on Databricks . Architect scalable data models and data vault/ dimensional schemas to support reporting, BI, and advanced analytics. Implement data quality, lineage, and governance practices; monitor data quality metrics and resolve data issues proactively. Collaborate with Data Platform Engineers to optimize cluster configuration, performance tuning, and cost management in cloud environments . Build and maintain data ingestion from multiple sources using modern data engineering patterns .Develop and maintain CI/CD pipelines for data workflows; implement versioning, testing, and automated deployments. Partner with data scientists and analysts to provision clean data, notebooks, and reusable data products; support feature stores and model deployment pipelines where applicable.Document data lineage, architecture, and operational runbooks; participate in architectural reviews and bestpractice governance.Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.Expertise with Apache Spark , Databricks notebooks, Delta Lake, and SQL.Experience with cloud data platforms and their Databricks offerings; familiarity with object storage . Proficient in building and maintaining ETL/ELT pipelines, data modeling, and performance optimization.Experience with CI/CD for data pipelines, and orchestration tools . Strong problem-solving skills, attention to detail, and ability to work in a collaborative, cross-functional team.Experience with streaming data .Knowledge of data visualization and BI tools .Certifications in Databricks or cloud provider platforms.
Data Engineering Databricks Data Pipelines ETL Cloud Computing
South Africa Latest News, South Africa Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Network Security EngineerSeeking a skilled Network Security Engineer to design, implement, and maintain a robust network security infrastructure, manage security platforms, monitor network traffic, conduct forensic analysis, and ensure compliance. Requires 6-8 years of experience, relevant certifications (CCNP Security, CISSP), and expertise in various security tools and protocols.
Read more »
Data Engineer (24 Months contract ) - Gauteng PretoriaIT Industry News. Daily.
Read more »
Data Engineer - IT-OnlineIT Industry News. Daily.
Read more »
Head AI and Data Engineer - IT-OnlineIT Industry News. Daily.
Read more »
Head AI and Data Engineer - Western Cape Cape TownIT Industry News. Daily.
Read more »
Synology Survey Highlights Data Management Challenges for EnterprisesA recent Synology survey reveals that IT teams face significant data management hurdles, including rising costs, ransomware threats, and complex data retention requirements. Over 90% of companies lack confidence in recovering critical data after a malware attack. Fragmented legacy solutions and budget constraints further complicate data protection, while increasing cyberattacks and global data residency laws add to the pressure. The survey points out the gaps in data protection strategies for businesses.
Read more »