Join our team as a Data Engineer to build and maintain scalable data pipelines and analytics solutions on Databricks. Responsibilities include designing data flows, optimizing performance, and collaborating with data scientists and business stakeholders. Experience with Spark, Delta Lake, Azure, and ETL/ELT processes is required.
We are looking for a skilled Data Engineer to join our team, someone who excels in hands-on development and has a strong track record of building scalable data pipelines and analytics solutions, particularly on the Databricks platform. The ideal candidate will be instrumental in the entire data lifecycle, from designing and implementing end-to-end data flows to ensuring optimal performance and collaborating closely with data scientists, analysts, and business stakeholders.
The primary goal is to transform raw data into reliable and actionable insights that drive informed decision-making across the organization. This position requires a proactive individual with a deep understanding of data engineering principles and a passion for leveraging the latest technologies to solve complex data challenges. The role will involve a wide range of responsibilities, including data pipeline development, data modeling, data governance, and collaboration with various teams to ensure the effective utilization of data assets.\Key responsibilities include designing, developing, testing, and maintaining robust data pipelines and ETL/ELT processes using Databricks technologies such as Delta Lake, Spark, SQL, and Python/Scala/SQL notebooks. This involves building and optimizing data ingestion pipelines from diverse data sources, including relational database management systems (RDBMS), Software-as-a-Service (SaaS) applications, files, and streaming queues. The candidate will be expected to utilize modern data engineering patterns, such as Change Data Capture (CDC), event-driven pipelines, and change streams. Furthermore, the role will require the architecting of scalable data models, including data vault and dimensional schemas, designed to support reporting, Business Intelligence (BI), and advanced analytics initiatives. A crucial aspect of the role is the implementation of data quality, data lineage, and data governance practices. This includes monitoring data quality metrics, proactively resolving data issues, and ensuring data integrity and compliance. Collaboration is also key, with the need to work closely with Data Platform Engineers to optimize cluster configuration, performance tuning, and cost management within cloud environments, specifically Azure Databricks. The development and maintenance of Continuous Integration/Continuous Deployment (CI/CD) pipelines for data workflows, including versioning, testing, and automated deployments, are also crucial responsibilities. The Data Engineer will partner with data scientists and analysts to provision clean data, notebooks, and reusable data products, and support feature stores and model deployment pipelines where applicable. This role also encompasses the documentation of data lineage, architecture, and operational runbooks, and participating in architectural reviews and best practice governance.\The successful candidate will possess a Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field. Extensive hands-on experience with Apache Spark (PySpark), Databricks notebooks, Delta Lake, and SQL is required. Deep understanding of cloud data platforms, specifically Azure and its Databricks offerings, is essential. Familiarity with object storage, such as Azure Data Lake Storage (ADLS), is also important. A proven ability to build and maintain ETL/ELT pipelines, perform data modeling, and optimize performance is critical. Experience with CI/CD practices for data pipelines, along with knowledge of orchestration tools like GitHub Actions or Databricks’ jobs, is highly valued. The role demands strong problem-solving skills, attention to detail, and the ability to thrive in a collaborative, cross-functional team environment. Experience with streaming data technologies, such as Structured Streaming, Kafka, and Delta Live Tables, is highly desirable. Knowledge of data visualization and BI tools, such as Splunk, Power BI, and Grafana, will be considered an advantage. Certifications in Databricks or relevant cloud provider platforms are a plus. This role is perfect for a data engineer looking to make a significant impact on data strategy and drive innovative data solutions. We offer a dynamic work environment with opportunities for professional growth and the chance to contribute to impactful projects
Data Engineering Databricks Data Pipelines ETL/ELT Azure Cloud
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Senior Data AnalystSeeking an experienced Senior Data Analyst to perform in-depth data analysis, manage reporting requests, design data models, and collaborate with stakeholders. The ideal candidate will have extensive experience with enterprise databases, data visualization tools (Power BI), and a strong understanding of data analytics principles. They will be responsible for extracting, transforming, and analyzing data to uncover trends and insights, contributing to data-driven decision-making. The position requires strong problem-solving skills, the ability to work independently, and excellent communication abilities.
Read more »
Systems Engineer for Company Mid Dish Systems and ProductsCompany Mid Dish is seeking a highly experienced Systems Engineer to lead the assembly, integration, and verification of its systems and products. The role encompasses the full system lifecycle, from requirements engineering and design to integration, testing, and operational support. The ideal candidate will have extensive experience in systems engineering, a strong understanding of electronic and mechanical systems, and the ability to lead and contribute to engineering tasks and analysis.
Read more »
Business Intelligence Engineer / Data Scientist Opportunity at Membership Management TrailblazerA leading membership management solutions provider is seeking a Business Intelligence Engineer / Data Scientist to analyze company and client data, develop BI dashboards (Power BI), design predictive models, and translate business needs into data-driven outcomes. The role involves mentoring junior analysts.
Read more »
Business Intelligence Engineer/Data Scientist (Remote)IT Industry News. Daily.
Read more »
Senior System Engineer – Gauteng JohannesburgIT Industry News. Daily.
Read more »
Wait goes on for Springbok under ‘doping’ cloudThe complicated case against Springbok and Lions prop Asenathi Ntlabakanye for 'doping' offences is nearing a conclusion.
Read more »