Recently added jobs Remote working jobs Apprenticeships & Graduates Education & Public Sector jobs Paid by the hour / Shift work

8 Contract, Tech / Digital / IT jobs in Lewisham

Senior Databricks Enginee...

Salary not available. View on company website.

Korn Ferry, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 2 days ago, 18 Jul

Full Stack Developer

Salary not available. View on company website.

CACI, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 4 days ago, 16 Jul

Automation Engineer

Salary not available. View on company website.

Korn Ferry, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 4 days ago, 16 Jul

Successful jobseekers create high quality email alerts

A great alert means less time searching & more time applying.

Managed File Transfer Arc...

Salary not available. View on company website.

GIOS Technology, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 5 days ago, 15 Jul

DV Cleared Data Scientist

Salary not available. View on company website.

Korn Ferry, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 1 week ago, 10 Jul

DV cleared Data Analyst

Salary not available. View on company website.

Korn Ferry, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 1 week ago, 10 Jul

Solution Architect

Salary not available. View on company website.

scrumconnect ltd, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 1 week ago, 7 Jul

Data Steward/Data Scienti...

Salary not available. View on company website.

Emagine Consulting, City of Westminster

  • Full time
  • Contract

Apply on company site

Posted 1 week ago, 7 Jul
Close

Senior Databricks Engineer - Data Streaming & Optimization (Remote Contract)

Salary not available. View on company website.

Korn Ferry, City of Westminster

  • Remote working
  • Full time
  • Contract

Posted 2 days ago, 18 Jul

Job ref: 2709c0e004784ed8bccf6a356514b75c

Full Job Description

Duration: 6 months (initial), with possible extension. Location: Remote/Hybrid (depending on candidates requirements) Rate: Flexible (inside IR35) We are seeking an experienced Databricks Engineer with deep expertise in data streaming pipelines, performance tuning, and cost optimization. You will work on an enterprise-scale crude and product data platform that processes large volumes of Real Time transactional data from on-prem Mainframe systems into Azure cloud. The current implementation leverages Databricks Delta Live Tables (DLT) for raw data ingestion and transformation. Key Responsibilities

  • Pipeline Diagnostics & Reliability
  • + Investigate and resolve data latency and inconsistency issues in streaming pipelines. + Troubleshoot root causes for intermittent DLT failures and optimize for stability.
  • Performance Optimization
  • + Tune Spark and Databricks configurations to reduce memory overflows and job crashes. + Improve cluster utilization, optimize job execution plans, and introduce partitioning/caching strategies.
  • Cost Optimization
  • + Analyze current compute cost drivers and propose strategies for efficient scaling. + Implement autoscaling, job parameterization, and cluster right-sizing to meet budget targets.
  • Architecture Enhancement
  • + Ensure the raw data layer is reliable and performant to support future silver and gold layers. + Collaborate with architects to define best practices for Delta Live Tables and streaming design patterns.
  • Operational Excellence
  • Introduce monitoring, alerting, and observability for data pipeline health.
  • Build trust in the platform by delivering predictable and consistent performance.
  • Required Skills & Experience
  • Deep experience with Databricks (including Delta Live Tables, Delta Lake, and Streaming Jobs).
  • Strong Spark tuning expertise for performance and memory optimization.
  • Solid knowledge of Azure Data Lake Storage (ADLS) and related Azure services.
  • Experience with data ingestion from on-prem systems and CDC tools (eg, Qlik Replicate).
  • Proven ability to diagnose and resolve latency, reliability, and scalability issues in large-scale pipelines.
  • Proficient in cluster sizing, autoscaling, and cost optimization strategies.
  • Familiarity with data architecture principles for bronze, silver, and gold layers.

    Deep experience with Databricks (including Delta Live Tables, Delta Lake, and Streaming Jobs).
  • Strong Spark tuning expertise for performance and memory optimization.
  • Solid knowledge of Azure Data Lake Storage (ADLS) and related Azure services.
  • Experience with data ingestion from on-prem systems and CDC tools (eg, Qlik Replicate).
  • Proven ability to diagnose and resolve latency, reliability, and scalability issues in large-scale pipelines.
  • Proficient in cluster sizing, autoscaling, and cost optimization strategies.
  • Familiarity with data architecture principles for bronze, silver, and gold layers.

Do you like this job?

We can email jobs like this to your inbox

  • Facebook

Copy the direct link to this job

www.jobs24.co.uk/job/senior-databricks-engineer-data-streaming-optimization-remote-125394640
Displaying results 1 to 8 of 8 found
Create a high quality job alert