Upload your CV
Go to job search

Principal Consultant_ Kafka Admin

Infosys Limited, City of Westminster

Principal Consultant_ Kafka Admin

Salary not available. View on company website.

Infosys Limited, City of Westminster

  • Full time
  • Permanent
  • Onsite working

Posted 1 week ago, 28 Oct | Get your application in now before you're too late!

Closing date: Closing date not specified

Job ref: 381575ce56c546f3ad13804cd646fc35

Location ref: City of Westminster

Full Job Description

  • Provision, configure, and manage Kafka clusters (Apache Kafka KRaft or ZooKeeper-backed if legacy), Confluent Platform components (Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center), and Confluent Cloud resources.
  • Ensure high availability (HA) and resilience across multi AZ/region deployments; manage partition replication, min.insync.replicas (ISR), and rack awareness.
  • Implement and maintain client quotas, broker quotas, throttling, and back pressure strategies.
  • Plan and execute upgrades/patching with zero/minimal downtime; maintain version currency against EOL and security advisories.
  • Own backup/restore workflows for metadata (e.g., cluster configs), Schema Registry, and Connect configs
  • Implement authentication (mTLS/SASL: SCRAM, OAUTHBEARER with IdP, or Kerberos if legacy) and authorization (Kafka ACLs, Confluent RBAC).
  • Enforce encryption in transit and at rest, secret management (Vault/AKV/SSM), secure endpoint exposure, and private connectivity (VPC peering/PrivateLink).
  • Govern topic naming conventions, retention policies, schema evolution rules (backward/fully compatible), and PII handling.
  • Maintain audit trails (broker, Schema Registry, Control Center, Confluent Cloud audit logs) and compliance artifacts (SOX, GDPR, HIPAA/PCI as applicable).
  • Coordinate vulnerability management (CVE watch, hardening baselines, CIS/STIG alignment).
  • Deploy and maintain metrics pipelines (JMX Prometheus/Grafana), logs (ELK/OPensearch), and tracing (OpenTelemetry where applicable).
  • Define and monitor SLOs/SLIs (produce/consume latency, end to end lag, broker CPU/heap/file handles, network throughput, GC pauses).
  • Analyze broker hotspots, partition skew, large message handling, compaction and retention settings, and tune GC/JVM for sustained throughput.
  • Implement consumer lag monitoring and alerting with actionable runbooks to prevent data loss or SLA breaches.
  • Model workload growth, topic/partition scaling, storage/IOPS, network egress/ingress, and broker sizing.
  • For Confluent Cloud, optimize environment cluster topic hierarchy, throughput tiers, retention costs, and data flow egress; right size Dedicated vs. Standard clusters.
  • Design partitioning strategies to meet throughput targets while balancing consumer concurrency and avoiding small partitions anti patterns
  • Automate cluster and topic lifecycle using Terraform (providers for Kafka/Confluent), Helm, GitOps workflows.
  • Operate and scale Kafka Connect with distributed workers; manage connectors (JDBC, Debezium CDC, S3/ADLS/GCS, Elasticsearch, etc.), transforms (SMTs), and converter settings.
  • Administer Schema Registry (compatibility levels, subject naming strategies, serializers/deserializers, schema size & references).
  • Operate ksqlDB and/or Flink for streaming SQL; enforce resource limits and multi tenant governance.
  • Engineer low latency and secure connectivity across data centers/VPCs/VNETs; manage DNS, TLS SANs, LB configurations, advertised.listeners, and inter cluster networking.

    5-10 years in platform/SRE/DevOps; 3+ years dedicated to Kafka/Confluent admin in production.
  • Strong hands on with Apache Kafka internals: brokers, controllers (KRaft), partitions/replication, log segments, compaction/retention.
  • Confluent Platform: Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs; or Confluent Cloud admin (environments, clusters, RBAC, audit logs, networking).
  • Proficiency with Kafka CLI tools and AdminClient usage; deep knowledge of ACLs, quotas, transactions, idempotent producers, and exactly once semantics scenarios.
  • Prometheus/Grafana, JMX exporters, alerting systems (Alertmanager etc).
  • JVM tuning, Linux performance (NUMA, file descriptors, page cache, disk/RAID, XFS), and network tuning (TCP buffers, MTU, jumbo frames where applicable).
  • Linux administration (systemd, journald, kernel params), virtualization/containers (Docker, Kubernetes).
  • Cloud networking (VPC/VNET peering, transit gateways, PrivateLink/Private Service Connect), multi region architectures.
    • Preferrable -
  • Confluent Certified Administrator for Apache Kafka (CCAAK), CKA/CKAD, HashiCorp Terraform Associate, major cloud certs (Azure/AWS/GCP).
  • Experience with Confluent for Kubernetes (CFK) for operator based deployments.

    Infosys is a global leader in next-generation digital services and consulting. We enable clients in 59 countries to navigate their digital transformation.
    • With over four decades of experience in managing the systems and workings of global enterprises, we expertly steer clients, in 59 countries, as they navigate their digital transformation powered by cloud and AI. We enable them with an AI-first core, empower the business with agile digital at scale and drive continuous improvement with always-on learning through the transfer of digital skills, expertise, and ideas from our innovation ecosystem. We are deeply committed to being a well-governed, environmentally sustainable organization where diverse talent thrives in an inclusive workplace.
  • All aspects of employment at Infosys are based on merit, competence and performance. We are committed to embracing diversity and creating an inclusive environment for all employees. Infosys is proud to be an equal opportunity employer.

Direct job link

https://www.jobs24.co.uk/job/principal-consultant-kafka-admin-127652448
Relevant jobs