Kafka Engineer

Role: Kafka Engineer

Function: DevOps Engineering

Based in India, Remote, Immediate, Contract 6-12 (extendable) months, Singapore/ Malaysia Working hours

Please state the candidate skill (below mentioned) levels from 1 to 10.  Options:  1) the candidates are on your payroll preferred, 2) freelancers are also ok 3) I can also take them on my payroll, if the CTC is realistic and pay you your margin month on month (with 30-45 days payment period). Good control over the candidates is mandatory.

Please also mention your preferred monthly rate in INR and note that your rates need to be realistic and you should feel that we have realistic chance of winning the opportunity.  If we fulfil, many more long term opportunities will come our way.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field (Master's degree preferred).
• Min 3+ years of professional experience as a Kafka Engineer or similar role.
• Strong understanding of distributed systems and event-driven architectures.
• Excellent problem-solving skills and the ability to troubleshoot complex issues.
• Strong communication and interpersonal skills to collaborate effectively with cross-functional teams.
• Kafka certification(s) would be a plus.

Core Responsibilities

• Kafka Topic Management: Design, create, and manage Kafka topics based on business requirements. Ensure proper partitioning, replication, and configuration settings for optimal performance and data distribution.
• Kafka Deployment: Deploy Kafka clusters, brokers, and related components in various environments (development, testing, production). Manage cluster configurations, scaling, and security settings.
• Kafka Infrastructure Maintenance: Perform routine maintenance tasks such as Kafka version upgrades, patch management, and security enhancements. Monitor cluster health, troubleshoot issues, and ensure high availability.
• Performance Optimization: Collaborate with development teams to tune Kafka configurations and parameters for optimal throughput, latency, and resource utilization. Identify and address bottlenecks in the Kafka ecosystem.
• Capacity Planning: Analyze current and projected data volumes to plan for Kafka cluster capacity. Scale up or out as needed to accommodate growing data demands.
• Integration and Middleware: Integrate Kafka with other systems, databases, and middleware components to facilitate seamless data exchange and event-driven architectures.
• Disaster Recovery and Backup: Implement and test disaster recovery strategies, backup procedures, and data retention policies. Ensure data integrity and recoverability in the event of failures.
• Documentation: Maintain thorough and up-to-date documentation of Kafka infrastructure, configurations, processes, and best practices. Create technical guides for internal teams.
• Cross-Functional Collaboration: Collaborate with cross-functional teams including developers, data engineers, DevOps, and QA to understand requirements and provide Kafka expertise.

Skills/Experience

Mandatory:

• Kafka Expertise: In-depth knowledge of Kafka architecture, concepts, and components. Proficiency in Kafka topic creation, partitioning, replication, and configuration management.
• Deployment Experience: Hands-on experience deploying Kafka clusters in various environments. Familiarity with cluster sizing, configuration, and security considerations.

Good-to-Have:

• Kafka Monitoring and Management: Familiarity with Kafka monitoring tools (e.g., Confluent Control Center, Prometheus) to track cluster health, performance metrics, and alerts.
• Scripting and Automation: Ability to write scripts (e.g., Bash, Python) for automating routine tasks such as deployment, configuration management, and monitoring
• Experience in ELK Stack configuration and development, comprising of Elastic Search, Logstash & Kibana