We are looking for a highly skilled professional with extensive experience in Elasticsearch, Logstash, and Kibana (ELK Stack), capable of managing complex data pipelines, transformations and advanced use-cases.
Technical Responsibilities :
1. Elasticsearch Architecture & Query Optimization
- Design and implement high-performance Elasticsearch architectures, optimizing query performance and index mappings for specific use-cases.
2. Data Pipeline Management
- Build robust, fault-tolerant data pipelines that clean, transform, and aggregate disjointed data streams into searchable indexes.
3. Real-Time Data Processing
- Implement real-time analytics solutions, taking advantage of Elastic's capabilities to perform operations at scale, possibly in a distributed manner.
4. API Development & Integration
- Build custom APIs that leverage Elasticsearch's capabilities for third-party integrations and integrate existing Elasticsearch APIs into the company's products/services.
5. Webhook & Trigger-Based Actions
- Develop event-driven architectures that make use of webhooks to trigger predefined actions based on specific events or anomalies.
6. Master Index Management & Data Consolidation
- Implement an advanced indexing strategy that includes the consolidation of multiple raw indices into master/golden indices to optimize for search and analytics.
7. Security & Compliance
- Implement security measures like encryption, role-based access, and compliance logging within the Elasticsearch ecosystem.
8. Scripting & Automation
- Develop scripts for automated tasks that relate to index management, data ingestion, and transformation, using languages like Python, Groovy, or Shell scripting.
9. Disaster Recovery & High Availability
- Design and implement strategies for backup, disaster recovery, and high availability, possibly involving cross-cluster search and replication.
10. Performance Tuning & Monitoring
- Continuously monitor system health, tuning performance, and ensuring that the Elastic solutions are optimized and healthy.
Non-Technical :
- Documentation : Create and maintain documentation related to Elasticsearch implementations, data models, and configurations.
- Team Collaboration : Work closely with various IT Teams (DevOps, RPA, EMS, InfoSec) to ensure smooth deployment and continuous improvement.
- Compliance : Ensure data security and compliance as per defined InfoSec standards.
Qualifications :
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- A minimum of 5 years of experience with Elasticsearch and the ELK Stack.
- Strong understanding of RESTful APIs, webhooks, and data transformation techniques.
- Knowledge of programming and scripting languages like Python, Ruby, or Shell.
- Familiarity with cloud platforms like Azure and understanding of on-premises data center operations.
Nice to Haves :
- Elastic Certified Engineer Certification
- Experience with Business Intelligence tools and techniques.
- Familiarity with other data ingestion and transformation tools.