Our client an established international research and development company are seeking to recruit an experienced Data Engineer.
You will work on the collection, storing, processing, and analysis of huge sets of data from heterogeneous domains.
The primary focus will be on researching for optimal solutions to appropriate for the aforementioned purposes, then maintaining, implementing, and monitoring them. The successful candidate will also be responsible for integrating the said solutions with the architecture used across the company.
Job Description
- Researching, designing and developing appropriate algorithms for Big Data collection, processing and analysis
- Selecting and integrating any Big Data tools and frameworks required to enable new and existing product capabilities
- Collaborate closely with product team to define the requirements and set milestones that relate to Big Data features
- Detect anomalies and perform audit on raw and processed data
- Monitoring performance and advising any necessary infrastructure changes
- Defining data retention policies
- Present data findings to internal and external stakeholders
Required Skills & Experience
- Proficient understanding of distributed computing principles
- Management of Elasic Search cluster, with all included services
- Proficiency with Hadoop v2, MapReduce, HDFS
- Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
- Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
- Experience with Spark
- Experience with integration of data from multiple heterogeneous data sources
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O
- Good understanding of Lambda Architecture, along with its advantages and drawbacks
- Experience with Cloudera/MapR/Hortonworks
Remuneration & Benefits
- Competitive base salary
- 13th salary
- Medical insurance
- Annual bonus
- In house complimentary dining