QUALIFICATIONS AND JOB DESCRIPTION
Do you have a passion for data, analytics, insights and technology? Do you want to be part of a team that is building N11’s data platforms?
We seek passionate engineers who have a proven record of successfully building and supporting data engineering platforms and products. Someone who can help us build and advance foundational data platforms that help deliver experimentation, analytics, data science and machine learning use cases, while balancing security and privacy of data.
What you will do:
- Translate business requirements into data models that are easy to understand and used by different disciplines across the company
- Design, develop, and maintain scalable and reusable streaming and batch data pipelines for analysis, reporting, optimization, data collection, management and usage.
- Maintain and enhance our data and computation platforms.
- Develop and implement solutions for data quality validation and continuous improvement.
- Manage, execute and monitor weekly and monthly production operations; resolve and escalate production issues as appropriate.
- Partner with business stakeholders, data analysts and engineering teams to understand business and technical requirements, plan and execute projects, and communicate status, risks and issues.
- Ensure security, privacy and compliance for all data assets.
- Perform root cause analysis of system and data issues and develop solutions as required.
Our ideal Data Engineer will have:
- Bachelor's degree in Computer Science, Electrical Engineering, Mathematics, a related field or equivalent practical experience.
- Understanding of data management fundamentals, data storage principles and distributed systems
- Experience working with cloud infrastructure (at least one of Azure, AWS, GCP)
- Experience with batch and streaming data pipelines, workflow engines, distributed computing technologies (Spark, Kafka, Hadoop, Kinesis, Flink, Airflow, etc.).
- Experience with Data modeling and ETL/ELT development
- Data Warehousing Experience with Oracle, BigQuery, etc.
- Excellent English verbal and written communication skills with a variety of audiences.
- Experience working in software engineering, and can demonstrate best practices for project management, quality control, and product development.
- Ability to communicate well with users, partner teams and senior management to collect requirements, explain data collection decisions and create data engineering strategy
- Development experience with at least one scripting language (Python, etc.)
Bonus points for having:
- Demonstrable skills and experience writing complex, highly optimized SQL queries across large data sets
- Experience in modern DevOps practices
- Knowledge of Lambda and Kappa architecture patterns.
- Knowledge of Master Data Management (MDM) and Data Quality tools and processes.
- Experience in designing and executing data pipelines using GDPR and private data
- Experience in building Data Lake
- Experience with containerization technologies like Docker, Kubernetes.
- Industry experience as a Data Engineer or related specialty
- Understanding the differences between different storage formats like Parquet, Avro, Arrow, and JSON