• Bachelor’s degree, preferably in Computer Science or Computer Information Systems
• 6+ years of software engineering experience
• 5+ years of data engineering experience with Apache Spark or Apache Flink
• 4+ years of experience running software and services in the cloud
• Proficiency in working with DataFrame APIs (Pandas and Spark) for parallel and single node processing
• Proficiency using advanced languages and techniques with Python, Scala, etc. with modern data optimized file formats such as Parquet and Avro
• Proficiency with SQL on RDBMS and data warehouse solutions like Redshift
• Hands on experience with Data Lake technologies like Delta Lake and Iceberg
• Experience with data acquisition from external APIs at large scale / in parallel processing
• Experience supporting ML/AI projects: deployed pipelines for computing features, using models for inference on large datasets

Desirable

• We're in search of an exceptional Principal Data Engineer who will play a pivotal role in designing, building, and optimizing scalable data infrastructure
• Develop and maintain scalable data pipelines, including:
• ETL pipelines, both single and multi-node solutions
• Build data quality assurance steps for new and existing pipelines
• Create derived datasets with augmented properties
• Work on analytics ready datasets to power internal and creator facing tools
• Troubleshoot issues when they arise, working directly with internal data consumers
• Automate pipeline runs with scheduling and orchestration tools
• Work with large scale datasets
• Work with/use various external APIs to enhance data
• Setup database tables for analytics users to consume the data collected by the Data Engineering team
• Work with big data technologies to improve data availability and data quality in the cloud (AWS)
• Lead development of projects involving other team members and act as a mentor
• Actively participate in team discussions about technology/architecture/solutions for new projects and to improve existing code and pipeline