Lead Developer - Greenfield Enterprise Data Warehouse

Lead Developer - Greenfield Enterprise Data Warehouse

Our client plans to build a new, large-scale (100+ TB) enterprise data warehouse (EDW) to aggregate data from multiple sources for comparison and reporting. In certain cases, this data will also be exported into other systems to support real-time web platform activity. We are looking for a developer with substantial experience designing, implementing, and maintaining EDW architectures to lead this initiative.

In this role, you will be there from day one, gathering requirements, designing architecture, evaluating and resolving security and compliance concerns, and ultimately implementing the build-out of the EDW. You will lead a right-sized team of developers and data engineers to deliver and ultimately support this platform, which will be used by many teams across many business units within our client’s organization.

Responsibilities:

  • Lead the design and implementation of the enterprise data warehouse architecture
  • Gather and analyze requirements from stakeholders to inform design decisions
  • Evaluate and select appropriate cloud or open-source columnar/distributed databases such as Google Big Query, Clickhouse, StarRocks, Hadoop, etc., ensuring cost-effectiveness and scalability
  • In conjunction with client’s sysadmin and network operations teams, build out development/testing and production environments of the EDW architecture with appropriate CI/CD automation and access controls
  • Design and implement data pipelines to ingest and transform data from various sources into the warehouse
  • Implement and maintain data security and compliance measures within the warehouse, including HIPAA-specific controls
  • Collaborate with cross-functional teams to ensure alignment of the EDW with organizational goals and objectives
  • Mentor and provide guidance to junior developers and data engineers on best practices for data warehousing.
  • In conjunction with client’s sysadmin and network operations teams, continuously monitor and optimize the performance of the data warehouse environment.
  • Stay current with industry trends and advancements in data warehousing technologies.

Experience and Skills:

  • Minimum of 5 years of experience designing and implementing enterprise data warehouse architectures.
  • Mastery of one or more cloud-hosted distributed databases such as Google BigQuery or Amazon RedShift.
  • Mastery of one or more open-source columnar databases such as Clickhouse, StarRocks, etc., as well as map-reduce systems such as Hadoop
  • Extensive experience with data engineering toolkits, either in Python or Java
  • Strong understanding of data modeling concepts and methodologies.
  • Experience with ETL (Extract, Transform, Load) processes and tools.
  • Knowledge of data governance principles and practices.
  • Familiarity with cloud-based data warehousing solutions is a plus.
  • Excellent communication and interpersonal skills, with the ability to effectively collaborate with cross-functional teams.
  • Strong analytical and problem-solving abilities.
  • Ability to thrive in a fast-paced, dynamic environment and manage multiple priorities effectively.
  • Bachelor’s degree in CS, mathematics, or physical sciences preferred

Application Process:

To apply, please submit your resume and cover letter detailing your relevant experience and why you are the ideal candidate for this position to [email protected]. We look forward to hearing from you!