About the engagement
Our client runs a modern but fragmented data landscape across Snowflake, Databricks, and a growing Microsoft Fabric footprint. Much of its enterprise data (SAP, Salesforce, and partner systems) arrives in batches with 3–24 hour refresh delays. As part of an independent data science and planning assessment, you'll review the client's data infrastructure and recommend how the platforms should integrate and where workloads should run.
This is a fixed, phased engagement (~124 hours, mid-November to mid-December 2026) with one concentrated on-site week in mid-November for a technical deep dive. Industry: large U.S. beverage manufacturing & distribution.
What you'll do
- Review the client's data infrastructure and cloud ecosystem: connectivity, integration, data flows, and technical redundancy across platforms.
- Assess Snowflake and Databricks integration, and how Microsoft Fabric fits in as the emerging enterprise data foundation.
- Map data flows from source systems (SAP ERP, Salesforce, planning platforms, and third-party feeds) through to the forecasting and analytics models.
- Evaluate ETL/ELT pipelines for reliability, refresh latency, data quality, and cost — and identify paths to fresher data.
- Assess the data engineering and MLOps foundations behind the custom AI forecasting platform, working alongside the Data Scientist.
- Recommend workload placement (Snowflake for governed data marts and serving, Databricks for ML and feature engineering, Fabric for integration and BI), backed by total cost of ownership (TCO) and performance evidence.
- Build light prototypes or proof-of-concept integrations where needed to validate recommendations.
- Document findings and contribute to the architecture blueprint and final roadmap.
What we're looking for
- 7+ years in data engineering, with strong hands-on Snowflake and Databricks experience (Delta Lake, Spark, Unity Catalog, Snowpark, data sharing).
- Proven experience designing and optimizing ETL/ELT pipelines at enterprise scale.
- Working knowledge of Microsoft Fabric and the Azure data stack (ADF, OneLake, Synapse, or similar).
- Experience integrating ERP and CRM data (SAP, Salesforce) into cloud analytics platforms.
- Strong SQL and Python; experience with orchestration (Airflow, dbt, ADF, or Databricks Workflows).
- Understanding of how ML pipelines are fed and deployed: feature stores, data versioning, and lineage.
- Ability to assess architecture and cost trade-offs across platforms and explain them clearly to technical and business audiences.
- Strong communication skills and comfort working on-site with client teams.
Preferred
- Experience with FastAPI, PostgreSQL, Docker, and React/Next.js for rapid prototyping.
- Background in supply chain, manufacturing, CPG, or distribution data.
- Experience with data governance, lineage, and data quality frameworks.
- Databricks and/or Snowflake certifications.
Out of scope
Building production pipelines. The engagement uses read-only access and makes no production changes.