Overview We are seeking an experienced and hands-on Databricks Specialist to join our partner’s team. In this role, you will be responsible for auditing current workloads and engineering robust, efficient data-processing pipelines across the Bronze, Silver, and Gold layers of the medallion architecture.
Key Responsibilities * Audit existing Databricks workspaces, notebooks, and jobs to ensure performance, security, and cost-efficiency. * Design and build modular batch and streaming sub-pipelines that land, cleanse, enrich, and publish data as physical tables across all medallion layers. * Implement robust exception handling, alerting, and automated rollback/retry mechanisms. * Create comprehensive documentation, reusable code samples, and lead knowledge transfer sessions for data engineers and analysts.
Required Skills & Experience * 4+ years of experience building production-grade ETL/ELT pipelines on Databricks (Spark 3.x, Delta Lake). * In-depth knowledge of the medallion architecture, performance tuning (e.g., Optimize/Z-Ordering), and cost-effective cluster configurations. * Proficiency in PySpark or Scala, along with advanced SQL skills — including job profiling and fine-tuning at the shuffle partition level. * Demonstrated experience in building reliable pipelines with data quality checks, unit testing, and CI/CD (e.g., Databricks Repos, GitHub Actions, Azure DevOps).
Nice to Have * Experience with Unity Catalog, serverless compute, or Databricks Photon. * Relevant certifications such as Databricks Data Engineer Professional or Databricks Champion Enablement (DBCE).
