Role: Senior Azure Databricks Engineer
Location Options (Onsite Required): Dallas, TX ( Locals in TX ) Pref
About the Role
We are seeking a highly experienced Senior Azure Databricks Engineer to support digital transformation initiatives for our client. The ideal candidate has strong expertise in Azure cloud services, Databricks, Spark, large-scale data engineering, and end-to-end pipeline development. This is an onsite role requiring candidates to work from any of the client locations listed above.
Responsibilities
- Design, build, and optimize end-to-end data pipelines using Azure Databricks and Apache Spark.
- Develop scalable ETL/ELT processes for structured and unstructured data.
- Integrate Databricks with various Azure services (ADLS, ADF, Azure SQL, Synapse, Key Vault, etc.).
- Implement Delta Lake features time travel, ACID transactions, schema evolution, and optimization.
- Perform advanced data transformation, cleansing, aggregation, and feature engineering.
- Optimize Spark jobs for performance, reliability, and cost efficiency.
- Analyze and troubleshoot complex data issues in production environments.
- Collaborate with data architects, analysts, and business teams to define requirements & deliver solutions.
- Follow CI/CD best practices for data engineering using Git, Azure DevOps, or similar tools.
- Document solutions, create technical specifications, and support code reviews.
Required Skills
5+ years of professional Data Engineering experience.
- Strong hands-on expertise with Azure Databricks and Apache Spark (PySpark / Scala).
- Strong experience with Azure cloud services:
- Azure Data Lake (ADLS)
- Azure Data Factory (ADF)
- Azure SQL / Synapse
- Azure Key Vault
- Strong SQL development skills (complex queries, window functions, tuning).
- Experience with Delta Lake features and Databricks workspace management.
- Experience building and running pipelines on CI/CD frameworks (Azure DevOps preferred).
- Strong understanding of data modeling, partitioning, performance tuning, and job orchestration.
- Experience working with large datasets in distributed environments.
- Excellent communication and problem-solving skills.
Education: Bachelor s or Master s degree in Computer Science, Data Engineering, or related field.