Data Engineer

Job description

BJIT Limited is seeking a skilled and experienced Data Engineer specializing in Databricks on Azure to join our dynamic team. As a Data Engineer, you will play a crucial role in designing, developing, and maintaining scalable data pipelines on Azure Databricks. Your expertise will be instrumental in building and optimizing ingestion, transformation, and loading processes using tools like Lakeflow, PySpark, SQL, Delta Lake, and Unity Catalog.

Responsibilities

• Design, develop, test, and maintain scalable data pipelines on Azure Databricks
• Build and optimize ingestion, transformation, and loading processes using Lakeflow, PySpark, SQL, Delta Lake, and Unity Catalog
• Integrate Databricks with Azure services like ADLS to ensure reliable data delivery
• Document technical designs and suggest improvements for the data platform
• Monitor pipeline execution, data quality, and system performance; proactively identify and resolve issues
• Provide production support, managing schedules, troubleshooting failures, and addressing root causes
• Offer Power BI support as needed

Requirements

• Proven experience in developing and supporting data pipelines on Azure Databricks
• Proficiency in PySpark, Spark SQL, SQL, Python, Lakeflow, and Unity Catalog
• Understanding of Delta Lake, ETL/ELT, and data pipeline design patterns
• Experience with Azure data services such as ADLS and related platform components
• Knowledge of data modeling, logging, alerting, data quality controls, and performance tuning

Benefits

• Festival Bonus (3 yearly)
• Performance Bonus
• Leave Encashment
• Medical Benefit
• Partially subsidized Lunch Facilities

Skills

• Azure
• Databricks
• Delta Lake
• Power BI
• PySpark
• Python
• SQL