Azure Databricks Engineer (PySpark) Job at Sira Consulting, an Inc 5000 company, Branchville, NJ

V2FXMGlKeEhVVlQrKzJSbDd4YjZ6ZDByS1E9PQ==
  • Sira Consulting, an Inc 5000 company
  • Branchville, NJ

Job Description

Job Role: Azure Databricks Engineer (PySpark)

Location: Branchville, NJ

Job Type: Full-time

Job Summary:

We are looking for an experienced Azure Databricks Engineer with expertise in PySpark to design, develop, and optimize big data solutions on Azure cloud. The ideal candidate will work with large-scale data processing, ETL pipelines, and performance optimization using Databricks, Spark, and Azure services.

Key Responsibilities:

  • Design, develop, and implement data pipelines and ETL processes using Azure Databricks and PySpark .
  • Optimize Spark jobs for better performance and cost efficiency.
  • Work with structured, semi-structured, and unstructured data to create scalable data solutions.
  • Collaborate with Data Engineers, Data Scientists, and Business Analysts to meet business requirements.
  • Implement best practices for data security, governance, and compliance on Azure Cloud .
  • Work with Azure services such as Azure Data Factory (ADF), Azure Data Lake (ADLS), Azure Synapse, and Azure SQL .
  • Troubleshoot performance bottlenecks and optimize query execution plans .
  • Develop and maintain CI/CD pipelines for data pipeline automation.
  • Write clean, reusable, and efficient PySpark code following industry best practices.

Required Skills & Qualifications:

  • 10+ years of experience in Azure Databricks and PySpark .
  • Strong expertise in Apache Spark (RDDs, DataFrames, Datasets).
  • Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Synapse Analytics .
  • Strong understanding of SQL and data modeling techniques.
  • Experience with version control tools (Git, GitHub, or Bitbucket) .
  • Hands-on experience with CI/CD pipelines using Azure DevOps .
  • Knowledge of Scala or Python for big data processing.
  • Strong analytical and problem-solving skills.
  • Experience working in Agile/Scrum environments .

Education:

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field .

Job Tags

Full time,

Similar Jobs

Employbridge

Maintenance Technician Job at Employbridge

 ...adoption, child care, schools and more ~ Flexible spending accounts ~ Relocation assistance (if applicable) Shift: ~ Rotating, Days, 5:30am to 2PM, Mid- 1:30PM to 10PM, Nights -9:30PM to 6:00PM (12 hr. shifts available) Pay: ~$33.00 - $44.00/hr.... 

Sanford Barrows Group

Customer Experience Specialist (Bilingual Spanish) Job at Sanford Barrows Group

 ...Job Title: Customer Experience Specialist (Bilingual Spanish Required) Location: Doral, FL (In-Office role, NOT hybrid / remote) Mon. - Fri 8:30am - 5:30pm Salary: $70k -- $75k base plus bonus and excellent benefits KEY POINTS ~ Bilingual (Spanish) for business... 

Gulfstream Strategic Placements

Tridium Project Manager Job at Gulfstream Strategic Placements

Job Title: Tridium Project Manager Location: Manassas, VA Job Type: Full-time Job Overview We are seeking an experienced Tridium Project Manager to oversee and manage the implementation of Niagara-based Building Automation Systems (BAS) projects. This role...

Aditi Consulting

Biomedical Device Engineer (Biotech/Biopharma/Medical device) Job at Aditi Consulting

 ...to personnel, develop and validate test methods, and develop SOPs for equipment use. Preferred Qualification: ~ Bachelors degree in science OR masters degree in science ~2-3 years of experience in development or manufacturing environment, working with Medical... 

San Gabriel Memory Care- Godfrey

LPN or RN Memory Care Job at San Gabriel Memory Care- Godfrey

 ...LPN or RN Full Time or Part Time $2,500 Sign on Bonus 12 hour shifts 8am-8pm Includes Every other weekend Do you want to be a part of a great team that has the passion for caring for seniors? If so, this is the place for you! At San Gabriel Memory Care...