Big Data Engineer

Job Description: •     Databricks, PySpark and Python project development work experience is a must •     Develop, design, tune and maintain PySpark scripts using Databricks Notebook. •     Expertise in Bigdata Eco systems like HDFS and Spark •     Experience in GitHuB repository •     Data Warehouse/Data Marts/Data Modelling/Analytics experience is a must •     Able to convert the SQL stored procedures to Python code in Pyspark frame work using Dataframes. •     Implementing data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools. •     Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries. •     Developing scalable and re-usable, self-service frameworks for data ingestion and processing. •     Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data. •     Processing performance analysis and optimization. •     Collaborate with business users, support team members, and other developers throughout the organization to help everyone understand issues that affect the data warehouse •     Good experience on customer interaction is required. •     Possesses good interpersonal and communication skills.