Data Engineer PySpark
__jobinformationwidget.freetext.LocationText__
Chennai, Tamil Nadu, India
- Sopra Steria
- Engineering, Development, Applications
- 6 to 10 years
- Standard
- India
- All Sectors
This vacancy has now expired. See all jobs
About Sopra Steria
Sopra Steria, a major Tech player in Europe with 56,000 employees in nearly 30 countries, is recognized for its consulting, digital services and software development. It helps its clients drive their digital transformation and obtain tangible and sustainable benefits. The Group provides end-to-end solutions to make large companies and organizations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a fully collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients in order to build a positive future for all. In 2023, the Group generated revenues of €5.8 billion.
The world is how we shape it.
We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Proficiency in below listed skills will be crucial in building and maintaining pipelines for training and inference datasets.
Responsibilities:
• Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines.
• Utilize PySpark for data processing, transformation, and preparation for model training.
• Leverage AWS EMR and S3 for scalable and efficient data storage and processing.
• Implement and manage ETL workflows using Stream sets for data ingestion and transformation.
• Design and construct pipelines to deliver high-quality training and inference datasets.
• Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities.
• Optimize and fine-tune pipelines for performance, scalability, and reliability.
• Ensure IAM policies and permissions are appropriately configured for secure data access and management.
• Implement Spark architecture and optimize Spark jobs for scalable data processing.
Requirements:
Mandatory
• Proficiency in Advanced SQL (Window functions), Spark Architecture, Pyspark or Scala with Spark, Hadoop.
• Proven expertise in designing and deploying data pipelines.
• Strong problem-solving skills and ability to work effectively in a collaborative team environment.
• Excellent communication skills and ability to translate technical concepts to non-technical stakeholder
Desirable
• Hands-on experience with Airflow, S3, and Stream sets or similar ETL tools. [ can be trained locally ]
• Understanding of real-time or near real-time inferencing architectures.
- •Basic Knowledge on Kafka ,AWS IAM, AWS EMR and Snowflake.
Total Experience Expected: 06-08 years
BE
At our organization, we are committed to fighting against all forms of discrimination. We foster a work environment that is inclusive and respectful of all differences.
All of our positions are open to people with disabilities.
Discover what working at Sopra Steria looks like...
Are you looking for a place where you can free your creativity and take initiatives, supported by tech experts?
Join us on this adventure where every idea counts and every talent steps up.
Job offers that might interest you
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
You will be working on the Merge & Acquisition process for new worldwide companies joining the group. Your primary role will be to liaise with these companies AD administrators to define AD/AzureAD ma
Reference
7ad77741-2249-49cc-90d2-a34e89e263d9
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
Responsibilities: Design and implement cloud and hybrid (mix of cloud and on-premises) based infrastructure solutions that includes both virtualized compute and storage. Design and implement high avai
Reference
8a22a284-430e-4486-b934-40469bccc2ac
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Mandatory: Proven experience as a Full Stack Developer with a focus on both front-end and back-end technologies with min 6 yrs experience Strong Technical Hands-on experience in Node.js, MongoDB, An
Reference
a1c60ba8-af2d-4110-a67e-6b2375ca5082
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
Should have at least 8+ years of experience in Java/Angular development, Design and Application development Proven experience as a Full Stack Developer with expertise in Spring Boot, Neo4j, Angular,
Reference
b8cf04da-e21a-475d-82df-b9bbdae8620d
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
Strong working experience on ABAP (RICEF) Objects on SAP S4 HANA system with at least 2 Implementation project experience in SAP. Experience in global template development and rollouts Experience in w
Reference
0f7b8561-51d5-4e1e-a817-6036c584458a
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Candidate should be able to - Design, configure, and implement SAP EWM solutions tailored to business requirements. Conduct workshops to gather business requirements and translate them into functional
Reference
432a9b80-08f1-4cd5-9838-b592eec4f1e6
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Primary skills : Requirements:- Candidate with 10+ years of experience in software development life cycle. Experienced professional in application development environment using languages Python, React
Reference
e5102d5f-69f7-42f6-bc83-0e67c2065a9e
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
Role: Automation Engineer Primary Skillset: Ansible, Ansible Tower, Ansible AWX, Ansible Automation platform Secondary Skillset: Terraform, CICD tools, AWS/Azure/GCP Experience: 6-10 years Location: N
Reference
d25ebc83-052d-430f-b449-69719bc897b7
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
8-10 years worked on full-cycle SAP MM implementation projects, including requirements gathering, design, testing, training, and post-implementation support. Minimum 1 full time participation in S4 Ha
Reference
737649ec-4ddc-473a-af44-e3a3e65ffa4f
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
Position: SAP S4HANA Data Migration Lead Location: Chennai/ Noida Experience: 8 -11 years Education: B.E./ B.Tech./ MCA Job Description At least one end to end experience as Data Lead on one projec
Reference
82c151fc-a971-4ebb-ae21-de2fec0c35b4
Expiry Date
Jan 1, 0001