Data Engineer PySpark
__jobinformationwidget.freetext.LocationText__
Bengaluru, Karnataka, India
- Sopra Steria
- Engineering, Development, Applications
- 6 to 10 years
- Standard
- Bengaluru, Karnataka
This vacancy has now expired. See all jobs
About Sopra Steria
Sopra Steria, major Tech player in Europe recognised for its consulting, digital services and software development, helps its clients drive their digital transformation and obtain tangible and sustainable benefits. It provides end-to-end solutions to make large companies and organisations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a fully collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients in order to build a positive future for all. With 50,000 employees in nearly 30 countries, the Group generated revenue of €5.1 billion in 2022.
The world is how we shape it.
We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Proficiency in below listed skills will be crucial in building and maintaining pipelines for training and inference datasets.
Responsibilities:
• Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines.
• Utilize PySpark for data processing, transformation, and preparation for model training.
• Leverage AWS EMR and S3 for scalable and efficient data storage and processing.
• Implement and manage ETL workflows using Stream sets for data ingestion and transformation.
• Design and construct pipelines to deliver high-quality training and inference datasets.
• Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities.
• Optimize and fine-tune pipelines for performance, scalability, and reliability.
• Ensure IAM policies and permissions are appropriately configured for secure data access and management.
• Implement Spark architecture and optimize Spark jobs for scalable data processing.
Requirements:
Mandatory
• Proficiency in Advanced SQL (Window functions), Spark Architecture, Pyspark or Scala with Spark, Hadoop.
• Proven expertise in designing and deploying data pipelines.
• Strong problem-solving skills and ability to work effectively in a collaborative team environment.
• Excellent communication skills and ability to translate technical concepts to non-technical stakeholder
Desirable
• Hands-on experience with Airflow, S3, and Stream sets or similar ETL tools. [ can be trained locally ]
• Understanding of real-time or near real-time inferencing architectures.
- •Basic Knowledge on Kafka ,AWS IAM, AWS EMR and Snowflake.
Total Experience Expected: 06-08 years
BE
At our organization, we are committed to fighting against all forms of discrimination. We foster a work environment that is inclusive and respectful of all differences.
All of our positions are open to people with disabilities.
Discover what working at Sopra Steria looks like...
Are you looking for a place where you can free your creativity and take initiatives, supported by tech experts?
Join us on this adventure where every idea counts and every talent steps up.
Job offers that might interest you
Salary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Good functional knowledge in standard SAP Production modules along with integration with other SAP modules.The relevant solution capabilities for the product are:Master Planning PP-MPDemand Management
Reference
70e35b0c-0a64-4ef5-bb1f-a58119d80908
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Primary skills :Candidate with 8 to 10 years of experience in software development life cycle.Experienced professional in application development using below tech stackPythonReact JSJavaScriptTypeScri
Reference
0df80d01-ad3e-4fa6-843d-602b70604358
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Design, configure, and implement SAP EWM solutions tailored to business requirements.Conduct workshops to gather business requirements and translate them into functional specifications.Integrate SAP E
Reference
bdc195fa-4e62-4cf0-9bca-c995b8db41db
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Key Responsibilities:ARAS Innovator Customization: Customize and configure ARAS Innovator solutions based on business requirements, including custom item types, workflows, and business logic.Developme
Reference
81cff6c0-f058-41cd-b351-1677a1bf9b87
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Primary Skills:6-8 years of experience in Node.js + Typescript development, maintenance & run mode operations.At least 2 years of experience in the area of product development using Node.js, TypeScrip
Reference
e2ff7395-5c95-4e33-9f1d-d1da44502730
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
We are seeking a skilled Automation Tester with AWS Cloud expertise to join our dynamic team in Bengaluru, Karnataka, India. In this role, you will be responsible for designing, developing, and mainta
Reference
2921f560-b63e-4d3d-8da4-eff7ed43076b
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Primary: Python, React, HTML, CSS, FastAPIGood knowledge of Agile/SAFe. Secondary: Microservice, Full stack frameworks Good to have:Knowledge on AWS (lamda & API gateway), Pytest is an advantageTotal
Reference
e09757b8-1815-4946-8bdd-a87774fb58cb
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Overview:We are looking for a talented and experienced ARAS Expert to join our team. As an ARAS Technical Lead, you will work with Aras Innovator, a leading PLM platform, to design, develop, and imple
Reference
5ced89c4-fbe0-46f7-ae52-aaa73a7ff38e
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Primary skills :Requirements:-Candidate with 10+ years of experience in software development life cycle.Experienced professional in application development environment using languages Python, React, J
Reference
e5102d5f-69f7-42f6-bc83-0e67c2065a9e
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah SmithSalary
Location
Bengaluru, Karnataka, India
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Location
Bengaluru, Karnataka
Description
Embedded Testing CMC - Experience 6 to 8 years Mandatory Skills: Expertise in EMBEDDED C++ Experience in Embedded C Experience in V&V as per DO178B/C guidelines Experie
Reference
b80c67c4-ba13-42a1-b3d1-36b861d38305
Expiry Date
Jan 1, 0001
Author
Sarah SmithAuthor
Sarah Smith