Data Engineer PySpark
__jobinformationwidget.freetext.LocationText__
Chennai, Tamil Nadu, India
- Sopra Steria
- Engineering, Development, Applications
- 6 to 10 years
- Standard
- India
- All Sectors
This vacancy has now expired. See all jobs
About Sopra Steria
Sopra Steria, a major Tech player in Europe with 56,000 employees in nearly 30 countries, is recognized for its consulting, digital services and software development. It helps its clients drive their digital transformation and obtain tangible and sustainable benefits. The Group provides end-to-end solutions to make large companies and organizations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a fully collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients in order to build a positive future for all. In 2023, the Group generated revenues of €5.8 billion.
The world is how we shape it.
We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Proficiency in below listed skills will be crucial in building and maintaining pipelines for training and inference datasets.
Responsibilities:
• Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines.
• Utilize PySpark for data processing, transformation, and preparation for model training.
• Leverage AWS EMR and S3 for scalable and efficient data storage and processing.
• Implement and manage ETL workflows using Stream sets for data ingestion and transformation.
• Design and construct pipelines to deliver high-quality training and inference datasets.
• Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities.
• Optimize and fine-tune pipelines for performance, scalability, and reliability.
• Ensure IAM policies and permissions are appropriately configured for secure data access and management.
• Implement Spark architecture and optimize Spark jobs for scalable data processing.
Requirements:
Mandatory
• Proficiency in Advanced SQL (Window functions), Spark Architecture, Pyspark or Scala with Spark, Hadoop.
• Proven expertise in designing and deploying data pipelines.
• Strong problem-solving skills and ability to work effectively in a collaborative team environment.
• Excellent communication skills and ability to translate technical concepts to non-technical stakeholder
Desirable
• Hands-on experience with Airflow, S3, and Stream sets or similar ETL tools. [ can be trained locally ]
• Understanding of real-time or near real-time inferencing architectures.
- •Basic Knowledge on Kafka ,AWS IAM, AWS EMR and Snowflake.
Total Experience Expected: 06-08 years
BE
At our organization, we are committed to fighting against all forms of discrimination. We foster a work environment that is inclusive and respectful of all differences.
All of our positions are open to people with disabilities.
Discover what working at Sopra Steria looks like...
Are you looking for a place where you can free your creativity and take initiatives, supported by tech experts?
Join us on this adventure where every idea counts and every talent steps up.
Job offers that might interest you
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
SAP EWM Functional experience between 6 and 8 years. Minimum 1 E2E implementation & 2 support projects. Work self-directed and independently. Able to lead and work as a subject matter expert in the
Reference
92c1bec5-1e6e-4847-8c32-aac79548b9d6
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Candidate should be able to: Design, configure, and implement SAP EWM solutions tailored to business requirements. Conduct workshops to gather business requirements and translate them into functional
Reference
c9608057-d123-4572-9612-6a1814be43fc
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Mandatory Certification: CBAP (Certified Business Analysis Professional) CCBA (Certification of Competency in Business Analysis) from IIBA PMI-PBA (PMI Professional in Business Analysis) Mandatory
Reference
b326dcfb-fdfb-44c4-987d-32e2d508b5db
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
Position : Java Fullstack Senior Technical Lead Experience: 10+ years Location: Noida/ Chennai Education: B.E./ B.Tech./MCA Skill Requirement: Exposure to all the phases of software lifecycle. O
Reference
74b23240-6f77-48a9-8e0d-02ee0da1fb38
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Good functional knowledge in standard SAP Production modules along with integration with other SAP modules. The relevant solution capabilities for the product are: Master Planning PP-MP Demand Managem
Reference
70e35b0c-0a64-4ef5-bb1f-a58119d80908
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
8-10 years design/implementation/consulting with SAP Production planning and Quality Management Minimum 1 full time participation in S4 Hana Greenfield implementation in the European project – full ti
Reference
453ed896-bae3-46ca-ac67-c0fc6008cd7c
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Mandatory Skills: Expertise in EMBEDDED C Expertise in V&V as per DO178B/C guidelines Experience in Requirement Analysis Expertise in Test plan design and Test development Expertise in Unit Testing, I
Reference
cfaacc33-1125-4fc4-ba5d-95d2ee0445b3
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
8-10 years design/implementation/consulting with Sap FI/CO Minimum 1 full time participation in S4 Hana Greenfield implementation in the European project – full time mean: participation in the Functio
Reference
be051668-5cf0-4219-bac8-9edc36ac5586
Expiry Date
Jan 1, 0001
Salary
Location
Bengaluru, Karnataka, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Bengaluru, Karnataka
Description
Nutshell JD: Must to Have Skills:- Python, PySpark, AWS (Lambda, RDS), Spark SQL Experience with Foundry (Slate, Workbook etc.) Schedulers (like Airflow) Good to Have: Java Script, Optimizing the q
Reference
8b500850-e464-4edf-b203-736c1ca59415
Expiry Date
Jan 1, 0001
Salary
Location
Noida, Uttar Pradesh, India
Vacancy language
English
Job Type
Standard
Experience Level
6 to 10 years
Department
Engineering, Development, Applications
Brand
Sopra Steria
Sector
All Sectors
Remote
No
Location
Noida, Uttar Pradesh
Description
What you'll be doing: Developing the Cloud Infrastructure assets and Solutions Analysing the business requirements and build the solution as per the plan Working as part of the Practice team to delive
Reference
376858b3-9ba3-4cac-9cfb-66e52084f7af
Expiry Date
Jan 1, 0001