Apache Spark Senior Technical Lead

About Sopra Steria
Sopra Steria, a major Tech player in Europe with 50,000 employees in nearly 30 countries, is recognised for its consulting, digital services and solutions. It helps its clients drive their digital transformation and obtain tangible and sustainable benefits. The Group provides end-to-end solutions to make large companies and organisations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients in order to build a positive future for all. In 2024, the Group generated revenues of €5.8 billion.
The world is how we shape it.

Job Title: Technical Lead/Senior Technical Lead

Location: Chennai

Total Experience Expected: 06-12 years

Role Overview:

We are looking for a skilled Data Engineer to build and maintain scalable data pipelines, manage large datasets, and support analytics initiatives. This role involves working with Big Data technologies like Spark, Scala, SQL/PLSQL, and cloud-ready environments.

Experience:

1. 8+ years of relevant experience in data engineering / big data development
2. Proven track record of handling large-scale data solutions

Key Responsibilities:

1. Maintain, develop, and design data ingestion and transformation pipelines using Spark and Scala
2. Process and optimize SQL and PL/SQL queries for large datasets
3. Integrate and process data from multiple sources for analytics and reporting
4. Collaborate with BI and Analytics teams to support reporting and dashboard needs
5. Ensure data quality, performance, and security
6. Provide innovative solutions to complex business/technical problems
7. Identify and resolve technical, integration and development issues
8. Implement best practices, standards and processes to ensure quality of the final product
9. Responsible for development of features independently
10. Continuous build, automated testing, and release management
11. Participate in status meetings and provide regular updates

Required Skills:

Technical:

1. Expertise in Spark and Scala for large-scale data processing
2. Proficiency in Python and SQL, including hands-on experience with PL/SQL
3. Experience in data integration/ETL workflows;
4. Strong knowledge of relational databases such as Oracle and PostgreSQL

Professional Competencies:

1. Excellent verbal and written communication skills
2. Result-oriented with strong analytical and problem-solving skills
3. Ability to work in a fast-paced, agile, and collaborative environment

Desired Skills:

1. Exposure to cloud data services (AWS preferred)
2. Proficiency in Linux/Unix shell scripting for automation and data operation
3. Experience working in Agile environments – Scrum and/or SAFe frameworks
4. Exposure to DevOps culture and practices, including CI/CD and automation
5. Understanding of software engineering lifecycle – Development, Testing, and Production Support
6. Experience in production issue handling and transformation support management

Good to Have Skills:

1. Exposure to Dataiku preferred
2. Awareness of cloud and big data architecture principles

BE / B.Tech/ MS/ M.Tech/ MCA

At our organization, we are committed to fighting against all forms of discrimination. We foster a work environment that is inclusive and respectful of all differences.

All of our positions are open to people with disabilities.