Join Insight Global as a Senior PySpark Engineer with a focus on framework development. Shape the way teams build scalable solutions in a collaborative setting. We are seeking senior-level candidates with extensive knowledge of Spark internals who can bridge gaps in data architecture.

The ideal candidate will enhance existing frameworks over standalone projects, emphasizing solid data engineering practices and effective communication with stakeholders. This hybrid role supports innovative engineering solutions.

Key Responsibilities

• Strengthen frameworks to allow “in the library” building
• Design reusable libraries and patterns for team use
• Optimize Spark SQL processing and transformation costs
• Resolve ambiguous production requirements effectively
• Facilitate seamless operations across shared data platforms

Requirements

• Strong understanding of PySpark internals and performance tradeoffs
...