Staff Data Engineer | London (Hybrid) | Python / Databricks / Spark | SC Clearance | £150 K

Want to build the data platforms governments use to understand global supply chains?

My client is a fast-growing AI company that maps global supply chains, from direct suppliers all the way down to raw materials. Governments, customs agencies, insurers and large enterprises use their platform to spot risks early, keep illegal goods and products made with forced labor off the market, and make supply chains more resilient.

As Staff Data Engineer, you join the Operations & Compliance team within the public sector business in the UK. The team, currently three data engineers, runs dedicated platform deployments for government clients, where customers bring in their own data and fuse it with the company's supply chain data. A flagship programme with UK central government is now being copied in other countries, and this hire supports new engagements across the UK and Europe.

You take a lead position: building services as real products, standardizing, building for scale and working closely with internal engineering on migrations. Your time is split about 50/50 between architecture and system design and hands-on work.

Your responsibilities:

• Working with customers to understand, map and standardize their data for integration
• Building and running pipelines for data ingestion and entity resolution
• Improving existing tooling and capabilities
• Managing migrations of internal product changes into secure customer environments
• Handling compliance work, such as penetration tests and security reviews
• Creating the designs, architecture and system designs for major initiatives
• Coordinating larger engagements and changes, as the bridge between internal engineering and customer-facing teams
• Bringing new ideas, leading on engineering best practices and helping the team grow their skills

Your profile:

• 8+ years of experience in data engineering or a closely related role
• Hands-on experience with Python and Databricks (or an equivalent stack); Spark and SQL
• A proven track record of putting data pipelines into production, from design to monitoring and maintenance
• Strong hands-on data wrangling: building complex data models from many sources and schemas
• Comfortable in cloud environments (AWS/Azure) and core services such as EC2, S3, IAM and Dockerized services
• A strong grasp of engineering best practices, data security and governance, and how pipelines affect the wider system
• Experience leading or mentoring engineers, with good communication across internal teams and with external stakeholders
• A flexible, self-driven startup mindset: happy to wear multiple hats
• Eligible for UK Security Check (SC) clearance, which requires 5+ years of UK residency. Already holding SC is a strong plus.
• Nice to haves - deploying ML pipelines, managing a data platform & experience in the public sector.

Hybrid: London office, 1 or two 2 a week at the office, the rest can be worked from home.

Salary: Max £150k base

Interested? Please contact me at j.vanbaalen@doghouse.nl