Gen AI Data Engineer - Pyspark/Python

​The Role: GenAI Data Engineer

Location: London (or) Edinburgh, UK

Position Type: Contract Inside IR35

Remote work option Available: Hybrid – 2 Days Onsite

Job Description:

Essential skills/knowledge/experience:

• Strong experience with PySpark, distributed data processing, and large scale ETL/ELT pipelines.

• Strong SQL expertise including star/snowflake schema design, indexing strategies, writing optimized queries, and implementing CDC, SCD Type 1/2/3 patterns for reliable data warehousing.

• Advanced proficiency in Python for data engineering, automation, and ML/GenAI integration.

• Hands‑on expertise with AWS services (S3, Glue, Lambda, EMR, Bedrock / custom model hosting).

• Practical experience with GenAI/LLM model creation, fine tuning, benchmarking, and evaluation.

• Solid understanding of RAG architectures, embeddings, vector stores, and LLM evaluation methods.

• Experience working with structured and unstructured datasets (documents, logs, text, images).

• Familiarity with scalable data storage solutions (Delta Lake, Parquet, Redshift, DynamoDB).

• Understanding model optimization techniques (quantization, distillation, inference optimization).

• Strong capability to debug, tune, and optimize distributed systems and AI pipelines.

Desirable skills/knowledge/experience:

• Pyspark, Python, SQL,AWS, GenAI