Senior Staff Applied AI Engineer - Context Retrieval

• 10+ years of software engineering experience, with significant time spent building production retrieval, search, or RAG systems at scale.
• Deep Information Retrieval (IR) expertise: lexical retrieval (BM25, Lucene/Elasticsearch/OpenSearch), dense retrieval (embeddings, ANN indexes — FAISS, ScaNN, HNSW), hybrid retrieval, and learning-to-rank.
• Hands-on experience with modern LLM-era retrieval: RAG architectures, query rewriting, re-ranking with cross-encoders, long-context strategies, and grounding techniques that reduce hallucination.
• Experience designing agentic systems on top of retrieval — search planners, multi-hop / iterative retrieval, self-reflection and sufficiency checks, tool-using agents that decide what to fetch and verify what came back.
• Strong grasp of relevance evaluation: nDCG, MRR, Precision@K, Recall@K; offline/online experimentation; LLM-as-judge frameworks; building human labeling pipelines.
• Experience working across structured and unstructured data — you've indexed and ranked over tables, code, and documents in the same system, and have opinions about how to do it well.
• Track record of building 0→1: you've stood up a retrieval system from an empty repo, made the foundational architectural decisions, and grown it into something that customers depend on.
• Demonstrated ability to operate as a technical leader: setting direction across teams, mentoring senior engineers, and influencing roadmap with research, product, and platform partners.
• Experience building retrieval over enterprise SaaS sources (permissions, freshness, multi-tenancy, ACL-aware indexing).
• Background in agentic systems, tool use, or multi-turn retrieval for LLM agents.
• Contributions to open-source IR/search projects, or publications at SIGIR, KDD, WWW, EMNLP, or similar venues.
• Experience training or fine-tuning embedding models, rerankers, or query understanding models.

Why This Role

• Foundational impact. Retrieval is the single biggest lever on agent quality. The stack you build will sit underneath every Databricks agent and every customer-built agent on our platform.
• Greenfield with scale. You get the rare combination of starting from a clean sheet and having immediate access to massive enterprise scale, real customer data, and a world-class research org.
• The right team. You'll work alongside engineers and researchers behind Lakehouse, Apache Spark™, Delta Lake, MLflow, MosaicML, and DBRX.

Location

This role is based in our Mountain View, CA or San Francisco, CA office. Hybrid in-office collaboration expected.

Pay Range Transparency

Databricks is committed to fair and equitable compensation practices. The pay range(s) for this role is listed below and represents the expected salary range for non-commissionable roles or on-target earnings for commissionable roles. Actual compensation packages are based on several factors that are unique to each candidate, including but not limited to job-related skills, depth of experience, relevant certifications and training, and specific work location. Based on the factors above, Databricks anticipates utilizing the full width of the range. The total compensation package for this position may also include eligibility for annual performance bonus, equity, and the benefits listed above. For more information regarding which range your location is in visit our page here.

Local Pay Range

$228,600—$342,800 USD