Role SOP and operating playbook
Search & Knowledge Systems Engineer Operating Playbook
This operating playbook follows a search and knowledge-systems feature from the information need through source and access checks, retrieval design, test evidence, release handoff and exception handling. Assign authority and cadence locally before use.
Explore the RAG & Enterprise Search course- Resource
- Role SOP and operating playbook
- Evidence
- United States
- Reviewed
- October 5, 2026
- Format
- Reusable professional guide
An evidence-derived operating playbook for a bounded enterprise search or RAG feature, from information task and source access through evaluation, handoff and exception handling.
Evidence scope: Evidence-derived role resources from a purposive review of 105 distinct U.S.-eligible employer requisitions, representing 67 employer clusters, observed on 4 October 2026. The corrected core sensitivity subset contains 81 requisitions. The study is not a representative estimate of U.S. hiring or a universal description of employer practice.
This evidence-derived operating model helps an AI Engineer, Search & Knowledge Systems move a bounded enterprise search or retrieval-augmented feature from an information need to a tested, supportable result. Adapt the activities, timing, tools and decision owners to the organization. It is not a universal employer policy. The model combines source preparation, retrieval design, application integration, evaluation and operations seen in distinct U.S. roles; one person does not automatically own every part. AbbVie; Glean; Pi Security; LTS.
Purpose and scope
Use this playbook when a team asks for new or changed search, retrieval or knowledge access across defined sources. The working objective is a result that is relevant to the user's task, traceable to approved source material, restricted to authorized identities and observable after release. The sequence includes design and tests even when another team operates the final service. Stop or hand off at the decision boundary assigned by the employer.
Start triggers: a new information task; a new or changed source; a retrieval-quality complaint; changed access rights; a release that alters indexing, ranking or answer assembly; or an operational incident. Close condition: the authorized owner accepts a bounded output and its test evidence, or records the unresolved decision and assigned next action. A quality failure is not closed by a convincing demonstration alone. Amazon retrieval engineering and evaluation; Exa evaluation role.
Roles and inputs
| Role in the work | Typical contribution | Confirm locally before starting |
|---|---|---|
| Requester or product owner | Defines the user task, acceptance use and priority. | Who accepts scope and release risk? |
| Source or domain owner | Identifies authoritative material, meaning, update process and permissible use. | Who approves new sources and resolves conflicting documents? |
| Search/knowledge engineer | Designs and implements the assigned ingestion, retrieval, API or evaluation work; records evidence and trade-offs. | Which components and decisions are within the engineer's assigned authority? |
| Identity/security owner | Defines or approves access rules and investigates denied/overexposed results. | Who may change source ACL mappings and stop exposure? |
| Platform/service owner | Owns deployment, reliability, monitoring and incident response where separated from engineering. | Who can release, roll back or rebuild an index? |
The interfaces are grounded in postings that explicitly name domain experts, product, backend, platform, security and evaluation colleagues. The table is a local assignment aid, not a claim that each employer uses these exact titles. AbbVie; Pi Security; Exa.
Minimum inputs before design: user question or search task; intended user groups; source list and owners; document formats and metadata; current access method; refresh requirement; sample questions and known wrong answers; operating targets if assigned; and the organization's release/incident route. Mark an absent input as “unknown,” ask the owner and keep it visible in the decision record. Do not replace missing access information with a technical assumption.
Trigger-to-close workflow
- Frame the task. Write one sentence naming the user, information need and action supported by the result. Record the source owner, authorized user groups, freshness need and what should happen when evidence is absent. Ask a domain expert to resolve terms with more than one meaning. Output: an agreed information-task brief. AbbVie requirements interface.
- Inspect sources and rights. Inventory each repository, format, update path and source identifier. Confirm permission to ingest or query it. Trace how user identity and source ACLs will be enforced at retrieval and result display. If an access rule is unclear, obtain a decision from the named identity/source owner before loading or exposing material. Output: source and access map. Amazon entitlement-aware retrieval; Pi Security metadata filters.
- Prepare and verify the information. Extract or connect the sources, preserve document identity and update time, and inspect difficult structures such as tables, headings and multi-page context. Validate a sample against the original; route malformed items for repair. Choose graph/entity modelling only when the task needs those relationships. Output: tested ingestion or federation path and source-quality observations. AbbVie graph-backed loading and validation; Amazon ingestion pipelines.
- Select retrieval behavior. Establish a simple baseline using the same judged questions and authorized identities. Compare lexical, vector, hybrid, structured-filter and ranking options only where appropriate. Document the selected path, alternatives, latency/cost implications and source trace. Output: decision note and working retrieval path. Amazon hybrid search; Pi Security chunking/reranking.
- Test relevance, support and access separately. Use representative and edge-case questions. Check whether the right source appears, whether the cited passage actually supports an answer, whether a denied identity receives no restricted passage, and whether insufficient evidence causes the agreed stop or escalation. Include changed/removed sources. Record failures, owner and retest result. Output: evaluation record and open-defect list. Exa evaluation frameworks; Amazon regression harness; LTS quality/security constraints.
- Integrate and hand off. Agree API fields, errors, versioning, logging and user-visible source references with the application or platform owner. Present measured trade-offs and unresolved risks to the person who can accept them. Do not infer release authority from a senior title. Output: interface contract, handoff and decision record. Glean API standards; Accellor design trade-offs.
- Release through the local process and observe. Run the agreed pre-release checks, retain the version of source/index/configuration tested, and verify initial behavior with authorized and denied identities. Watch the assigned quality, error, latency and cost signals. The designated service owner decides rollback or recovery. Output: release evidence, monitoring owner and next review date. LTS production troubleshooting; Amazon runbooks and dashboards.
- Close or escalate. Summarize what changed, which tests passed, open limitations and who owns each follow-up. An unresolved access failure, unsupported consequential answer or broken source trace goes to the locally designated owner and remains open until that owner records a decision and retest. Output: concise handoff or incident record.
Working rhythm to assign locally
| Rhythm | Review or action | Evidence and limitation |
|---|---|---|
| Daily, when assigned | Check ingestion failure, source freshness, quality/error alerts and user-reported defects; implement and test bounded changes. | One Amazon posting describes starting with dashboards; Acquia names daily production coding. Neither sets every role's schedule. |
| Weekly, when useful | Triage recurring failed questions, review source-owner feedback, update the defect list and agree upcoming source or release changes. | A customer-facing Amazon role names weekly feedback syncs; Smartsheet names a weekly support rotation. These are different roles. |
| Monthly, if agreed | Reconfirm source ownership, access mappings, sample quality, refresh reliability, costs and retired content. | Suggested governance rhythm for adaptation; the vacancy evidence does not establish a universal monthly cycle. |
| At each release or source/access change | Re-run relevant retrieval, citation, permission and regression tests; document the version and approver. | NetSpeek explicitly names per-release evaluation. |
| During an incident | Contain exposure or failure through the local incident route; preserve evidence; identify source, connector, index, ranking or identity cause; verify recovery. | Amazon names on-call and runbooks; LTS names production issue troubleshooting. |
Most reviewed U.S. advertisements leave task cadence unspecified. Use the table to negotiate actual responsibilities, not to impose an unsupported rota.
Exception or disruption decisions and escalation
| Observation | Engineer's immediate action | Decision or handoff |
|---|---|---|
| A restricted source appears for a denied identity | Stop the affected output path, preserve the test identity/query/source and alert the designated incident channel. | Identity/security owner decides containment and rule correction; service owner controls release or rollback. |
| A citation exists but does not support the answer | Mark the case failed, retain the retrieved span and route the answer for correction or abstention. | Product/domain owner decides whether the answer may be shown and what level of evidence is sufficient. |
| Source content is stale, conflicting or malformed | Record source ID, timestamp and failure; avoid silently changing meaning during extraction. | Source owner resolves authority and content; platform/data owner repairs connector or parser as assigned. |
| Retrieval quality improves but latency or cost exceeds target | Compare the same judged questions and workload measurements, then document trade-offs. | Accountable service/product owner selects the approved operating envelope. |
| A new connector or federated source is proposed | Check source rights, identity propagation, freshness and failure behavior in a bounded test. | Source, identity and platform owners approve their respective boundaries. |
These handoffs are a conservative model. In a real organization, one named owner may cover several columns; use the actual approval chain.
Records and quality measures
Keep a compact record for each change: task and requester; source owner and version; user/identity class; ingestion or federation route; retrieval design; test-set version; relevance and source-support results; authorized/denied checks; latency/cost observations if assigned; defects; decision owner; release/rollback reference; and next review. Store only permitted data in the organization's approved location.
Useful measures include judged source retrieval at a declared cutoff, proportion of answers whose cited span supports the claim, denied-access test pass rate, stale-source incidents, ingestion failures, latency distribution and cost per workload unit. Define the population, denominator and review period before comparing values. A green aggregate cannot excuse a failed restricted-source test. LTS evaluation dimensions; Amazon operational measures.
Reusable operating record
| Field | What to enter |
|---|---|
| Information task and authorized users | The real question or workflow, user groups and decision supported. |
| Sources and owners | Repository names, source identifiers, approved uses, refresh rules and contacts. |
| Method and alternatives | Baseline, selected retrieval path, measured comparison and rejected options. |
| Access boundary | Identity propagation, source ACL behavior and denied-user test cases. |
| Quality evidence | Judged questions, citation-span checks, missing-evidence handling and defects. |
| Operating limits | Locally approved latency, reliability, freshness and cost targets. |
| Decisions and handoffs | Named owner, decision, date, unresolved question and next action. |
| Release and review | Tested version, release/rollback owner, result and review date. |
Worked example
Worked example
Fictional example for learning purposes.
A U.S. equipment-service company asks for a search feature that helps authorized support agents locate current troubleshooting guidance. The source owner identifies product manuals and approved service bulletins; a restricted engineering repository must remain unavailable to the support role. The engineer first records sample agent questions and source owners, then checks PDF extraction, bulletin update dates and access metadata. A baseline lexical index retrieves exact part numbers well but misses symptom descriptions. The engineer compares a hybrid route on the same reviewed question set and keeps part-number filtering intact.
The test set includes a request for a current repair step, an obsolete bulletin, an ambiguous symptom and a query whose supporting document is restricted. The team measures judged top-five source coverage and checks each proposed citation against the exact passage. The restricted query returns no engineering document for a support identity. One ambiguous answer lacks enough support, so the application displays the approved “insufficient evidence” path and sends the case to the domain owner. The engineer hands the API response shape, source IDs, failure cases and monitoring signals to the application and service owners. The source owner signs off content scope; the identity owner signs off the access mapping; the service owner approves the release and rollback plan.
After release, a connector fails to refresh one bulletin. The monitoring alert identifies the source ID and last successful update. The engineer records the failure, blocks stale content from the affected result path under the team's approved rule, and routes connector repair to the platform owner. A retest checks freshness, retrieval and denied access before the incident is closed.
Quick reference
| Before work | During work | Before release | After release |
|---|---|---|---|
| Name task, source owners, users and access rules. | Preserve source identity; compare methods on the same judged cases. | Test relevance, claim support, denied access and failure behavior. | Watch assigned signals; document defects, owners and retests. |
Quick reference
Use the resource in five moves
- Read the role purpose and expected outputs.
- Compare the model with the local role and authority boundaries.
- Select only statements supported by real evidence.
- Adapt the reusable fields without inventing experience or approvals.
- Review the result with the accountable person before operational use.