Build a Secure RAG System for Your Company

Your company already has an internal training system. It is called asking the same three people the same twelve questions in Slack.

The answers exist. They are just trapped across Google Drive, Notion, PDFs, old proposals, recorded calls, support tickets, and the head of the employee who has been there longest. Search finds filenames. It does not reliably turn scattered knowledge into a useful answer.

A RAG system does. RAG stands for retrieval-augmented generation: before an AI answers, it searches your approved documents, retrieves the most relevant passages, and uses those passages as context. The model is not expected to remember your handbook. It reads the right part of the handbook at question time.

What you are actually building

The useful version is not “ChatGPT, but with our PDFs.” It is a permission-aware company knowledge layer with five parts:

1. Sources: the systems where approved knowledge lives.
2. Ingestion: a pipeline that extracts, cleans, labels, and updates that knowledge.
3. Retrieval: search that finds the right passages, not merely similar words.
4. Generation: an LLM that answers only from retrieved evidence.
5. Controls: identity, permissions, audit logs, retention, and evaluation.

Employee question
      ↓
Identity + permission check
      ↓
Keyword search + vector search
      ↓
Rerank the best passages
      ↓
LLM answers with citations
      ↓
Feedback + audit log

If the security layer is bolted on after the chatbot works, you will rebuild half the system. Design identity and document permissions before choosing an embedding model.

Step 1: decide what the system is allowed to know

Start with one high-value use case: onboarding, sales enablement, support, compliance, or operations. “Search everything” sounds ambitious and usually produces a swamp of duplicate, stale, and contradictory documents.

For every source, assign an owner, a sensitivity level, an audience, and an expiry or review date. A public product guide and a compensation spreadsheet should never enter the same index with the same access rules.

• Public: website copy, public documentation, published policies.
• Internal: SOPs, onboarding guides, meeting notes.
• Restricted: finance, legal, HR, customer data, security procedures.
• Excluded: passwords, API keys, raw authentication tokens, and anything without a legitimate retrieval use.

Delete and revoke flows matter too. When a source document is removed or an employee changes teams, the search layer must reflect that quickly. An index is another copy of your knowledge, so it needs the same lifecycle discipline as the original.

Step 2: ingest documents with useful metadata

The ingestion pipeline connects to approved sources, extracts text, removes repeated headers and navigation, splits the text into meaningful chunks, creates embeddings, and stores both the chunks and their metadata.

Do not chunk every file into arbitrary 500-token rectangles. Keep headings with their paragraphs, table rows with their headers, and procedures with their steps. A chunk should be independently understandable when the model sees it.

Attach metadata that the retrieval layer can enforce:

{
  "source_id": "drive://handbook/leave-policy",
  "title": "Leave Policy",
  "department": "people",
  "allowed_groups": ["all-employees"],
  "owner": "people-ops",
  "version": "2026-07-14",
  "classification": "internal",
  "section": "Parental Leave"
}

Keep the original source reference and document version. Every answer should be able to link back to the exact page or section it used. Citations are not decoration; they let employees verify an answer and notice when the source itself is wrong.

Step 3: use hybrid retrieval

Embeddings are good at meaning. Keyword search is good at exact identifiers: product names, policy codes, ticket numbers, and acronyms. Use both, then rerank the combined candidates before sending a small set of passages to the LLM.

The sequence looks like this:

1. Rewrite the employee's question into a clean search query.
2. Filter candidates by the employee's permissions.
3. Run semantic and keyword retrieval.
4. Rerank the results for relevance.
5. Send only the best permitted chunks to the model.
6. Return a concise answer with source links and an “I don't know” path.

The permission filter must happen before generation. Asking the model to “ignore confidential passages” after retrieving them is not access control. Data the employee cannot open should never enter that employee's prompt.

Step 4: make security part of every request

A private deployment does not automatically make a secure product. The complete path matters: source connector, ingestion worker, vector store, application database, LLM provider, logs, analytics, backups, and admin tools.

Identity and authorization

Use company SSO and map employees to groups. Carry that identity into retrieval filters on every query. Test document-level and, where needed, row-level access. An HR manager and a new sales hire can use the same interface without searching the same corpus.

Encryption and network boundaries

Encrypt data in transit and at rest, keep secrets in a proper secret manager, use private networking where your risk profile requires it, and separate development from production. Do not copy real restricted documents into a developer laptop just to make testing convenient.

Provider and retention controls

Choose vendors whose data handling, retention, region, and model-training terms match your requirements. Minimize what leaves your boundary: retrieve fewer passages, redact unnecessary personal information, and avoid placing raw documents into application logs.

Prompt-injection resistance

Documents are untrusted input too. A file can contain instructions telling the model to reveal secrets or ignore policy. Treat retrieved text as evidence, not instructions. Separate system rules from document content, restrict tool access, validate outputs, and require human approval for consequential actions.

Step 5: turn search into employee training

A search box answers questions. A training system helps someone become competent. Use the same retrieval layer to create role-specific learning paths grounded in current company material.

• Onboarding guide: “Teach me our delivery process over five days and quiz me after each module.”
• Scenario coach: “Role-play a customer objection, then score my response against our sales playbook.”
• Process companion: “Walk me through issuing a refund one step at a time, citing the current SOP.”
• Knowledge checks: Generate questions from approved passages, record weak areas, and recommend the next source to read.
• Manager view: Show aggregate gaps without exposing private employee conversations unnecessarily.

Keep high-stakes decisions out of autopilot. The assistant can explain a policy; HR owns the policy. It can coach a support reply; an employee remains accountable for what gets sent. The system should make judgment better, not make judgment disappear.

Step 6: test answers before employees trust them

Build an evaluation set from real questions, not questions invented by the team building the chatbot. Include easy lookups, ambiguous requests, conflicting documents, outdated policies, permission traps, unanswerable questions, and malicious instructions hidden inside files.

Measure at least four things:

• Retrieval quality: did the right source appear in the top results?
• Groundedness: is every factual claim supported by a cited passage?
• Permission accuracy: did any result cross an access boundary?
• Usefulness: could an employee take the next step from the answer?

Ship to one team first. Put a “wrong, outdated, or unsafe” button beside every answer. Route feedback to the document owner, because many apparent AI failures are actually knowledge-management failures: two policies disagree, a process changed, or nobody owns the source.

A sensible first version

Week 1  Pick one use case and inventory approved sources.
        Define classifications, owners, and access groups.

Week 2  Build ingestion, chunking, metadata, and deletion sync.
        Index a small clean corpus, not the whole drive.

Week 3  Add hybrid retrieval, reranking, citations, and refusal rules.
        Connect SSO and enforce permissions before retrieval.

Week 4  Test with real employee questions and adversarial cases.
        Pilot with one team, collect feedback, and fix the sources.

After   Add role-based lessons, scenario practice, and knowledge checks.
        Re-evaluate whenever models, prompts, or source systems change.

The stack matters less than the boundaries

You can build this with managed AI services or assemble your own application, object storage, parser, vector-capable database, reranker, and model gateway. The right choice depends on scale, latency, compliance, existing cloud, and the skills of the team operating it.

But no vendor choice rescues bad boundaries. A great embedding model cannot tell which policy is authoritative. A private vector database cannot fix an over-shared Drive. A clever prompt cannot replace authorization. The durable work is deciding what the system may know, who may retrieve it, how answers prove their source, and who owns corrections.

The short version

Start narrow. Clean the source documents. Preserve permissions in metadata. Filter before retrieval. Combine keyword and semantic search. Make the model cite its evidence and admit when the evidence is missing. Test access boundaries harder than answer fluency. Then use the same trusted layer to coach employees with the procedures your company actually follows.

That is the difference between a demo that chats with PDFs and a company system people can safely learn from.

Want it built for you? Partner with NeuralArc to turn your private documents into a secure, searchable knowledge and training system.