How to Build a RAG Pipeline for Financial Services in Hong Kong

RAG (Retrieval-Augmented Generation) combines a vector database of your institution’s proprietary knowledge with a large language model’s reasoning capabilities. For financial services, this solves three critical problems that generic LLMs cannot address alone:

Auditability and Regulatory Traceability

Unlike a black-box LLM, RAG systems retrieve and cite their source documents. When a relationship manager presents a product recommendation to a high-net-worth client, or a compliance officer generates an AML alert narrative, regulators can trace the decision back to the original regulatory circular, policy document, or transaction record. The HKMA’s new AI governance framework (covered in detail in Section 2) explicitly requires institutions to demonstrate model explainability and data provenance. RAG provides this by design—every answer includes not just reasoning, but the specific document and section that informed that reasoning.

This traceability is material during regulatory examinations. When an HKMA inspector asks “Why did your AI system recommend this product to this customer?”, you can produce the exact policy document, client profile, and decision logic that the system relied on. This documentation transforms the AI system from a liability into an asset—proof of compliance, not a gap waiting to be closed.

Knowledge Currency and Compliance Drift Prevention

Financial regulations change constantly. The SFC updates AI guidelines, the HKMA issues supervisory circulars, and internal policies evolve. A model trained on a static dataset becomes a liability—its knowledge is frozen at a point in time, and staff relying on it may be acting on superseded guidance. RAG solves this elegantly: update the document store, and the system’s knowledge is current immediately, with no model retraining required.

You can deprecate outdated documents, track when policies changed, and prove to auditors that your AI system never used stale regulatory text. This is particularly critical in Hong Kong’s fast-moving regulatory environment. When the HKMA issues a new circular on stress testing or the SFC updates its guidance on AI-assisted investment advice, your RAG system can ingest the new guidance within hours. Your team isn’t scrambling to retrain models; you’re simply updating the vector database.

Reduction of Hallucination in High-Stakes Contexts

Hallucination—when an LLM invents plausible-sounding but false information—carries severe consequences in financial services. A mortgage advisor confidently quoting a loan rate that doesn’t exist, or a compliance system generating fictitious AML regulations, can result in regulatory fines or customer harm. By constraining the model to cite only retrieved documents, RAG dramatically reduces hallucination. You can further enforce confidence thresholds and flag uncertain answers for human review, creating a safety valve that stops the system from confidently getting things wrong.

Hong Kong’s regulatory framework for AI is multi-layered, involving the HKMA (banking), SFC (securities), and data protection laws. Cross-border firms must also consider MAS (Singapore) and EU AI Act equivalencies.

HKMA Supervisory Policy Manual and AI Governance Requirements

The HKMA’s supervisory manual includes dedicated sections on model risk management, operational resilience, and AI governance. For RAG systems, key regulatory requirements include:

• Model governance and validation: Institutions must document model development, testing, approval, and ongoing monitoring. Your RAG pipeline requires a model risk governance framework that covers the embedding model, the LLM, and the retrieval logic. Document training data, validation procedures, and performance benchmarks. The HKMA will ask for this during examinations.
• Data quality and lineage: The HKMA expects institutions to track data sources, quality assurance steps, and version control. RAG systems must log which documents were retrieved for each query and when those documents were last updated. Maintain a data lineage map showing the journey from source document through ingestion, chunking, embedding, and retrieval.
• Explainability and interpretability: Financial models must be explainable to supervisors. RAG’s citation mechanism directly satisfies this requirement: each answer includes the source documents, allowing a human to verify the model’s logic.
• Operational resilience: RAG systems must include fallback mechanisms. If your vector database fails, you need a manual override or alternative retrieval path. The HKMA expects resilience plans for AI systems that support critical processes.
• Third-party risk: If you use managed cloud vector stores (e.g., Azure AI Search) or external embedding models, the HKMA requires third-party due diligence documentation. You must ensure your vendor has appropriate audit controls and SOC 2 certifications.

SFC AI Governance Circulars and Securities Regulation

The SFC has issued guidance on AI use in investment advisory, algorithmic trading, and risk management. For RAG systems supporting securities operations:

• Suitability and recommendation transparency: If your RAG system assists in generating investment recommendations, the SFC requires that clients understand how recommendations were derived. Document your retrieval sources and keep audit trails showing which documents informed each recommendation.
• Conflict of interest disclosure: If your RAG pipeline is trained on internal documents that contain information about the firm’s products, the system could unconsciously bias recommendations. Implement conflict-of-interest filters in your retrieval logic. For example, if a client is considering both an in-house product and a competitor’s product, ensure your retrieval treats both equally.
• Algorithm testing and ongoing review: The SFC expects regular backtesting and review of algorithmic systems. Your RAG evaluation framework (covered in Section 9) must include metrics for recommendation quality, compliance drift, and fairness analysis.

PDPO Compliance and Data Privacy in RAG Systems

The Personal Data Protection Ordinance (PDPO) governs how financial institutions handle personal data. For RAG systems, this creates specific requirements:

• Data minimization: Avoid storing customer PII in your vector database. If you ingest documents containing customer names, account numbers, or identification numbers, apply redaction before embedding. Design your retrieval to filter PII from results before they’re shown to users.
• Purpose limitation: If you ingest customer correspondence or transaction narratives, ensure your RAG system only retrieves them for authorized uses (e.g., AML compliance) and never for marketing or unrelated purposes. Document the intended use of each document collection in your governance framework.
• Data access logs: The PDPO requires institutions to track who accessed what personal data. Your RAG logging must record which users queried the system, what was retrieved, and whether PII was exposed. These logs are your defense against PDPO violations.
• Right of access and data portability: If a customer requests their data, you must be able to identify all personal data stored in your RAG system. Maintain data catalogs and lineage for audit trails.

Comparison: MAS (Singapore) and EU AI Act

For firms operating across borders, alignment with broader frameworks accelerates multi-market compliance. MAS guidance on AI governance is closely aligned with HKMA but emphasizes fairness, explainability, and accountability. If you build a RAG pipeline compliant with HKMA standards, MAS compliance is largely achieved by adding fairness audits.

The EU AI Act classifies AI systems by risk. Financial advisory RAG systems fall into the “high-risk” category, requiring impact assessments, transparency, human oversight, and quality management. Firms planning European expansion should design compliance from day one.

A production RAG system for financial services must handle data governance, ingestion, embedding, retrieval, generation, and compliance logging simultaneously. Here is the reference architecture:

Layer 1: Data Governance and Source Management

Before a single document is embedded, define your governance layer. This is not optional—it’s your compliance foundation.

• Document taxonomy: Classify documents by type (regulatory circulars, internal policies, product guides, transaction records) and sensitivity (public, internal, confidential). Only expose appropriate documents to each user role. A junior analyst should not retrieve confidential proprietary trading strategies.
• Version control: Use a document management system (Confluence, SharePoint, or custom solution) as the source of truth. Each document gets a version number, approval date, effective date, and expiration date. Deprecated documents remain indexed (for historical query support) but are flagged as obsolete.
• Audit trail: Log every document ingestion, update, and deletion. Record who approved the document, when it was published, and when it was removed from the RAG system. This is your compliance trail.
• Access control mapping: Define which user roles can retrieve which document types. Build this into your vector database filters so sensitive documents are never even retrieved for unauthorized users.

Layer 2: Data Ingestion and Chunking

Ingestion pipelines must handle diverse financial document formats while preserving semantic structure. Key components:

• Connectors to ingest PDFs, Word documents, HTML, and structured data (CSV, databases).
• Multilingual processing for English and Traditional Chinese with language-aware tokenization.
• Semantic chunking that respects document structure (preserve clauses, sections, and table context).
• Metadata attachment (source document ID, section number, publication date, security clearance level, language, document type).

Layer 3: Embedding and Vector Store

The embedding layer converts chunks into vectors for similarity search. Production requirements include:

• Use domain-appropriate embedding models (detailed comparison in Section 5).
• Store vectors in a vector database with ACID guarantees, backup capabilities, and compliance-ready audit logs.
• Implement data residency controls. If your firm must keep data on-shore, deploy Qdrant or pgvector in your own data center.
• Version your embeddings. If you upgrade embedding models, re-embed all documents and track which model version generated each vector.

Layer 4: Retrieval Strategy and Ranking

Raw similarity search is insufficient for financial contexts. Production retrieval requires:

• Hybrid search (dense + BM25): Combine dense vector search with BM25 lexical search. Some queries (e.g., searching for a specific regulatory reference like “HKMA Circular 2024-01”) are better served by keyword match than semantic similarity.
• Metadata filtering and role-based access control: Filter retrieved chunks based on user role, security clearance, and query intent. A junior analyst should not retrieve confidential proprietary trading strategies.
• Cross-encoder re-ranking: After retrieving candidate chunks, re-rank them using a small, precise cross-encoder model. This improves relevance without increasing latency significantly.
• Query routing: Some queries benefit from multi-source retrieval. A query about “interest rate swaps and hedging” might require documents from product guides, risk policies, and regulatory circulars. Implement intent detection to route queries appropriately.

Layer 5: Generation and Guardrails

The generation layer takes retrieved context and produces an answer. Financial guardrails are critical:

• Model selection and fine-tuning: Choose models known for financial reasoning and reduced hallucination (Claude 3.5 Sonnet, GPT-4o, or Llama 3.1 for on-prem deployment).
• Citation enforcement: Force the model to cite every claim. Use prompt engineering or custom tokenizers to require [Source: Document X] syntax.
• Confidence thresholding: If retrieved context is below a confidence threshold (e.g., all similarity scores < 0.5), instruct the model to refuse the query rather than hallucinate. Log low-confidence queries for human review.
• PII detection and redaction: Before returning results, scan for leaked customer names, account numbers, or identification numbers. Redact or block retrieval if PII is detected.
• Prompt injection and adversarial defense: Financial professionals may attempt to manipulate the system via prompt injection. Validate user inputs and use model guardrails to prevent jailbreaks.

Layer 6: Observability and Compliance Logging

Logs are your proof of compliance. Every query must be recorded with:

• Query content and context (what did the user ask, user role, clearance level).
• Retrieved documents (which chunks were returned, similarity scores, whether they were filtered or re-ranked).
• Generated response (what answer was produced, did the model cite sources).
• User interaction (did the user accept, challenge, or escalate the answer).
• Latency and cost (how long did retrieval take, how many API calls were made).
• Error and risk flags (was low confidence detected, was PII nearly exposed, was adversarial input detected).

Store logs in a tamper-proof system (append-only database or blockchain-backed audit log) to satisfy regulatory audits. Retention must match regulatory requirements (typically 7 years for financial records).

Hong Kong’s financial sector operates in English and Traditional Chinese. Your ingestion pipeline must handle both seamlessly, plus diverse document formats.

Document Format Handling

• PDFs: Most regulatory circulars, policies, and compliance documents are PDF. Use libraries like PyPDF2, pdfplumber, or commercial services (Unstructured.io) that preserve layout, detect tables, and extract text with high fidelity. PDF processing is non-trivial—some PDFs have scanned images, others have embedded text; your pipeline must handle both.
• Word documents (.docx): Internal policies and procedure manuals are often Word files. Extract text while preserving section structure, headings, revision metadata, and tracked changes. This metadata is valuable for understanding document evolution.
• HTML and web content: Regulatory announcements, news, and product pages are HTML. Parse with BeautifulSoup or Readability libraries to extract main content and filter boilerplate (navigation, footers, ads).
• Structured data: Product comparison matrices, fee schedules, and compliance matrices exist in CSV or database tables. Convert to markdown or JSON before chunking to preserve tabular structure. Table structure is semantically important in financial documents.

Multilingual Processing (English + Traditional Chinese)

Dual-language processing is non-trivial and requires careful design:

• Language detection: Use langdetect or TextBlob to identify whether chunks are English or Traditional Chinese. Some documents are mixed-language, requiring chunk-level language detection.
• Separate or unified embeddings: You can either (a) embed English and Chinese in the same vector space (using multilingual models like BGE-M3 or Cohere Embed v3) or (b) maintain separate vector databases. Unified embeddings are simpler operationally but may sacrifice precision. Our recommendation: use multilingual embeddings but add a language metadata filter so bilingual queries can be routed appropriately.
• Query translation: If a user queries in English about “利率互換” (interest rate swaps in Chinese), your system should understand both. Add a query expansion step that translates queries to the other language and merges results.
• OCR for scanned documents: Some older regulatory documents or archived materials are scanned images. Use OCR (Tesseract for open source, Azure Read API for enterprise) to extract text. OCR quality on dense financial tables is often poor, so consider hybrid approaches: extract text where possible, fall back to image-to-text for complex layouts.

Chunking Strategies for Financial Documents

Standard chunking (e.g., split every 512 tokens) damages financial documents. Financial context spans clauses, definitions, and cross-references. Three proven approaches:

• Clause-level chunking: For regulatory documents, each clause is a natural chunk. A clause in an HKMA circular is a complete unit of meaning. Parse clause boundaries (often numbered like 1.1, 1.2) and use these as chunk boundaries.
• Semantic chunking: Use a small embedding model to cluster similar sentences, then chunk at semantic boundaries. Tools like LangChain’s SemanticChunker or custom approaches with sentence transformers can detect topic shifts and chunk accordingly.
• Hierarchical chunking: Create chunks at multiple levels. Level 1: whole section (e.g., “Section 3: AML Requirements”). Level 2: subsections (e.g., “3.1 Customer Identification”). Level 3: sentences or clauses. Store hierarchical relationships in metadata. When retrieving Level 3 chunks, provide Level 1 and 2 context to the LLM.

Version Control and Document Deprecation

Financial policies evolve. Your ingestion must track versions meticulously:

• When a document is updated, do not delete the old version from your vector store. Mark it as “deprecated_on: [date]” in metadata.
• New queries retrieve only current documents (filter where deprecated_on is null).
• Historical queries or audits can access deprecated documents for context (“what was the rule on March 2024?”).
• Maintain a change log: when document X was updated on [date], what changed? Store diffs in your document management system and reference them in queries about policy changes.