Data Ingestion & Knowledge Base Management

To answer questions and return accurate search results, Visperity transforms your web pages, product catalogs, and help articles into semantic AI vector embeddings.

This guide explains how vector search works in plain English, how to import and manage your content, and how to configure custom web scraper exclusion rules and end-of-page cutoffs to exclude recurring site headers, footers, sidebars, and referenced cross-sell SKUs.

1. How AI Vector Search Works

Unlike traditional keyword search engines that only look for exact character matches (e.g. typing "raincoat" won't find a product titled "waterproof jacket"):

• Understands Meaning: Visperity converts the meaning and context of your products and articles into mathematical concept maps.
• Synonyms & Context: A visitor searching for "how do I return a damaged shirt?" will instantly match your "30-Day Refund & Exchange Policy", even if the word "shirt" doesn't appear in that document.
• Exact SKU & ID Support: In addition to semantic AI matching, Visperity also checks exact product SKUs, tags, and External IDs for instant lookup.

2. Ingestion Methods

Navigate to the Data Import section (section=ingestion) on your dashboard to choose from four ingestion options:

┌───────────────────────────┬───────────────────────────┬───────────────────────────┐
│   Single URL Crawler      │    XML Sitemap Importer   │     CSV Bulk Upload       │
│ Crawl 1 page instantly    │ Index entire website      │ Upload catalog spreadsheet│
└───────────────────────────┴───────────────────────────┴───────────────────────────┘
                              │
                              ▼
               Manual Custom Document / URL Ingestion
               (Add bespoke text, price, image & SKUs)

Method 1: Single Web Page Crawler

Best for quickly indexing individual product pages, blog posts, or newly published support articles.

1. In the Single Page Crawler card, enter the full web page URL (e.g. https://example.com/products/deluxe-headphones).
2. Click Crawl & Vectorize URL.
3. Visperity's web crawler fetches the page, automatically applies your dataset's exclusion, region, and end cutoff rules, generates an AI search summary, creates the vector embedding, and saves it to your dataset within seconds.

Method 2: XML Sitemap Ingestion

Best for indexing your entire website or online store catalog automatically in the background.

1. In the XML Sitemap Ingestion card, enter your sitemap URL (e.g. https://example.com/sitemap.xml or https://example.com/sitemap_index.xml).
2. Click Parse Sitemap in Background.
3. Visperity reads your sitemap (including nested sitemap indexes), discovers all page URLs, and queues them into the Knowledge Ingestion Queue for automatic background processing.

Method 3: CSV Bulk Upload

Best for importing large product catalogs, offline knowledge bases, or content with explicit thumbnail image URLs and product prices.

1. Prepare your spreadsheet in standard CSV format with the following columns:

2. In the CSV Bulk Upload card, click the Example CSV button to download a sample reference spreadsheet.
3. Choose your .csv file and click Upload & Ingest CSV.
4. The records will be queued and processed automatically.

Method 4: Manual Document & Custom URL Ingestion

Best for custom FAQ answers, internal company policies, or creating records where you want full manual control over the text, price, image, and SKUs.

1. Click + Manually Add Document / URL to open the manual ingestion form.
2. Fill in the fields:
   • Target Resource URL: The link visitors will navigate to when clicking the search result.
   • Document / Product Title: The headline displayed in search results.
   • Document Text Content: The full text used to generate the AI vector embedding.
   • Product / Resource Price (Optional): Optional price (e.g. $49.99).
   • External ID (Optional): Custom system identifier or API record ID.
   • Associated Product SKUs (Optional): Comma-separated SKUs for instant exact matching.
   • Thumbnail Image (Optional): Enter an existing image URL or click Upload to S3/CDN to upload a product photo directly from your computer.
3. Click Save & Vectorize Document.

3. Content Indexer & Web Scraper Customization Options

Web pages often contain repetitive layout elements (such as site navigation headers, footer menus, cart drawers, and cookie notices) or cross-sell accessory SKUs that you do not want indexed for a product page. Visperity provides fine-grained scraper controls configured per dataset in Datasets & Knowledge (section=datasets).

Dual-Purpose Segmentation: HTML Comments & CSS Selectors

The custom_start and custom_end fields feature smart dual-purpose support:

• 1. HTML Comments (Priority #1): If you embed comment tags such as <!-- START_CONTENT --> and <!-- END_CONTENT --> in your website code, Visperity extracts strictly the content between those comments.
• 2. CSS Selectors & End Cutoffs (Smart Fallback): If no matching comment tags exist in the page HTML, Visperity treats custom_start as a container selector (e.g. #main-content) and custom_end as a page cutoff element (e.g. .footer_container or #footer).

Configuring Extraction, Exclusion & Cutoff Rules

You can configure these rules either by clicking Configure Rules in the Content Indexer & Scraper Options panel or inside the Edit Dataset Settings modal:

Practical Example: Top-to-Bottom Scraper Filtering

You can combine multiple exclusion methods simultaneously. For example, if you want to remove the header via a class exclusion, start from the top of the page, and chop off everything at the footer:

• Exclude Divs by Class (custom_styles): .site-header, .nav-menu
• Start Marker (custom_start): (Leave Blank)
• End Cutoff (custom_end): .footer_container or #footer

┌────────────────────────────────────────────────────────┐
│ ❌ .site-header (Removed by custom_styles)              │
├────────────────────────────────────────────────────────┤
│                                                        │
│ ✅ Clean Body / Product Content                         │
│    (Vectorized & Indexed into AI Knowledge Base)       │
│                                                        │
├────────────────────────────────────────────────────────┤
│ ❌ .footer_container (Cut off by custom_end)           │
│ ❌ Sub-footer, copyright & legal notice (Also Cut off) │
└────────────────────────────────────────────────────────┘

Result: Visperity strips the header at the top, parses the clean product and article content in the middle, and immediately halts and discards everything from .footer_container down to the bottom of the page.

[!TIP] Automatic Global Application: Once saved on a dataset, these scraper rules are automatically applied to Single URL Crawls, XML Sitemap batch imports, and REST API v4 auto-indexing requests.

4. The Ingestion Queue Monitor

When importing large sitemaps or CSV spreadsheets, tasks are managed in the Queue Monitor (section=queue):

• Pending: URLs queued and awaiting AI vector generation.
• Completed: Successfully vectorized and active in live search.
• Failed: Pages that could not be reached (e.g. 404 Not Found or server timeout).

Action Controls:

• Step 1 Batch: Processes the next batch of queued records immediately.
• ⚡ Process ALL: Continuously processes all remaining queue items automatically.
• 🔄 Retry Failed: Resubmits any failed items for another crawl attempt.
• 🧹 Clear Completed: Cleans up the queue log of finished items.

5. Multi-Language Knowledge Base Translation Manager

Directly above the indexed records table in Datasets & Knowledge (section=datasets), Visperity includes a dedicated Multi-Language Knowledge Base Translation Manager:

• Language Enable Checkboxes: Toggle which secondary languages to maintain. Only enabled languages are queued for translation.
• Live Progress Counter: Displays real-time translation completion for each language (e.g. 142 / 150 (95%) Translated vs base English source records).
• Sync Status Badges:
  • ✓ Up to Date: All English source records have corresponding translated vector embeddings.
  • ⚠️ X Untranslated: New English records were added or imported and need translation.
  • ⚡ Z in Queue: Translations currently pending in the background Ingestion Queue.
• Incremental "Translate Untranslated" Action:
  • Clicking Translate Untranslated detects newly added English articles and queues only the missing delta into the Ingestion Queue.
  • Generates an instant toast notification with the count of queued records.

6. Managing & Editing Knowledge Records & Child Translations

In the Datasets & Knowledge section (section=datasets), scroll down to view the Indexed Vectors Table:

Filter Controls: [ 🌐 All Languages (Hierarchical) ▼ ] [ 🔍 Search title, content, SKUs, price, ID... ]

Expandable Child Translation Hierarchy

• English primary records display an expandable toggle: ▶ 🌐 X Translations.
• Clicking the toggle reveals all child language variants (e.g. Spanish, French, German) indented directly underneath.

Record Actions:

• ✏️ Edit Translation: Open the Edit modal to adjust the translated title, content, excerpt, price, or thumbnail image, and immediately re-vectorize the embedding in that language.
• ⚡ Retranslate & Re-Embed: Instantly re-run AI translation from the parent English record and generate a new 1536-dimensional vector embedding for that specific language.
• 🗑️ Delete Translation: Delete an individual language translation without affecting the primary English record. (Deleting a primary English record automatically cleans up all associated child translations).

Filtering Records by Language

Click the language dropdown to filter your records:

• 🌐 All Languages (Hierarchical): Displays base English records with expandable child translation rows.
• 🇺🇸 English (Default Only): Displays only base English records.
• Secondary Languages (e.g. 🇪🇸 Spanish, 🇫🇷 French): Displays translated records specific to that language.

7. Category Tags: Visual Labeling vs Dataset Segmentation

Visual Category Labels for User Experience

Visperity provides a simple Category Tag system allowing administrators to assign intuitive category badges to any indexed record (e.g. PRODUCTS, FAQ, SUPPORT, ACCESSORIES).

• Display across Widgets: If a category tag is present, it appears in bold uppercase text directly above the result title in:
  • Live Search Autocomplete dropdown suggestions
  • Embedded Search result cards
  • AI Virtual Assistant recommended resource links
• Real-Time Tagging: Admins can edit or assign category tags directly in real-time from the indexed records table via the inline category tag input, or inside the Edit Knowledge Record Modal.
• Multi-Language Translations: When an English record with a category tag is translated, the category tag is automatically included in the translation queue and localized into each target language.

[!IMPORTANT] Category Tags are a Labeling Feature, NOT a Segmentation Tool: Category tags do not restrict or filter which results are returned by a search query—they provide visual clarity for visitors scanning results.

True Content Segmentation via Multiple Datasets

If you need to segment your content so that a specific widget on your site only searches a specific portion of your content (for example, searching only store products on /shop and only help articles on /support):

1. Create a Dedicated Dataset: Go to Datasets & Knowledge -> + New Dataset and create a new dataset (e.g. "Store Products Only").
2. Ingest Targeted Content: Add only the URLs/products relevant to that specific section.
3. Embed the Unique Dataset Script: Copy the embed script generated for that dataset into the target pages.
4. Shared Content Across Datasets: You can freely index the same URLs into both your primary site-wide dataset and your segmented dataset. Visperity provides generous URL quotas on all plans to support multi-dataset configurations with zero penalty.