Data Ingestion & Knowledge Base Management
To answer questions and return accurate search results, Visperity transforms your web pages, product catalogs, and help articles into semantic AI vector embeddings.
This guide explains how vector search works in plain English, how to import and manage your content, and how to configure custom web scraper exclusion rules, title delimiter trimming, end-of-page cutoffs, automated sitemap schedules, and record-level indexing protections.
- • Method 1: Single Web Page Crawler
- • Method 2: XML Sitemap Ingestion & Scheduler
- • Method 3: CSV Bulk Catalog Upload
- • Method 4: Manual Document & Custom URL
- • Active Status (API & Live Search Visibility)
- • Exclude from Auto-Indexing (Crawler/API Lock)
- • Protection Across Ingestion Channels & REST API
How AI Vector Search Works
Unlike traditional keyword search engines that only look for exact character matches (e.g. typing "raincoat" won't find a product titled "waterproof jacket"):
Visperity converts the meaning and context of your products, articles, and documents into mathematical concept maps.
A visitor searching for "how do I return a damaged shirt?" will instantly match your "30-Day Refund & Exchange Policy", even if the word "shirt" doesn't appear in that document.
In addition to semantic AI matching, Visperity also checks exact product SKUs, tags, and External IDs for instant lookup.
Ingestion Methods
Navigate to the Data Import section (section=ingestion) on your dashboard to choose from four core ingestion options:
┌───────────────────────────┬───────────────────────────┬───────────────────────────┐
│ Single URL Crawler │ XML Sitemap Importer │ CSV Bulk Upload │
│ Crawl 1 page instantly │ Index entire website │ Upload catalog spreadsheet│
└───────────────────────────┴───────────────────────────┴───────────────────────────┘
│
▼
Manual Custom Document / URL Ingestion
(Add bespoke text, price, image & SKUs)
Method 1 Single Web Page Crawler
Best for quickly indexing individual product pages, blog posts, or newly published support articles.
- In the Single Page Crawler card, enter the full web page URL (e.g.
https://example.com/products/deluxe-headphones). - Click Crawl & Vectorize URL.
- Visperity's web crawler fetches the page, automatically applies your dataset's title delimiters, element exclusions, and end cutoff rules, generates an AI search summary, creates the vector embedding, and saves it to your dataset within seconds.
Method 2 XML Sitemap Ingestion & Automated Scheduler
Best for indexing your entire website or online store catalog automatically in the background on an ongoing schedule.
- In the XML Sitemap Ingestion card, enter your sitemap URL (e.g.
https://example.com/sitemap.xmlorhttps://example.com/sitemap_index.xml). - Manual Sitemap Crawl: Click Parse Sitemap in Background to discover all page URLs and queue them immediately into the Knowledge Ingestion Queue.
- Automated Sitemap Scheduler: Keep your vector knowledge base synced with your live store automatically:
- • Recurrence Frequency: Choose between Weekly or Monthly crawling.
- • Day of Week: Select the designated run day (e.g. Sunday, Monday, Saturday) to run during off-peak traffic hours.
- • Execution Time: Set the preferred execution time (e.g.
02:00 AM). - • Protection for Custom Edits: Any record marked with "Exclude from Auto-Indexing" is automatically skipped, preserving manual customizations across recurring batch runs.
Method 3 CSV Bulk Upload
Best for importing large product catalogs, offline knowledge bases, or content with explicit thumbnail image URLs and product prices.
| Column Header | Required? | Description | Example |
|---|---|---|---|
| url | Yes | Web link or product page URL | https://example.com/products/headphones |
| title | Yes | Product or article title | Deluxe Bluetooth Headphones |
| content | Yes | Full description, product specs, or article text | Premium over-ear noise-cancelling wireless headphones with 40-hour battery... |
| excerpt | Optional | 2–3 line summary displayed in search results (if blank, AI generates one automatically) | Premium noise-cancelling wireless headphones with 40h battery. |
| skus | Optional | Comma-separated product SKUs or search tags | HP-100, AUDIO-DLX, B09XYZ |
| image_url | Optional | Direct link to a product thumbnail image | https://example.com/images/headphones.jpg |
| external_id | Optional | Your custom ERP, CRM, or API catalog ID | PROD-1001 |
| price | Optional | Product or service price | $149.99 |
- In the CSV Bulk Upload card, click the Example CSV button to download a sample reference spreadsheet.
- Choose your
.csvfile and click Upload & Ingest CSV. - Visperity queues and processes the records while automatically respecting existing protected records.
Method 4 Manual Document & Custom URL Ingestion
Best for custom FAQ answers, internal company policies, or creating records where you want full manual control over the text, price, image, and SKUs.
- Click + Manually Add Document / URL to open the manual ingestion form.
- Fill in the fields:
- • Target Resource URL: The link visitors will navigate to when clicking the search result.
- • Document / Product Title: The headline displayed in search results.
- • Document Text Content: The full text used to generate the AI vector embedding.
- • Product / Resource Price (Optional): Optional price (e.g.
$49.99). - • External ID (Optional): Custom system identifier or API record ID.
- • Associated Product SKUs (Optional): Comma-separated SKUs for instant exact matching.
- • Thumbnail Image (Optional): Enter an existing image URL or click Upload to S3/CDN to upload a product photo directly from your computer.
- Click Save & Vectorize Document.
Supported Formats & Sectional Indexing (Addendum)
In addition to standard HTML web pages, Visperity provides native indexing capabilities for hosted binary documents and granular section-level anchor tags:
📄 PDF & Word Document Indexing (.pdf, .docx, .doc)
When you crawl or submit URLs pointing directly to Adobe PDF or Microsoft Word documents (or when your XML sitemap contains document links), Visperity extracts the text natively without requiring HTML conversion:
| Format | Extensions & Content-Types | Extraction Capabilities |
|---|---|---|
| Adobe PDF | .pdf application/pdf |
Extracts text across multi-page files, FlateDecode compressed streams, kerning operators (TJ / Tj), and document title metadata. |
| Microsoft Word | .docx, .doc application/vnd.openxmlformats... |
Parses OpenXML document structures (word/document.xml), preserving paragraph breaks (<w:p>), tabs, table cells, and core title properties. |
| Plain Text & Markdown | .txt, .md, .json, .csv text/plain, text/markdown |
Normalizes plain text, technical documentation, markdown guides, and data feeds. |
- • Direct Search Links: Search results link directly to the hosted document URL (e.g.
https://example.com/downloads/spec-sheet.pdf). - • AI Answers from Documents: The Virtual Assistant reads specifications, tables, and guides inside your PDF/Word files to generate direct natural-language answers.
⚓ Sectional & Anchor Tag URL Indexing (#section)
If you have long single-page guides, comprehensive documentation, or multi-topic FAQ pages, you can index specific sections of the same page independently using URL fragment anchors (e.g. https://example.com/docs/api#authentication or https://example.com/faq#returns):
https://example.com/manual#setup ──► [AI Vector: Initial Setup & Installation] https://example.com/manual#wifi-config ──► [AI Vector: Wi-Fi & Network Setup] https://example.com/manual#troubleshoot──► [AI Vector: Troubleshooting & FAQ]
- • Independent Knowledge Records: Each anchor URL is indexed as its own distinct vector record with its own semantic context.
- • Deep Linking: When visitors click a search result or AI recommendation, the browser opens the page and scrolls directly down to the specific section anchor (
#section-name).
Content Indexer & Web Scraper Customization Options
Web pages often contain repetitive layout elements (such as site navigation headers, footer menus, cart drawers, and cookie notices), brand suffixes in <title> tags, or cross-sell accessory SKUs that you do not want indexed for a product page. Visperity provides fine-grained scraper controls configured per dataset in Datasets & Knowledge (section=datasets).
Configuring Extraction, Exclusion, Title Delimiter & Cutoff Rules
You can configure these rules either by clicking Configure Rules in the Content Indexer & Scraper Options panel or inside the Edit Dataset Settings modal:
| Scraper Setting | Database Field | How It Works | Syntax Examples |
|---|---|---|---|
| Exclude Characters After Title Delimiter | custom_title_exclude_after | Strips everything to the right (including the delimiter character) from page titles and trims trailing whitespace. Prevents repetitive brand suffixes like " | StoreName" from cluttering search headings. |
| , - , or — ("About Us | Acme" → "About Us") |
| Exclude Elements by ID / Name | custom_ids | Strips any HTML element with matching id, name, or tag name prior to text vectorization. |
#header, #footer, #sidebar, #cookie-banner, #cart-drawer, promo-banner |
| Exclude Elements by CSS Class | custom_styles | Removes any DOM node containing one of the specified CSS classes. | .header, .footer, .site-nav, .no-index, .cart-dropdown, .related-posts |
| Exclude SKUs & Product IDs | custom_sku_field | Comma-separated exclusion list of accessory SKUs, related product IDs, or element selectors to strip from the indexed text and SKU index so that referenced items don't skew search results for the primary product. | SKU-999, PROD-102, ACC-*, .related-skus, #accessories |
| Target Single Region (Selector or Start Marker) | custom_start | Two Modes: 1. Single CSS Selector (e.g. #main-content or .product-body) — extracts ONLY the body content inside that container.2. Comment / String Delimiter (e.g. START_CONTENT) — extracts content following <!-- START_CONTENT -->. |
#main-content, .product-body, or START_CONTENT |
| End Class / ID Cutoff & Delimiter | custom_end | Two Modes: 1. End Class or ID Cutoff (e.g. .footer_container or #footer) — cuts off and excludes everything starting from that element down to the end of the page (including cart slideouts, sub-footers and legal notices).2. Closing Delimiter (e.g. END_CONTENT) — closes matching <!-- END_CONTENT --> tags. |
.footer_container, #footer, footer, or END_CONTENT |
| Secondary Region Delimiters / Cutoff | custom_start2 / custom_end2 | Optional secondary content delimiter pair or secondary cutoff element for scraping supplementary tabs or technical specifications. | START_SPECS / END_SPECS (or #specs-footer) |
Practical Example: Top-to-Bottom Scraper Filtering
You can combine multiple exclusion methods simultaneously. For example, if you want to clean page titles, remove the navigation header, start from the top of the body, and chop off everything at the footer:
- • Title Delimiter (
custom_title_exclude_after):| - • Exclude Divs by Class (
custom_styles):.site-header, .nav-menu - • Start Marker (
custom_start): (Leave Blank) - • End Cutoff (
custom_end):.footer_containeror#footer
┌────────────────────────────────────────────────────────┐ │ ✂️ Title Filter: "About Us | Acme" -> "About Us" │ ├────────────────────────────────────────────────────────┤ │ ❌ .site-header (Removed by custom_styles) │ ├────────────────────────────────────────────────────────┤ │ │ │ ✅ Clean Body / Product Content │ │ (Vectorized & Indexed into AI Knowledge Base) │ │ │ ├────────────────────────────────────────────────────────┤ │ ❌ .footer_container (Cut off by custom_end) │ │ ❌ Sub-footer, cart drawer & legal (Also Cut off) │ └────────────────────────────────────────────────────────┘
Result: Visperity trims the brand suffix from the title, strips the header at the top, parses the clean product and article content in the middle, and immediately halts and discards everything from .footer_container down to the bottom of the page.
Knowledge Record Controls: Two Distinct Operational Toggles
Each indexed knowledge record in Visperity features two independent toggle switches designed for total administrative control:
| Toggle Setting | Default | Purpose & Scope | Behavior when Toggled |
|---|---|---|---|
|
Active Status (API Calls & Searches)
is_active |
ON (1) | Controls whether this record is returned in live search queries across the AI Virtual Assistant, Live Search autocomplete, and Embedded Search widgets. | Setting to OFF (0) temporarily pauses/hides the document from all visitor searches without deleting the vector embedding or losing translation links. |
|
Exclude from Auto-Indexing
exclude_indexing |
OFF (0) | Protects custom edits and bespoke content from being overwritten by automated crawlers, sitemaps, CSV re-uploads, or API pushes. | Setting to ON (1) locks the record. Automated sitemap re-crawlers, Single URL crawls, CSV imports, and REST API update calls will skip this URL. The record can only be re-indexed manually via the Edit Knowledge Record modal. |
Channel Enforcement of Exclude from Auto-Indexing
When exclude_indexing = ON is set on any record, protection is strictly enforced across every automated ingestion channel:
- • Single Page Web Crawler: Crawl attempts on this URL are halted with a dashboard alert explaining that the URL is locked against auto-indexing.
- • XML Sitemap Ingestion & Scheduled Crawls: The sitemap parser skips the protected URL and all its translated variants, ensuring automated batch syncs will not overwrite manual edits.
- • CSV Bulk Upload: Spreadsheet rows referencing a protected URL are skipped during queue generation, preserving the current title, body, and vector embedding.
- • REST API Calls (
POST,PUT,PATCH):- -
POST /api/v4/datasets/{api_key}/records: If the submitted URL matches a protected record, the API returns a422 Unprocessable Entityerror. - -
PUT/PATCH /api/v4/datasets/{api_key}/records/{record_id}: If the targeted record hasexclude_indexing = ON, update attempts via API are rejected with a422 Unprocessable Entityresponse.
- -
- • Dashboard Edit Modal (Sole Manual Interface): Administrators can safely update the document title, content, price, image, or translations by clicking ✏️ Edit in the dashboard, which regenerates the vector embedding while respecting all manual settings.
The Ingestion Queue Monitor
When importing large sitemaps or CSV spreadsheets, tasks are managed in the background by the Queue Monitor (section=queue):
Action Controls:
- • Step 1 Batch: Processes the next batch of queued records immediately.
- • ⚡ Process ALL: Spawns an isolated background CLI worker to process all remaining queue items continuously without holding up dashboard tabs.
- • ⏸ Pause Processing: Cleanly suspends the background worker.
- • 🔄 Retry Failed: Resubmits any failed items for another crawl attempt.
- • 🧹 Clear Completed: Cleans up the queue log of finished items.
Multi-Language Knowledge Base Translation Manager
Directly above the indexed records table in Datasets & Knowledge (section=datasets), Visperity includes a dedicated Multi-Language Knowledge Base Translation Manager:
- • Language Enable Checkboxes: Toggle which secondary languages to maintain. Only enabled languages are queued for translation.
- • Live Progress Counter: Displays real-time translation completion for each language (e.g.
142 / 150 (95%) Translatedvs base English source records). - • Sync Status Badges:
- -
✓ Up to Date: All English source records have corresponding translated vector embeddings. - -
⚠️ X Untranslated: New English records were added or imported and need translation. - -
⚡ Z in Queue: Translations currently pending in the background Ingestion Queue.
- -
- • Incremental "Translate Untranslated" Action:
- - Clicking Translate Untranslated detects newly added English articles and queues only the missing delta into the Ingestion Queue.
- - Generates an instant toast notification with the count of queued records.
Managing & Editing Knowledge Records & Child Translations
In the Datasets & Knowledge section (section=datasets), scroll down to view the Indexed Vectors Table:
Quick Inline Actions & Expandable Hierarchy
- • Inline Quick Actions: Duplicate ✏️ Edit and 🗑 Delete buttons are placed directly adjacent to the document title on every row for instant one-click editing.
- • Expandable Child Translation Hierarchy: English primary records display an expandable toggle:
▶ 🌐 X Translations. Clicking reveals all child language variants indented in a highlighted container card directly underneath.
Record Actions:
- • ✏️ Edit Record / Translation: Open the Edit modal to adjust the document title, content, excerpt, price, thumbnail image, or operational toggles (Active Status & Exclude from Auto-Indexing), and immediately re-vectorize the embedding.
- • ⚡ Retranslate & Re-Embed: Instantly re-run AI translation from the parent English record and generate a new 1536-dimensional vector embedding for that specific language.
- • 🗑️ Delete Record: Delete an individual translation without affecting the parent, or delete a primary record to automatically clean up all associated child translations.
Filtering Records by Language
Click the language dropdown to filter your records:
- • 🌐 All Languages (Hierarchical): Displays base English records with expandable child translation rows.
- • 🇺🇸 English (Default Only): Displays only base English records.
- • Secondary Languages (e.g. 🇪🇸 Spanish, 🇫🇷 French): Displays translated records specific to that language.