Knowledge Scraping / Knowledge Sources

Overview

Mainstay's generative AI features are able to provide institution-specific responses based on content we scrape from your websites and documents. This tool allows you to indicate which pages we should pull information from, and lets you test the generative AI by asking it questions.

Ask the AI

After selecting and scraping knowledge sources (see below), you can test your coverage by asking the AI a question. This tool uses the same AI settings/prompt as KB Response Generation and AI-Assisted Live Chat.

When the suggested response was crafted using your scraped Knowledge Sources, those will be indicated for reference:

Knowledge Sources

Adding Sources

To add a new knowledge source, click + New Source:

There are four types available:

• Specific Webpages: one or more URLs. Mainstay will scrape the content from these individual pages.
• Domain/Site Section: a full URL. Mainstay will scrape the content from this page and other pages on the site that include this URL.
  • So for example, if you input "https://example.com/a", we will scrape that page, as well as "https://example.com/a/b" and "https://example.com/a/b/c".
  • However, we would not scrape "https://example.com/x", "https://something.example.com/a" or even just "https://example.com" by itself.
• PDF: a URL to a hosted PDF document. At this time, Mainstay does not allow you to upload a document directly, but you can reference a PDF document that is available online, such as one on your domain or one you upload and make public on Google Drive or similar.
• Google Document: a URL to a Google Doc that is publicly accessible, or shared with an @mainstay.com email address.
  • 

A new knowledge source will be scraped immediately. You can also trigger a re-scrape of all existing sources by clicking Scrape All.

Scraping Queue

If you add or rescrape multiple sources at once, they will go into a Queue:

The system will process these one at a time, oldest to newest:

When a source is done scraping, its Status will change to either "Success" or "Fail", its Last Scraped will change to a timestamp indicating when the scrape completed, and the header text may change to the webpage's title.

The page will automatically update as these statuses change, meaning you will see a pattern of statuses like this gradually moving upwards:

Viewing Sources

Below these options is the list of all knowledge sources you have selected. Each includes:

• Title: the <title> element from the webpage or the name of the PDF file.
• URL: the full link to the webpage, Google Doc, or PDF file
  • Note: if you selected "Domain/Site Section" above, that will become multiple knowledge sources, where each one represents a specific page we've scrape.
• Last Scraped: the date and time that Mainstay last scraped the webpage or document
• Status: Queued | Started | Success | Failed

From the ... menu on each knowledge source, you can take the following actions:

• Edit: Update the Title and add an optional Description.
• Scrape: Trigger a re-scrape of this knowledge source. This is helpful if that page or document was recently updated.
• Delete: Remove this knowledge source and all content we've scraped from it.

Note: Only admin users are able to add, edit, and delete knowledge sources.