Overview
The Engagement Insights Dashboard includes several visual graphs and downloadable reports to help you understand your contacts' engagement with your communication.
Filtering
The Insights graphs and reports can be filtered by date range, communication Channel (sms, web, etc.), and/or Audience. At the top of the page, you can see the currently applied filters.
The default time frame for all data shown is the last 90 days. All timestamps are shown in the institution's timezone.
In addition to specific date ranges, you can also use relative date ranges:
Click the filter button to edit a filter, or click on an already applied filter to change it.
For all graphs and reports on the page, data will only be shown for the selected time range.
- For example, the Individuals Contacted graph will show messages received during the selected days, and the Campaigns graph will show Campaigns sent during the selected days.
- Note that all date ranges are inclusive, meaning that the first and last day selected are included in the returned data set.
Data is further filtered by only finding contact records that meet the selected Channel(s) and Audience criteria. This is evaluated in real-time, meaning we find contact records who currently meet this criteria, and show historical data for those individuals.
- For example, if you have an Audience of students whose classyear is freshman, that will surface your current freshmen class. If you then apply a date range from 2 years ago, you will not see data for freshmen from 2 years ago (ie, juniors)... you'll see data from 2 years ago for your current freshmen class... probably nothing!
Note: If your institution has User Groups enabled, the graphs will not automatically filter by the current user's User Group. However, the downloaded file will be restricted to only contacts that match the current user's User Group.
Downloading
When clicking to Download a report, you can choose which default, custom, and report-specific fields to include. You can also save your preferences for next time, and customize the name of the CSV file.
Note: If your institution has User Groups enabled, the downloaded file will be restricted to only contacts that match the current user's User Group. However, the graphs will not be filtered in this way.
Copying
Click the Copy button to copy an image of this graph into your clipboard. The image will contain the name of the graph, the labeled axes, and all data series, as well as a legend. This image can then be pasted into an email, another web page (like Google Docs or Slides), or an external program (like Microsoft Word or PowerPoint). You can also use an image-editing tool, like Preview on a Mac, and "Create new from clipboard" to save as an image file.
Note: Firefox prevents adding images to the clipboard. For that browser, we instead copy an HTML <img> element, which can be pasted in most places (email, web pages, some desktop applications.)
Individuals Contacted
Graph
This displays the number of unique contacts who were sent a message during each week bucket. This excludes test contacts.
Report
The downloaded report identifies all successfully received messages associated with non-test contacts and counts the number of incoming and outgoing messages for each contact. If a contact does not have any incoming or outgoing messages successfully received during the selected timeframe (for example, a campaign that was undelivered due to spam issues or a blocked or invalid number), then that contact will not be included in the report. The file contains the following fields:
- standard contact fields & custom contact fields
- Total # of Incoming Messages: number of messages received from this contact
- Total # of Outgoing Messages: number of messages sent to this contact
- Campaign Names: list of campaigns this contact received
Engagement
Graph
This displays the number of messages sent (outgoing) and received (incoming) during each week bucket. This excludes test contacts.
Report
The downloaded report contains the number of outgoing and incoming messages during each week bucket and contains the following fields:
- Range: the start and end date for each week bucket
- Outgoing Messages: the number of messages sent to non-test contacts during this time
- Incoming Messages: the number of messages received from non-test contacts during this time
Opt Outs
Graph
This displays the number of opt-out events grouped by week.
"Opt Outs" refer to events that opt out a contact permanently. These include STOP messages, the #pause dialog when pause forever is selected, opt out script actions, and manual opt-outs.
"#pause" refers to instances of a contact messaging #pause in order to temporarily block messages at the Mainstay level. (See Opt Out Commands.)
Report
The downloaded report contains rows for each opt-out event, meaning each instance of a contact opting out, such as with a carrier keyword (STOP) or a Mainstay keyword (#pause), with the following fields:
- standard contact fields & custom contact fields
- Date of Opt Out Event: the date that this contact opted out
- Script: id (name) of Script if opt-out event occurred while contact was in a Script
- Campaign: id (name) of Campaign if opt-out event occurred while contact was in a Campaign
- Type: Pause|Stop|Start - the type of status change event
- Source: the reason for the opt-out event, and one of the following:
- #Pause - the contact triggered the #pause command
- STOP - the contact used a carrier keyword to opt out
- START - the contact used a carrier keyword to opt in
- Contact Import - the contact's status was changed due to a CSV import
- Public API - the contact's status was changed via the API or integration
- Contact Panel - a user manually changed this status on the Contacts or Conversation page
- Script Action - the contact reached an Opt-out Action in a Script
- Delivery Failure - a message was not delivered because the contact was already opted out, according to our SMS provider
Campaigns
Graph
This displays the number of campaigns sent during each week bucket. This excludes test campaigns, archived campaigns, and campaigns that did not complete (either because they are still in progress or failed to send).
Note that this displays when campaigns were actually sent, not when the platform user created or edited them. (For example, a campaign that was created in January and scheduled for March will not appear here until it is sent in March.)
Interactive campaigns are those with one or more questions, such as yes/no, multi-choice, or open-text prompts. Nudge campaigns are those without these interactive elements. See Campaign Types.
- Note: If you apply an Audience filter to this page, the Campaigns graph will show any Campaigns received during the selected time frame by any 1 or more contacts who meet the Audience criteria.
Reports
Prior to 7/10/23, the downloaded Campaign History report contains these same campaigns with information about each, presented with the following fields:
- Date Sent (UTC): the timestamp of when the campaign was actually sent
- Campaign Name: the display name given to this campaign
- Campaign ID: the randomly generated unique identifier for this campaign
- Campaign Type: Interactive|Nudge - whether this campaign included interactive questions
- # of Contacts Sent To: the number of intended recipients for this campaign (Active + Passive + Resulting Opt-Out + Resulting Pause + Delivery Failures + Mainstay Failures)
- # of Opt-Outs: the number of contacts who did not receive the campaign because they opted out
- # of Delivery Failures: the number of contacts who did not receive the campaign because it could not be delivered (see Reasons for Message Delivery Failure)
- # of Contacts Who Received Successful Send: the number of contacts who did receive the campaign
- # of Contacts Replied: the number of contacts who sent in one or more incoming messages during this Campaign, if it's interactive (not necessarily valid responses)
- % Response Rate: (# of Contacts Replied / # of Contacts Sent To) * 100
- Import Name Sent To: if a manual CSV import was used to send this campaign, the name of that file, or else empty.
- Audience Name Sent To: if an Audience was used to send this campaign, the name of that Audience, or else empty.
As of 7/10/23, the downloaded Campaign History report contains these same campaigns with information about each, presented with the following fields:
- Date Sent (UTC): the timestamp of when the campaign was actually sent
- Campaign Name: the display name given to this campaign
- Campaign ID: the randomly generated unique identifier for this campaign
- Campaign Type: Interactive|Nudge - whether this campaign included interactive questions
- Import Name: if a manual CSV import was used to send this campaign, the name of that file, or else empty. (renamed from "Import Name Sent To")
- Audience Name: if an Audience was used to send this campaign, the name of that Audience, or else empty. (renamed from "Audience Name Sent To")
- Audience ID: if an Audience was used to send this campaign, the id of that Audience, or else empty. (new!)
- Script Name: the name of the Script used to send this campaign (new!)
- Script ID: the id of the Script used to send this campaign (new!)
- Sent To: the number of intended recipients for this campaign (Active + Passive + Resulting Opt-Out + Resulting Pause + Delivery Failures + Mainstay Failures) (renamed from "# Of Contacts Sent To")
- Prior Opt-Out Status: the number of contacts who did not receive the campaign because they were previously opted out (renamed from "# of Opt-Outs")
- Prior Pause Status: the number of contacts who did not receive the campaign because they were previously paused (new!)
- Prior Invalid Phone Status: the number of contacts who did not receive the campaign because they have an Invalid Phone number (new!)
- Received Successful Send: the number of contacts who successfully received the campaign (renamed from "# of Contacts Who Received Successful Send")
- Replied: the number of contacts who sent in one or more incoming messages during this Campaign, if it's interactive (renamed from "# of Contacts Replied")
- Response Rate: (# of Contacts Replied / # of Contacts Sent To) * 100 (renamed from "% Response Rate")
- Active Recipients: the number of contacts in the Active bucket after the campaign ends (new!)
- Passive Recipients: the number of contacts in the Passive bucket after the campaign ends (new!)
- Resulting Opt-Outs: the number of contacts in the Opt-Out bucket after the campaign ends (new!)
- Resulting Pauses: the number of contacts in the Pause bucket after the campaign ends (new!)
- Recipients Impacted By Failures: the number of contacts who did not receive one or more messages because they could not be delivered (see Reasons for Message Delivery Failure) (renamed from "# of Delivery Failures")
- Total Delivery Failures: the number of total message failures experienced across all contacts; note that some contacts may have multiple failing messages (new!)
- Mainstay Sending Issues: the number of messages Mainstay was unable to send due to internal issues (new!)
- SMS Failures: the number of messages that could not be delivered, as reported by carriers (ie, Twilio error codes) (new!)
Note: If you apply an Audience filter to this page, the Campaign History Report will include any Campaigns received during the selected time frame by any 1 or more contacts who meet the Audience criteria.
The downloaded Contact Engagement By Campaign report contains these same campaigns, but rows represent contacts engaging with each campaign, including information about the contact and their response or deliverability issues, through the following fields:
- standard contact fields & custom contact fields
- Date Campaign Received: the date that this particular was sent to the contact
- Campaign Name: the display name of the particular campaign this contact received
- Campaign ID: the randomly generated unique identifier for that campaign
- Campaign Type: Interactive|Nudge - whether this campaign included interactive questions
- Contact Opted Out: Yes|No - whether the contact has used the keyword STOP to block messages at the carrier level
- Message Delivery Failure: Yes|No - whether the contact experienced any issues receiving messages from this campaign
- Contact Successfully Received Campaign: Yes|No - whether the contact received messages from this campaign
- # Of Messages Received From Contact (For Interactive Campaigns): the number of messages received from the contact as part of this campaign, such as answers to questions
- # Of Messages Sent To Contact: the number of messages sent to the contact as part of the campaign
Incoming Messages by Day of Week
Graph
This displays the number of messages received from contacts during the given time frame, grouped by the day of the week, in the institution's timezone.
Incoming Messages by Time of Day
Graph
This displays the number of messages received from contacts during the given time frame, grouped by the hour of the day, in the institution's timezone.
Individual Question and Answer Report
Report
For this report, all incoming messages that were handled by the AI bot / Knowledge Base are shown with the corresponding response. This does not include Live Chat messages, nor messages that were responses to a Script (such as a Campaign, #Command, web-chat Intro, etc.).
The downloaded report contains the following fields:
- standard contact fields & custom contact fields
- Incoming Message (Question): the actual message received from the contact
- Outgoing Message (Response): the actual message sent in response; note that this message may have been constructed using organization attributes or embedded contact fields.
- Note: if the understanding is configured to respond with an Interactive Response, then the name of the corresponding script is shown instead.
- Log ID: a randomly generated unique identifier for this message
- Timestamp: the timestamp that the message was received from the contact
- Category, Subcategory, Topic: the location of the selected understanding in the Knowledge Base
Escalated Messages Report
Report
The downloaded report indicates instances of the bot matching to understandings configured with escalation rules. Each row is an escalation event, including the incoming message and other information included in the escalation email, and contains the following fields:
- standard contact fields & custom contact fields
- Escalated Message Id: a randomly generated unique id for this message
- Escalation number: the number of rounds of escalations for this particular message; the initial escalation is 1, and then if the contact replies to the staff member's response, that appears as 2, and so on
- Escalation date: the timestamp that the message was escalated via email
- Escalation email address: the email recipient of the escalation notification
- Escalation prefix: the body of the escalation notification above the actual content
- Escalation type: Manual|Sensitive Match|Script Action - the reason for the escalation. See Email Forwarding & Escalations and Script Email Actions.
- Escalated message body: the message from the contact that was escalated
- Replied: True|False - whether the contact has been responded to
- Reply date: the timestamp that the contact was responded to
- Reply email address: the email address of the respondent
- Reply body: the body of the email escalation response, and thus the message the contact received from Mainstay
Bot Summary Report
Report
This report is a holistic overview of the bot. See Bot Summary Report for a detailed breakdown of the many fields in this report.
- Note: If you have any Channel or Audience filters applied, this report will be inaccessible, as it is intended to give a summary of the entire account.
In addition to the tiles on the default Engagement dashboard, several tiles are available to add to Custom dashboards.
Conversations
Graph
This displays the number of Incoming messages, as well as the number of Outgoing messages for each week bucket within each of 4 categories:
- AI: responses sent by the bot based on Knowledge Base matching or fallbacks
- Live Chat: messages sent by a platform user during a Live Chat session
- Script: messages that were part of a script, such as a Campaign, #Command, or Intro
- Other: communication outside of these modes, including:
Report
The downloaded report contains rows for each message, both Incoming and Outgoing, with the following fields:
- standard contact fields & custom contact fields
- Message ID: the unique ID of the message itself
- Message Channel: SMS | Web - the channel that the message went through
- Message Direction: Incoming | Outgoing - whether the message came in from a learner or went out to a learner
- Message Timestamp: the time and date the message was sent or received
- Message Body: the original text of the message
- Message Source: AI | Live Chat | Script | Other - for outgoing messages only; the communication mode that generated the message
- Campaign ID: if the source is "Script" and it was part of a Campaign, the ID of that Campaign
- Campaign Name: if the source is "Script" and it was part of a Campaign, the Name of that Campaign
- Script ID: if the source is "Script", the ID of that Script
- Script Name: if the source is "Script", the Name of that Script
- Understanding Topic: if the source is "AI", the topic/subtopic label for that understanding
- Live Chat User: if the source if "Live Chat", the email address of the platform user who sent the message
Important Topics
Graph
This displays the top 5 most-matched Knowledge Base topic/subtopic areas. For each one, it indicates the Total number of incoming messages that matched to that topic/subtopic (for the filtered data), and the % of all bot-handled responses that this represents.
To see more than the top 5, click View more important topics to go to the Knowledge Insights page.
Answered Messages
Graph
This displays the number and percentage of incoming messages that matched to an understanding, or to a #Command.
To see more details, click on Review X Marked Incorrect to go to the Knowledge Maintenance page.
Missed Questions
Graph
This displays the number and percentage of incoming messages that did not match to an understanding, either because there was no good match in the Knowledge Base or because there was a matching understanding but it did not have an approved answer.
To see more details, click on Review X Missed Questions to go to the Knowledge Maintenance page, or click on Approve X Answers or Add Answers for X Understandings to go to the Knowledge Base.
Webchat Feedback
Graph
This displays the number of contacts who were asked to provide feedback to web-chat during the selected time range, and the number who provided positive and negative feedback.
To see more details, click on View All Webchat Feedback to go to the Knowledge Maintenance page.
Comments
0 comments
Article is closed for comments.