Retrieval-Augmented Generation - The metasearch engine in the age of AI

What is Retrieval Augmented Generation / RAG?

A nice analogy that also makes it clear what RAG is, is the concept of a metasearch engine. Here, the search query is forwarded to several other search engines. The results of all the requested services are then collected, processed and made available to the user. RAG is a technique in which an AI model is combined with other data sources in addition to the data in the LLM (Large Language Model) in order to generate more precise and contextually relevant answers. This is therefore a very similar approach to the metasearch engine. Even the two schematic diagrams of the technologys are similar:

RAG is used in this way in Microsoft 365 Copilot. To extend the capabilities of the AI, information is retrieved from various data sources and integrated into the response generation. This enables Copilot not only to access pre-trained data, but also to use current and specific information from other sources, including the data in the M365 Tenant. Access is via the Microsoft Graph. This also ensures that the underlying permission concept is always respected by the AI.

Source and further details: How Copilot for Microsoft 365 works: A deep dive

Copilot in Microsoft 365 uses RAG - this cannot be customized

In Microsoft 365 Copilot, RAG is used to improve responses to user queries. Copilot can access various data sources, such as documents, emails, Teams chats, etc., to provide well-grounded and accurate answers.

This also determines which functions / roles Copilot provides in the respective apps.

Examples:

• Word: Generate text with and without formatting in new or existing documents.
• Excel: Suggestions for formulas, chart types and insights for data in Excel sheets.
• PowerPoint: Create a presentation from a prompt or a Word file.

Complete overview:

Source: Copilot integration with Graph and Microsoft 365 Apps

Now we have GraphRAG - that can be customized

The article Unlocking LLM discovery on narrative private data describes GraphRAG, a new method from Microsoft Research that extends the capabilities of large language models (LLMs) to access and analyze your data.

GraphRAG combines LLM-generated knowledge graphs with machine learning to improve document analysis performance, for example. This method shows significant improvements in answering complex questions compared to standard approaches.

A key benefit of GraphRAG is its ability to identify and understand topics and concepts in large data sets, even if the data was not previously known to the LLM. Here are some practical use cases for this technology:

• Information extraction: GraphRAG can be used to extract specific information from large document collections or databases.
• Content generation: GraphRAG helps to create content that requires in-depth contextual knowledge.
• Customer support: GraphRAG can improve customer support by accessing a knowledge base and providing accurate answers to customer queries.
• Knowledge management: In large organizations, GraphRAG can help to make efficient use of existing knowledge by retrieving and consolidating relevant information from different departments and documents.

Quickstart

To get started with the GraphRAG system (https://github.com/microsoft/graphrag), it is recommended to use the Solution Accelerator package (https://github.com/Azure-Samples/graphrag-accelerator). This offers a user-friendly end-to-end solution based on Azure resources, quote: One-click deploy of a Knowledge Graph powered RAG (GraphRAG) in Azure

The graphic shows, for example, the following sources for own solutions and GraphRAG:

• Azure Blob Storage
• Cosmos DB
• Azure OpenAI
• Azure AI Search / Vectorstore
• Container Registry
• Application Insights

As described on GraphRAG's GitHub page, Prompt Tuning options can also be used to customize the solution to your needs and use cases:

Source: GraphRAG Prompt Tuning

How many Copilots are there? Fewer than you think

Why does Copilot in Microsoft 365, for example, also use data that is integrated into the M365 search via a connector, but Copilot in Word does not? Or to be more general: What is the difference between Copilot in Word and Copilot for Microsoft 365? Let's just ask him ourselves.
Prompt in Copilot in Word: What makes you different from Copilot in Microsoft 365?

The prompt was used with the "Draft with Copilot" feature:

The detail "Copilot in Word is more suitable for users who need more specific and detailed suggestions and feedback on their writing" in the generated text makes the difference clear. It says that Copilot in Word is a writing assistant. The specification that Copilot should act as a writing assistant here is like telling a prompt in ChatGPT, that the AI should act as a travel guide and that the focus should be on suggestions for a weekend in Rome.

The technology behind this is called Retrieval Augmented Generation. In general, every prompt that a user enters is pre-processed by the Copilot Orchestration. This is where the Retrieval Augmented Generation architecture is used. Although this step is not mentioned by name in the architecture diagram, it is explained in this video by Mary Pasch, Principal Product Manager at Microsoft for Copilot: How Copilot for Microsoft 365 works: A deep dive.

A term that is also frequently used is "grounding". In the end, it means the same, i.e. adapting the prompt entered by the user, respecting the context of the used app an combine these factors to achieve the best possible result. Details on this are shown in this overview:

Source and further details can be found in this article from Microsoft: Microsoft Copilot for Microsoft 365 overview.

Retrieval Augmented Generation (RAG)

Depending on the app from which the prompt is sent, the Retrieval Augmented Generation technology decides which information sources are used to generate an answer with the help of the LLM. This aspect, called Information Retrieval System, can also be used with Azure AI Search. The article Retrieval Augmented Generation (RAG) in Azure AI Search describes how the technology works, which is also used by Copilot Orchestration.

How Copilot for Microsoft 365 decides which plug-in to use

There is also a similar effect when using plug-ins. How does Copilot decide which Plug-Ins to use to respond to a prompt? This also happens in the Copilot orchestration. The article How Copilot for Microsoft 365 decides which plugin to use describes what should be done when creating Plug-Ins so that Copilot Orchestration can evaluate them properly. The app description is located in the app manifest:

Roundup

Although this does not answer the initial question of how many Copilots are there. But it is now clear that there are far fewer than you might think. Copilot in Microsoft 365, Copilot in Word, Copilot in Outlook etc. are all the same engine, which is only used differently through orchestration and pre- and post-processing. Solutions such as Copilot in GitHub etc., however, are independent solutions.

Size matters - Large documents and Copilot for Microsoft 365

UPDATE

Problem solved - at least an improvement is on its way!

As described in my article “Size matters - Large documents and Copilot for Microsoft 365”, Copilot is currently reaching its limits with documents longer than 20 pages / 15,000 words.

Roadmap ID 399413 now announces that this limit is to increase significantly: “Copilot in Word will be able to fully summarize documents that it could previously only partially summarize. The upper limit increases to about four times more words.”

The Microsoft page linked in the article below: Keep it short and sweet: a guide on the length of documents that you provide to Copilot has also been updated. It now speaks about 80,000 words.

--

Microsoft has published an article named Keep it short and sweet: a guide on the length of documents that you provide to Copilot. It describes how Copilot for Microsoft 365 reaches its limits when it has to work with large documents or very long emails.

The reason for this is that Copilot works with data from the Microsoft Graph, which means that the search in M365 also has a role here. Documents, emails and all other content must first be indexed by the search before they are available for Copilot. At least for the search in SharePoint Online, the limits are documented: https://learn.microsoft.com/en-us/sharepoint/search-limits.

The exact limits that apply for processing by Copilot in Microsoft 365 are currently unclear. The article Keep it short and sweet: a guide on the length of documents that you provide to Copilot gives the following recommendations:

• Shorter than 20 pages
• Maximum of around 15,000 words

The example shows how it behaves when relevant information is after these limit recommendations. The relevant information to be used via Copilot are as followed. These are on page 49 of a Word document that contains a total of 27,208 words.

If you ask Copilot “What can you tell me about Snabales Total liabilities?” you get the following answer:

If you use Copilot in Word and ask the same question, the answer is: “This response isn't based on the document: I'm sorry, but the document does not provide any information about Snabales Total liabilities...”

One option you now have here is not to use Copilot for Microsoft 365 natively, but to create your own solution based on Azure AI-Search and Azure OpenAI. In Azure AI-Search, a vector search can be used that splits large documents into so-called chunks. This article describes the details: Chunking large documents for vector search solutions in Azure AI Search

Prepare your organization for Microsoft 365 Copilot

Make sure that all permissions are set correctly. Check that all technical requirements are met and assign Copilot licenses to your user.

All of these points are certainly part of rollout planning. However, they are not the only ones and, above all, the implementation is not done quickly for many companies.

In this article from September 21, 2023, it was announced that Microsoft 365 Copilot will be generally available on 1. November: https://blogs.microsoft.com/blog/2023/09/21/announcing-microsoft-copilot-your-everyday-ai-companion/ 

A classic public preview, as known from other products, was not available for Copilot. Therefore, the usual method of evaluating features with a small pilot group in the company, as soon as a feature is available as a public preview, was not available.

Currently, it is still the case that at least 300 licenses have to be purchased in order to use Microsoft 365 Copilot:

Source: https://cloudpartners.transform.microsoft.com/download?assetname=assets%2FMicrosoft365-Copilot-GA-Partner-FAQ.pdf 

This article describes what you can do today to prepare for Microsoft 365 Copilot, even if you don't have the license available yet.

Copilot, Bing Chat Enterprise, Azure OpenAI – When to use what

The Microsoft 365 Copilot license is an add-on to an existing Microsoft 365 E3/E5 license and is currently quoted at $30 per user/month. Many companies are therefore planning a mix, which may look like this, for example:

• Bing Chat Enterprise is available to all users who have a Microsoft 365 E3/E5 license. Bing Chat Enterprise can also be used with Office Apps, SharePoint, Teams etc. via the features in the Microsoft Edge browser / Bing Sidebar.  (How this works is described here for example: https://m365techtalk.blogspot.com/2023/08/bing-chat-enterprise-office-online.html ) 

• Approximately 20 - 30% of the employees get a Microsoft 365 Copilot license. These are mainly the so-called power users. 

• Own solutions based on Azure OpenAI are only implemented for users and requirements where it is really about providing very specific solutions. 

In general, the checking scheme is structured like this:

• Should the AI have access to own data? - If the answer is YES, the solution with Bing Chat Enterprise is not working
• Are the features of Microsoft 365 Copilot suitable to cover the requirements? - If the answer is NO, then the answer is to create your own solution based on Azure OpenAI and possibly customized LLMs.

How Microsoft 365 Copilot is designed

Microsoft 365 Copilot is available as a plugin in Office Apps or as M365 Chat in Teams. When a user enters a request, it is tailored through an approach called "grounding". This method makes the user's input more specific and ensures that the user receives answers that are relevant and usable for his specific request. To obtain the data, a semantic index is used. This is also where security trimming takes place, ensuring that a user only receives answers generated based on data they are allowed to access. This is done via the Microsoft Graph. The response generated in this way is then returned to the user. Microsoft Copilot can also be extended. To do this Graph Connectors (https://www.microsoft.com/microsoft-search/connectors) can be used.

Example in Word and M365 Chat:

Get ready for Microsoft 365 Copilot

Before Microsoft 365 Copilot can be used, some prerequisites have to be fulfilled. For example, the solution is only available from a minimum Office version or only in the Office Apps for Enterprise. This also applies to Outlook.  Microsoft 365 Copilot requires the new Outlook, which is available for Windows and Mac. All details about the requirements for Copilot are described in this article: Microsoft 365 Copilot requirements

Preparing for the launch of Copilot

As mentioned above, it is recommended to make Copilot available to a selected test group first. Even if at least 300 licenses have to be purchased, the actual license can then only be assinged to selected users. The feedback from this test group can then be used to plan the further rollout. Microsoft provides the following information and guidelines for this tasks:

• Microsoft AI help and learning 
• Get started with Microsoft 365 Copilot

The most important task of this test group is to check that the access- rights-concept has been implemented properly, using representative scenarios. A user's request for information to which he does not have access must provide no answers. In general, the Search in Microsoft 365 can also be used independently of Copilot for such tests. The search at https://www.office.com/search returns results from all Microsoft 365 services and connected sources. This includes Teams chats, emails in Outlook, and posts in Viva Engage. The example shows that searching for sensitive information should not bring up any matches:

Create a Copilot Center of Excellence

In the article How to get ready for Microsoft 365 Copilot, Microsoft recommends creating a Center of Excellence for Copilot. This Center of Excellence can then be used to provide training materials, updates regarding the rollout in the company, FAQs and other information. The Center of Excellence is intended to be a central place for users to find everything related to the topic. Microsoft provides extensive material for this:

Source and more details: Microsoft Copilot – Microsoft Adoption

The Center of Excellence can then also provide information on where the limitations of AI and Copilot are and what needs to be considered from a regulatory perspective The EU has published the EU AI ACT for this reason.

Graph Insights Adjustments

Microsoft Viva takes over the functionality of the My Analytics and Workplace Analytics features step by step. And there is also a lot going on in the backend. The MeetingInsights and ItemInsights endpoints have been customizable for some time and now the PeopleInsights endpoint can also be configured. This affects the following areas:

• Profile Cards
• User Profile in Delve
• Listing relevant people / working with overview

I have already posted about the options to customize MeetingInsights and ItemInsights in the article "My Analytics, Workplace Analytics & Delve - or from the backend perspective Office Graph & Microsoft Graph".

Customize Insights

In general, the ItemInsights as well as the PeopleInsights can now be customized. The respective endpoint can be completely disabled or enabled / disabled for an Azure-AD Group. The third category, MeetingInsights, can only be globally enabled or disabled.

Details on ItemInsights and MeetingInsights: Customizing item insights privacy in Microsoft Graph

Customize PeopleInsights

PeopleInsights show relationships between people. This evaluates public signals such as joint Microsoft 365 Groups / Teams or shared Exchange distribution lists. This data is displayed in Delve and the profile cards in Microsoft 365 and can be queried via the API:

By default, this feature is enabled in a Microsoft 365 tenant. The following options are available for customization:

• Disable PeopleInsights for all users:

PATCH, beta:

{

  "isEnabledInOrganization": false

}

• Disable PeopleInsights for an Azure AD Group by additionally using the disabledForGroup paramter: 

PATCH, beta:

{

  "isEnabledInOrganization": false,

  "disabledForGroup": "f742839d-2321479-4de9-9616-e61877675c88c"

}

 To do this, for example, with Microsoft Graph Explorer, the following steps are performed:

• Open Graph Explorer and log in: https://developer.microsoft.com/en-us/graph/graph-explorer 
• GET, beta: https://graph.microsoft.com/beta/organization returns the OrganizationId that is needed in the next step:

• Get the current settings:

GET, beta: https://graph.microsoft.com/beta/organization/{organizationId}/settings/peopleInsights

• Customize the settings as needed:

PATCH, beta:

{

  "isEnabledInOrganization": false,

  "disabledForGroup": "f953948d-3423-4ac7-9616-d6187975c00c"

}

 

Effect: No more users are displayed in the "works with" section:

Important:

• The settings for PeopleInsights are currently only available in BETA.
• The update process does not check if an Azure AD Group exists. If the Group does not exist, no changes will be made for any user.
• It can take up to 24 hours for insightsSettings adjustments to take effect, or longer in the BETA version.

 

My Analytics, Workplace Analytics & Delve - or from the backend perspective Office Graph & Microsoft Graph

The analytics features in Microsoft 365 remain a sensitive topic for works councils and data privacy officers. This article describes the options currently available to customize these apps.

Overview and history of Office Graph & Microsoft Graph

At the time of its initial release in 2014, Office Graph was the backend for Delve, among other things. The Office Graph has since evolved and the Microsoft Graph was joined. To provide a coherent Microsoft Graph schema, Microsoft introduced an itemInsights entity that inherits all the properties of the existing officeGraphInsights resource. Currently, for backward compatibility reasons, the officeGraphInsights entity is still available. Generally, the "insights" generated by the Microsoft Graph can be disabled for the entire tenant or per Azure AD group in the Admin Center:

More details about this and the options with PowerShell / API are described in this Microsoft article: Customizing item insights privacy in Microsoft Graph. Currently, August 2021, the options described in the article are still in "Preview" status.

The apps My Analytics, Workplace Analytics & Delve

All three apps aggregate and analyze users' actions in Microsoft 365 to give individual users a system-wide view of what's happening in Microsoft 365.

Delve: The Delve app mines all documents that a user is actively working on and attempts to analyze correlations between their own work and that of other users. The goal is to show users correlations and common work areas with other users. The following always is true:

• Security by Design. Everyone only sees what they have access to.
• Private signals can only be seen by the person himself, e.g. which content might be interesting for him. This cannot be impersonated in the sense of: please show me what is interesting for John Doe.
• Public Signals only show what can also be found in other ways, e.g. via search.

Delve can be customized by each user: The "Show Documents in Delve" setting is available at the following link: https://delve.office.com/ 

If a user deactivates this function, however, his signals are still collected by the Office Graph / Microsoft Graph and are available to other users who also have access to the affected documents. Only the user himself will not see any reports about the documents he is working on or has access to after deactivation.

The Office Graph / Microsoft Graph also creates the overview and reports for the user under office.com and in apps like Outlook or Microsoft Teams. The following functions / customizations are available here:

• Meeting Insights: When a user calls up an appointment in their calendar, Outlook displays other content relevant to that appointment. This can include mails and files in the mailbox, files from OneDrive or SharePoint for which the user has at least read permission. Meeting Insights can currently only be switched on or off for the entire tenant.
• Item Insights: This feature creates recommendations based on user interaction/common tasks in Microsoft 365. These recommendations can include documents or other types of content and are displayed in People Cards (Contacts), Delve, Office.com, and other locations. Item Insights can be turned on or off per Azure AD group.

The Office Graph can also be disabled completely, as described here: Control access to Delve. This will then affect many other features such as:

• SharePoint Home
• SharePoint Activities
• OneDrive Suggestions
• Copy / Pats Functions
• etc.

My Analytics: My Analytics creates an overview of the complete working day and provides reports on how long the user has spent on mails, meetings and other things. The My Analytics function can be controlled via the license. If a user is not assigned the license for this app, the function is not available.

My Analytics, like the Delve app, cannot be impersonated and reports are only available to the individual user to view

The My Analytics app can also be configured per user:

The following applies here:

• If the app as a whole is deactivated, the user will no longer be able to access his dashboards in My Analytics. The Insights Outlook add-ins are then also no longer available.
• Inline suggestions and weekly digests will no longer be generated.
• Email activity will no longer be included in the count for other users' email open rates.

Workplace Analytics: In Workplace Analytics, data from daily work in Microsoft 365 is used to identify patterns. Data protection is considered here by default, by only presenting data anonymized and aggregated at the group level.

The Workplace Analytics function can also be controlled via the license. If a user is not assigned the license for this app, his data will not be collected and analyzed. Workplace Analytics can currently only be licensed via an EA. For details, see: Requirements for Workplace Analytics

Workplace Analytics offers the following options for configuration:

• Sources: Administrators and analysts use these to verify that Microsoft 365 and organizational data has been uploaded correctly to Workplace Analytics.
• Upload: Administrators use this to prepare and upload organization and customer data.
• Administrator Settings: Administrators use this to configure system default settings, privacy settings, and manager settings.
• Analytics Settings: Analysts use this to customize meeting exclusion rules that help ensure data accuracy.

Details: Workplace Analytics Settings

By the way: The Viva Insights feature also uses data from the Microsoft Graph, combines it with its AI and creates the corresponding dashboard. The apps My Analytics and Viva Insights will be merged in the future:

Source: https://docs.microsoft.com/en-us/workplace-analytics/myanalytics/mya-landing-page