How many Copilots are there? Fewer than you think

Why does Copilot in Microsoft 365, for example, also use data that is integrated into the M365 search via a connector, but Copilot in Word does not? Or to be more general: What is the difference between Copilot in Word and Copilot for Microsoft 365? Let's just ask him ourselves.
Prompt in Copilot in Word: What makes you different from Copilot in Microsoft 365?

The prompt was used with the "Draft with Copilot" feature:

The detail "Copilot in Word is more suitable for users who need more specific and detailed suggestions and feedback on their writing" in the generated text makes the difference clear. It says that Copilot in Word is a writing assistant. The specification that Copilot should act as a writing assistant here is like telling a prompt in ChatGPT, that the AI should act as a travel guide and that the focus should be on suggestions for a weekend in Rome.

The technology behind this is called Retrieval Augmented Generation. In general, every prompt that a user enters is pre-processed by the Copilot Orchestration. This is where the Retrieval Augmented Generation architecture is used. Although this step is not mentioned by name in the architecture diagram, it is explained in this video by Mary Pasch, Principal Product Manager at Microsoft for Copilot: How Copilot for Microsoft 365 works: A deep dive.

A term that is also frequently used is "grounding". In the end, it means the same, i.e. adapting the prompt entered by the user, respecting the context of the used app an combine these factors to achieve the best possible result. Details on this are shown in this overview:

Source and further details can be found in this article from Microsoft: Microsoft Copilot for Microsoft 365 overview.

Retrieval Augmented Generation (RAG)

Depending on the app from which the prompt is sent, the Retrieval Augmented Generation technology decides which information sources are used to generate an answer with the help of the LLM. This aspect, called Information Retrieval System, can also be used with Azure AI Search. The article Retrieval Augmented Generation (RAG) in Azure AI Search describes how the technology works, which is also used by Copilot Orchestration.

How Copilot for Microsoft 365 decides which plug-in to use

There is also a similar effect when using plug-ins. How does Copilot decide which Plug-Ins to use to respond to a prompt? This also happens in the Copilot orchestration. The article How Copilot for Microsoft 365 decides which plugin to use describes what should be done when creating Plug-Ins so that Copilot Orchestration can evaluate them properly. The app description is located in the app manifest:

Roundup

Although this does not answer the initial question of how many Copilots are there. But it is now clear that there are far fewer than you might think. Copilot in Microsoft 365, Copilot in Word, Copilot in Outlook etc. are all the same engine, which is only used differently through orchestration and pre- and post-processing. Solutions such as Copilot in GitHub etc., however, are independent solutions.

Lessons learned - Copilot and his attention deficit

Not quite on the ball, impulsive, too creative - this is how Copilot is often described by users. In the following example, this is exactly how the Copilot app behaves and how you can still achieve good results.

Starting point

Dona Sarkar spoke on "This Is Why We Can't Have Nice Things" at the European Cloud Summit 2024. According to the agenda:

So all we have heard about for the past year and a half is AI , AI and more AI. How do we know if this is something that will stick around or be yet another fad, for example: Bitcoin, blockchain, NFTs, augmented reality, etc. The key to being able to identify hype vs reality around AI is to develop tech intuition. This is the way to structure your career investments around things that will pay off versus wasting your time. Let's find out together how you can build this intuition step-by-step and develop this skill of seeing around corners.

What was explained in the presentation was 100% comprehensible and reflected the hype topics of recent years. Among other things, Donna Sarkar mentioned the 100 Bad Ideas method as an approach to analyzing hype topics for their future potential. The method has a lot in common with the Design Sprint and Repertory Grid methods. Copilot should now compare the 3 approaches and clarify when which of the approaches should be used.

The initial prompt

After the title for the document, Copilot should take care of the rest. The following initial prompt was used for this: 

Create an overview of the Design Sprint, 100 Bad Ideas and Repertory Grid methods. Explain the details of each method. Compare the methods in a matrix and explain the advantages of each method.

The generated text was good. However, the comparison of the methods was a little short and the column “Advantages” in the generated table was empty:

This is why additional prompts were used.

Additional prompts

Copilot offers the choice of accepting the generated text or adding a further prompt:

This results in the entire text being recreated. However, as the text should be retained, a new prompt was used:

Create a new chapter. Explain when which of the methods Design Sprint, 100 Bad Ideas or Repertory Grid should be used. Create a matrix in which the methods are compared. The matrix should be followed by a text that uses examples of each method to explain when which of the methods should be used.

Copilot then creates the chapter " Chapter: Comparing Methods for Creative Problem Solving". However, the "Disadvantages" and "When to use" columns of the table he generates do not contain any values. And there is a formatting error in the examples he lists, as if someone had typed too fast and mixed up the keys.

Copilot aborts

Finally, there should be a chapter explaining how the three methods can be used to work out use cases for generative AI solutions. Prompt:

In a new chapter, explain which of the three methods can be used to develop use cases for the topic of generative AI. Create a matrix with the advantages and disadvantages of each method. Create a summary.

Without an error message, Copilot terminated before the prompt was completely fulfilled. In addition, the table he created was incomplete and therefore useless:

The exact same prompt was entered again and Copilot generated the chapter. It is remarkable that even though the identical prompt was repeated in both runs, a different heading was created for the chapter. In the first run, wich was aborted, the heading was " Use Cases for Generative AI". In the second run, it was then " Chapter: Developing Use Cases for Generative AI".

Conclusion

The comparison with an assistant, who is not always fully attentive and somewhat impulsive, fits very well. Copilot produces useful results if you tell him very explicitly what to do. This also means that you may have to enter a prompt repeatedly without it being clear why it was aborted or why not all aspects of the prompt were fulfilled equally. To create good prompts for Copilot, you can use this structure as a guide:

Source and further details: Learn about Copilot prompts

Further good tips can be found in the article Best practices for custom instructions. In addition to the aspects "Be specific" and "Keep it simple", the following aspects are also explained here:

• Use examples: Provide examples to illustrate your instructions and help the copilot understand your expectations. Examples help the copilot generate accurate and relevant responses.
• Give the copilot an “out”: Give copilot an alternative path for when it's unable to complete the assigned task. This alternative path helps the copilot avoid generating false responses.

Here the result in German and in English:

 Übersicht über Design Sprint 100 Bad Ideas und Repertory Grid

 Overview of Design Sprint 100 Bad Ideas and Repertory Grid

Don't make me think - The challenges with Microsoft Copilot and Azure OpenAI

The answer is: Copilot

...but what was the question?

It's true, the hype is very present and, similar to Microsoft Teams before it, it affects all areas of daily work if you work in a Microsoft-influenced environment.

The title "Dont make me think" is borrowed from Steve Krug's book with the same name about web design. If you look at the topics covered in the book, you will quickly find parallels to the current situation with generative AI. This is even more impressive considering that the book was first published over 20 years ago.

Or as Wictor Wilén - Product Leader @ Microsoft described it so well in his post on LinkedIn:

If you need to be an engineer to use ChatGPT or Microsoft Copilot - then we failed!:

https://www.linkedin.com/feed/update/urn:li:activity:7168538964692353027/

Even if Wictor's context here was the topic of prompt engineering, the conclusion also applies to the topic of AI in general.

Topics / chapters in the whitepaper:

• Metadata: The hidden heroes of AI
• Managing metadata in SharePoint
• Example with metadata in SharePoint and Copilot in Microsoft 365
• The semantic index
• In a nutshell - What is a semantic index?
• Copilot in Microsoft 365 & data from other sources
• Copilot in Microsoft 365 - Lessons Learned
• Copilot Studio
• Copilot - A field report
• Hijack Copilot in Microsoft 365

Download English version: LINK

Download German version: LINK

Lessons Learned – pdf, docx & Copilot in M365

If you save a document once as a Word / *.dox and once as a PDF / *.pdf file, Copilot provides significantly different responses to the content of the files, even if they have identical content.

Technically, the *.docx format and the *.pdf format differ significantly in their structure and details. And that is why the information available to Copilot via the M365 Graph / the index of the search is different, even though it is the same document / the same content.

Details on pdf

Texts are determined by the following attributes: font, font size, character string, color, position on the page and the type of display, e.g. italics, etc. Information such as line breaks, headers, paragraphs, indents etc. are not included. This means that information on the formatting of paragraphs, as used in *.docx files, is not included. The text itself is divided into fragments, which can be as small as individual characters or as large as an entire line. These fragments are stored randomly and are like pieces of a puzzle that form an overall picture only when they are positioned correctly.

As a result, the actual text has no structure, as it is the case in a *.docx file. A *.pdf file essentially has a header, a body and a trailer. The header contains information about the pdf file, such as the version of the pdf file format, the creation date and the author of the file. The body of the pdf file contains the actual content of the file, e.g. text, images and other media. The trailer of the pdf file contains information about the file, e.g. the size of the file, the checksum of the file and the location of the file on the hard disk.

Structure of a pdf file:

• Header
• Body
• xref Table
• Trailer
• Incremental updates

Source and further details on the structure of a pdf file: https://www.save-emails-as-pdf.com/news/pdf-file-format-internal-document-structure-explained/

Details on docx

*.docx files are based on XML. This means they save the layout in individual XML files and structures and can contain text, images, tables, diagrams and other data. They also support extensive formatting and document structure elements such as headers, footers and page numbers.

Structure of a docx file:

• document.xml
• Content_Types].xml
• docProps
• _rels
• text
• Media
• Topic
• user-definedXml
• Settings.xml
• FontTable.xml
• webSettings.xml
• Styles.xml
• header.xml
• footer.xml
• footnotes.xml
• endnotes.xml
• comments.xml

But the user doesn't want to know, doesn't realize and doesn't care about that. Software that should work must compensate for this. As things stand today, Copilot in M365 has potential for improvement.

An example

The following example is about the file “A quick guide to secure Office 365”. It describes details for securing Office 365 and monitoring it. The file is from 2018 and, in addition to the description of the features, it contains several tables with different levels of security settings.

The following prompt was used for this example:

Please find the document “A quick guide to secure Office 365_ENG.docx”. In the document there is a chapter “Cloud security strategy with Office 365”. Please summarize what is described in this chapter. Describe what there is about the “Default” level, the “Medium” level, the “High” level and the “Very High” level. All other chapters of the document are not relevant. Refer to the table with the levels and describe the different levels.

In both cases, the first answer is not quite what you would expect. With the *.docx file, however, it is much better than the answer based on the *.pdf files, where only one level is described:

Followup Prompts:

• Prompt for the pdf file:
• Copilot tells: To provide a detailed description of the “Medium,” “High,” and “Very High” levels, I would need access to the full content of the document. If you can provide the document or direct me to it, I would be able to summarize the different levels as requested.
• Prompt: Here you can finde the document: %Link%

Then the result is much better, even Copilot suggests in the first answer that he has found and used the correct file. See screenshot above.

• Prompt for the docx file:
• Prompt: Please describe in more detail what the levels mean. Create a list with the details for each level.

The result is slightly better than the initial prompt, but still different from the example with the pdf file with the same content.

Conclusion

Both approaches deliver what the user wants in the end. However, the prompts and the details are very dependent on the file format. The outputs are also different depending on the file format. In the end, it is up to the user to decide which output is better or worse. In any case, they differ, even though the initial prompt and the content of the two files are the same.

 

 

Size matters - Large documents and Copilot for Microsoft 365

UPDATE

Problem solved - at least an improvement is on its way!

As described in my article “Size matters - Large documents and Copilot for Microsoft 365”, Copilot is currently reaching its limits with documents longer than 20 pages / 15,000 words.

Roadmap ID 399413 now announces that this limit is to increase significantly: “Copilot in Word will be able to fully summarize documents that it could previously only partially summarize. The upper limit increases to about four times more words.”

The Microsoft page linked in the article below: Keep it short and sweet: a guide on the length of documents that you provide to Copilot has also been updated. It now speaks about 80,000 words.

--

Microsoft has published an article named Keep it short and sweet: a guide on the length of documents that you provide to Copilot. It describes how Copilot for Microsoft 365 reaches its limits when it has to work with large documents or very long emails.

The reason for this is that Copilot works with data from the Microsoft Graph, which means that the search in M365 also has a role here. Documents, emails and all other content must first be indexed by the search before they are available for Copilot. At least for the search in SharePoint Online, the limits are documented: https://learn.microsoft.com/en-us/sharepoint/search-limits.

The exact limits that apply for processing by Copilot in Microsoft 365 are currently unclear. The article Keep it short and sweet: a guide on the length of documents that you provide to Copilot gives the following recommendations:

• Shorter than 20 pages
• Maximum of around 15,000 words

The example shows how it behaves when relevant information is after these limit recommendations. The relevant information to be used via Copilot are as followed. These are on page 49 of a Word document that contains a total of 27,208 words.

If you ask Copilot “What can you tell me about Snabales Total liabilities?” you get the following answer:

If you use Copilot in Word and ask the same question, the answer is: “This response isn't based on the document: I'm sorry, but the document does not provide any information about Snabales Total liabilities...”

One option you now have here is not to use Copilot for Microsoft 365 natively, but to create your own solution based on Azure AI-Search and Azure OpenAI. In Azure AI-Search, a vector search can be used that splits large documents into so-called chunks. This article describes the details: Chunking large documents for vector search solutions in Azure AI Search

Sharing link „People in your organization“ & Copilot for Microsoft 365

Microsoft Copilot for Microsoft 365 only surfaces organizational data to which individual users have at least view permissions. Source: https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy#how-does-microsoft-copilot-for-microsoft-365-use-your-proprietary-organizational-data

The sharing function in SharePoint and OneDrive can be used to share content with users. It is then also possible to define which persons or groups are granted access and with which rights:

The following applies:

• Anyone gives access to anyone who receives this link, whether they receive it directly from you or forwarded from someone else. This may include people outside of your organization.
• People in <Your Organization> with the link gives anyone in your organization who has the link access to the file, whether they receive it directly from you or forwarded from someone else.
• People with existing access can be used by people who already have access to the document or folder. It doesn't change any permissions and it doesn't share the link. Use this if you just want to send a link to somebody who already has access. 
• Specific people gives access only to the people you specify, although other people may already have access. This may include people outside of your organization. If people forward the sharing invitation, only people who already have access to the item will be able to use the link.

Source and further details: https://support.microsoft.com/en-us/office/share-sharepoint-files-or-folders-1fe37332-0f9a-4719-970e-d2578da4941c

For the options “Anyone ”, “People with existing access” and “Specific people”, everything described above also applies for Copilot => Microsoft Copilot for Microsoft 365 only displays organizational data for which individual users have at least display permissions.

The situation is slightly different with the option “People in <Your Organization> with the link”. The following applies here: 

Creating a People in your organization link will not make the associated file or folder appear in search results, be accessible via Copilot, or grant access to everyone within the organization. Simply creating this link does not provide organizational-wide access to the content. For individuals to access the file or folder, they must possess the link and it needs to be activated through redemption. A user can redeem the link by clicking on it, or in some instances, the link may be automatically redeemed when sent to someone via email, chat, or other communication methods. The link does not work for guests or other people outside your organization. 

Source and further details: https://learn.microsoft.com/en-us/sharepoint/deploy-file-collaboration#control-sharing

This can lead to non-transparent effects for users. For example, a user who has shared content in this way may assume that this information is now available to all users in the tenant and is therefore also accessible via Copilot. In the following example, the user Stan Laurel shares the files RefDoc.docx and SnabelesSnowball.docx via the “People in your organization” link.

Another user has received and clicked the link to the RefDoc.docx file, but not the link to the SnabelesSnowball.docx file.

This leads to the following result in Copilot although the user generally has access to both files:

Question to Copilot about the contents of the SnabelesSnowball.docx file for which the share link has not yet been clicked:

Question to Copilot about the contents of the RefDoc.docx file from which the sharing link has already been used at least once:

This effect, that a user is only granted access to the file once he has clicked on the sharing link, is also confirmed in another example. Here, Copilot is asked to create a list of all files that have been shared via the link type People in <your organization>. The file RefDoc.docx, whose sharing link has already been clicked, appears in the list. The file SnabelesSnowball.docx, for which this is not yet the case, is not mentioned.

There is also another side to this topic. If the default sharing link is “People in <Your Organization> with the link”, and this link is further promoted by people, for example by posting it in a Teams post or sending it by email, this can lead to all users suddenly receiving replies from Copilot to the content behind the link, even if this information was not intended for everyone in the company. So caution and a solid concept for dealing with the topic is necessary.

To see what kind of sharing links are used, the “Data access governance reports for SharePoint sites” can be used:

Source and further details: https://learn.microsoft.com/en-US/SharePoint/data-access-governance-reports?WT.mc_id=365AdminCSH_inproduct#sharing-links-reports

Case study on the use of Microsoft AI solutions

Currently, not a day goes by without news in the area of AI. The following article is about a project with a company from Germany that has used features from the Microsoft AI stack to implement solutions for employees' daily work.

Highlights

• Added value of AI technologies in daily work
• How can AI be used in the context of customer projects 
• Requirements related to the AI Act and GDPR
• How can AI be used effectively in harmony with the human factor
• How does the secure use of AI solutions look in terms of an IT security strategy?

Challenges

As a leading consultancy strategy projects, the company's focus is on what their customers need. The human factor remains one of the most important aspects here. AI solutions must be easy to use and deliver reliable, reproducible results if they want to add value in day-to-day work. This made it even challenging to classify and use artificial intelligence correctly.

Statement taken during the project:

• With the solutions based on Azure OpenAI, we speed up our qualification process and the final validation activities in consulting projects, which is a significant advantage - explains the management.
• We use the Microsoft Azure solution architecture to meet the high requirements of our customers for the secure collection, storage and analysis of data - says the data protection officer.
• By using the Azure tools for SecDevOps, i.e. the interaction between security, development and IT operations, we can automate and standardize processes. - This is how the CISO summarizes the framework for the AI solution.

These design principles, as outlined by the CISO, are the foundation for fulfilling the requirements of the GDPR and the AI Act / AI Regulation without any problems.

Even sensitive data with aspects relating to the Geschäftsgeheimnis-Gesetz / German Trade Secrets Act can now be processed by Microsoft's AI solutions.

Example / quote from the project: One of our customers has well over 100 existing patents. The customer now wanted to know what new product suggestions the AI would generate based on the existing patents.

Objectives and solutions in the project

The driver of successful companies is not exclusively digitalization. Our markets have been saturated for a long time now, and only a few sectors are still focused on real growth, but more often just on shifting market share. The pace of innovations is becoming ever tighter. However, speed is only effective if there is a strategic idea behind the innovations. Finding out whether an idea meets the actual requirements of the market takes time. The aggregation of survey results and feedback from beta phases takes time to complete. 

Beta testing for new processes and products always has to survive against established expertise and experience. This specialist knowledge exists either in the heads of employees or in countless internal company repositories, databases and knowledge sources. Making this knowledge usable when testing new solutions was a challenge that could be solved with AI.

Quote from the project: With the Microsoft 365 Chat function, we can answer questions relating to data in our M365 environment in seconds. As we store data and information from our customers in secure project rooms in Microsoft Teams, this function is also available to us there.

The Microsoft 365 Chat solution is a feature in the context of Microsoft Copilot. Here, the quality of the prompt that a user enters determines the quality of the result that the AI generates. It was therefore a key factor during the rollout to ensure employee empowerment through a training concept.

Data accuracy was necessary, especially in the area of quality management of results. Here, the typical hallucination of generative AI solutions was prevented by teaching existing large language models in Azure OpenAI and using predefined prompts.

The solutions:

• Storing project data and information in Microsoft Teams / SharePoint means that this information can be analyzed using Microsoft 365 Chat.
• Training concept for the use of AI solutions / Prompt Engineering
• Special processes and quality control based on AI were made possible with customized Large Language Models in Azure OpenAI
• Solutions relating to specific topics were implemented using predefined prompts/prompt extensions with Microsoft Copilot Studio. 

Benefits

A data-supported approach to developing new and innovative processes and products can be applied much more efficiently, quickly and scalably with AI. The definition of new solutions, as well as the associated testing and quality management, is also supported by AI solutions. 

• Evaluating the current situation / evaluating exsisitng data pools in consulting projects
• The creativity of generative AI solutions is used to identify new processes and approaches for product innovations
• The human factor is seen as an initial component of the consulting approach, but can now focus on the essentials