GPT - here to stay

GPT = Generative Pre-trained Transformer

• G = Generative -> An output is generated
• P = Pre-trained -> The model was pre-trained
• T = Transformer

OpenAI published Chat GPT in November 2022. In June 2023, I wrote my first blog post on this topic: Next Level AI. A lot has happened in the meantime, there have been further versions and new models such as GPT-4, GPT-4o and small language models such as Phi-3. Nevertheless, it is still true that ChatGPT and therefore services such as Microsoft Copilot are not intelligent in the true sense of the word. Nevertheless, they are very helpful and that is why they are here to stay.

To ensure that the models generate the most useful results from the start, i.e. the “G” for generative, in the name GPT, they are pre-trained, i.e. the “P” for pre-trained, in the name GPT.

These models are trained using deep learning. Random values are used to generate an output. This calculated output is then checked against an output that should ideally have been calculated. The model contains a feature that can be used to return the error/deviation from the ideal result as a correction. This means that it is trained in such a way that the correct solution is now likely to be produced if the same input is used again. It is therefore transformed. The “T” in the name GPT.

Details and further information: https://en.wikipedia.org/wiki/Generative_pre-trained_transformer

According to Gregory Bateson's learning theory, this corresponds to so-called “Zero-Order”, also known as Try & Error. But deep learning sounds better 😉.

Users report that Microsoft Copilot answers them in a friendly way when they ask nicely and in a rude way when their prompt was rude. This effect can also be explained by the way this technology works. Pre-processing takes place before the prompt is sent to the LLM. Details are described here: Microsoft Copilot for Microsoft 365 overview. Something similar happens with OpenAI / ChatGPT. The prompt, as entered by the user, remains as the baseline. So if the prompt is formulated in an unfriendly way, the response will also correspond to this tenor. The orchestration / grounding has no influence on this. Quote in the context of Copilot:

Copilot then pre-processes the input prompt through an approach called grounding, which improves the specificity of the prompt, to help you get answers that are relevant and actionable to your specific task. The prompt can include text from input files or other content discovered by Copilot, and Copilot sends this prompt to the LLM for processing. Copilot only accesses data that an individual user has existing access to, based on, for example, existing Microsoft 365 role-based access controls.

This phenomenon, that the Copilot answer is based on the language of the prompt entered by the user, is therefore not related to the next stage of learning according to Gregory Bateson, protolearning or even deuterolearning.

• Protolearning can be regarded as simple association.  I learn that when I see green, I go, and when I see red, I stop.
• Deuterolearning is a learning of context.  If you reverse the association, how long does it take for the organism to adapt? 

Source: https://www.aaas.org/taxonomy/term/9/protolearning-deuterolearning-and-beyond 

In fact, you can even tell ChatGPT and Copilot which role and style it should use. Example: Please formulate a reply to this e-mail and use a very friendly style.

Here to stay

Together with LinkedIn, Microsoft has published the 2024 Work Trend Index Annual Report. It identifies the following four key points:

1. Employees want AI at work - and they won’t wait for companies to catch up.
2. For employees, AI raises the bar and breaks the career ceiling.
3. The rise of the AI power user - and what they reveal about the future.
4. The Path Forward

The first point here is the most important and clearly differentiates AI from pseudo-trends such as Blockchain or Virtual Reality. ChatGPT was disruptive at the time of its release in November 2022. Just like Apple with the first iPhone in 2007, OpenAI created something that did not exist at this level before. This version of generative AI could be used by a normal user who had no special knowledge of the technology and produced meaningful and useful outputs. Example: Act as a travel guide and tell me what I should see in Rome. The output is certainly helpful when it comes to planning a trip to Rome.

Employees want AI at work

Just like the iPhone, generative AI applications are currently mostly going viral in companies. Employees are familiar with solutions such as ChatGPT or the video creator HyGen from their private lives. They have heard about them from friends or played around with them at home. HyGen's claim sums it up: “In just a few clicks, you can generate custom videos for social media, presentations, education and more.”

Unless you work in marketing or in the PR department, social media usually refers to a private context. Presentations, education and more - is the bridge to business.

And they won’t wait for companies to catch up

The 2024 Work Trend Index Annual Report describes the phenomenon that every user knows: Professionals aren't waiting for official guidance or training - they're skilling up. In other words: What works is also used. It doesn't matter whether the company has officially introduced such a solution or whether you have to use your private access to OpenAI, HyGen or other apps.

The 2024 Work Trend Index Annual Report also sums up the impact of these trends: “For the vast majority of people, AI isn't replacing their job but transforming it, and their next job might be a role that doesn't exist yet”

The report also provides examples and scenarios from users:

How I use AI

• I research and try new prompts
• I regularly experiment with different ways of using AI
• Before starting a task, I ask myself, “could AI help me with this?

How AI impacts my experience at work

• AI helps me be more creative
• AI helps me be more productive
• AI helps me focus on the most important work

The path to the future

The opportunity for companies is to channel employee’s enthusiasm for AI into corporate success. This will look different for every company, but there are some general starting points:

• Identify the business context of a problem or challenge and then try to use AI to solve it.
• Take a top-down and bottom-up approach. Ask both your employees and the management in the company about their use cases with AI.
• Empowering employees: AI in a business context is not intuitive. Factors such as the AI Regulation / the EU AI Act, the GDPR and topics such as who has access to which information are important here.

How many Copilots are there? Fewer than you think

Why does Copilot in Microsoft 365, for example, also use data that is integrated into the M365 search via a connector, but Copilot in Word does not? Or to be more general: What is the difference between Copilot in Word and Copilot for Microsoft 365? Let's just ask him ourselves.
Prompt in Copilot in Word: What makes you different from Copilot in Microsoft 365?

The prompt was used with the "Draft with Copilot" feature:

The detail "Copilot in Word is more suitable for users who need more specific and detailed suggestions and feedback on their writing" in the generated text makes the difference clear. It says that Copilot in Word is a writing assistant. The specification that Copilot should act as a writing assistant here is like telling a prompt in ChatGPT, that the AI should act as a travel guide and that the focus should be on suggestions for a weekend in Rome.

The technology behind this is called Retrieval Augmented Generation. In general, every prompt that a user enters is pre-processed by the Copilot Orchestration. This is where the Retrieval Augmented Generation architecture is used. Although this step is not mentioned by name in the architecture diagram, it is explained in this video by Mary Pasch, Principal Product Manager at Microsoft for Copilot: How Copilot for Microsoft 365 works: A deep dive.

A term that is also frequently used is "grounding". In the end, it means the same, i.e. adapting the prompt entered by the user, respecting the context of the used app an combine these factors to achieve the best possible result. Details on this are shown in this overview:

Source and further details can be found in this article from Microsoft: Microsoft Copilot for Microsoft 365 overview.

Retrieval Augmented Generation (RAG)

Depending on the app from which the prompt is sent, the Retrieval Augmented Generation technology decides which information sources are used to generate an answer with the help of the LLM. This aspect, called Information Retrieval System, can also be used with Azure AI Search. The article Retrieval Augmented Generation (RAG) in Azure AI Search describes how the technology works, which is also used by Copilot Orchestration.

How Copilot for Microsoft 365 decides which plug-in to use

There is also a similar effect when using plug-ins. How does Copilot decide which Plug-Ins to use to respond to a prompt? This also happens in the Copilot orchestration. The article How Copilot for Microsoft 365 decides which plugin to use describes what should be done when creating Plug-Ins so that Copilot Orchestration can evaluate them properly. The app description is located in the app manifest:

Roundup

Although this does not answer the initial question of how many Copilots are there. But it is now clear that there are far fewer than you might think. Copilot in Microsoft 365, Copilot in Word, Copilot in Outlook etc. are all the same engine, which is only used differently through orchestration and pre- and post-processing. Solutions such as Copilot in GitHub etc., however, are independent solutions.

Lessons learned - Copilot and his attention deficit

Not quite on the ball, impulsive, too creative - this is how Copilot is often described by users. In the following example, this is exactly how the Copilot app behaves and how you can still achieve good results.

Starting point

Dona Sarkar spoke on "This Is Why We Can't Have Nice Things" at the European Cloud Summit 2024. According to the agenda:

So all we have heard about for the past year and a half is AI , AI and more AI. How do we know if this is something that will stick around or be yet another fad, for example: Bitcoin, blockchain, NFTs, augmented reality, etc. The key to being able to identify hype vs reality around AI is to develop tech intuition. This is the way to structure your career investments around things that will pay off versus wasting your time. Let's find out together how you can build this intuition step-by-step and develop this skill of seeing around corners.

What was explained in the presentation was 100% comprehensible and reflected the hype topics of recent years. Among other things, Donna Sarkar mentioned the 100 Bad Ideas method as an approach to analyzing hype topics for their future potential. The method has a lot in common with the Design Sprint and Repertory Grid methods. Copilot should now compare the 3 approaches and clarify when which of the approaches should be used.

The initial prompt

After the title for the document, Copilot should take care of the rest. The following initial prompt was used for this: 

Create an overview of the Design Sprint, 100 Bad Ideas and Repertory Grid methods. Explain the details of each method. Compare the methods in a matrix and explain the advantages of each method.

The generated text was good. However, the comparison of the methods was a little short and the column “Advantages” in the generated table was empty:

This is why additional prompts were used.

Additional prompts

Copilot offers the choice of accepting the generated text or adding a further prompt:

This results in the entire text being recreated. However, as the text should be retained, a new prompt was used:

Create a new chapter. Explain when which of the methods Design Sprint, 100 Bad Ideas or Repertory Grid should be used. Create a matrix in which the methods are compared. The matrix should be followed by a text that uses examples of each method to explain when which of the methods should be used.

Copilot then creates the chapter " Chapter: Comparing Methods for Creative Problem Solving". However, the "Disadvantages" and "When to use" columns of the table he generates do not contain any values. And there is a formatting error in the examples he lists, as if someone had typed too fast and mixed up the keys.

Copilot aborts

Finally, there should be a chapter explaining how the three methods can be used to work out use cases for generative AI solutions. Prompt:

In a new chapter, explain which of the three methods can be used to develop use cases for the topic of generative AI. Create a matrix with the advantages and disadvantages of each method. Create a summary.

Without an error message, Copilot terminated before the prompt was completely fulfilled. In addition, the table he created was incomplete and therefore useless:

The exact same prompt was entered again and Copilot generated the chapter. It is remarkable that even though the identical prompt was repeated in both runs, a different heading was created for the chapter. In the first run, wich was aborted, the heading was " Use Cases for Generative AI". In the second run, it was then " Chapter: Developing Use Cases for Generative AI".

Conclusion

The comparison with an assistant, who is not always fully attentive and somewhat impulsive, fits very well. Copilot produces useful results if you tell him very explicitly what to do. This also means that you may have to enter a prompt repeatedly without it being clear why it was aborted or why not all aspects of the prompt were fulfilled equally. To create good prompts for Copilot, you can use this structure as a guide:

Source and further details: Learn about Copilot prompts

Further good tips can be found in the article Best practices for custom instructions. In addition to the aspects "Be specific" and "Keep it simple", the following aspects are also explained here:

• Use examples: Provide examples to illustrate your instructions and help the copilot understand your expectations. Examples help the copilot generate accurate and relevant responses.
• Give the copilot an “out”: Give copilot an alternative path for when it's unable to complete the assigned task. This alternative path helps the copilot avoid generating false responses.

Here the result in German and in English:

 Übersicht über Design Sprint 100 Bad Ideas und Repertory Grid

 Overview of Design Sprint 100 Bad Ideas and Repertory Grid

Don't make me think - The challenges with Microsoft Copilot and Azure OpenAI

The answer is: Copilot

...but what was the question?

It's true, the hype is very present and, similar to Microsoft Teams before it, it affects all areas of daily work if you work in a Microsoft-influenced environment.

The title "Dont make me think" is borrowed from Steve Krug's book with the same name about web design. If you look at the topics covered in the book, you will quickly find parallels to the current situation with generative AI. This is even more impressive considering that the book was first published over 20 years ago.

Or as Wictor Wilén - Product Leader @ Microsoft described it so well in his post on LinkedIn:

If you need to be an engineer to use ChatGPT or Microsoft Copilot - then we failed!:

https://www.linkedin.com/feed/update/urn:li:activity:7168538964692353027/

Even if Wictor's context here was the topic of prompt engineering, the conclusion also applies to the topic of AI in general.

Topics / chapters in the whitepaper:

• Metadata: The hidden heroes of AI
• Managing metadata in SharePoint
• Example with metadata in SharePoint and Copilot in Microsoft 365
• The semantic index
• In a nutshell - What is a semantic index?
• Copilot in Microsoft 365 & data from other sources
• Copilot in Microsoft 365 - Lessons Learned
• Copilot Studio
• Copilot - A field report
• Hijack Copilot in Microsoft 365

Download English version: LINK

Download German version: LINK

Lessons Learned – pdf, docx & Copilot in M365

If you save a document once as a Word / *.dox and once as a PDF / *.pdf file, Copilot provides significantly different responses to the content of the files, even if they have identical content.

Technically, the *.docx format and the *.pdf format differ significantly in their structure and details. And that is why the information available to Copilot via the M365 Graph / the index of the search is different, even though it is the same document / the same content.

Details on pdf

Texts are determined by the following attributes: font, font size, character string, color, position on the page and the type of display, e.g. italics, etc. Information such as line breaks, headers, paragraphs, indents etc. are not included. This means that information on the formatting of paragraphs, as used in *.docx files, is not included. The text itself is divided into fragments, which can be as small as individual characters or as large as an entire line. These fragments are stored randomly and are like pieces of a puzzle that form an overall picture only when they are positioned correctly.

As a result, the actual text has no structure, as it is the case in a *.docx file. A *.pdf file essentially has a header, a body and a trailer. The header contains information about the pdf file, such as the version of the pdf file format, the creation date and the author of the file. The body of the pdf file contains the actual content of the file, e.g. text, images and other media. The trailer of the pdf file contains information about the file, e.g. the size of the file, the checksum of the file and the location of the file on the hard disk.

Structure of a pdf file:

• Header
• Body
• xref Table
• Trailer
• Incremental updates

Source and further details on the structure of a pdf file: https://www.save-emails-as-pdf.com/news/pdf-file-format-internal-document-structure-explained/

Details on docx

*.docx files are based on XML. This means they save the layout in individual XML files and structures and can contain text, images, tables, diagrams and other data. They also support extensive formatting and document structure elements such as headers, footers and page numbers.

Structure of a docx file:

• document.xml
• Content_Types].xml
• docProps
• _rels
• text
• Media
• Topic
• user-definedXml
• Settings.xml
• FontTable.xml
• webSettings.xml
• Styles.xml
• header.xml
• footer.xml
• footnotes.xml
• endnotes.xml
• comments.xml

But the user doesn't want to know, doesn't realize and doesn't care about that. Software that should work must compensate for this. As things stand today, Copilot in M365 has potential for improvement.

An example

The following example is about the file “A quick guide to secure Office 365”. It describes details for securing Office 365 and monitoring it. The file is from 2018 and, in addition to the description of the features, it contains several tables with different levels of security settings.

The following prompt was used for this example:

Please find the document “A quick guide to secure Office 365_ENG.docx”. In the document there is a chapter “Cloud security strategy with Office 365”. Please summarize what is described in this chapter. Describe what there is about the “Default” level, the “Medium” level, the “High” level and the “Very High” level. All other chapters of the document are not relevant. Refer to the table with the levels and describe the different levels.

In both cases, the first answer is not quite what you would expect. With the *.docx file, however, it is much better than the answer based on the *.pdf files, where only one level is described:

Followup Prompts:

• Prompt for the pdf file:
• Copilot tells: To provide a detailed description of the “Medium,” “High,” and “Very High” levels, I would need access to the full content of the document. If you can provide the document or direct me to it, I would be able to summarize the different levels as requested.
• Prompt: Here you can finde the document: %Link%

Then the result is much better, even Copilot suggests in the first answer that he has found and used the correct file. See screenshot above.

• Prompt for the docx file:
• Prompt: Please describe in more detail what the levels mean. Create a list with the details for each level.

The result is slightly better than the initial prompt, but still different from the example with the pdf file with the same content.

Conclusion

Both approaches deliver what the user wants in the end. However, the prompts and the details are very dependent on the file format. The outputs are also different depending on the file format. In the end, it is up to the user to decide which output is better or worse. In any case, they differ, even though the initial prompt and the content of the two files are the same.

 

 

Size matters - Large documents and Copilot for Microsoft 365

UPDATE

Problem solved - at least an improvement is on its way!

As described in my article “Size matters - Large documents and Copilot for Microsoft 365”, Copilot is currently reaching its limits with documents longer than 20 pages / 15,000 words.

Roadmap ID 399413 now announces that this limit is to increase significantly: “Copilot in Word will be able to fully summarize documents that it could previously only partially summarize. The upper limit increases to about four times more words.”

The Microsoft page linked in the article below: Keep it short and sweet: a guide on the length of documents that you provide to Copilot has also been updated. It now speaks about 80,000 words.

--

Microsoft has published an article named Keep it short and sweet: a guide on the length of documents that you provide to Copilot. It describes how Copilot for Microsoft 365 reaches its limits when it has to work with large documents or very long emails.

The reason for this is that Copilot works with data from the Microsoft Graph, which means that the search in M365 also has a role here. Documents, emails and all other content must first be indexed by the search before they are available for Copilot. At least for the search in SharePoint Online, the limits are documented: https://learn.microsoft.com/en-us/sharepoint/search-limits.

The exact limits that apply for processing by Copilot in Microsoft 365 are currently unclear. The article Keep it short and sweet: a guide on the length of documents that you provide to Copilot gives the following recommendations:

• Shorter than 20 pages
• Maximum of around 15,000 words

The example shows how it behaves when relevant information is after these limit recommendations. The relevant information to be used via Copilot are as followed. These are on page 49 of a Word document that contains a total of 27,208 words.

If you ask Copilot “What can you tell me about Snabales Total liabilities?” you get the following answer:

If you use Copilot in Word and ask the same question, the answer is: “This response isn't based on the document: I'm sorry, but the document does not provide any information about Snabales Total liabilities...”

One option you now have here is not to use Copilot for Microsoft 365 natively, but to create your own solution based on Azure AI-Search and Azure OpenAI. In Azure AI-Search, a vector search can be used that splits large documents into so-called chunks. This article describes the details: Chunking large documents for vector search solutions in Azure AI Search

Sharing link „People in your organization“ & Copilot for Microsoft 365

Microsoft Copilot for Microsoft 365 only surfaces organizational data to which individual users have at least view permissions. Source: https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy#how-does-microsoft-copilot-for-microsoft-365-use-your-proprietary-organizational-data

The sharing function in SharePoint and OneDrive can be used to share content with users. It is then also possible to define which persons or groups are granted access and with which rights:

The following applies:

• Anyone gives access to anyone who receives this link, whether they receive it directly from you or forwarded from someone else. This may include people outside of your organization.
• People in <Your Organization> with the link gives anyone in your organization who has the link access to the file, whether they receive it directly from you or forwarded from someone else.
• People with existing access can be used by people who already have access to the document or folder. It doesn't change any permissions and it doesn't share the link. Use this if you just want to send a link to somebody who already has access. 
• Specific people gives access only to the people you specify, although other people may already have access. This may include people outside of your organization. If people forward the sharing invitation, only people who already have access to the item will be able to use the link.

Source and further details: https://support.microsoft.com/en-us/office/share-sharepoint-files-or-folders-1fe37332-0f9a-4719-970e-d2578da4941c

For the options “Anyone ”, “People with existing access” and “Specific people”, everything described above also applies for Copilot => Microsoft Copilot for Microsoft 365 only displays organizational data for which individual users have at least display permissions.

The situation is slightly different with the option “People in <Your Organization> with the link”. The following applies here: 

Creating a People in your organization link will not make the associated file or folder appear in search results, be accessible via Copilot, or grant access to everyone within the organization. Simply creating this link does not provide organizational-wide access to the content. For individuals to access the file or folder, they must possess the link and it needs to be activated through redemption. A user can redeem the link by clicking on it, or in some instances, the link may be automatically redeemed when sent to someone via email, chat, or other communication methods. The link does not work for guests or other people outside your organization. 

Source and further details: https://learn.microsoft.com/en-us/sharepoint/deploy-file-collaboration#control-sharing

This can lead to non-transparent effects for users. For example, a user who has shared content in this way may assume that this information is now available to all users in the tenant and is therefore also accessible via Copilot. In the following example, the user Stan Laurel shares the files RefDoc.docx and SnabelesSnowball.docx via the “People in your organization” link.

Another user has received and clicked the link to the RefDoc.docx file, but not the link to the SnabelesSnowball.docx file.

This leads to the following result in Copilot although the user generally has access to both files:

Question to Copilot about the contents of the SnabelesSnowball.docx file for which the share link has not yet been clicked:

Question to Copilot about the contents of the RefDoc.docx file from which the sharing link has already been used at least once:

This effect, that a user is only granted access to the file once he has clicked on the sharing link, is also confirmed in another example. Here, Copilot is asked to create a list of all files that have been shared via the link type People in <your organization>. The file RefDoc.docx, whose sharing link has already been clicked, appears in the list. The file SnabelesSnowball.docx, for which this is not yet the case, is not mentioned.

There is also another side to this topic. If the default sharing link is “People in <Your Organization> with the link”, and this link is further promoted by people, for example by posting it in a Teams post or sending it by email, this can lead to all users suddenly receiving replies from Copilot to the content behind the link, even if this information was not intended for everyone in the company. So caution and a solid concept for dealing with the topic is necessary.

To see what kind of sharing links are used, the “Data access governance reports for SharePoint sites” can be used:

Source and further details: https://learn.microsoft.com/en-US/SharePoint/data-access-governance-reports?WT.mc_id=365AdminCSH_inproduct#sharing-links-reports