Microsoft 365 TechTalk: Lessons learned - Copilot and his attention deficit

Not quite on the ball, impulsive, too creative - this is how Copilot is often described by users. In the following example, this is exactly how the Copilot app behaves and how you can still achieve good results.

Starting point

Dona Sarkar spoke on "This Is Why We Can't Have Nice Things" at the European Cloud Summit 2024. According to the agenda:

So all we have heard about for the past year and a half is AI , AI and more AI. How do we know if this is something that will stick around or be yet another fad, for example: Bitcoin, blockchain, NFTs, augmented reality, etc. The key to being able to identify hype vs reality around AI is to develop tech intuition. This is the way to structure your career investments around things that will pay off versus wasting your time. Let's find out together how you can build this intuition step-by-step and develop this skill of seeing around corners.

What was explained in the presentation was 100% comprehensible and reflected the hype topics of recent years. Among other things, Donna Sarkar mentioned the 100 Bad Ideas method as an approach to analyzing hype topics for their future potential. The method has a lot in common with the Design Sprint and Repertory Grid methods. Copilot should now compare the 3 approaches and clarify when which of the approaches should be used.

The initial prompt

After the title for the document, Copilot should take care of the rest. The following initial prompt was used for this:

Create an overview of the Design Sprint, 100 Bad Ideas and Repertory Grid methods. Explain the details of each method. Compare the methods in a matrix and explain the advantages of each method.

The generated text was good. However, the comparison of the methods was a little short and the column “Advantages” in the generated table was empty:

This is why additional prompts were used.

Additional prompts

Copilot offers the choice of accepting the generated text or adding a further prompt:

This results in the entire text being recreated. However, as the text should be retained, a new prompt was used:

Create a new chapter. Explain when which of the methods Design Sprint, 100 Bad Ideas or Repertory Grid should be used. Create a matrix in which the methods are compared. The matrix should be followed by a text that uses examples of each method to explain when which of the methods should be used.

Copilot then creates the chapter " Chapter: Comparing Methods for Creative Problem Solving". However, the "Disadvantages" and "When to use" columns of the table he generates do not contain any values. And there is a formatting error in the examples he lists, as if someone had typed too fast and mixed up the keys.

Copilot aborts

Finally, there should be a chapter explaining how the three methods can be used to work out use cases for generative AI solutions. Prompt:

In a new chapter, explain which of the three methods can be used to develop use cases for the topic of generative AI. Create a matrix with the advantages and disadvantages of each method. Create a summary.

Without an error message, Copilot terminated before the prompt was completely fulfilled. In addition, the table he created was incomplete and therefore useless:

The exact same prompt was entered again and Copilot generated the chapter. It is remarkable that even though the identical prompt was repeated in both runs, a different heading was created for the chapter. In the first run, wich was aborted, the heading was " Use Cases for Generative AI". In the second run, it was then " Chapter: Developing Use Cases for Generative AI".

Conclusion

The comparison with an assistant, who is not always fully attentive and somewhat impulsive, fits very well. Copilot produces useful results if you tell him very explicitly what to do. This also means that you may have to enter a prompt repeatedly without it being clear why it was aborted or why not all aspects of the prompt were fulfilled equally. To create good prompts for Copilot, you can use this structure as a guide:

Source and further details: Learn about Copilot prompts

Further good tips can be found in the article Best practices for custom instructions. In addition to the aspects "Be specific" and "Keep it simple", the following aspects are also explained here:

Use examples: Provide examples to illustrate your instructions and help the copilot understand your expectations. Examples help the copilot generate accurate and relevant responses.
Give the copilot an “out”: Give copilot an alternative path for when it's unable to complete the assigned task. This alternative path helps the copilot avoid generating false responses.