Does refining prompts really improve the accuracy of generative AI?｜禁術おじさん

This article is about this topic

Organizing your prompts makes it easier to get the answers you want from generative AI.
Specifying details also helps you clarify what you are looking for.
But honestly, it is a hassle to refine every little casual question.

While studying for the Generative AI Passport, the topic of prompts comes up.

Organize and write down the purpose, conditions, output format, and restrictions.

By doing so, it becomes easier for the generative AI to understand your intentions and easier to draw out the model’s capabilities.

I see.

While I think that is certainly true, I also thought of something else.

Isn’t it quite a hassle to do that every time?

In this article, I will consider whether refining prompts really improves the accuracy of generative AI. And, now that generative AI itself has become better at grasping our intentions, I will think about whether there is still any point in going out of our way to refine prompts.

Does refining prompts improve accuracy?

First, “Does refining prompts improve the accuracy of generative AI?”

Basically, it seems safe to assume that it does.

In fact, companies that provide generative AI also officially release information on how to write prompts.

Anthropic’s official documentation introduces giving clear and direct instructions to Claude, specifically conveying the desired output format and constraints, and providing necessary context and examples.

OpenAI, in its official guide for ChatGPT, recommends making prompts “clear and specific” and providing necessary background information. It also introduces a method of adjusting prompts while looking at the answers rather than trying to complete it in one go.

Reference link: https://help.openai.com/ja-jp/articles/10032626-prompt-engineering-best-practices-for-chatgpt?utm_source=chatgpt.com

In other words, the idea that “refining prompts improves accuracy” seems to have a reasonable basis even from the perspective of those providing generative AI.

However, “improving accuracy” here does not mean that the generative AI itself becomes smarter.

For example,

“Summarize this article”

Compared to asking like this,

“Summarize this article in about 300 characters, divide it into three key points, and avoid using technical jargon as much as possible.”

The latter makes it much clearer what is being requested.

How long should it be?
What should be emphasized?
What kind of writing style is desired?

By communicating such conditions, you can reduce the gap between what the generative AI produces and what you actually wanted.

That is what I mean when I say that “accuracy improves” in this context.

So, how do you refine them?

When organizing a prompt, there are several elements to consider.

What do you want it to do?
What is the purpose?
What are the conditions?
In what format should it be output?
What should it not do?

If necessary, you can also provide reference texts or output examples.

Looking at it this way, it seems less like giving instructions to a generative AI and more like creating a work order.

And I, too, decided to start doing this yesterday.

Actually, I only started using detailed prompts yesterday

Although I have been writing about prompts up to this point, I only started using prompts where I carefully organized the conditions yesterday.

I was studying for the Generative AI Passport, and I rediscovered the concept of organizing objectives, conditions, output formats, and prohibitions.

So, I decided to actually try it out.

The result was a prompt for writing note articles.

Article theme.
Content to include.
My personal thoughts.
Writing style.
Article structure.
Prohibitions.
Output format.

I have used generative AI to write note articles before, but I had never specified conditions in such a consolidated way.

And this article you are reading right now was also created together with generative AI using that prompt.

I am refining prompts to have generative AI write an article about prompts.

It is a bit of a complicated story.

Refining prompts also organizes your own thoughts.

What I felt when I actually created a prompt for Note was that I wasn’t just organizing instructions for the generative AI.

What do I want to write in this article?
What is the main thing I want to convey?
What kind of writing style do I want?
Conversely, what kind of writing style do I not want?

To write these in a prompt, you need to decide these things yourself.

I thought I was organizing what I wanted the generative AI to output, but as a result, what I was looking for was also being organized.

This was a benefit I felt after actually creating one.

It not only makes it easier for the generative AI to answer, but it also organizes the mind of the person asking the question.

The meaning of refining prompts might surprisingly lie here as well.

Are prompt templates found everywhere actually useful?

When researching generative AI, I see various prompt templates.

Assign a role.
Write the purpose.
Write the prerequisites.
Write the constraints.
Specify the output format.

I think this framework itself is useful.

Because you don’t have to think from scratch.

The prompt for Note that I created this time can also be used for the next article once it is created.

Every time,

Use ‘I’ (boku) as the first person.
Do not make up episodes on your own.
Do not force a positive conclusion.

I don’t have to write these things.

If you are repeating similar tasks, it seems to make sense to make them into templates.

However, as templates become more detailed, other problems also arise.

Creating the prompt itself is a hassle.

To get generative AI to do something, you first have to complete a request form for the generative AI.

In some cases,

“I want to do this, what kind of prompt should I use?”

You end up creating the prompt to give to the generative AI while asking the generative AI itself.

You end up using generative AI to use generative AI.

For everyday use, would you really go that far?

This is the part that bothers me the most.

I understand if you are using it for work.

You want to standardize the output format.

You want to repeat the same task many times.

You want to minimize answers that fall outside the conditions as much as possible.

In those situations, I think there is great value in properly creating a prompt at the beginning.

This note article is also, in a sense, close to this.

But what about everyday interactions with generative AI?

For example,

“What does this word mean?”
“I have eggs and cabbage in the fridge, what can I make?”
“I don’t feel like working today”
“Can you make this text a bit softer?”

Even for things like this, every time,

“You are an expert in XX”
“Please answer according to the following conditions”
“The purpose is XX”
“The output format is XX”
“The following is prohibited”

…is not something I do very often.

At least, I find it troublesome.

Even if I just want to ask a quick question, starting by designing a prompt beforehand feels like it defeats the very convenience of generative AI.

Even if generative AI becomes smarter, is there any point in refining prompts?

Recent generative AI models can understand our intentions to some extent even with short instructions.

If that’s the case, will there be no need to go out of our way to write detailed prompts?

As someone who has only been using detailed prompts for two days, I don’t know the answer.

However, for now,

instead of “you should always refine your prompts,” I think “refine them only when there is a reason to do so”

is probably fine.

If it’s a quick question, just ask it as is.

If you don’t get the answer you expected, provide additional conditions.

On the other hand, for things used in work, repetitive tasks, or when you want to control the output to some extent like in this note article, it’s better to refine the prompt from the start.

That’s the extent of how I differentiate my approach.

By refining prompts, it becomes easier to get the desired answers from generative AI.

At the same time, you can also clarify what you are looking for yourself.

But refining them takes effort.

So in the end, I think “how much you should write in a prompt” depends on what you want the generative AI to do.

For simple things, just ask directly.

If you want something proper, ask properly.

And if you really want to refine your prompts, you should just create the prompt itself together with the generative AI.

Now, on my second day of using detailed prompts, that is how I feel.

By the way, the prompt used to create this article

Since I have written about the “meaning of refining prompts” up to this point, I would like to include the actual thing at the end.

This is the prompt I created yesterday while interacting with the generative AI to write my note articles.

And this article itself was also created using this prompt.

It is quite long.

I even wonder myself if it is necessary to be this detailed.

However, what kind of conditions are actually specified? And what kind of article was created from those conditions?

I think that if you look at the prompt after reading the completed article, you will be able to compare them to some extent.

Below, I will post exactly what I am actually using.

*Paid section starts here

Originally Appeared Here