Image: Kaylee Walstad, EDRM with AI- hat tip to Ralph Losey’s Visual Muse.
Did you know that talking nicely to your Generative AI (“GenAI”) platform can improve its performance? This may not be intuitive, but it is true. And there are other aspects of effectively prompting your GenAI bot that merit consideration. Today’s topic is prompts. Lawyers who use GenAI need to develop the distinct skill of creating effective prompts and carry out the responsibility of critically evaluating the output. A “prompt” is the text input that provides context and instructions to the GenAI app. Prompts begin the conversation with GenAI and carry it on in an ordered, iterative fashion to achieve the desired output.
Fortunately, skillful prompting is not complicated or difficult and may be mastered with a reasonable amount of study and practice. Lawyers, who mastered library research skills, Boolean and keyword searches, and critical analysis of research products, should have no problem adapting to producing excellent and sound output with GenAI apps. Interaction with these apps is fundamentally conversational rather than complex in nature, which is the whole point of GenAI. GenAI apps are created using complex programming, applying algorithms, machine learning, testing, and refining so that the user can communicate with the machine in a manner similar to human communication. However, effective communication with GenAI apps needs to be orchestrated in a way that plays on the strengths of the GenAI apps while avoiding potential issues of Large Language Models (LLMs) such as automation bias, hallucinations, and source of training data.
Do not treat GenAI apps like an internet search. To unleash the full power of GenAI apps, prompts must be more specific and directive than a keyword or typical internet search.
The Hon. Judge Ralph Artigliere (ret.)
Investment in developing effective prompts pays off in efficiency, accuracy, and effectiveness of outputs from Gen AI. Fundamentally, the prompting process is like giving clear instructions to someone who works for you. If you ever had an assistant or partner who understood you well and complemented your work product such that the two of you together could routinely maximize output, this is what can happen with effective prompts. Simply put, the better you and the program understand each other, the better the result.
DISTINCTION BETWEEN GENERATIVE AI AND SEARCH ENGINES
Search engines like Google, Microsoft Bing, and Ask.com identify websites that provide answers to questions or locate something the user wants on the internet, like a specific product or service. GenAI is so much more. Search engines direct the user to a number of sites that may or may not be useful and may be ordered in priority based on factors other than utility or quality of the site. Then, the user must review the list and search for the desired information. GenAI apps, on the other hand, are designed to extract information from billions of bits of training information well beyond internet websites and to conduct analysis and specific tasks at the user’s direction. Do not treat GenAI apps like an internet search. To unleash the full power of GenAI apps, prompts must be more specific and directive than a keyword or typical internet search.
UNDERSTAND HOW LLMs WORK
Learn how LLMs work. Knowing why the accuracy of LLM’s is dependent on the quality of the prompts you use makes it easier to learn effective prompting. For a concise and understandable discussion of this topic, see Casetext, “What makes large language models tick?”.
Learn enough about LLMs to appreciate their weaknesses. Then conduct your prompting in a way to maximize the strengths and minimize the weaknesses. For example, here are some key weaknesses of GenAI apps that effective prompting aims to avoid:
Limited Training data: GenAI is limited to its training data, so may lack common sense or detailed knowledge about how the world works. Prompts should provide key facts and context.
Hallucinations and pig trails: GenAI will attempt to generate plausible-sounding responses even when lacking knowledge. GenAI apps may make up convincing but false responses, called hallucinations. Prompts should be specific to avoid misinformation, and responses should be verified against reliable sources.
Bias: Gen AI apps reflect biases in their training data and by virtue of biased prompts. Prompts should avoid assumptions and be neutrally framed.
Tendency to be overconfident: GenAI results appear authoritative and plausible even when incorrect. Prompts should request caveats, citations, or accuracy ratings.
Lack of memory: GenAI apps treat each prompt in isolation. Prompts should repeat key details instead of saying “as mentioned”.
Literal interpretation: GenAI apps interpret prompts in a literal, rigid way. Prompts should be free of idioms, metaphors and ambiguity.
Diverting responses: GenAI apps may provide tangential or irrelevant text. Prompts should be laser focused on the information sought. Iterative efforts should clearly delineate the focus.
Limited prompt length: Different models have varied limits on length or size of prompts. Organization and step by step prompting are needed to overcome limits.
The goal of good prompting is to work within and around these limitations to elicit useful, accurate responses by providing targeted context and accurate direction in prompts.
CHOOSE YOUR INPUT CAREFULLY–
7. Most GenAI apps have limited amount of space for prompts and context. Do not overshare because you may not have room for more important information and oversharing can lead to misunderstanding about what you need.
8. Avoid excessive detail that is not directly relevant to the desired output. GenAI apps can sometimes get distracted or misled by too much background information.
9. Biased input can lead to biased results. If you provide a prompt that is a leading question, you will get a biased answer. A more unbiased prompt asks for both sides of an issue. Open-ended questions yield a more robust and unbiased result. For a discussion of this topic and an example of biased product, see How To Write Effective Prompts for Generative AI Tools, Center for Faculty Excellence, Montana State University found at https://www.montana.edu/facultyexcellence/teaching-advising/genai/prompts.html.
The Hon. Judge Ralph Artigliere (ret.)
QUICK TIPS ON PROMPTS
Here are a few quick tips for learning better prompting. Apply these and see how quickly your work with GenAI will be more efficient and effective.
- It is essential to pose specific, unambiguous, and clear instructions in your query. Avoid conflicting terms and use positive language. Tell the GenAI app exactly what you want it to do and give it context and information that focuses the search and analysis. Just as you would with an assistant, provide detail on the specific task you want done. Use action terms like propose, summarize, list, compare, outline, or predict.
- Use an iterative approach to refine the output. Have a discussion and tell the program what you like and do not like about an answer. GenAI apps learn along with you in an iterative process of prompting. Providing feedback to the program allows your tool or copilot to adjust to your needs and respond accordingly. And unlike the work of a law clerk or associate, the product of your prompt is available in a matter of seconds, making iterative prompting efficient and economical.
- Some products allow you to input fundamental information about you and your needs that will be applied to all your searches unless you direct otherwise. For example, ChatGPT from OpenAI allows the user to customize your interactions with ChatGPT by providing specific details and guidelines for your chats. The custom instructions ask: “What would you like ChatGPT to know about you to provide better responses?” and “How would you like ChatGPT to respond?” Whenever you edit your custom instructions, they will take effect in all new chats you create. Take advantage of this to provide foundational information and establish the tone, specificity, and accuracy of the responses you need, which is important context for your queries. This feature may not be available on all products, but it is useful where present.
- Be as specific as possible, contextualizing such specifics as the intended audience, tone, and timeframe.
- You may establish parameters for GenAI apps such as word count or numbers of items on a list. If necessary, when there are limitations, instruct the program to prioritize. It is possible to ask later what was excluded from the product.
- Format prompts clearly by separating the context from the actual question or instructions. Visual separation with line breaks or bullets points can help the GenAI app understand the different parts of the prompt.
- Most GenAI apps have limited amount of space for prompts and context. Do not overshare because you may not have room for more important information and oversharing can lead to misunderstanding about what you need.
- Avoid excessive detail that is not directly relevant to the desired output. GenAI apps can sometimes get distracted or misled by too much background information.
- Biased input can lead to biased results. If you provide a prompt that is a leading question, you will get a biased answer. A more unbiased prompt asks for both sides of an issue. Open-ended questions yield a more robust and unbiased result. For a discussion of this topic and an example of biased product, see How To Write Effective Prompts for Generative AI Tools, Center for Faculty Excellence, Montana State University found at https://www.montana.edu/facultyexcellence/teaching-advising/genai/prompts.html.
PROMPTING IS THE BEGINNING OF THE PROCESS; VERIFICATION IS ALSO REQUIRED
Applying good prompt engineering is an essential beginning to effective use of GenAI apps. However, critical analysis of output before using the information is essential to ethical and accurate lawyering. Just like you would check a brief prepared by a legal assistant or clerk, verification of work done with the assistance of GenAI apps is essential to achieving accuracy and suitability for submission as your work.
MORE RESOURCES ON PROMPTS
In addition to the resources mentioned above, the following articles and sources are helpful in learning to properly prompt:
CONCLUSION
Prompt engineering may sound intimidating or formal, but it is actually not as complicated as it sounds. Lawyers, in particular, should be able to master effective prompts because they are already experts at linguistics, context, logic, and communication. All of the discussion in this piece is really common sense based on a limited but sufficient knowledge of how LLMs are constructed and how they work. Learn what you need to know, and “happy prompting.”
____________________________________
Judge Artigliere (ret.) gratefully acknowledges the assistance of Rose Jones of King & Spalding, Jeanne Somma of Lineal for their help with this article.
