Prompting Large Language Models for Legal Work
Last updated August 2023. LLMs move fast, so treat the tool list as a snapshot.
This page is based on a talk and workshop I held for the SLTA Chapter Basel on January 18th 2023. Both the presentation and the handout are available here. This document was updated for the presentation in Bern on the 15th of August 2023.
First and foremost: not every task should be solved with Large Language Models (LLMs). LLMs require significant computational resources and can be expensive to run, which may not suit all applications. As the models are trained on a diverse range of text from the internet, they may contain biases and inaccuracies that lead to biased or misleading results. Many tasks can be solved with simple rule-based methods or smaller models, making a large language model overkill. Furthermore, LLMs may not work well on certain languages and specialised domains they haven’t seen enough of during training. Last but not least, LLMs are often proprietary rather than open source, which makes your project more dependent on the LLM operators.
To see whether and how a project could use LLMs, you can work through this checklist:
- Does this problem need to be solved by an LLM? (Maybe there are rule-based methods, or smaller models are available.)
- If yes: is the problem solvable with zero-shot prompting (instruction or zero-shot)?
- If no: does the model’s performance improve when given expert status?
- If no: is the problem solvable with few-shot prompting?
- If no: can the model solve the problem with chain-of-thought prompting?
- If no: can a smaller model, or the LLM itself, be fine-tuned on larger amounts of data?
- If no: the problem may be too specific to solve with LLMs in their current state.
Choosing an LLM
Many companies offer Large Language Model inference. Here are a few examples:
| Name | Company | Registration | Frontend (playground) | Link |
|---|---|---|---|---|
| ChatGPT | OpenAI | yes | yes | Open |
| GPT-3, GPT-4 (Playground) | OpenAI | yes | yes | Open |
| Luminous | Aleph Alpha | yes | yes | Open |
| Llama 2 (hosted) | Hugging Face / Meta | yes | yes | Open |
| Jurassic | AI21 Labs | yes | yes | Open |
| Cohere Generate | Cohere | yes | yes | Open |
| GPT-J (hosted) | NLPCloud | yes | yes | Open |
| GPT-J (hosted) | Forefront | not needed | yes | Open |
| Bloom (hosted) | Hugging Face | not needed | yes | Open |
| FLAN-T5 (hosted) | Hugging Face | not needed | yes | Open |
There are a lot of open-source LLMs out there. Most of them are too big to host without a data centre or GPU cluster. Through Hugging Face, many models are available for inference or can be hosted on Hugging Face infrastructure via inference endpoints or Spaces. Some amazing models are BLOOM from the BigScience project, FLAN-T5 from Google, the OPT and Llama models from Meta AI, GPT-J and GPT-NeoX from EleutherAI, Falcon 40B from TII, and many more. I encourage everyone to visit the Hugging Face model hub and click through the many open-source models.
Few-shot prompting an LLM
Few-shot prompting is a technique in which a model is given only a few examples and is then asked to solve a task based on them. It’s useful when there isn’t enough data for traditional training (with exceptions such as SetFit and the like) or when the task is very specific and the model needs some context.
A simple example: you want a language model to classify sentences as positive or negative, and you only have a few examples. Traditional machine learning (training or fine-tuning a model) would struggle to give acceptable results. By few-shot prompting a large language model, you can get good results with very little data:
Sentence: I love having to wait for my coffee for 30 Minutes.
Sentiment: Negative (Sarcasm)
###
Sentence: I do not dislike drinking coffee early in the morning.
Sentiment: Positive
###
Sentence:
This applies to the legal sector as well. Say you want a table of each argument and its corresponding evidence: give the model one or a few examples, like so:
Text: lorem ipsum dolor.
evidence 1.
dolor sit amet.
evidence 2.
Output in Markdown:
| Argument | Evidence |
|lorem ipsum dolor. | evidence 1 |
###
Text:
A problem with complex few-shot prompting can be the length of the document (too many tokens for the model). Legal documents in particular tend to be long, and large language models can only work with a limited number of tokens. There are special models that handle longer texts (Longformers, for example), but they are limited as well. If length is a problem, try a fine-tuned model that can solve the problem with zero-shot prompting.
Zero-shot prompting an LLM
With zero-shot prompts, the model is asked to solve a problem without any examples. Zero-shot capabilities vary a lot. Good zero-shot results mostly come from fine-tuned models such as OpenAI’s InstructGPT models, T0pp, or Google’s FLAN-T5. These models have been trained to follow instructions and generate much better zero-shot results than non-fine-tuned models.
The two examples from above as zero-shot prompts:
Classify the following sentence into one of the following categories: positive, neutral or negative.
Sentence: [YOUR INPUT]
Generate a markdown table from the text. The table should have two columns: Argument and Evidence.
Text: [YOUR INPUT]
Markdown Table:
Zero-shot prompting needs a lot of experimentation. A few hacks:
- Give the model expert status, like “You are a specialised lawyer in the field…” or “The output is generated by a legal expert.”
- If the model tends to hallucinate (making up false information), you can adjust the hyperparameters. Adding text like “The legal expert AI only answers when it’s sure. If it isn’t sure about an output it answers with ‘I don’t know’” to the prompt can help, too.
- Add the first few tokens of the completion. In the example above, you could enter the table’s title in the prompt, which helps the model understand what comes next.
- Make the prompt as specific, descriptive and detailed as possible. This can include context, outcome, length, format, style and so on.
More examples can be found in OpenAI’s best practices for prompt engineering and the OpenAI Cookbook. All of the above also holds for other zero-shot-capable models, even though these links are from OpenAI.