It is very important to engage with the court's various stakeholders to ensure the successful planning, development, and launch of an ODR project. Bringing as many stakeholders into the design process as possible will result in a more robust end product, fewer surprise objections or issues down the road, and a higher likelihood of buy-in and use of the ODR platform.
Sample ODR Talking Points
NCSC ODR Fact Sheet for Courts
Pew ODR Fact Sheet for Courts
These videos were created by the National Center for State Courts and are available for your use. They come with English and Spanish subtitle files. The Legal Language 101 video was designed so that it could be cut into just a specific definition or section, should a jurisdiction wish to embed a video where a user might first encounter a term. Lastly, there is room at the end of each video where a jurisdiction could add a link to a website of their choosing. If you would like copies of the files or to learn more, please use the Contact button on the top of the page to get in touch.
[California] Stakeholder Invitation Template
[Florida] Online Case Resolution Flyer
[Florida] Media Advisory
[Florida] Online Case Resolution (OCR) Pilot Project Report
[New Mexico] ODR Poster
[New Mexico] ODR Rack Card
[New Mexico] Consumer Debt Flowchart
NCSC-Designed Examples
These are examples of process maps that NCSC made for a jurisdiction. They are designed to be given to a court user early on (either as a stand alone document, or as part of an ODR platform or self-help website) so that the user has a high-level overview of what a certain case type entails in terms of required steps, costs, and timing. If your jurisdiction is interested in making one of these maps, please use the Contact page to get in touch.
[Hawai'i] Small Claims Process Map
[Illinois] Divorce Process Map
If your court is producing materials that need to be read and acted upon by the general public, it is always good to double-check that your content or websites are clear and easy to use. User testing can give you information about content that needs to be rewritten, tasks that are confusing for users when navigating your website, and even identify which content is most frequently used. User testing is most effective when it is used frequently across every stage of content development and service implementation.
If your court has the resources, it is possible to engage external consultants or purchase software that can perform sophisticated analysis on your written content and facilitate user tests of your websites. But even if your court does not have dedicated resources for formal user testing, it is possible to connect with your constituents and test your written or digital content.
Testing as few as three users over 90 minutes each month can give you valuable information about the effectiveness of your materials and digital services. Here are the steps you can use to conduct simple user tests using a “think aloud” protocol :
- Identify a task, or piece of content you would like to test. The easiest place to start can be making a list of all of the actions you would like a user to accomplish when visiting your website for example: find a specific piece of information, set up and log into a user account, navigate between pages, upload/download a document, and understand how information is displayed and communicated.Once you’ve made your list, discuss with your team which tasks or content are most important to immediately. Try to identify 2 or 3 priority items.
- Now that you have an idea about which tasks you’d like to test first, turn those tasks into scenarios that you can discuss with a user. Scenarios are prompts for the user to try out your website. A scenario can give the user information about what to do without getting specific guidance from you about how to do it. For example: Task 1: find the new forms to file for a divorce.
Scenario: Imagine that you would like to divorce your partner. You both agree to the divorce and have already agreed on how to split up your property and assets. You have no shared children. Starting from the homepage of this website, where would you look for the forms to file for your divorce?
- Time to set up your space. Informal user testing does not require an elaborate technical setup, but you will want to have a private space that is set up with an internet-connected computer (if you are testing a website) or paper copies of draft content. Your testing space should be comfortable and quiet enough to chat.
If available, you can use screen recording software like Zoom or Castima to record the session. If you don’t have recording software, you can have a college join you and take notes.
- Recruit Volunteers. Identifying potential testers for your website or online content can be as easy as walking the halls of your courthouse! Unless your content is very specific to a particular population, such as documents translated into another language, almost any person who has an additional half-hour can be an effective usability tester. You can also find testers by recruiting with flyers, asking for referrals from colleagues who have ongoing contact with members of the public, using a pop-up on your current website, or even producing handouts to be included in self-help materials.
Here’s what a walk-up script might look sound like:
“Hi! We’re working on a website (app, prototype, whatever it is) and want to see if it works as intended. That’s why we’re asking visitors to the court to test this site for a few minutes and tell us what they think about it.
Would you like to do one of these tests? Before you answer, there’s absolutely nothing you can do wrong. Plus, you can enjoy a cup of coffee, muffin, whatever, at my expense as a thank you.” (adapted from this site)
You are looking for volunteers who are available, who meet any specific user requirements you might have (like language ability), would agree to have their voice recorded (if you are recording), and who might be comfortable thinking aloud with you while they are using your website.
If you are having a hard time recruiting testers, you may want to consider offering a small incentive like a $5 coffee card, or perhaps validated parking.
- Conduct the test. Now that you’ve identified a few volunteers (no more than three) it’s time to get testing! Your volunteer should be seated at the computer with your chair next to them. If someone is taking notes they should be inconspicuously off to the side. After your volunteer is seated and ready to go, you will need to share a little about how the test will go. There are a few key concepts you will want to be sure to share (and there is a sample script in the resources section of this page):
We are testing our website to see if it works for the people who visit our courthouse.
We are testing our website, NOT YOU. You don’t need to worry about making any mistakes, you are pointing out our mistakes!
We are going to ask you to try to do a few things on the site. While you are using the site, we’d like you to talk out loud about what you are thinking, what you expect to happen, if you are confused or if the site did something you didn’t expect.
Don’t worry about our feelings! We want to make sure that this site works for people who visit our courthouse and you are helping us make it better!
After you share these ground rules, it’s time to share your scenarios (one at a time). While your user is navigating the website, avoid the impulse to explain or guide. Try not to lead or guide users. If they have questions, tell them that you appreciate the question but you are interested in seeing how they use the site without someone sitting next to them and that you will answer their question once the session is over. Ideally, there are only times you should speak during the test session: to introduce the scenarios, to prompt the user to think aloud if they stop speaking, and to encourage them to expand on thoughts and reactions. Check the resource section of this page for a list of handy responses that might help encourage a user to give you a little more detail.
While your user is navigating the site, concentrate on their words and their body language. If you take notes, it might be distracting to the user so It’s best to rely on your note-taker or recording to capture content. Make sure you are also watching the clock; each question shouldn’t take more than 10 minutes.
Check the resource section of this page for a testing checklist to help you plan for a usability session.
Next Steps. After the session has concluded and you’ve thanked your users and sent them on their way, debrief your findings. What worked? What didn’t work? Did users have the same set of issues? What might you be able to do / revise to address the issues raised during testing? Do these tests raise issues you should explore for your next round of testing?
Resources :
All about usability testing from usability.gov (US Department of Health and Human Services)
https://www.usability.gov/how-to-and-tools/methods/usability-testing.html
Rocket Surgery Made Easy: the Do-It-Yourself Guide to finding and Fixing Usability Problems http://sensible.com/rsme.html
Guerrilla Usability Testing / DIY Usability Testing Method https://userbrain.net/blog/7-step-guide-guerrilla-usability-testing-diy-usability-testing-method
How to Conduct a Usability Test in Six Steps from Start to Finish https://uxplanet.org/how-to-conduct-a-usability-test-in-six-steps-from-start-to-finish-4082e8d57858
Downloads:
Demo Videos
Sample User Testing Session
Think-Aloud Example Session with a Young User Using a Paper Prototype:
https://www.youtube.com/watch?v=9wQkLthhHKA
More About "Think-Aloud" Protocols
Start by identifying the problem(s) you collectively want to solve. Next, identify the core and periphery stakeholders and prioritize your engagement. See the Stakeholder Outreach List on this page for ideas. Then, decide how you want to interact. Tell your stakeholders the problem(s) you are trying to solve, why you need their help, and how they can help. Do you want them to assist in user testing the ODR platform? Do you want their feedback on a process? Can they help you create legal information to incorporate into the platform? Can they help you conduct outreach to court users who might take advantage of ODR?
- Designing Effective Conversations (Article from SustainAbility)
- Stakeholder Invite Template from City of San José
- Leading Change: Improving the Court and Community’s Response to Mental Health and Co-Occurring Disorders (NCSC) [see the stakeholder section]
- City of LA Public Planning Process (Overview Video) https://youtu.be/hTcSWeTckWQ
- Meaningful Public Engagement (American Planning Association) https://youtu.be/8POVTOpie9E
- Questions Change Everything in Community Engagement (TedX St.Kilda) https://youtu.be/JViKndm2y9E
- Community Engagement: Ensuring a Role For Courts’ Most Valuable Partners (Center for Court Innovation) https://youtu.be/3PlRnmrOcPQ
- Domestic Violence and Court Community Engagement (NCJFCJ) https://youtu.be/tv2QqmyARaw
Examples from Outside the Courts