Thresh’s LLM & ElasticSearch Search Experience
ongoing research, recommendations and design of Thresh Power’s search experience
Summary
Thresh is a diligence tool for buyers of renewable energy projects that uses AI to organize, rename, and summarize files, detect risks, and streamline task management. I established a plan and preliminary designs for implementing new LLM search and file search.
getting a more complete picture
Understanding User’s Needs
I began by learning as much as I could about user needs through contextual inquiry and user interviews, and dove into learning more about search types like RAG, Fuzzy Search and ElasticSearch.
-
I learned everything I could about designing search, and how different types of search work: Retrieval Augmented Generation (RAG), ElasticSearch work and Fuzzy Search.
I surveyed how companies like Google Docs, Instagram and Facebook design their search experience. I paid special attention to sites that incorporated LLM query to get ideas.
-
Using Hotjar, I was able to spy on how users were navigating and what they were searching for in Thresh. This wasn't hugely illuminating, but I think Hotjar will be once we have more user behavior to analyze.
-
Most importantly, I learned the key data points that the M&A Analyst is looking for through the due diligence process and how they organize it. This helped me decide how to best display and prioritize search results that users would need.
Considering all the options
3 User Journeys
I settled on three user journeys that focused on the 3 most important search functions:
Generating LLM answers to queries like “summarize the progress of this project for me”
Pulling key information from files, like an expiration date of a permit
Retrieving files & folders
Evaluation with 6 core principles
I established 6 core principles in order to measure and evaluate different approaches to designing Thresh’s search experience.
-
A UI with a way to discern between keyword search and LLM search, and for engineering to know if it's an LLM or keyword query.
-
LLM answers can be inaccurate.
Thresh is due diligence software. We need to do our own due diligence in delivering the most accurate answers possible to queries. These are huge deals with a lot of risks, and small mistakes could kill a deal or lead to lawsuits. If we aren't reliable, we are unsuccessful.
-
Retrieval Augmented Generation (ex. asking ChatGPT a question) is expensive. Each question that the user needs an LLM to answer costs Thresh money. My goal is to have the user get the information they need with the least amount of queries, and not encourage back and forth chat, which would ramp up cost.
-
The UI of the search experience fits with the user's mental model of search and is intuitive to use.
-
LLM answers take time to generate, keyword results don't. If we combined LLM and keyword search in the search bar without having the user choose between search type, and have both keyword and LLM results on the search results page, they would have mismatched latency, and it would be clunky.
-
Again, Thresh is due diligence software, so our search experience needs to feel like due diligence software.
Recommendations
Information Retrieval
Toggle for ‘Ask Mode’ vs ‘Search Mode’ next to search bar
I recommended that our search bar defaults to file search, and the user has the ability to switch into ‘ask Thresh’ mode by either clicking an ‘Ask Thresh’ button or toggling on ‘ask mode’ instead of ‘search mode’.
‘Ask Thresh’ button next to search bar
Search Results
I recommended a solution similar to google drive where all search results, both keyword search for title and content, are displayed together based on relevancy.
I consulted with a software engineer and understood that combining keyword search results for content and title it isn’t too much more work than displaying them separately.
Search bar drop down with bolded matching file content
Search results that were keyword matches for both file title and file content
Results
Choosing direction
It was decided that Thresh’s LLM queries would be answered by a chatbot, at least at first. I had considered a chatbot in my reporting of options because it fit with user’s mental model of AI search, but I vetoed it because it violated my two most important core principles for an AI Search.
They are:
Cost: a chatbot encourages back and forth correspondence which is expensive
User trust: chat bots don’t encourage user trust, they don’t feel like due diligence software
However, immediately implementing something quick and easy like a chatbot will give us early user feedback to work with. This will give us insight into the direction we want to head longer term for AI search.
What I learned
Build simple, first.
On this project, I was excited that I disagreed with a design decision made, because being a new designer, it showed me that I had enough experience to have a rationale. It sounds simple, but it was a moment for me! From dialogue on the differing in opinion, I came to realize that in fact, implementing the simplest thing early on would give important insights for the direction of a feature instead of spending crucial time on building something more elaborate to begin with. Early on, keep it simple.
Recommendation, then reasoning
Giving a preview of your recommendation, then go into your thought process. It helps give the listener the context for where they are going—basically giving context for context.
I shifted to this intuitively and then found it written about in the book How to Castrate a Bull by Dave Hitz.