About the development of genAI assistants AskARev and Searchbuddy

General

Context

The blog post shall tell the story of the development of AskARev (“Ask a (customer) Review”) and Searchbuddy. Both are generative AI based products. They were developed by Team PIT from Otto Group data.works; their colleague Andreas Lattner provided support with AskARev.

We started with AskARev when generative AI was still in early phase and few models were available. Nevertheless, a working product was developed in only two weeks. We took some time after AskARev, then developed Searchbuddy leveraging from all the knowledge we had gathered from the development from AskARev. 

Note: The space of LLMs currently develops quite fast so it is important to test/benchmark newly released models for the specific use case.

Detailed information about both services

AskARev

• started in July 2023
• summarizes product details and reviews to answer customer question about product
• data requirements: high (reviews + product details)
• model: text-bison@001 (Google), at that time the only available model in the European Union, required for data privacy
• larger model possible as is no “live” service
• harder to benchmark due to unstructured text output
• in rare cases, hallucination has the model make up product details

Searchbuddy

• started in October 2023
• suggests products based on non-specific customer input (e.g. gift ideas or fashion suggestions for events), returns up to 5 products in a list
• data requirements: low (only product assortment needed)
• model: gpt-3.5-turbo (Microsoft/OpenAI)
• small/fast model (<1s) required due to “live” service
• easier to test due to structured list output
• hallucination has the model return products which don’t match the assortment
• implemented a benchmark tool for baseline tests of new models
• currently only available in German

Both services

• hosted on Google Cloud Run
• Google DLP used to remove private data
• Google cloud armor used for protection of the service
• Use multiple security layers to guard against prompt injection, LLM misuse etc.

Further

As can be seen from the comparison LLM based products can be developed in no time thanks to easily usable APIs! LLM’s summarization and inspirational features offer real customer benefit. 

Also, a status with relatively good quality answers can be achieved quickly as the LLM generalizes well to new tasks. But the fine-grained improvements that make up the last mile take the longest because it is not simple logic you’re playing with but bare language! An empty string (“ “) can change a lot. 

Similarly setting up a robust benchmark set is helpful (to compare different LLMs and prompts), but also really hard as the text output of the service is almost unlimited compared to the old school products where you know the output will e.g. be a float between 0 and 1.