Insight of the Week: 5 Tips to Get GPT-Powered Prototypes into Production
The abundance of powerful generative AI APIs along with the ease with which these models can be prompted to produce useful outputs has democratized access to artificial intelligence. Solutions that in the past would have taken months or years to deploy can now be prototyped and tested in hours, if not minutes. These are exciting times but the gap between compelling prototype and production deployment is still larger than you might think.
Here's a quick rundown of the things you should be planning for to make sure you have a pathway from prototype to production:
1) Quotas: OpenAI, Azure and other AI platforms strictly limit the number of requests that you can make to their APIs. Whilst prototyping and testing the taskGPT solution I introduced a couple of weeks back my team quickly hit usage limits on Open AI, and it took a little while before we could get our quotas increased on Azure. So plan ahead!
2) Cost: it is faster and cheaper to build generative AI powered conversational self service, but the API usage costs can mount up quickly. The example dialog I shared when introducing task GPT cost $1.17 in API usage alone. That's about $0.17 for each response to the user. We can certainly optimize usage, and we will probably switch to cheaper, fine tuned models in production, but just like human intelligence, artificial intelligence does not come cheap.
3) Integration: existing chat widgets and voice infrastructure are not well suited to the needs of generative AI powered self-service. We had to roll our own chat widget, and jump through some hoops to make the telephony layer dance to the tune of generative AI.
4) Quality assurance: with generative AI, small changes to the inputs can have significant impact on the response of the models. The usual scripted test plans will only get you so far. We're leveraging automated testing and large scale usability testing as a way to overcome this challenge.
5) Acceptance: leadership and regulatory bodies are used to a level of certainty with regards to the business rules applied and behavior expected from IT system deployment. But Generative AI solutions have the benefits and limitations of human creativity and fallibility. The acceptance bar needs to align with what could be expected from well-trained, managed, and monitored human teams. Your business and /or industry regulators may or may not be ready to embrace this.
There's five considerations to get you started on how to bridge the gap from prototype to production, and ultimately a return on investment.
Kerry Robinson is an Oxford physicist with a Master's in Artificial Intelligence. Kerry is a technologist, scientist, and lover of data with over 20 years of experience in conversational AI. He combines business, customer experience, and technical expertise to deliver IVR, voice, and chatbot strategy and keep Waterfield Tech buzzing.