Discover more from The Growth Catalyst Newsletter by pmcurve
ChatGPT for Product Managers (PMs)
PMs need to know how to build products using large language models like GPT. We cover where PMs need to focus build expertise and how to integrate it in their own products.
👋 Hey, I am Deepak and welcome to another edition of my newsletter. I deep dive into topics around building products and driving growth.
I started writing about ChatGPT and LLMs for Product Managers recently. In the first post on this topic, we discussed Understanding Large Language Models - The Force Behind chatGPT. In this post, we are going to discuss the what PMs need to do when it comes to building products with LLMs.
For the new ones here, do check out the popular posts that I have written recently if you haven’t
9,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇
Let’s dive in the topic now!
If you are a product manager, you might be falling in one of the three categories right now:
You are building a product like Jasper.ai, which used large language models (LLMs) - the tech behind ChatGPT. You are aware of the advantages and disadvantages of this new tech on the block.
You are working in a product team where leadership wants to use GPT, and has given the mandate for people to come up with ideas. Some have even set up a small task force to figure out where all they can benefit from it.
You are working in a product team which doesn’t believe GPT has immediate benefits for them. They have bigger things to worry about. You, on the other hand, still want to learn and possibly surprise the team by showing a good use-case.
If you are in the first category, there is a very good chance the post doesn’t have much to offer you. You may continue reading this post for fun. If you are in second/third category, this post will help.
For the sake of simplicity, we would focus on ChatGPT. Usually, what’s true for ChatGPT for Product Managers would be true for others like Bard as well. Let’s get started.
There is just too much noise around GPT. The goal, then, should be not to cover as much ground as possible. It should be separating signal from the noise.
The best place to start is to list down where all GPT helps/affects PMs. There are two ways in which GPT affects PMs:
Improving their productivity by answering questions, writing emails, summarising, etc. We will not spent much time here since there is good literature around it on the Internet. We will do brief intro and share few good resources here.
Using ChatGPT and underlying LLM (GPT) to build products. We will spend majority of time here debating which methods are good for building products using this new technology.
Let’s cover productivity in brief first.
Productivity of a PM
Understanding how ChatGPT is helpful for Product Managers when it comes to productivity is quite useful. Productivity improvement would happen by learning how to write good prompts so that you can get relevant information from ChatGPT. It is also known as prompt engineering.
Here is one example of what OpenAI tells us on how to write better prompts.
Prompt engineering is useful to learn for everyone, not just PMs. But you don’t need to spend a lot of time on it. Just like Google search, most smart people would come to write good prompts themselves through trial and error.
PM productivity isn’t what we are focussed on in this post, and maybe we will dedicate a future post to it. For now, if you want a good place to start as a PM, here are some good free resources:
We are onto the second point. To build products using GPT, you need to build a good thesis/understanding of what’s possible and what’s not possible with this new technology.
There are many ways to go about building this thesis:
Reading papers behind the LLMs to understand them deeply
Blog and social media posts to understand its potential and risks
Going through the trusted sources like what experts have to say
Looking at plugins/current applications
Let’s get into details of all 4, and see where they help.
Some PMs want to go to the core and start with the research paper by Google in 2017 that started it all — Attention is All You Need. The paper introduced transformers and how they created a way better NLP model as compared to anything that had come before.
Unless you are building a large language model yourself, I would not advice you to go through research paper for a couple of reasons:
If you don’t have a data science background or built AI products in the past, it would take a lot of time and energy to understand these highly technical papers. To understand the whole thing, you would also need to learn about the historical evolution of NLP and how we reached here.
Even after understanding the papers, you would find it hard to map all applications. The tech behind LLMs got built first, and now we are off to figuring out applications. We keep discovering new advantages and limitations every passing day.
So starting to read research papers isn’t the right way to go about it unless you are building a LLM like GPT yourself. What about reading about what blogs and social media posts have to say what it can and can’t do?
Blogs and Social Media Posts
There are a lot of people who don’t have background in AI/ML and still writing about LLMs heavily on social media and blogs. These people are just repurposing the content for likes and followers. And because they are writing it to garner likes on social media platforms, they end up writing sensational things, most of which may not be true in its entirety.
That makes social media and blogs a very unreliable source of information — the noise is way more than the signal. Be sure to fact check everything that you read there, unless it’s coming from an PM/engineer/researcher in the AI/ML field.
What about going after what top experts have to say?
Experts are focussed on the high level discussions like future risks and rewards of Generative AI. The downsides of generative AI (LLMs) going wrong could be very high, so it’s a legitimate concern.
Further, these experts are divided on what the future of LLMs is. There is great debate happening whether LLMs would lead to human-level intelligence, aka general AI. Some people including OpenAI CEO Sam Altman believe it’s possible. Others don’t believe so. Here is a thread worth reading from the chief data scientist of Meta. This is what he says,
“On the highway towards Human-Level AI, Large Language Model is an off-ramp. To clarify: LLMs that auto-regressively & reactively predict the next word are an off-ramp. They can neither plan nor reason.”
In my opinion, builders should be focussed on what’s possible to start with. You can read about future risks and rewards, but as of now, there is little evidence to end this debate. So start focussing on what’s already happening, which brings us to current plugins/applications.
Starting with current plugins/applications allows you to quickly see how different products are using LLMs. A bunch of smart founders and PMs are already ahead of the curve, and they have built products over GPT. You need to learn about these use-cases and limitations to get upto speed. This is what we should be most excited about.
Let’s start with plugins of ChatGPT that product managers should understand.
Tech knowledge is essential to perform well in a PM job. If you are a PM who doesn’t come from a software background, you can checkout my book ‘Tech Simplified for PMs and Entrepreneurs’ which has been immensely useful (readers’ word, not mine) in getting them to understand tech well :) 250+ people have rated it 4.5+ on Amazon.
How Plugins Work
OpenAI plugins connect ChatGPT to third-party applications like Instacart, Expedia, etc. Understanding how the plugins work will help you decide whether you can use ChatGPT for your product or not.
If you have been following the news, you may have heard about ChatGPT plugins from popular companies like Instacart, Expedia, etc. Here is a list of ChatGPT plugins in Beta.
While using ChatGPT, limited users would be able to see and interact with these plugins. For example, when you ask the integral of x^2cos (2x), the ChatGPT used Wolform plugin, and answers it pretty accurately. You should note that the user is typing the query in ChatGPT and not on Wolform website.
So how does the plugin shown above work? We have to understand the flow to see where all can we apply in our product. Here is an explanation step by step:
When asked to perform computations, ChatGPT interprets the user’s question and formulates it as a query.
The query is can now hand it off to Wolfram Alpha instead of attempting to generate the answer by itself.
Wolfrom computes and passes the response to ChatGPT.
ChatGPT structures its response based on the response.
ChatGPT is great at understanding user natural language, and Wolfrom is great a performing computations. Both sides use their strengths in this arrangement to create an amazing user experience.
Current Plugins’ Use-cases
To build a holistic understanding, let’s look at few of the plugins one-by-one:
Instacart: Instacart is a grocery shopping app in the US. The Instacart ChatGPT plugin helps people figure out what they would need to make a particular meal, create an instant shopping list based on ingredient needed, and get ingredients delivered to their door so they can start cooking. Like Wolform, user interacts with this plugin on ChatGPT website, and is redirected to Instacart for modifying the cart and payment.
Expedia: Expedia is a travel platform. The users can get recommendations on places to go, where to stay, how to get around, and what to see and do. The plugin automatically saves hotels discussed in the conversation, and adds on flights, cars or activities. The payment for the booking still happens on the main Expedia website.
Opentable: Opentable does restaurant reservations. The plugin will help with open-ended questions such as — “I have a date I want to impress. What’s a restaurant with oysters and great cocktails in the Upper West Side NY?”. These sort of questions are hard to search on Opentable website.
Zapier: You can automate tasks that require data to flow from one place to another using integrations in Zapier. And that is possible now from within ChatGPT's interface. It saves you time and the hassle. Here is a sample use case.
Fiscalnotes: Fiscalnotes shares information about global policy and governance to various companies. ChatGPT can quickly find the right information users are looking for, thus saving time.
We can look at more plugins, but we have gotten a broad idea of where ChatGPT is helping users. The products seems to use the plugins for the following use-cases:
Processing natural language queries — One thing that GPT does very well is understand the natural language query. When a user provides a query for real-time information, ChatGPT can understand what the user is looking for and retrieve that information pretty quickly. This is happening across apps in beta.
Maintaining context for conversations just as humans do — GPT can retain the information from past when the user asks the next question. This wasn’t possible in regular search on websites and apps, and that is where ChatGPT shines. Look at this example below - ‘what’s the salary’ is a pretty generic question. ChatGPT is able to retain context from the previous question like humans.
Searching and filtering to show the most relevant results from structured database — The plugin isn’t accessing the data itself, it is hitting search endpoint with keywords and filters. The particular app will process the keywords and filters to send a response with relevant results.
Klarna, OpenTable, Kayak, Expedia — all are using the search and filter to produce relevant results.
Automating tasks for users — This is where things get interesting. For Instacart, the input provided by the user is a meal name. ChatGPT is fetching the recipes, and ingredients. Further, it is taking into account the portions of ingredients, and creating an automated shopping list.
For a user, creating a shopping list is a pretty mundane task. It’s much easier to have a starting shopping list of 15 items, and remove/modify it. The experience of grocery shopping actually improves significantly. The same can be said about the Zapier integration.
Finding relevant information from unstructured data — This is yet another example where experience becomes significantly better. When applied to Fiscalnotes, which would have thousands of policy documents lying around, GPT can bring the relevant data pretty quickly. Searchability in unstructured data is a big problem in policy research, market research, law, etc. ChatGPT would play a massive role in reducing the effort required to search and summarise in these areas.
Other applications of this capability for product companies is in searching knowledge base and answering support queries from the customer. Stripe has already started using it to improve the experience of reading support documents. They have heavy documentation for developer APIs. When ChatGPT sits on the top of documentation, it can help developers finding the right piece of code/information pretty quickly.
Translation: Speak is a language learning app, and one of the current plugins. Speak plugin provides a tailored language learning experience whenever a user is looking for a translation or explanation across languages.
Solving questions and explaining them — Wolfram can be used for computation and finding answers to math/science questions. What’s better, ChatGPT can write simpler explanations because of its NLP capabilities. So if a student isn’t able to understand the answer, they can request ChatGPT to explain it in simpler language.
Your Product Plugins
No matter which product you are working on, one or more of such capabilities can be put to use. Providing support post-sales by simplifying documentation, or building a sales bot are few obvious use-cases for all transactional and productivity apps out there.
Search and discovery experiences are also quite horizontal in nature across apps that ChatGPT seems to improve.
One thing that can stop you from building a ChatGPT plugin at this point is whether your customers are already on ChatGPT or not. If they are already spending time in ChatGPT, building a plugin can give you an advantage over a competitor who is not open to the idea. What’s better, it can also become a source of new user acquisition given that more than 100 million users are active on ChatGPT. You have to evaluate for yourself on whether a plugin makes sense or not. It will come down to how much the experience improves, and whether ChatGPT has your current/future customers on its platform.
This post is already quite long. I plan to write about ‘understanding applications built on the top of LLMs’ in the next post.
Meanwhile, let me know how you found this post by liking or commenting.
Have a good day!