ChatGPT as a Companion to PM Interviews and Real-Work
ChatGPT as a Companion to PM Interviews and Real-Work
On popular demand, we are having a free event on Getting to Product-Market Fit next Saturday. You can check the details here
Off to the topic at hand,
I was going to start writing about Product Sense after covering market sizing and guesstimates in the last few posts (here, here and here). But analytical seems like a topic that’s a natural extension of market sizing and guesstimates.
Analytical skills aren’t just important for PMs. They are also quite important in adjacent roles — business or product analyst, data scientists, consulting, program management, etc. Every one of these people have to work with data to figure out insights.
But the burden of using data to create insights, and translating that insight into the impact lies with product teams. To do that, product team needs to have a deeper understanding of:
Defining the metrics
Product dashboards, and defining various levels of metrics in the dashboards
NSMs (north star metrics)
Success metrics for a product launch, and guardrail metrics
Merging user research and data merge to create powerful insights
Root cause analysis of a problem using data, like a metric going down (or up)
Creating a plan to move a metric
The above skills are in the increasing order of complexity. For example, item #4 (moving a metric) is something that is the hardest to do among all the others.
While PMs require all the skills listed above on the job, when it comes to the interviews, it’s hard to display #2 (merging research and data to create insights). So those questions don’t appear in the interviews. Further, the type of skill tested also changes with levels. At an APM/PM level, you would get more questions around defining metrics (success metrics, NSM, dashboards) and RCA, whereas at an SPM/GPM level, you start getting problems of creating a product strategy to move a metric.
We are going to cover these items in the next set of essays. But one of the more interesting things that has happened in the past 1.5 years is large language models (LLMs). And one of the notions I had around analytics is —
Analytics is also lot more structured and objective than product sense or strategy interviews, the response of LLMs might be pretty effective here just as it happens in coding.
In other words, can we start using these LLMs as a product companion when it comes to analytics? Since ChatGPT is available at your service 24x7, it’s an amazing companion to have. If the answer to the first question is yes, to what extent should we be using them?
Let’s kickstart this post with the first item: Defining the product metrics!
Defining the Product Metrics with LLMs
Defining the product metrics is the core responsibility for a PM at every level. And it appears quite frequently in the PM Interviews. Here are some sample questions —
If you were the PM for Lyft, what dashboard would you build to track health of the app? (Google)
Imagine you're a PM for Netflix. Your team launched Netflix Podcasts 6 months ago. How do you measure success? (Meta)
What should Airbnb's key north star metrics be? (Airbnb, Google)
Let’s start with the simplest problem for the LLMs — building a product dashboard.
Here is the prompt —
Imagine that you get this question in the interview as a PM candidate. How will you answer the following question?
”You are the PM of the food marketplace product - Food Delivery App like Zomato or DoorDash. You are tasked to create a product dashboard that can be used by product team to track and monitor the important metrics and trends of the product. How would you do that?”
Here is the response from ChatGPT-4. You should surf through the response below and evaluate positives and negatives for yourself before moving on:
As the Product Manager for Food Delivery, my approach to creating a product dashboard involves several key steps to ensure it effectively tracks and monitors the most important metrics and trends. Here’s how I would go about it:
1. Understanding the Business and Product Goals
First and foremost, it’s crucial to align the dashboard with the overarching business and product goals. These typically include:
User Acquisition and Growth: Attracting new users to the platform.
User Retention: Ensuring users continue to engage with the app.
Revenue Growth: Increasing overall revenue through more orders and higher order values.
Operational Efficiency: Streamlining delivery operations to ensure timely and cost-effective deliveries.
Customer Satisfaction: Providing excellent service to keep customers happy and loyal.
2. Identifying Key Stakeholders and Their Needs
I would work closely with various stakeholders to understand their specific needs and how the dashboard can support their roles:
Product Managers: Need insights into user engagement, feature usage, and retention rates.
Marketing Team: Focused on user acquisition metrics and the effectiveness of marketing campaigns.
Operations Team: Interested in delivery efficiency, partner performance, and logistics.
Finance Team: Requires detailed revenue, cost, and profitability metrics.
Customer Support Team: Needs data on support tickets, resolution times, and customer satisfaction.
3. Defining Key Metrics
Based on the goals and stakeholder needs, I would define key metrics for the dashboard:
User Metrics
Daily Active Users (DAU) / Monthly Active Users (MAU): To track engagement.
User Retention Rate: To understand how well we retain users over time.
New User Sign-ups: To monitor user acquisition.
Order Metrics
Total Orders: To gauge platform usage.
Average Order Value (AOV): To measure revenue per order.
Order Frequency: To track how often users place orders.
Delivery Metrics
Average Delivery Time: To ensure timely deliveries.
On-time Delivery Rate: To measure delivery reliability.
Delivery Partner Efficiency: To track orders per delivery partner and distance traveled.
Operational Metrics
Restaurant Performance: Including order acceptance rates, preparation times, and user ratings.
Cancellation Rate: To understand why orders are canceled.
Customer Support Metrics: Number of support tickets, resolution times, and satisfaction ratings.
Financial Metrics
Revenue Metrics: Gross merchandise value (GMV), net revenue, and profit margins.
Cost Metrics: Delivery costs, customer acquisition costs, and operational expenses.
4. Designing the Dashboard
I would design the dashboard with the following sections:
Overview Section: Key Performance Indicators (KPIs) for quick insights.
User Engagement Section: Graphs and charts for DAU/MAU, retention rates, and new user sign-ups.
Order Analytics Section: Visualizations for total orders, order value, and order frequency.
Delivery Performance Section: Metrics for average delivery time and on-time delivery rate.
Operational Efficiency Section: Restaurant performance and cancellation analysis.
Financial Performance Section: Revenue and cost metrics.
5. Tools and Technologies
To build the dashboard, I would leverage:
Data Integration Tools: For gathering data from various sources.
Visualization Tools: Such as Tableau, Power BI, or Looker for creating interactive dashboards.
Real-Time Monitoring: Ensuring the dashboard reflects the most current information.
User Access Controls: To provide appropriate access to different team members.
6. Implementation Plan
Requirement Gathering: Collaborate with stakeholders to define metrics and dashboard requirements.
Data Collection: Identify data sources and establish data pipelines.
Dashboard Design and Development: Build and test the dashboard for accuracy and reliability.
Launch and Training: Deploy the dashboard and train the product team on its usage.
Continuous Improvement: Regularly update the dashboard based on feedback and changing business needs.
Summary
By aligning the dashboard with business goals, understanding stakeholder needs, defining key metrics, and using the right tools and technologies, I can create a comprehensive and actionable product dashboard. This approach ensures the dashboard provides valuable insights that drive informed decision-making and contribute to the success of Food Delivery App.
I am happy with the answer that ChatGPT provides. The positives are:
It is well-structured, and covers all the important stages of building a product dashboard. For example, it talks about items in planning stage as well as implementation stage.
It covers all 3 sides of the marketplaces — users, delivery partners, restaurants. So the dashboard output covers all the important aspects.
The negatives are:
It misses the fact highlighted in the question that the dashboard will be used by the product team, and not for all teams.
As a result, it doesn’t grasp the finer details of a product experience like user activation rate, search versus recommendation behaviour, feature usage, product performance, etc.
I went ahead with asking additional follow-up questions to refine the response, and based on what I saw, here is what I would recommend when it comes to using ChatGPT.
Using ChatGPT as Companion
There are two key issues with using ChatGPT as a companion.
The biggest issue lies in the fact that if you don’t know what a right answer looks like, ChatGPT may push you down a wrong path. For example, if someone misses that the Zomato interview question had Product Teams as an end user for the dashboard, they would likely go ahead with the first answer provided by ChatGPT and miss on the metrics relevant for the answer.
The second issue is that sometimes these LLMs will hallucinate, especially when you ask them a question belonging to a niche product. Since we don’t know when and where a particular LLM will hallucinate, the ability to separate right from wrong answer becomes even more important in this case.
And because of these two issues, I won’t recommend using ChatGPT on an unknown problem, especially the unknown problems on the job. Making a mistake in an interview might be acceptable, but doing so on the job can have disastrous effects.
The best use of ChatGPT as a companion is to practice and finding your blindspots in the interview prep. It can be used as a tool once you have gone through frameworks and some good practice problems elsewhere. That way, you aren’t going in with many unknowns.
The last caution is that it’s hard to get to a quite nuanced answer from ChatGPT. These nuanced answers are pretty important for the PM job, and product interviews at senior levels.
But should we just rely on a sample answer to conclude all this? The conclusions feel like a stretch given that we have just taken a problems and seen some answers.
That’s a good question. To fully appreciate and get convinced about the recommendations, you need to understand a thought experiment known as the Chinese Room Argument.
The Chinese Room Argument
Alan Turing, the genius often considered the father of modern computer science, proposed Turing test in 1950. The test proposes that if a computer could answer questions posed by a remote human interrogator in such a way that the interrogator could not distinguish the computer’s answers from those of a human subject, then the computer could be said to be intelligent and to think.
The Chinese Room Argument was a thought experiment proposed by John Searle in 1980s, and it came after the validity of Turing Test.
Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program).
Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input).
And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output).
The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.
The point of the argument is this: if the man in the room does not understand Chinese on the basis of implementing the appropriate program for understanding Chinese then neither does any other digital computer solely on that basis because no computer, qua computer, has anything the man does not have.
The Chinese Room Argument undermines the validity of the Turing test.
Now there is debate around this thought experiment. There continues to be significant disagreement about what makes a computer similar to humans. The debate around LLMs is similar.
Sam Altman (CEO of OpenAI) believe that LLMs would lead to artificial general intelligence (AGI) because they (ChatGPT) seem like the way humans would answer these questions. Over time, we will get a breakthrough on this path and reach AGI.
For others (like Meta Chief Scientist, Yann Lecun), LLMs are just a more complex set of rules running on much faster hardware. He argues that LLMs are not a sufficient path to AGI because they lack essential capabilities for intelligent beings, such as understanding and reasoning about the physical world.
The debate has been going on for some time.
Just a few days back, Satya Nadella spoke about how we should ‘Stop treating AI Like Humans, and start treating it like a Tool’. Here is a 3-minutes clip where he talks about how AI is more like an assistant than a human-like intelligence.
Satya would be one of the few people to know the limits of LLMs, given that he has invested in Open AI and manages on the largest tech companies across the globe. And he seems to be thinking that a Chinese Room which can behave like a human isn’t akin to human intelligence.
The Verdict
You need to keep Satya’s comment in mind as a PM while using ChatGPT. They definitely improve the productivity of PMs, but as for doing a PM job even in simpler areas like defining metrics, they can do a poor job at times.
If we merge the Chinese Room Argument for LLMs, and the analysis of sample responses we received, we can now see why it’s hard to rely LLMs for generating insights like a human can. They are definitely better than humans in many regards, but so are calculators.
So use them as a productivity tool on the job, but not as the sole source of research and insights.
As for interview companion, it can do a fantastic job if you want some good practice after preparation. With the voice and video capabilities added to ChatGPT 4-o, you can even do mock interviews and analyse your response later.
That would be all for this week! Do join the Saturday event on Product-Market Fit if you can.
Thanks,
Deepak