I’m attending your meetup today in Lisbon (funny enough, it’s a 2 minute walk from our office) and would love to chat about the topic of AI (particularly gen AI) with you. I think we agree more than we disagree and talking human to human is the best way to check this.
I have a theory about AI and curious to get your thoughts, not really on product management specifically but just AI broadly - I totally agree about the diminishing returns piece, but I think it has very little to do with architecture, compute, the models, etc. I think it's mostly the training data.
LLMs are not trained on expert data/knowledge, they're trained on whatever's posted online. Deep expert knowledge in anything non-mainstream is often proprietary and very closely guarded, or maybe it's simply that true domain experts are too busy doing actual work that they don't tend to post about it online. Not to mention LLMs also rely on weights where commonality and popularity of information is also important. So inherently it all tends towards the mean, or like you say "vanilla".
Maybe Gen AI will always be bound by the Pareto principle, where sure they're good enough to do a convincing job for many things because - let's face it - 80% is just fine for plenty of tasks. But anything requiring the hard 20% will be no easier - and require no less human ingenuity or expert knowledge - than before.
But I guess the question is, if everyone relies on AI to build up the 80% without actually learning anything in the process or knowing what it did, who will still bother to do (or be capable of doing) the 20%?
Basically, thinking of the old adage "nothing easy is worth doing" - but now AI makes everything "easy". So what's worth doing?
Nice perspective. I agree that GenAI will not replace any product managers. You mention LLMs don't output anything you wouldn't check first - there are use cases where full autonomy is already the norms eg customer support chatbots, JP Morgan reports, natural language data analysis, etc
I am more excited about the untapped potential to improve the products we build by creatively embedding GenAI. From decades of working on products with traditional data science, ML, I expect the most significant win is creative problem-solving by combining ML & GenAI.
But as with all new tech (mobile, blockchain, AR) it has to solve real problems and not just serve the hype.
Unlike AR or BlockChain, If GenAI were taken away from me today, I would miss it - I use perplexity every day and really like Gmails Smart Reply feature.
Just a small note - could you fix the hyperlink here "compelling arguments that LLMs are already peaking,"? I'm trying to get up to speed on the topic and I was excited about this link, haha.
Bee-you-tee-ful. Yes, AI can help you be better but it cannot replace you. Your German language example is why.
(But if all you're doing as a product manager is typing Jira tickets, well, AI really *is* going to take your job.)
PS. I don't think an AI would applaud you with "Bee-you-tee-ful."
Sure, I'm all for PMs not having to do so much grunt work and, if that's all they're doing, that's a problem.
Hi Jason, Synthetic Users founder here.
I’m attending your meetup today in Lisbon (funny enough, it’s a 2 minute walk from our office) and would love to chat about the topic of AI (particularly gen AI) with you. I think we agree more than we disagree and talking human to human is the best way to check this.
Love this piece, so thoughtful and detailed.
I have a theory about AI and curious to get your thoughts, not really on product management specifically but just AI broadly - I totally agree about the diminishing returns piece, but I think it has very little to do with architecture, compute, the models, etc. I think it's mostly the training data.
LLMs are not trained on expert data/knowledge, they're trained on whatever's posted online. Deep expert knowledge in anything non-mainstream is often proprietary and very closely guarded, or maybe it's simply that true domain experts are too busy doing actual work that they don't tend to post about it online. Not to mention LLMs also rely on weights where commonality and popularity of information is also important. So inherently it all tends towards the mean, or like you say "vanilla".
Maybe Gen AI will always be bound by the Pareto principle, where sure they're good enough to do a convincing job for many things because - let's face it - 80% is just fine for plenty of tasks. But anything requiring the hard 20% will be no easier - and require no less human ingenuity or expert knowledge - than before.
But I guess the question is, if everyone relies on AI to build up the 80% without actually learning anything in the process or knowing what it did, who will still bother to do (or be capable of doing) the 20%?
Basically, thinking of the old adage "nothing easy is worth doing" - but now AI makes everything "easy". So what's worth doing?
Nice perspective. I agree that GenAI will not replace any product managers. You mention LLMs don't output anything you wouldn't check first - there are use cases where full autonomy is already the norms eg customer support chatbots, JP Morgan reports, natural language data analysis, etc
I am more excited about the untapped potential to improve the products we build by creatively embedding GenAI. From decades of working on products with traditional data science, ML, I expect the most significant win is creative problem-solving by combining ML & GenAI.
But as with all new tech (mobile, blockchain, AR) it has to solve real problems and not just serve the hype.
Unlike AR or BlockChain, If GenAI were taken away from me today, I would miss it - I use perplexity every day and really like Gmails Smart Reply feature.
Well written, thanks for sharing!
Just a small note - could you fix the hyperlink here "compelling arguments that LLMs are already peaking,"? I'm trying to get up to speed on the topic and I was excited about this link, haha.
Updated the links!