Introduction

Thomas Joy 0:00

Hello, my name's Thomas, and you're listening to the Pensions and AI Podcast, the show where we explore the way that generative AI is being used across the pensions industry. In every episode, we talk to people in pensions who are working at the frontier of generative AI. You'll hear from super users, builders, advisors, and big thinkers, the people who are shaping the future of how our industry could look and how it can deliver better outcomes for members. This is not a show for people to come and sell you things. Far from it. We're here to share, to have open and honest conversations, talking about what's really going on, what actually works, and the lessons we're all learning along the way. If we're doing a good job, then you'll leave each episode feeling more informed and confident with generative AI, and you'll be better equipped to use the technology both for yourself and also to shape how it's used across your organization. In this first episode, we're going to be talking about how generative AI is changing the world of pensions communication. And I can think of no better guests for that topic than two friends who I had the pleasure of working with for many years. So I'm very excited to welcome to the podcast Cath Collins and Rhys Williams from the Pensions Communication Consultancy Quietroom. Cath and Rhys, welcome to the pod.

Cath Collins 1:09

Pleasure

Who are Quietroom?

Cath Collins 1:10

to be here.

Thomas Joy 1:10

Thanks for having us. So for anyone who hasn't been lucky enough to work with you guys yet, or maybe hasn't come across Quietroom, tell everybody listening who you are and what Quietroom do.

Rhys Williams 1:19

So Quietroom is an insight-led communication consultancy. What does that mean? It means we work with trustees, pension schemes, pension providers, the whole kind of ecosystem of advisors and consultants that surround them on the way they communicate. Most of that is the way they communicate with members. So it will be things like helping them set direction for how that, you know, what they want their communication to achieve, how they want it to work. It will be things like delivering lots of communication for them. So creating content, building journeys, shaping experiences for their for their members. And there's also an element of capability building. So we work with those client organizations to create frameworks, guidance, standards, training, coaching, all of the above. But it when you put all that together, what does it mean? Well, I hope it means that members understand their pensions enough to do whatever it is that we're asking of them, that they feel good about their future, that they feel a greater sense of their own ability to engage with the topic, that they feel more confident, they feel less anxious, they're more likely to trust us and the people who are looking after them. And then ultimately, if there's an action that they need to take, that they're able to do that with as little time and effort as possible. And of course, we hope that that all adds up to better outcomes for them.

Cath Collins 2:43

I don't know if I can top that really.

Rhys Williams 2:45

Just don't try.

Thomas Joy 2:47

So that's Quiet Room. Now tell me a bit about the both of you.

Cath Collins 2:51

I will start,

Meet Cath and Rhys

Cath Collins 2:52

give Rhys a rest. So I'm Cath. I joined Quietroom in 2022, shortly after Thomas, maybe a year after Thomas. I'm a lead consultant at Quiet Room. What that really means is that I have a team of brilliant content people who deliver work for our clients. We love it when it's a particularly gnarly topic. Gnarly is a word we band around a lot at Quiet Room. When someone comes to us with a problem that's just too difficult for them to work out how to get through how they're going to communicate that to their members, because the nuance that exists in pensions is ridiculous, often not useful to members. So we spend a lot of time working out the best strategy to get a message across to members, really. And then the best way to get that to them. And when Rhys talked about the insight-led piece as well, where possible, kind of speaking to the people, putting communications in front of the people that we design those communications for. Because without exception, every single bit of communication that we do, every bit of user testing that we do surprises us in some way. So yeah, kind of bringing all that together to write really, really good communications, hopefully.

Rhys Williams 3:58

And just to sing Kat's praises for a second, so that um when you've created a modeller for your members who work in a factory, she'll go to the factory and watch them use it and then come back and tell you how it could work better. So about me, I joined Quiet Room in 2006 when I think Thomas, you were 14 years old. The so I was Quiet Room's first ever employee after after the two founders. So we were a very strange three-person, three-person team in those in those days. And so I've seen the whole company kind of grow up from there. When you start as early as I did in a business, you become an all-rounder because everybody who works in a startup business has to be an all has to be an all-rounder. So I do a little bit of everything. I'm on the company's board. I spend, I like to spend as much of my time as possible talking to clients, so trying to understand what are the things they want to do, what's the stuff that gets in the way, and therefore what sort of help should we be giving them? I still do loads of hands-on work as well, so I write things, I train people, and so on.

How generative AI is changing the way Quietroom works

Thomas Joy 5:07

So 2006 was a momentous year. I would argue 2022, though, Cath, is an even bigger year. And let me give you a couple of reasons why. A bit a big year for lots of reasons. 2022 was the year that the Lionesses won the Euros. It was the year that Charles became King Charles. The year that Elon bought Twitter. Oh God. It was also the year that Liz Truss became Prime Minister and gave us Gilted Geddon. And it and it was the year that she stopped being Prime Minister too, but despite all of those things, you might argue that the most momentous thing that happened in 2022 was the launch of ChatGPT. 30th of November 2022, OpenAI brought us ChatGPT with the Wii model, bless it, GPT 3.5. The numbers have gone up a little bit since then. I'm really curious. It's been three years since we've had tools like ChatGPT in our hands. How has generative AI changed the way that you're working at Quietroom?

Rhys Williams 6:05

I think it's changed everything and nothing. Explain what I mean. That I see it. I was I came I was at university when email became a thing. I I was at work when mobile phones became a thing. I presided over the death of the facts. I didn't kill it personally, but I saw it die. So I I've seen quite a few changes to the world, to the world of work. And I think this is up there with those, maybe even bigger than those in terms of the impact it's had. I think there's not a part of my work now that it doesn't touch. So, you know, as simple as having meetings with people, those meetings are recorded, they are transcribed. We used to have a transcription service that we paid for. They used to send us brownies at Christmas. They used to turn around transcripts overnight if you paid a premium. And if not, there was a sort of standard 48-hour window. And now it happens not quite instantaneously, but not far off. I mean, you could have you could have, I had a three-hour meeting on Wednesday and 20 minutes later have the whole transcript. I can take that transcript from the app that we use and I can put it into a into a generative AI tool built on a large language model, and I can say summarize the top five themes, what are the next steps for me? It can write minutes, it can write actions and all that kind of stuff. And that's just, think about it, that's just one meeting I've just talked about. Now imagine that multiplied. So we we do, as we were talking about when I was saying, you know, Cath goes to the factory and watches people try and use the modeler. We do tons and tons of research, tons and tons of interviews with stakeholders, with members, with with everybody. Every time I did an hour of that in the past, I would have I it would have generated 20 plus pages of transcripts, which would in turn have taken me about an hour to read and analyse them. When I'd read them all, I'd have to then spend time distilling, synthesizing, and doing all of that kind of stuff. And I would do all of this before I ever wrote anything. That that can now be compressed into a considerably shorter amount of time. So what that's created for me is just a couple of things. One, it's made me better because I'm only a human and I didn't used to catch everything. And sometimes I used to read things, and because I'm a human, you get kind of entranced by novelty, you know, you sort of go, Oh, that's not never heard it expressed that way before, but you'd miss the really obvious thing that everybody said. AI doesn't do that. AI will replay the obvious to you in a way that I find really, really helpful and miss things that I'm that I'm sorry, pick up things that I might otherwise have missed. So makes me makes me better, creates more time for me. As I've discovered, is it's not made me any less busy. In fact, if anything, I'm a little bit more busy than I was before. Because the question then becomes, well, what do you do with all this time you've saved? And the answer is you do more of the really good stuff. So more of the stuff where you think you're adding most value, more of the stuff where you think that's the best use of your sort of talent, skills, experience, and and and what have you. And so it's given me the ability to sort of reshape my working life in a way that is genuinely wholly positive, like that I can see no downsides. So it's changed everything. It's also, as I said, changed nothing because I still turn up to work trying to do exactly the same thing as I did before. I just have unbelievable help with that job now that I didn't have before. But I'm still me. I st I still care about the things I care about. I'm still good at the things I'm good at. I still like doing the things I do. So it's just cleared away everything else so that I can just be more the kind of person and the kind of do the kind of work that I think is most valuable and most interesting. So yeah, very it's just it's changed everything, but also nothing.

Cath Collins 10:01

I think it's it's given us the opportunity to have richer insights into the work that we do as well. So I am a comms person, I have an English and drama degree, I'm a bit scared of data, I'm a bit scared of numbers. I've did my math GCSE, it was fine, but I don't intend to do any more maths in my life. Looking at member data though is really, really valuable, and a lot of our clients have access to loads and loads of data that tells you quite a lot, and can you can segment it in many different ways? In the old days, you would have had to have somebody who was really, really good on Excel to kind of build you these really magical pivot tables and pivot charts, but now you can use this. I must flag anonymized data before our clients start panicking. You can use that anonymized data to really look at what kind of behaviours are happening and to work out exactly where you need to focus your efforts. And I wouldn't have been able to do that without AI tools. So it's given me the opportunity to delve a lot deeper into insights that would have been more impenetrable in the past.

Thomas Joy 11:04

You've

What clients are saying about AI

Thomas Joy 11:05

touched on clients there. Do you find clients are coming to you and talking to you about how you're using AI? Are they pushing you to use it? Are they saying they don't want you to use it? What do you hear from people?

Rhys Williams 11:15

Nobody that I know of is either pushing us to use it or telling us that we can't. What we're seeing much more often now is is a genuine sort of open inquiry into how we're using it and and what we what we're using it for. There are some things that our clients don't want us to do with AI, but without exception, they're all things that our AI policy says we shouldn't be doing anyway. So it's we we're not apart from the anonymised data that that we analyze, we're not working with lots of raw raw member data, so that we're not putting sensitive or personally identifiable information into large language models. We're not training any models ourselves, we're not doing any automated decision making. Nothing that we do would ever see the light of day without a human being heavily involved. So the actually the level of risk is is really, really low. And most of our clients' reaction to that is really proportionate, as you might imagine. It's not like, you know, we're not um a high street bank making or a building society making lending decisions or an insurance company making underwriting decisions using AI with no humans. We're not doing all automated, uh computer generated content for people. That's not the kind of business we are, it's not the kind of work we do. So yeah, I think they're very, very interested. I think, but not prescriptive either way. And I think they're just genuinely interested, and I think it's important to say as well that our clients are not all at an identical level of development in terms of their own AI use. So so some clients are much more much more advanced, some are much more ambitious, some are cautious and want to wait and see. And I think that that's totally to be expected because every business is different, they work operate in a different context, different leadership, different risk appetites, and all that kind of stuff. So it's no surprise to me that that's that's the case, and it's certainly no criticism of anyone. But what it what it means is that it's hard to give a straight answer to a question like that because they're all quite different from one another. Yeah.

Thomas Joy 13:32

It sounds

Finding the line between human and machine

Thomas Joy 13:33

like you've got some red lines then on the things that you can use AI for and the things that you might not use them for. I'm interested, where do you draw the line and how did you decide on that as a team?

Rhys Williams 13:44

The the the big one, I mean, obviously all the stuff I've just talked about is when I say forbidden, it makes it sound much, much more like a sort of authoritarian dictatorship than it is in real life. But we're very, very clear with people, all the stuff I've talked about that is genuinely risky or reckless, we don't do any of that. The other the other thing, and I think this is really important, is that it I think that so we do a lot of writing, and one of the use cases for AI is is writing things. But if you try and deconstruct writing into its different components, only really five percent of it is is stringing words together in order to make it to make a sentence and putting sentences together to make a paragraph and so on. All the rest of it is thinking, planning, researching, distilling, synthesizing, and all and all structuring and all that kind of stuff. And I I find AI invaluably useful at all of that. But when it comes down to the, and now what am I gonna write, that's still a that's still a me thing. And I don't really want to lose that because that's what I like doing. I like half I've got an English degree, I like writing things, and and perhaps I'm colossally arrogant, but I still think I'm much better than the than the bots at it. And I think what the what the bots can do is is lift someone who really has no writing experience, skill, or confidence to a place where something that would have been quite ropey is now perfectly serviceable. And in lots of situations that might be good enough, but for us, in our situation, that's not going to be good enough because we're we're employing people here who are who are very skilled, very experienced, very talented, and none of us turn up at work to just do the average or to or to do something that's just okay. So so for me, it helps in the process, but I I would personally never use it to write something that a client was paying me to write.

Cath Collins 15:46

And I also think a lot of the context of particularly in the world of pensions, lives in in your head and in your experience with writing. So you know that you can't use a particular word, but actually you've had four conversations with different sets of lawyers and trustees and pension managers who have, you know, you've you've got got to a place, or you know actually that expressing something in a jovial tone that Chat GPT has written for you isn't going to work because it's a serious topic, or it builds upon a language model that you've used for years with that particular scheme. And I don't think you can teach that and shouldn't teach that because that's really, really important.

Thomas Joy 16:28

You just made me think of there a meeting I was in a couple of days ago where it was 15 people in a room looking at and shaping a web page full of copy with feedback going in lots of different directions, lots of different competing priorities. And I was thinking of poor co-pilot sweating in the corner, being given 15 lots of feedback and being told to make all of that work in the way that's still gonna be the best possible outcome for the end user.

Rhys Williams 16:50

Yeah, and like let's be totally real, it would absolutely mess that up.

Cath Collins 16:56

That's a really polite way of saying that, Reece.

Rhys Williams 16:58

I really had to think really hard. Because if you think about what do you add when you're in that room, you add, you're bringing all of your judgment and experience from all of the all of your previous work experiences. You can appreciate things that are not just unknown but unknowable to a large language model. It's never looked anyone in the eye, never seen someone, you know, that's sweating on their upper lip because they've got tension. You never see that someone, oh my god, she's got a bit angry, you know, those sorts of things. So all of the ability to kind of sit in a room and understand other humans and make judgments and reach consensus and kind of arbitrate, negotiate, because that that's the kind of undertaking we're talking about. There's lots and lots of different stakeholders coming together to try and do something really good. And you can't you can't expect a virtual thing to to step into that breach that wouldn't normally be filled by a human, and nor can you hold it accountable. So you have to be able to say, okay, this was your decision. Why did you make the decision you make? What information did you look at? What did you consider? What did you rule out? Why did you go for this? And at the end, at the end of the day, in this regulated world, someone has to be held accountable because you don't just have regulated organizations, you have regulated people. And I think it let's imagine potential negative consequences. And if something happens and you go, okay, well, whose idea was this? You cannot say the computer told me to do. That's not acceptable. So so I think that that's a that for me is a that's a really good example of what I was saying before when I was saying it's changed everything and nothing. Because that that sort of humanity, whether it's the you know, imagination, creativity, emotional intelligence, or whether it is the ability to sit in a room and debate, discuss, uh challenge constructively, negotiate, arbitrate, all of that kind of stuff. I think that that's the that will be the final frontier that that um hopefully for the rest of my working career in the 15 and a half years that I have before I reach my state retirement age, I hope that that will not change.

How to get good results from AI tools

Thomas Joy 19:14

I was at a conference a couple of months ago. It was a conference with a panel talking about AI, and one of the questions to the panel was: Will something like Copilot just be like Microsoft Excel? In that we all get the same tool, and so therefore, is it the case that nobody has a competitive advantage because we're all using the same technology? And I was took a step back and reflected on that, and I thought, well, uh when I talk to people across the industry, I hear people achieving very different things with the same sets of tools. So, Cath, I'm curious. You look after a team of brilliant content people. When you see really great uses of AI coming from the team, what are the things that people are doing and thinking about that helps them get more from those tools?

Cath Collins 19:54

We've been very lucky. We've had quite a lot of time at Quiet Room to kind of invest in learning about these things. Some of that was guided by you, lots of us guided by Rhys as well. But we have spent a long time kind of learning about the best use of prompts, the best use of different types of tools. We have access to a range of different tools, but I think with any any of this, context is all. So if you fire in a ropey prompt that's a sentence long, you're going to get a ropey response back. The more context you give an AI tool, the better the response is going to be. And then you can use things like project knowledge to give it the background information, you can provide it with the details that it needs, you can look at different drafts. There's so many different versions of things that you can do. But really being clear about what the intent is, what the context is that you come to this from, because any communications challenge has a massive amount of context to it. It's not just write this thing to tell these people this, you're also considering what's going on in the business, how much knowledge they have, where they work, what their access to IT is, whether they have engaged in any process before. And all of that is really, really important to kind of shape what your approach is, and then all of the user testing that you've done and all of that. So bringing in that context, making it really clear what the purpose is, and also I think the most important thing for me that I've learned here is kind of chunking tasks. If you give an AI tool too much to do at once, it's going to massively mess it up, as Rhys said. We would define that slightly differently in normal language, in our day, everyday language, but chunking something. So giving it the background to something, setting it off, checking that it understands what you've asked it to look at as well is really, really important because if they don't, or if they it needs further clarification, asking it what that clarification is, and then giving it a task one by one rather than all at once, so that you can see those outputs, you can eyeball them as a human as well to check that it hasn't gone off completely on the wrong direction, and then yeah, just making sure that you bring that. altogether. But if you can do it in that way, it's really, really useful.

Why communication skills are a prompting superpower

Thomas Joy 22:04

And it feels like we've gone a bit meta here because we're talking about communicating with AI tools in order to work in the field of communication. And I wonder to what extent all the skills you have kind of mean that you've got a superpower when it comes to getting more.

Rhys Williams 22:17

Yeah. I mean I I really think that's right because you know you can practically go and do a degree in prompt engineering now. As in as in like how do you create a a prompt that gets a gets a good result, you know, whether it's first time or whether with minimal iteration. And I think ex exactly the same skills that you would deploy to make a message clear and actionable for a human are incredibly useful when it comes to prompting. Because you know large language models don't have the same kind of cognitive constraints that that we have, but they have different constraints that they're operating within. And so you have to navigate and negotiate those constraints. You know, crudely how much information can this thing eat in one go? You know, how many tasks can you ask it to do before it's going to start to to mess them up. I mean if you imagine you were asking you know you're asking a a waiter silver service waiter who's bringing out a meal at someone's wedding and the tablecloth is really white and the the sauce on the plate is particularly dark and you were saying also do you think you could juggle with fire while you do that I mean that's basically what we're what we're talking about here. And I had a really amusing conversation with a with an unnamed pensions lawyer. Was it this week or last week? I forget. But anyway he he was saying and this is I am caricaturing his position here but he was saying oh you know AI is rubbish it keeps hallucinating and he he then said that he had spoken to the chief technology officer of his of his business who had gone no no no it's not the problem you are and and I have a lot of and he I think he was looking to me to try and validate his position and I ended up going look sorry I've got a lot of sympathy with this CTO guy. I think he's he's right everything Cath said that the the context is really important because without context you're just asking for effectively a gigantic traw of the internet you know the whereas if you control the information that the the tool can access and work work from you will you will shape the output in a way that's really positive and then just deciding how you chunk up and manage manage that task so that you you get the best results. And I think the only thing I'd add to what Cath said was that you also have a really good checker of its own work. So so that kind of final layer of and it's not just the same as putting in a prompt please make sure you check your work although that is quite useful but it it is okay okay do the do the research now synthesize the research now structure the structure the points now I've gone I've written something read what I've written tell me tell me what I've missed you know go back over your research find me the citations you know and we Cath and I worked on a project literally this week where we did more desk I mean it would have taken us a month to to do this desk research but but we still went through and read all the pe read all the underlying pieces because because of that risk of it being distorted you know crucial details being missed out and what have you so if that sounds really convoluted that's no more convoluted than what I would have done in the past except I would have done it all manually and I probably would have printed loads of documents and I would have got a pencil or a pen and and scrawled on it and written in the margins and it would have taken me forever. I would have found it intensely enjoyable but but there was a limit on how much of it I could do and how good I could make it and now those limits have been removed which is what I find most exciting.

How members are using AI to interact with their pension

Thomas Joy 25:50

So we've been talking about how you guys are using AI. The pensions world is obviously far bigger than just the people working in it. It's members who this industry was set up to serve. What are you learning about how members are using AI to interact with their pension?

Cath Collins 26:06

So we're already seeing more AI referrals to pension websites. That's quite an interesting one. So that is obviously where an I say obviously if you don't know this you don't know this that's where an AI tool has intercepted a prompt usually Google AI overview for example and rather than sending someone to a website it might just finish there end in a zero click search or if they have unusually very unusual behaviour if they've unusually clicked something and ended up on that website that's been tracked there. We're seeing more more people using those tools in that way. Not as much as we thought but I think it's going to be on the rise and it's something we're definitely looking at. But we're also investigating the quality of that output as well in different AI tools kind of auditing different websites looking at the value of those answers that AI tools are serving because what we have to remember as well is that people are using the bog standard versions of these tools we're super lucky we have the top tools at Quietroom they can do really sophisticated things. Basic chat GPT the free version or free for now is not as good and we'll just use the use the information that it already holds to provide those answers and within that all that key nuance is missed. We aren't hearing a lot about members per se using it. What we are hearing about and Rhys you might want to talk about this a bit more is about an influx in complaints from members written by AI tools.

Rhys Williams 27:37

Yeah this

The rise of AI-written member complaints

Rhys Williams 27:38

is really interesting so it's um pretty much every trustee or pensions lawyer I speak to says there is a noticeable rise in the volume of can I if I can get technical in a in a minor way the IDRP. So it's basically you know a a dispute between an individual and the and the pension scheme. And so not only is the number of them going up but the nature of them is surprisingly articulate. So it it's that kind of um oh wow telling you what this um this person who used to make and fit tires for a living knows a surprising amount about pensions law. You know, that kind of thing and you can take kind of two and it's using up lots and lots of trustee time is what is what I'm hearing because they've all got a process that they have to follow and they're following it despite the fact that the number of these communications is going up and the nature of them is changing a lot. So we're here we're talking to quite a lot of people about changing up your processes and procedures to to to respond to that. And you know rather ironically maybe there's a use for AI in that. So we don't want to kind of necessarily preside over a world where you know members use AI to to to write complaints and then trustees use AI to respond to them and you kind of go, what a terrible waste of human time and effort that is but but it is more on the sort of triage triage side that quite a lot of these very very similar to what Cath was saying about you know if you ask a Google or you ask ChatGBT for advice on something to do with your pension it may well give you an answer that's a bit true and a bit mangled and you know sometimes that's really quite dangerous. You know we might advise you to do something that's really against your best interests. Most of the time it's just sort of wrong in a way that's quite hard to unpick. And we've seen examples like mmxing up two pension schemes with the same initials you know quoting information from a the Dutch arm of a company rather than a UK arm of the company drawing information not from not from our client's website but from an 18 year old union letter on the on the subject you know all of this kind of stuff. But in the world of the IDRP there's a load of stuff where reference referencing US law could you imagine if it's going to its there's kind of two ways it can get an answer. It can go to its training data so at some point some people have made it swallow the internet or it can do it do a web search. But it's very easy for it to to mix up some nuances from from you know different regulations, leg legislation and so on and and it will be stuff that is oh yeah that is true but not true in this situation or or it is true but not true for you and and these kinds of things then taking lots of trustee time to unpick. Although I would say I spoke to someone yesterday who said actually it helps me a lot because I used to have to try to work out what people are on about and now it seems much more obvious because they're much better at communicating than ever before. He said I said what's the mo what what are you doing to deal with that? And we were on a on a Teams call and he picked up his mobile phone and he waved his phone at me said I just ring them up and and I think that that goes back to what I was saying earlier that that um you could get really into a tit for tat, you know, AI enabled slug fest, you know, with these kind of communications or or you could just call someone up and he said when you speak to somebody you can usually get to the nub of it very quickly and it's quite easy to solve the problem.

How schemes can influence the answers that members are getting

Thomas Joy 31:18

Cath I'd want to come back to something you were saying about how members are using tools to ask questions about their pension and then to interact with pension information that way. Is it possible for trustees and schemes to do anything to influence the answers that members are getting it feels like telling members no you shouldn't you shouldn't use these transformative tools. That feels like that's not likely to be something that works but what are you finding is working with clients?

Cath Collins 31:41

That's a really good question. I think they have to do something to make sure that their members are getting the right answers and all of that lies in making sure their underlying content on their website actually works correctly. By that I mean it's accessible for humans. The heading on the website on the webpage tells you about the thing it answers that it's going to describe the answers users' questions straight away in the language that they need it's structured clearly so that a user but also an AI tool can navigate that because what we've found in our kind of AI testing in our AI research is that things that work for humans also work really well for AI tools because it's about finding the information that you need really quickly. And it's really important particularly if there's any kind of content that you prioritize it's anything that carries a high risk of a bad member outcome. So we've seen a lot kind of journeys around transfers particularly in DB, the DB world anything around retirement journeys if you don't present that information clearly we're seeing kind of nefarious advisors popping in the same ones might I add across different schemes schemes testing that will offer DB transfer services or will offer to you know give you the retirement plan that you need. Whereas actually in a lot of these schemes they've got advisors that are already set up that know their scheme inside out that offer preferential rates to members but these these other advisors are coming in and intercepting that traffic in the AI answers. And what they're doing which is very very clever is they're mirroring the language that users would ask about a question. For example should I transfer my DB pension large heading on that page swoops in straight away because a lot of trustees wouldn't be comfortable asking that question on their website but maybe they need to get comfortable with that asking those difficult questions that they know members are asking anyway. They've got the trustees and pension schemes have got access to so much data about member queries to call centres, about what people are typing into the search box on their website, about things they're hearing in in feedback generally that they could really shape their content around those users' needs and if that content meets those users' needs they don't go elsewhere and their information will appear in those AI searches much higher up anyway.

Thomas Joy 34:08

Wow that's really shocking so it feels like what you're saying is people are asking questions to AI. AI is just trying to be helpful and find the answer and if you're not answering that question the answer might be taken from someone else instead and you've kind of lost control of your message at that point.

Cath Collins 34:24

Yeah and I think AI also can be a little bit too helpful. There are certainly a couple of AI tools that will really quickly offer to build you a modeler for example and if anyone's ever been involved in pensions modeller building the underlying assumptions and calculations that goes into those things take multiple teams across different industries to get that right but these AI tools are doing it immediately they're going oh I can tell you what you need to do for your retirement I can advise you on what you should be saving and if you're on track. So I would say you need to solve your members' problems on your own website in a controlled manner otherwise these tools will do it for you probably wrong.

Rhys Williams 35:05

I've

The hidden risk of old content

Rhys Williams 35:06

I've got just a quick thing to add which is that um sometimes in trying to be super helpful we think more content about pensions is a good thing. You know oh well we've created this guide let's put that up on the website or we've done this let's put that up on the website but nobody ever retires the content when it's out of date and you know you go well it doesn't matter because it's on page 48. Nobody will ever nobody will ever click back to page 48 and you know you go to page 48 and it and it's you know the pension scheme booklet 2018 you know draft six final final you know that kind of thing and it's and that was doing nobody any harm where it was and people thought it was useful and it was just a task that we didn't need to do. And yet now all of that old information is still accessible you know one of the things I've learned about the internet is it's pretty hard to get rid of things once they're on there. That there's always some sort of way you can access this information. And and I think what before seemed like a sort of a pointless admin task of content governance, you know, of going going through and deleting, rationalizing, removing things that are old or out of date is it is now no longer that sort of inconsequential bit of administration is now something that you really need to do to protect your members because otherwise they will be getting this information that was right at the time but is but is wrong now.

Thomas Joy 36:34

So we're

Quickfire questions

Thomas Joy 36:35

coming to the end of the podcast. I would like to ask you some quick fire questions though I suppose they could be quick but we could also talk about them for a while. I'm interested to hear both of your thoughts on these so question one for you favorite AI tool that you use?

Rhys Williams 36:47

Claude without a doubt uh Claude Opus 4.7 with the maximum thinking toggled on is AI as you might imagine it would work rather than what you normally get. You're a man of good taste and opus 4.7's a new model what is it about what you're getting back from that that makes you go wow so oh god this is meant to be quickfire so did the one the real breakthrough one was actually the one before for me so it's Opus 4.6 you had the ability with the last two 4.5 and 4.6 to toggle on extended thinking which means it just thinks for longer it's looking at more information thinks for longer. As a result it can do it can do more complex tasks particularly complex reasoning. So you know look at these 10 documents compare them to these other 10 documents a job that is very very difficult it can handle that kind of stuff. And I think also the it seems to be much better at saying oh no no no I can't do that or or or I know you've asked me to do that but I don't think that's a very good idea. I think this is what you actually meant and and and that sort of stuff. So so when I mean it's AI like you think it should be it's it really does feel like it is at a sufficient level of sophistication to deal with the the more complex bits of your job versus the other models.

Cath Collins 38:09

Yeah and I think it also won't get going on one of those extended thinking tasks until it's made sure that it's definitely got the right end of the stick. So it will ask you loads and loads of clarifying questions now, whereas before you would have to ask those clarifying questions just to make sure it knows exactly what it's doing before it sets off, which is a bit annoying sometimes because you think that you've set Claude off on a task and you'll come back in a couple of hours and you're like I missed the clarifying questions.

Rhys Williams 38:34

Yeah and and I think the other thing is that it's it's um so the the newest model has got adaptive thinking what that really seems to mean is it it won't overthink if it doesn't need to and but it will really think when it needs to and sometimes I used to because my preference was for the most sophisticated model turned up to the maximum setting and then you'd go oh can you just extract the text from this PDF and he goes yeah let me really think through this task really deeply and you know about four hours later and I'm like I didn't want you to like emboss it on gold. Do you know what I mean? It's like the most unbelievable answer. So I I think that is as well as you can ask a quick question get a quick answer. But if you ask a really difficult question or pose a difficult problem you'll get something much more I'm gonna say thought through even though I know that that's a metaphor but you know what it you know what I mean. I think the other thing to say about Claude if anybody hasn't tried it is so Claude at the moment has got kind of three arms the one is your standard chat you know that you would know from any AI tool that you use and then there's a couple of others. So one is Claude Code which is you know as far as I can tell absolutely revolutionising the world of coding and software development you know it can do vast volumes of pretty reliable code create things for you you know at the drop of a hat. I'm not a software developer so I don't use it a great deal but what anthropic the the makers of Claude have have done is create Claude co-work which is effectively Clawed code for normals and it is so it it it's um absolutely extraordinary can orchestrate and then spawn a whole load of agents that can do things locally on your computer. This is a really trivial use case and yet made such a difference to me. I recently changed laptops so I changed from one laptop which had given up the ghost to to a new one and I'm going what do I need to keep in the new world I don't just want to do what I've done for the last 20 years every time I change a laptop which is just moved the whole laptop onto a new laptop and if you just need progressively bigger and bigger laptops until it's like the size of a house. But the the so what so what I did was look I've just got all these screenshots I've just got all these files I've got everything I've ever downloaded. Can you just look through, categorize it into home and work, delete all the duplicates, organize it into folders based on what the subject is anything that you can't anything you can't categorize, come back and present to me. Anything that you can categorize, just just put them in put them in folders. And after a few clarifying questions, I go off and have a sandwich and come back and 40 minutes later it's done and and I'm like that is unbelievable. Now it's very very token hungry. So so you know if every single week now I am rationing myself and going oh uh oh you know I've worked out how you divide a hundred a hundred by seven a hundred percent by seven in to get to seven days or to or you know or by five to get there that's an easy sum. But it it it's I'm always at the edge of my limits of usage now just because of kind of things things like this. So yeah the other one is I feel like I'm selling clawed now. I should be on some kind of clawed kickback. There there are extensions for for for Chrome internet browser so you just go can you do this job on the read this website and do this job for me plugins for Excel so you know coming back to ki Cath's pivot tables and charts and everything that's the limit of my Excel skill claw just sort of wangs the numbers into Python and it's just a brp like that and so just unbelievable analysis capability that would have been the preserver of a data scientist or a real or an Excel specialist and now is me typing in natural language into a box and it just doing it for me. There's one for PowerPoint as well can you build me a slide and it builds you a slide what can I say so so yeah the very very long answer to your quickfire question which one do you like?

Thomas Joy 42:31

Claude it's Claude Okay we are going to make these next two super quick fires I'll be I'll be the judge of that and this is this is my yeah my poor choice of question. Alright one secret to a really good prompt just one secret context context chunks is that quick for you quick enough for you super quick good like giving one word answers to this next question however okay final thing I want to ask you everyone's at different stages of their AI journey just because you're not far along now doesn't mean that you should give up on it. It means you've just got an opportunity to to get started quicker and to learn from everybody else. If a client came to you today and said what's the thing that I need to do in order to have a really good start on my AI journey what one thing would you recommend to them?

Rhys Williams 43:21

Start with the problem you're trying to solve the biggest mistake I I hear from people is oh yeah I've I've booked um I've booked two hours on Friday to do my AI you know and basically I'm just gonna now just play with it and try and see what it can do. And I'm like no think about what you have to do and all of the things that you struggle with or find difficult or or might find a workaround for set it going set it going on that. And beyond that trial and error. Like the the very very best thing we did as a business was give everyone the tools and encourage them to experiment and let them solve their own problems for themselves.

Cath Collins 43:57

And if that makes you nervous find a buddy So I have many AI buddies at work, and shadowing someone who is very good at that kind of thing is worth its weight in gold. So two years ago, I didn't know how to write an AI prompt. I just watched really good people get even better at it and copied them. And now I do the same for the more kind of higher order thinking as well. So yeah, just find a buddy, somebody who's an evangelist, and just follow them. Even five, ten minutes every now and then with them will be really, really valuable.

Thomas Joy 44:35

Those are such great answers. I'm really sad that we're gonna have to bring this one to an end and say goodbye. It's been so lovely to be back in a room with both of you. This has been a really good chat. Thank you so much for your time. Thank you. Thank you, Thomas.