Transcript: Inside US Foods' Journey to Scale a Generative AI Sales Tool

Generative AI continues to be a significant area of interest and investment in the consumer goods space, but wider adoption still faces challenges. Companies are having difficulty transitioning from concepts and pilots to real-world, scalable applications. In this webinar transcript, David Falck, VP of machine learning engineering at US Foods, shared how his team overcame this hurdle by developing and launching an AI-powered sales-optimization tool. This tool not only helped boost sales productivity but also reduced operational costs. Learn the strategies and tactics Falck used to quickly turn generative AI ideas into practical, growth-minded solutions.Lisa Johnston: Hello everyone. Welcome to “From POC to Production: US Foods’ Generative AI Journey.” My name is Lisa Johnson. I'm the editor-in-chief of CGT, and I'm very excited to welcome you here today. If generative AI isn't on your company's agenda, it's certainly on many of your leaders’ minds. But many — I probably might even have to say most — are struggling to move out of the idea stage and into wide-scale use. So today, David Falck, VP of machine learning engineering at US Foods, is going to share how he and his team have built and launched a sales-focused tool that's not only improving sales productivity but is also reducing costs. And he's going to tell us some of the tactics that they use to get this moving quickly and into real-world applications. We're going to first start out with a brief presentation from Aparna Galiasso, head of North American retail and CPG industry strategy and BD at Amazon Web Services (AWS), which is going to set the stage with some key considerations about GenAI. After that, Aparna and Dave are going to talk through the US Foods journey, including what it's been like to roll this tool out to US Foods teams around the country. Aparna Galiasso: Great. Thank you, Lisa, and thank you all for your time. Let me kick us off with a memory from my childhood reading. Lewis Carroll the Red Queen explains to Alice a very peculiar fact about the race that she's in. She shares it takes all the running you can do to keep in exactly the same place. Now, if we come back to where we are in our current day, how many of you feel like Alice; that you're running in circles around GenAI, but that you're actually not getting anywhere? What makes this even harder is that you can't stop running because, like the red Queen's race, if you stop running, your team stops running, the org stops running, you fall behind. Now you could say that the question really is, “What does it take to get ahead and how do we evolve faster than the market, faster than your competitors, and faster than the rate of change of technology?” The question really isn't just about what it takes to do GenAI, but how do you do more GenAI? How do you do more POCs, perhaps? How do you do GenAI better? How do you move more POCs to production? And truly, it probably is a mix of both. How do you do more GenAI and how do you do it better? In my role as the head of North American business development for AWS, I have the privilege of working with brands and retailers on this journey from AOP to POC to production. And while at this point, there's really no singular formula for success, I'm going to share with you a couple of the key considerations that we've picked up on the path to production for GenAI. So let's get into it. Step one, the first thing is really for moving from art possible into tangible POCs. And this stage is largely about ideation. The biggest call out we have for this stage is it's all about moving quickly, however, not at the cost of quality. And you are really at the top of your funnel with your GenAI ideas. And if you don't have quality input at this point, you certainly will not have quality outputs. Now what do you do to move quickly? Well, at Amazon, we are customer-obsessed, and step one for process is to work backward from that customer and really find those durable and persistent needs. These are the needs that consumers have, not just today, but will continue to have in the future, and focus on solving those now with something as pervasive as GenAI, you will find yourself with a long list of potential POCs that you can do. And here the call out really is to focus the way we say it is. You want to prioritize and really start to think about a common, company-wide framework that you can use for doing so. Step two is where things start to get really fun. And in our discussion with Dave, we'll be spending a lot of time in this stage. This is the path from POC to production. This is where you, like Alice, will have to figure out your competitive advantage, and in doing so, we really recommend focusing on customization. This is where you take a generic application of GenAI that anyone could do, and then you start to get very specific with it around your organization and needs for this. One of our core beliefs at AWS is that data is your differentiator. GenAI is just the tool that unlocks the rich insights from your data. What this does also mean, however, is that to get any meaningful application of GenAI, your data has to be in order. And of course, while data can be your competitive advantage, the question really is, “How do you make that advantage sustainable, given how quickly things are changing with GenAI?” And with what your competitors are doing, “How do you really start to foster a culture of agile experimentation and design for flexibility from the get-go?” You need an approach to GenAI that lets you have choice, and let me just give one example of that. It might be the choice of which model to use as your use case matures, you might need a different model and the most advanced, feature-rich model might not be what you need for every use case, and we'll dig into that with Dave in a little bit. Now, step three is really about scaling, and arguably, we see the least activity here since it's still early days. But given what we know about digital innovation in companies, we know that scale is all about measuring impact, and managing metrics, and in the case of GenAI, this will require doing things like balancing cost, speed, accuracy, basically the cost and the metrics that you're balancing around, doing inferences and then trying to get to desired business results. Now we also know from our deep experience in AI and ML innovation that you have to use AI responsibly, safely, and securely, and that becomes imperative in this stage. You have to think about governance and guardrails more in this stage. Arguably, you already should have been thinking about security and privacy from day one, and that, at its simplest form, are some of the patterns we're seeing. Now let's jump into the conversation with Dave and talk about this in the real-world context of US Foods. So let me fans. Hi, Dave, how are you doing?Galiasso: Let's do this, Dave, before jumping into the GenAI side, can you tell us a little bit about your background and the journey that got you to where you are and the role you're in at US Foods today?David Falck: Yeah, happy to. So I started out with a degree in chemistry, which makes a lot of sense, right? Why I'm doing what I'm doing today. But I've always been a programmer. Why I think I got a degree in chemistry originally — I really loved experimentation. I loved creating new things. What chemistry didn't provide was a quick way to experiment, so I quickly gave that up and went into programming. Luckily, this was the 90s. I was able to talk my way into a consulting job. So I was a Java developer for a very long time, and then took on different roles at very, very small companies, even a two-person startup that luckily got purchased by a company in California, Autodesk. They make AutoCAD, and that was where I fell in love with AWS. When AWS only had five or six services, so it was really early days for AWS, but I saw the value because I spent enough time in the industry by then to know how hard it was to create even in the software world, it was still hard to create something brand-new, because you had to acquire hardware and size it and stuff all the networking AWS just took care of all that for me. So I fell in love with it, and I've been in love with it ever since. And then I got to US Foods in 2016 — they acquired a startup, and they needed some new leadership. I came in, and that's how I came into US Foods world. It's been a great journey ever since,Galiasso: Yeah, I remember the first time meeting you and walking into that room and still seeing that big check that startup won at some competition in the background, so really that you're still sort of living that startup feel within the bigger US Foods, and that's very interesting to see. Okay, so now let's start from the beginning of this GenAI journey for you. When did you start thinking about generative AI, and how did you decide which use case to prioritize for the POC?Falck: Yeah, because we kind of came from a startup kind of perspective within US Foods, I think the rest of the organization kind of saw us as more of an R&D Shop for the longest time, and we were, we've progressed through that, and we were actually running all of our machine learning operations, pipelines, and processes for all of our data scientists. So we're in a really interesting spot. And we're we're producing a lot of really valuable insights and analytics, and hosting models in production using Sagemaker and all the great stuff that comes with AWS. But because we are kind of known as this R&D AI/ML shop, we started getting a lot of inbound kind of questions or requests like, “Hey, is this a good idea for GenAI?” We are interested in generative AI just like everybody else was when it came out almost two years ago, into the like and we really hit the ground. We spent a lot of time, even before we started getting those inbound requests from around the enterprise, around use cases or potential use cases. We spent a lot of time working with coding tools, coding copilots. We've tried just about every kind of coding copilot out there, and we saw a lot of value there. The interesting thing that really stuck with me was I thought it would be so interesting if we could find a way to do what those copilot tools are doing; they're not necessarily reading your code and learning from that per se, but your actions within that coding editor like the code you write and suggest the suggestions that Y=you accept or reject as you're using it, those indirectly contribute to the refining of the model and providing feedback on the code and what's relevant what's useful. And I thought, if we could find a use case and build an app that we could provide an audience within US Foods, that could help us train the tool and learn from it and provide more consistent results and feedback. I thought that would be really interesting, and luckily, we did. I'll say too, a year ago and a half ago, a lot of the consultancies that we were working with offered a lot of POC time and kind of pre-sales investments to experiment with us, and that was great. We learned a lot from all of that, and they helped us refine our use cases around generative AI. But what really came out of that was the realization that generative AI is so new that no one really had a real advantage in terms of experience. So that really gave me and the team the boost of confidence that we could build this ourselves, we don't need to bring in a bunch of consultants to do this, necessarily. Galiasso: When I was thinking through what to share in terms of what it takes to win in GenAI, it's really hard to say what it takes to win. We have some considerations. You have some learnings, and I'm so glad you're willing to share them. But there are no answers yet, and we're really discovering as we go along. So it'll be interesting to look back at this a year from now and see where you get with us, too. Dave, okay, so coming back to the questions you identified this sort of durable customer need for your salespeople, which is this idea of doing more with less time. Can you tell us about the sales tool you've built, and how does that address their needs?Falck: I can't go into all the details, but I can talk broadly about it. Going back to the ideas that we had with all the use cases we had a lot of ideas around chatbots and summarizing tons of documents and even some crazy ideas of generating images of products and food and menus. And so there's a lot of interesting ideas out there. The thing that we found, too, in kind of curating this list of generative AI ideas, they weren't all generative AI ideas. Some of them were more suited to machine learning or more traditional AI kind of approaches. So that actually helped us, the insights and analytics team really prioritize those use cases because we had those conversations. You know, it was initially around generative AI, but we found other ways to satisfy those needs for our clients, our customers, internally. Back to generative AI. We wanted to make sure that if we were going to build something around generative AI, we didn't want to spend a lot of time and effort on something that was going to be given to us through another tool or subscription, right? We didn't want to build something that might come with SharePoint someday, or with Outlook someday, or some kind of Microsoft copilot or a Salesforce.com generative AI function. We were trying to be careful — who knows what's going to come in the future, but we were trying to find ways that at least guarded us from some of those possibilities. US Foods is one of the biggest food distribution companies in the country. We've got hundreds of thousands of customers, and my team has been focused for the longest time. I've been here for almost eight years, and all that time, we've been focused mostly on helping our local sellers, our local salespeople. And these are sellers that focus on our independent restaurants, kind of mom-and-pop restaurants. These are our customers, right? And we're here to help them make it and help them succeed. Being a restaurant owner is not for the faint of heart. It's hard work. It's even harder work for our sellers, right? Our sellers will manage dozens and dozens of accounts and our customers or prospects in the restaurant world aren't necessarily very digital, right? Despite all the POS systems that you see and all the QR codes on tables at restaurants, there's still a lot of just unstructured data and documents and pictures and videos that these restaurants use to communicate with our sellers and with our digital team to really hone in on the products that they need and the services that they might need from our check business tools.So there's a lot going on in this kind of prospecting and onboarding phase, tons and tons of manual work to bring on a new customer. It can take hours to just get this process kicked off. And that big chunk of time can be thought of as taking all of that unstructured data, all that manual work, all the notes, all the discussions, and all the thoughts that kind of came up during these conversations with these prospects, and pulling them all together and pitching and proposing something to these customers. So we found a way to provide our sellers an app that allows them to put all of that in, and then we process all of that. And there are lots of calls to more traditional AI as well as generative AI. In fact, we've got about four or five calls to generative AI. We're using Bedrock, so that makes it all a lot simpler. But this application has proven to save somewhere between 75% and 90% of our sellers’ time, and this is per restaurant, right? Not, not every restaurant comes on board and becomes a customer right away. And so there's a lot of work, a lot of investment and time to get all this manual work into something more digital, so we can pitch and propose to these restaurantsGaliasso: So it really starts to allow you to let the salesperson have more time and having those conversations and building that relationship and earning trust, instead of having them do the busy work of creating some of the manual things that you need to serve that customer. That's great. I think a couple of other things you said there that is really worth noting is this idea of having choice and essentially right, you're using Bedrock, and you're able to make these calls right now, but over time, perhaps as the use case matures and you need something different, you can switch out to rethink about how you serve the same need, solve the same problem in new in new ways. So that's interesting as well. Let me bring up another point about your journey on this. I think a lot of people have the perception that it takes months or years and very large budgets to do GenAI. I know that wasn't the case for you. So can you share a little bit more about that?Falck: I mentioned my team with the ML ops team, and we're relatively small but mighty. I like to say it's our little tagline that we use internally. We don't tell anybody, but we've got some really talented folks and we go way back six, seven years, working on really gnarly natural language processing problems. Right back in the day, we were trying to find data from venues and anything out there — lots of external data sources, lots of fuzzy matching of data. So we had a lot of kind of muscle built already to be able to take a lot of this kind of unstructured or semi-structured data and find ways to make sense out of it and build signals that can help our sellers sell find new products, prospects, or leads or opportunities within those accounts, and we've done a lot of work around that. So we had that kind of muscle built, but we also had a very talented gentleman who had some full-stack development experience and knew how to call APIs. And that's really all we needed to build out the first version of the proof of concept of this app. He did a phenomenal job. He got it done in about six weeks. He'll say eight weeks, but I'm gonna stick with six, but a very short period of time.We got a lot of support from the very beginning, our leadership team was phenomenal. They immediately saw the value, the potential of value of it right? Because we've been working with the sellers for such a long time, they just totally brought it on board. Started telling us immediately what was right, and what was wrong. And there were no filters between the sellers and us, because we've been working with them for a long time. We know what they need to some degree, and they're not afraid to tell us when we get something wrong or when something's right and drill into what they want to see. So a lot of it's just about relationships, not surprisingly, and we've got an amazing IT and digital team. They've been very supportive of our CISO team, our legal. Everybody's been really supportive of this and helping us along the way, which has been a fantastic journey.Galiasso: You found this sort of area where everyone had a passion to solve, right from the sellers wanted to solve the IT team wanted to solve legal, looking for everyone sort of got behind the vision, and you got them involved early in solving that durable need. Then were able to sort of scale it through — that's impressive. Six to eight weeks is super impressive. That's incredibly quick, and I think that comes back to this idea of building something that's minimally lovable, and then you're able to iterate on it after that, right? So what has that approach been to production? As part of that, you have all of these users that already bought in and giving you feedback. How has it been? In involving the executives. And how are you rolling this out further as you move to production?Falck: Our leaders, our executive team, saw the potential, so that was helpful. That doesn't always happen, as we all know, and because we had these relationships with our salespeople, we knew that we needed to get some champions. And change champions here, behind us. We did a couple of things. We formed kind of an ad hoc, generative AI steering committee. We pulled in a whole bunch of representatives from across the organization, our strategy team, in fact, our the new leader of our insights and analytics team, came up with one of the original ideas for this. It was part of the list. And we're like, oh, this is interesting. We connected that with all of our experience and with our relationships with our sellers. This one actually feels feasible, like we could get this done in the next six to 12 months, and it's high value and we've got some support here from even before the PSC that this is an interesting idea from everybody in the steering committee. So let's go for it, and we went for it, and we tested it with some of our best and most tenured salespeople. They haven't always been the ones that adopted technology. We've had lots of experiences over the years in rolling things out and they luckily saw the value, and they were able to understand where we were going and what we were trying to do, and gave us a lot of great feedback. Because of all that, a lot of sellers look up to these more tenured folks — they became just awesome change champions. We also worked with our sellers' tech council. So our CRM team and others within the organization will often throw ideas at them, and they immediately loved it, too, and gave us tons of feedback. But at the end of the day, we worked with our strategy team, and they were very helpful in helping us build the business case behind this. And there are cost savings components and there are revenue components, because, we're saving hours a week of our sellers' time, and they could essentially be in front of our customers and our leads and prospects talking to them and building the relationship and closing more deals. So we're comfortable that this has a return on investment of at least 10x — maybe even more — but we're really comfortable with that, and that helped us build the case to get some investment and really build this out for the next year. Galiasso: I think that partnership with the strategy team is interesting and very important here, because it's building that business case, and it's giving you that ability to say, “Hey, this is not just short term going to help us be more efficient and effective, but that it has this long-term revenue potential, and that really adds a multiplier to things.” Let's continue thinking about the strategy and that path forward. So do you have a roadmap for where you're going to go with this in the future? Is there more optimization? Is there iteration and talk a little bit more as part of that process, also on how you will be tracking metrics?Falck: Our transformation office has been really helpful in helping us roll out these tools as a pilot. We have 70-plus markets across the country. They helped us choose which ones to work with. First, our head of sales pointed us in the right direction in terms of markets or areas that were doing well, that probably had some time to give us some feedback, and that worked really well. And then we're still in the pilot phase. We're about to go to production, but we're getting inbound requests from other markets that weren't part of the pilot necessarily, like, “When do we get this? We've heard about it.” And that's fantastic. That doesn't happen very often in software development or machine learning or AI development. So very lucky. I think the key thing here is we spent a lot of time with our sellers. They've driven the development of the entire project, and they give us the ideas we have in our backlog to help us prioritize. And as we rolled out the pilot across the country, we realized different parts of the country have different needs, right? We rolled out West and we realized, oh, wow, there's a lot of non-English speakers that we need to take into account that we hadn't. In the Midwest, for example, different languages, different kinds of ways of interacting with our sellers, taking all that into account, and finding ways to just include enough knobs and dials to tune this for every market across the country has been really helpful.Galiasso: Yeah, I really like this idea of knobs and dials and that being sort of what you've built into the design to allow you to sort of iterate as you learn more. And the power of the fact that you have those salespeople and your teams align with you from the start again, because you're solving something that is a pain point that drives value across the company. I think another area that'd be interesting to get your point of view on. You're using GenAI to provide something as input to your salespeople, who can then sort of review it. So you have this human in the loop that looks at what the output is. Has that driven your model choice? And what have you thought about that journey? Where did you start? And where are you now in finding the sweet spot as you think about production between that balance of cost, speed, and accuracy?Falck: That is one of the reasons we went with Bedrock, because we knew we'd have the ability to switch from model to model, or to include many models in our process, as we do today. And it's also allowed us too that maybe we start working with the most advanced, most comprehensive model. And then we realized that we can actually do this with the kind of the medium version of the the small model version of this, and it's good enough, right? But we actually had a lot of interesting experiences in the past, in our previous life, as more of the innovation or R&D arm, we did a lot of work with unstructured data. We actually used Amazon's Mechanical Turk service a lot, and we built up all, pipelines and workflows to take into account. You know, Mechanical Turk, if you're not familiar with it, is still an amazing service. It's one of my favorites. I think it's Jeff Bezos's favorite. It allows you to crowdsource work. You can ask simple questions and real people behind an API will tell you the answer, and you don't always get the same answer from everybody. So we had to take that into account. That's also what clicked as I was thinking about this webinar. What made us so comfortable with generative AI's outputs? There are some hallucinations, sometimes it's a little non-deterministic. You ask the same question, and you might not get the same answer every time, but we felt really comfortable in that world because we came from that world with Mechanical Turk and were able to find signal in the noise there. And what we realized, too with the first version of this app was the seller, it wasn't perfect. We were wrong maybe 30, 35% of the time in what we were showing the sellers, but the sellers saved two or three hours of their time, and they're presented with a list that they have to review, and they didn't really care if things were wrong, if they had a way to fix it really quickly. We gave them that capability, and that all came from our experience with Mechanical Turk back in the day. Building these tools for our Turkers out in the community, we kind of took a lot of those learnings and built it into this app for our sellers. So they're going in, they're making changes. Go around this coding copilot tool, gathering the feedback, what's accepted, and what's rejected. We have a tool that allows us to do that with our sellers. So they're accepting things, they're rejecting things, they're deleting things, they're adding things, and we're learning from all of those things. It's just, it's fascinating.Galiasso: Yeah, that's awesome. Okay, so I'll close out my line of questioning with one last one, and then we'll see what questions are coming in through the chat. What are your learnings? What would you advise others who are on this journey with you?Falck: There's a lot of talk about generative AI. A lot of hype. But there's a lot of value there. You know, there's a lot of noise, but there's a ton of signal too. So you just got to figure out what that means for your company. And you don't have to do anything unique. You don't have to build a chatbot either if you don't want to. You can think of generative AI as just a software module, right? Especially with Bedrock, we have APIs that we can call. We can switch those out. We can put them into a step function, and have different calls to the foundational models. And one thing to keep in mind, too, as you're building with generative AI, the models and the capabilities of these models are just going to continue to change and continue to broaden in scope. And that was one of the first realizations we had about maybe four or five months into this journey. I was like, oh my gosh, this new model, we could actually do all five steps of what we're doing, individual calls to the model or two different models, we could do all five steps in one call. We have one big prompt, really complicated prompt, and and we could do the whole thing in one go. And we're like, this is really cool. This will save us so much time. But then we realized, no, it won't really, at least for us, for our application. We realized that having things chunked out, different calls, again, more of a module architecture, was more advantageous, because we could control the quality of the output from each of those steps, and again, switch out models or versions of models in each one of those steps. And just because you can do like, let's say, optical character recognition with Claude, doesn't mean that you shouldn't use Textract Right? Textract has been around for a long time. It's been tuned. It's been used by millions and millions of people for just millions of hours and you have the ability to take on, kind of build on all the work that everyone's done before you, and again, keeping these things kind of modular, and having the ability to not only test in between, but also have a human in the loop when needed, right verify the output that You're getting from each of these steps has been just crucial for us.Galiasso: And just to build on that, I totally agree with you on this. You know, fine-tuning isn't just fine-tuning a model. Fine-tuning could be of the way you're structuring the entire project and the piece components that make the end result, just working back from that, as long as you always keep a line of sight to being secure and making sure privacy standards are met. You can sort of play within that frame as much as you want. So that's that's great to hear. And, of course, the ability to make different choices as you learn. So have options, keep them on ice, and execute on them in the future. That's always great to have with that. Lisa, can we open this up and see what questions might be coming through from the audience?Johnston: Yeah. So thank you so much. David and Aparna, a great conversation. You've already shared a lot of learning. So let's go through some of these questions to tease out a few more. So I have a question about resistance to user adoption. So you had mentioned, Dave, that not all of your sellers are exceptionally digital. There's certainly a lot of excitement surrounding GenAI, but that's not always coming from every direction. So can you share some tips that you have for overcoming resistance?Falck: It really was just finding these sellers that could be good change champions and we were surprised too, by some of our more tenured sellers that they took to this. It's not that they're not digital. They're just very busy, right? And they don't have the ability to take on everything that gets thrown at them. They get a lot of stuff thrown at them. They're sellers. That's that's their job, and they have the hardest job in the company. The ability to show them something and kind of paint the vision of where we wanted to take this, but then also show them that we're not perfect, but we can also save you a lot of time even though we're not perfect. It was a game-changer for them. So the change champions saw it. Some of our younger sellers saw it, less tenured sellers saw it, and everybody in between, again, has been just really helpful in helping us gather feedback. Like, why aren't we doing this? It's a great idea. We put it on the list, we prioritize it. And we we turn around the changes and and the updates to our applications very, very quickly. And so we try to be as responsive to our sellers as we can. They need something. We try to get it in the next sprint, if possible, if not the next sprint. And so we've just been trying to make sure that we have the trust of our sellers. We want to continue to earn that trust and make sure that we are saving them as much time as possible so and they get that message, and it's been very successful.Galiasso: We're seeing that across all of our customers when they're starting to do these pilots and moving to production. It's really this idea of having involved that user or that consumer of the application from the start, but then keeping them involved throughout it. See you’ve got to engage them early and often, and that often part cannot be forgotten. You can't just get input at the start and then sort of say at the end, “Here is the rock I've been polishing forever,” and constantly using them as one of the loops in the iteration is where we see the most success. Johnston: So we have a question again. They're asking you, Dave, are there any other parts of the business that you can talk about where you're looking to use GenAI again, and if so, how are you identifying those use cases?Falck: We've matured since our kind of ad hoc GenAI steering committees, and we've got some fantastic senior VP, senior leadership folks that have put together a kind of center of excellence and clearing house. We're keeping on top of these use cases and gathering as many as we can as we go. And also keeping track of what's coming from our SaaS providers and from our other vendors, trying to make sure that we're not going to build something, or spend time that's going to be duplicative work come three months from now or six months from now. And the other, the other thing I'll say is, it really depends on the use cases. There's a dependency on whether or not we have the right data in place. Luckily, US Foods has been investing a lot of effort, resources, and time on our data warehouse, and our modernization efforts. So we have a lot of muscles and kind of energy already in place around in that world. So especially if we have a use case that kind of falls into that realm where we have really clean, reliable data. And luckily, we have a lot of that now. I'd say a few years ago that might not have been the case. So it might have been a little bit harder, but we do have a very strong enterprise data team, and we work with them to make sure that we're finding ways to take advantage of that. Even if you don't need your enterprise data as input to your application, you're gonna need to collect a lot of data, and you're gonna need to put it somewhere. Like Aparna said earlier, if you don't have your data house in order you really need to. It’s step zero before you go down this generative AI journey.Galiasso: Yeah, that makes a lot of sense. And Dave and I have not discussed it, but I want to put it out there, because in my role, one of the things that gives me the most pleasure is when retailers and brands start collaborating and working together and coming together and US Foods is sort of in that interesting space where you're doing this for salespeople. It'd be interesting to see what on the supplier side, with your CPG partners, you could start doing to simplify that front end of the process that eventually feeds down to that restaurant tour. So food for thought, let's do something on that side. Falck: There's lots of applications that we've already kind of talked about, and we're prioritizing around that world. We were focused on our local sellers — right now they are independent kind of mom-and-pop facing sellers. There's a whole other side of the company focused on health care and hospitality and more national chain accounts and just bigger kinds of organizations. There's a lot of interesting ideas around how we can apply some of the functionality that we've already built. And because we've modularized our work, some of these modules are already being used and experimented with in some of these other applications. So this idea of having an asset library with these kinds of generative AI modules, or at least, kind of a rough sketch or an architecture or a way of working with it already, we're already seeing some benefits in that architecture, in these other applications of generative AI.Johnston: Could you talk a little bit about how you're quantifying the impact?Falck: We have our tools, our applications that we've built, instrumented pretty well. It's very important for us to understand when something's accepted or rejected, so we can use that in future iterations of the tool. But we're also keeping track of how long our sellers spend in the tools, right? So we can say, okay, we know you started here, you went to this page, you went to this page, you did all of these things, and you spent 15 minutes in the tool, or 20 minutes in the tool. And we know how long, on average, they would have spent doing this work, this particular part of the process, without this tool. We can do some rough calculations on the time savings. There are a lot of other things that we can do in terms of sales in the future, and understanding our effect on future sales or new restaurants coming on board as customers, but right now, we're just focused on getting this pilot completed and getting to production in the next couple of weeks. But yeah, rough metrics, but we're able to show some real value, at least in the time-saving or cost-savings category,Galiasso: And just to build there generally, one of the things that becomes tough is sometimes customers aren't quick to establish baselines before. And it doesn't have to be robust, necessarily, but having some sort of baseline to say this is what it would look like if we made no change, and then being able to attribute to the best of your ability with some assumptions, what that delta is that this new pilot, or innovation generally, is solving. With US Foods right now, it is time savings, but I'm very excited to think about the time when we're going to start talking about revenue generation with it.Johnston: Okay, so I think we have time for one more, and then I'm gonna wrap things up. You mentioned you gather user feedback through this. Could you share just some of the ways that you're doing that?Falck: Some of our sellers just call me directly. You know, I see a missed call. It's probably a seller just calling me telling me what they don't like about the app right now. And that's great. I love that we also have a feedback tab in all of our apps, and they can write and take screenshots of what's right, what's wrong, and give us feedback through that. So we always make sure that we have that in all of our kind of seller-facing functions. And we have a team channel. We have office hours every week, and. That our sellers come to and sometimes we'll even talk about the app, and they'll give us feedback and ideas on what to focus on next. But it's try to have a couple of different channels, kind of an omni-channel approach to feedback.Johnston: Okay, great, Aparna and Dave, I want to thank you both so much for giving us your time today and all of your insight. Great conversation, great learnings for our audience.

Related Topics