Embracing AI: A Chief Learning Officer’s Journey: Part 2 in a 3-Part Series
This article was written in partnership with two colleagues I get the pleasure of experimenting with everyday: Joshua Ehrenreich and James Larcus
A few weeks ago, I wrote a post on how Generative AI has become a workplace staple, transforming how we work, learn, and operate. As learning and talent leaders, we have three primary responsibilities to ensure we are staying ahead and even leading the way with AI:
Upskill on AI literacy: Understand what AI is and is not, how generative AI works, and the principles of prompt engineering.
Courageously Experiment: Discuss how we leverage AI with our team and highlight these stories across the organization as learning experiences and new, more productive ways of working together.
Lead through Change: Set your vision for leveraging AI, managing risk, building sound governance systems, and recognizing employees for learning and using GenAI in their work.
As part of a 3-part series, I’ll go into detail with each one of these areas. I shared about upskilling with AI in a previous blog post. The second area is about courageously experimenting. Being “Courageously Experimental” is one of our core values at Udemy. The behavior connected to that value that we work to exemplify more is: We take risks, celebrate failures, and learn fast.
My team is leveraging GenAI to be more productive in their day-to-day work, thinking differently about potential solutions to problems we are trying to solve. And we are all learning fast. Specifically, the two use cases in which we leveraged GenAI were 1) assessments and 2) developing practical and contextual learning exercises. I’ll go into detail for each one.
Revolutionizing Assessment Development with Generative AI
In organizational learning and development, agility and efficiency are paramount. When we sought to revamp our manager development program, we encountered a common roadblock: creating a high-quality skills assessment within a limited timeframe and budget. Traditional approaches—outsourcing or relying solely on internal resources—proved too expensive or time-consuming. We needed an innovative solution to deliver 30 well-crafted assessment items without jeopardizing the project's timeline or financial constraints.
The Challenge: Quality, Time, and Cost
Developing a robust assessment bank typically involves a complex process. It necessitates subject matter expertise, coordination across multiple stakeholders, and significant lead time. Often, these assessments also incur vendor costs and protracted development cycles. With a mere four months between project initiation and the desired launch date, the traditional six-month development cycle was not an option.
Exploring AI-Powered Solutions
Before turning to AI, we explored various options. Outsourcing, the standard approach, was deemed infeasible due to cost and timeline limitations. Internal subject matter expert (SME) authoring, while possible, would burden us with unplanned work and potentially extend the timeline due to the need for upskilling. A proposed internal item-writing retreat presented logistical challenges and risked not yielding the desired quality.
A Breakthrough Solution
We recognized the need for a more efficient and cost-effective approach, so we turned to GenAI. By leveraging ChatGPT's advanced text generation capabilities, we developed a prompt that equipped the AI tool with precise learning objectives and customized requirements like our learning principles and standards. This enabled the AI to rapidly generate an initial set of assessment items we reviewed and refined.
Generating the 30 items took approximately one day, a fraction of the time required by traditional methods. The real-time review process significantly streamlined feedback and iteration, saving countless hours compared to asynchronous collaboration with multiple cross-functional authors. Moreover, by avoiding outsourcing, we bypassed vendor fees, contributing to substantial cost savings.
The success of this generative AI implementation has fueled our enthusiasm to explore further applications. We plan to pilot the skills assessment within the revamped Udemy Manager development program and gather feedback for future enhancements. Sharing how we are experimenting has sparked ideas across the organization to think differently about how they might address old problems with new solutions, leveraging AI.
From Abstract to Actionable: AI-Powered Decision-Making Learning
The second use case was inspired by one of the pioneers in GenAI, Ethan Mollick. He is an Associate Professor at the Wharton School of the University of Pennsylvania, where he examines the effects of artificial intelligence on work and education. He is also the Co-Director of the Generative AI Lab at Wharton. He embodies “courageously experimental”, writing and playing around with AI before most of us knew about it, sharing his learnings in his blog, “More Useful Things.”
One of the many contributions Mollick has made at the intersection of learning and AI is a prompt library. We leveraged the library for instructors as we worked to personalize and contextualize application exercises beyond the standard templates, worksheets, and job aids that we have always provided.
We leveraged prompt engineering and a "CIDI " framework to create dynamic, AI-powered learning experiences to achieve this. CIDI stands for:
Context: Setting the stage for the AI interaction. For example, "You are a coach helping managers improve their delegation skills."
Instructions: Provide step-by-step guidance for the AI, like a recipe it follows to produce the desired outcome.
Details: Adding specific requirements or constraints, such as "Focus on techniques for delegating tasks to remote team members."
Input: Supplying the AI with relevant data, documents, or information to work with.
The result was 18 experiences for nine management skills, which allow managers to practice in a safe “sandbox” environment and receive real-time feedback relevant to their situation.
One of those experiences involved developing the skill of decision-making and leveraging our decision-making framework, DACI. We designed an interactive exercise where a bot guides learners through a real-life decision scenario. The bot prompts them to identify key stakeholders, consider cross-functional impacts, and generate a customized responsibility matrix based on their input. This approach reinforces learning and provides a tangible output that learners can immediately apply to their work.
With these in-the-flow-of-work GenAI learning exercises, the team created a toolbox of learner-relevant, self-driven support activities. We have been able to repurpose these across different learning programs and experiences just in time. As one of my colleagues, Paul Kent, Sr. Manager at Pepsico, calls it, “Work in the flow of learning.” By leveraging AI, we've more easily bridged the gap between theory and practice, enabling learners to develop and apply critical decision-making skills that are more engaging and impactful.
A recent HBR article about using GenAI by Gabriele Rosani and Elisa Farri referenced research that only 15% of leaders and managers consistently use generative AI. While we know AI won’t replace managers, the most effective managers will use AI consistently in their work. What better way to encourage managers to leverage AI than to give them an experience of interacting with it that benefits them? As Ethan Mollick points out in his book, Co-Intelligence, humans working with an AI co-intelligence can achieve previously unattainable outcomes, even by highly skilled individuals working alone.
The Future of Learning: AI-Powered, Human-Centered
These examples demonstrate how we're moving beyond whether AI will transform learning and development. Instead, we're focused on how to guide that transformation to create a more human-centered, impactful learning experience. By embracing AI, we can break down traditional barriers, personalize learning at scale, and empower our workforce to thrive in a rapidly changing world. This is the future we're building—one where AI and human expertise work together to unlock the full potential of every learner.
You can’t experiment with AI unless you learn more about it. Want to learn more about how we are courageously experimenting at Udemy? Check out some additional resources here: