Designing the machine that makes the machine

In the past decade, I was hired as a product leader to execute multiple assignments in the human capital management, health benefits and financial services industries. Over the years I learned that when an organization hired me as a product leader they not only hired me to design the machine (the product) but also to design the machine that makes the machine. 

Elon Musk famously pointed out that “The machine that makes the machine is vastly harder than the machine itself”. He added "The extreme difficulty of scaling production of a new technology is not well understood. It is 1000% to 10,000% harder than making a few prototypes."

A product leader is hired to design the machine that makes the machine

Based on my conversations with a few CEOs and business leaders who hired me or wanted to hire me, I determined that the most important value a product leader is expected to bring to the table is the ability to define what the machine that makes the machine is, build it and run it. Very few job descriptions for a product leader (Director, Vice President or Chief Product Officer) call this out clearly, let alone define what that is. Even the hiring managers, who were CEOs of small companies or business leaders in a big company, were not able to articulate this very clearly to me. Instead they said one of the following things. These are real scenarios and quotes from CEOs of companies that hired me or wanted to hire me.

1. Scenario 1: We believe we have a good idea to solve a customer need and have smart engineers who have some solutions. We need a leader to architect the product to ensure that it brings value for customers, validate it with customers and then rapidly scale it for hundreds of customers.

2. Scenario 2: We have identified a major opportunity in our industry. We have smart people who have extensive experience in the industry. We are working with two customers to provide solutions. They are excited about what they see. But every project requires our engineering team and client services team to be involved every day. Our CEO, Client Services and Sales can’t agree on what is priority. We think a product leader can step in and provide a roadmap and deliver high quality products reliably.

3. Scenario 3: We have a product that a few companies bought. We poured smart people and money into it. However we are struggling with delivering the product because there are competing priorities for our customers and we have a hard time delivering anything.
   
   
   Scenarios A, B and C are companies usually at the Series A funding stage if they are private. They may be small divisions within a larger public company. Such divisions resemble a Series A company in terms of purpose, autonomy, responsibility and investment.

4. Scenario 4: We have delivered a very successful product. Millions of users are using it. They love it. We grew rapidly and hired many smart individual contributors who are great at defining and delivering features. However, we need product leaders who make the team work together well to serve customers at a scale we are not used to. The processes, skills, tools, artifacts and frameworks that worked for 1 Million users may not work for the number of users we have now. We need a product leader who can bring experience, hire new talent, mentor them, help them develop professionally, develop new leaders, bring wisdom, and execute with discipline in a regulated industry.  

Let’s now look at how a machine could be designed to meet these needs.

Ray Dalio, the leader of the most successful hedge fund in the world said that the role of a leader is to design a machine that can produce outcomes that meet an organization’s goal. He defined the machine as people and culture. I extended that framework a few years back to include what a product leader’s machine for delivering outcomes for a product company might look like. This is based on my experience executing about 4 assignments in the past decade. It has evolved over time.

Figure: The Product Management Machine

This is how a product leader can go about using this framework when starting a new assignment within the company or in a new company. This could be the first 90 day plan for a product leader.

Before accepting the assignment.

1. Determine the maturity level of product management in the organization. Are you the first product leader? Is product management at an experimental stage? Is it valued and respected? The importance of product management for a company may change over time.

2. Understand the mission of the company. For example, the mission could be to ‘democratize financial advice’, or ‘simplify navigating the healthcare system’. Ensure that you believe in that mission and its impact on society.

3. If you are not familiar with the industry or subdomain, learn and get certified in the specific subdomain. For example, a year before I joined Castlight Health, I got certified in data science and machine learning from Johns Hopkins. When I joined the financial services industry, I called a few of my expert friends and learned the domain through a few work sessions. Recently I added some certifications on Financial Technology from Wharton University. Certifications and learning by conversations are effective ways to understand the domain. Now-a-days, you can learn almost any topic from top universities in a matter of weeks. One of the main skills a product leader needs to have is to know how to learn ten times faster than normal. I highly recommend this course that teaches you how to learn.

After accepting the assignment but before starting it.

1. Listen  to engineering leaders to build credibility. Meet them for coffee or over the phone to understand their point of view and challenges. Convey that you are there to serve them well and you value engineering.

2. Listen to client services leaders or user growth leaders and the support team leader to understand their challenges. Since they are responsible for retaining customers, they can give you raw insights into what is happening on the ground.

Month ONE

1. Understand the top three priorities for the year from the CEO or the business leader responsible. For example, when I joined Jemstep, my manager and Jemstep CEO, Simon Roy made it clear to me that delivering the product to the first major bank customer was the top priority. At Castlight Health, my manager and current CEO Maeve O’Meara made it clear to me that delivering the predictive analytics and programmatic marketing product to Walmart and Mondelez was the top priority for the first year.
   
   

2. People: It is important to understand people, their motivations and their character as quickly as you can.  Once you get the right people in the bus, the bus can be taken to the right destination even if it went in the wrong direction for a few days or weeks. But the wrong people in the bus will ensure that you fail. 
   
   

1. Focus on people. Understand their current assignments and individual feature roadmaps. Recognize overlaps on assignments or features without a clear custodian.

2. Understand their aspirations and mental models.

3. Learn about their prior experience and place them in the pragmatic product management framework. For example, some may be closer to customers and not very close to engineering. Some may have deep experience in analytical thinking but not much experience in day to day execution. Some may be good at design thinking.

4. Look at the artifacts they produce.

5. Assess their mastery of the craft of product management.

6. Design a learning map for everyone based on skill levels and gaps.

7. Do not make any changes to people or organizations at this time.

Unfortunately there is no college degree for product management. It is likely that you will have a team or hire a team of people with varied backgrounds. All product managers irrespective of their responsibilities need to have a basic set of skills. For example, they needed to know how to write a user story. Verify the artifacts of even the most experienced product managers. You would be surprised by the lack of current skills. I normally put everyone through Pragmatic Product Management and key product managers through IDEO’s Design Thinking training programs.

3. Process: The purpose here is to understand if there is a commonly understood way of doing things to collaborate with customers or users, define features, design them, define them for engineering and deliver them.
   
   

1. Understand if there is a process for getting things done.

2. See actual artifacts delivered by the product and design teams. A person’s writing pretty accurately reflects their thinking, knowledge, communication skills and attention to detail. Written communication is critical in today’s world of distributed teams.

3. Understand how the product team collaborates with engineering. Impress upon them that they need to partner with engineering and earn their trust to be successful. 

4. Understand the sharing of responsibilities between product managers and designers. 

5. Make  minor process changes if needed. But do not change the process dramatically in the first month.

MONTH  TWO

4. Platform: The product management team and the design team need effective tools to do their work. These tools do not have to be expensive. Many of these tools can be simple document creation tools or spreadsheet based tools. Some will be off the shelf tools. Ensure that everyone uses them and is skilled at using them.

1. Understand tools used for product definition and design

2. Understand the tools used to deploy or implement the product.

3. Understand the underlying technology platform from the engineering leaders.

5. Product Design: You have to be familiar with the product in month 1. However, in month 2 you need to develop a deeper understanding of every persona, every user Journey, and every possible alternative flow. You may have to work with product managers and designers to accomplish this.

1. Review every persona and the job they want the product to do.

2. Review every user journey in the product. Ask the team for the user flow diagrams. Ask them to create it if needed.

3. Review an entity relationships diagram if available with engineering. If not ask senior product managers to create an entities and logical relationships diagram. Train them to create one if required.

4. Visit three top customers or prospects and conduct co-innovation workshops . If possible, spend a day at your customer support team’s offices and listen to 50 calls. I have not done this in my previous jobs. Instead I listened to recordings of calls that my colleagues made at Castlight. However, colleagues who did this in-person told me that it is very valuable. I plan to do this for future assignments.

MONTH THREE

6. Product Strategy: By month three it will be clear if the team is executing towards the company’s goals. However month 3 is the time to take a deep look at data to verify if the current and planned roadmaps bring value to customers. It is also time to verify if the company is getting back some of the value or will get back some of the value in the future.

1. This is the time to ask senior product managers to perform ROI analysis on major functionality if they have not already done so. 

2. This is also a good time to have a workshop with the senior product managers to review if the features under development align with the goals of the company. Shreyas Doshi of Stripe has done a very good job of defining good and bad product strategy. It is worth a read to understand what needs to be done.

Communicating your work to the organization.

It is important to communicate what you are doing as a product leader to those who do the work following your directions and senior leaders who expect you to produce the desired outcomes. I normally send a weekly work plan email to critical leaders and colleagues outlining the main priorities of the week and what I plan to do and who I plan to work with. It serves many purposes. It helps the product leader earn credibility with product, engineering and key business leaders. It keeps the CEO and senior leaders informed about the areas of focus for the week. More importantly it also points out what the product leader is not planning to do. It gives others an opportunity to point out important tasks if they don’t see it. This is hard to do. But very valuable.

Applying this framework

This framework is based on my experience. It may not apply to every scenario. I am certain it won’t apply to any scenario and organization without some modification. I also want to point out that this is a machine that requires continuous monitoring and nurturing. It is not a self-driving car. It needs a driver, the product leader.

Product leaders could use this framework as a starting point when they start an assignment or a new job. Product managers who aspire to become product leaders could use this framework to understand how their product leaders might be approaching their work. CEOs and business leaders considering hiring product leaders (Directors, Vice Presidents, Chief Product Officers) can use this framework to understand what a product leader can and should bring to their organization. If you use it, let me know. I look forward to learning from all of you.

Ideation and Innovation In The Product Design and Development Life Cycle

For the the past 8 years or so, I have been designing, developing and delivering cloud based enterprise software in a quarterly cadence for use by employees and customers of organizations. This model is normally referred to as business to business to consumer (B2B2C) model. I have done this in the human capital management, health benefits and recently in the financial technology industries. In all these organizations, I followed a quarterly development cycle where senior executives and product leaders set the business goals for a year and product managers defined features that need to be developed to meet those goals with quarterly planning and delivery cycles. 

When developing functionality rapidly quarter after quarter, I noticed that sometimes product teams become delivery oriented and lose their ability to innovate. They stop meeting users and customers in a planned and systematic way. Market research and innovations becomes the responsibility of a few individuals. No matter how smart those individuals are, they eventually run out of energy, ideas, and lose touch with the capabilities of the organization. When this happens, product managers who deliver a product, start looking up to senior executives to point them to the next big thing. If they do not hear a clear direction from senior executives, they become demotivated and may leave the organization or become less engaged with the with the mission of the company. Slowly but steadily, talented people leave and the organization fails.

There is another extreme scenario that happens in some startup organizations where there is no successful product yet. In such organizations, every work session and every meeting becomes a new product brain storming session with no frameworks or business processes to listen to the voice of the market, identify the pain of a customer, test the feasibility of a solution, confirm a hypothesis, clarify the details of a solution, pilot it for a customer and build a product in a phased manner. In this scenario, the organization makes poor investment decisions, product decisions and commercialization decisions. The organization either fails or takes a very long time to become successful.

These are not  good results for any organization.

I believe it is necessary for the sustainability of an enterprise software business to incorporate the process of ideation and innovation in the product design and development process. It is also important to tap the creativity and talent of many people in the organization rather than rely on the vision of one or two people in the company. I developed the following framework a few years back to do this. It has worked well for me and has helped me created products that bring multiple millions in annual recurring revenue. Please note that this framework has been tested only in business to business to consumer software models. This is designed to work in an existing medium sized software organization with 100 to 500 employees. Smaller organizations may take a different more nimbler approach. 

1. Proof of concept : This is the stage where executives have a hypothesis about a market need and a product manager or engineer has an idea to build a solution. At this stage, one senior product manager working with a senior engineering architect builds a proof of concept to test the feasibility of the solution. They may show the solution to end consumers and get their inputs at this stage. No formal customer co-innovation or user testing is done. At the end of this session they produce a functioning proof of concept that proves that their idea works. I normally prefer to make this a project to be completed within a quarter. This proof of concept is formally tracked as a quarterly objective. It should take about 25 percent of an experienced and motivated product manager's time and about 25 percent of an engineering architect's time.. That translates to about 200 hours of work in a quarter.

2. Co-Innovation: Once a solution is proven to be feasible with the capabilities available in the organization, it is time to go meet 2-3 customers for 2 hours work sessions. Product managers will have to create detailed concept stories with low fidelity mock-ups to tell the story to customers. usually 2-3 concept stories explaining 2-3 user cases will work. Once they complete the co-innovation sessions with 2-3 customers or prospects, they should be able to define a possible solution and prototype it using medium fidelity mocks. It is important that they prototype almost 90 percent of the functionality to ensure that prototype conveys the user experience and value created for a customer. Co-Innovation sessions require senior product leaders who understand the design thinking process, a user experience designer and a sales executive. The sales executive's role is to guide the team to work with appropriate prospects and customers who are most likely to co-innovate and most likely to buy the proposed solution. The co-innovation session should be led by the product team and not the sales team. If done well, this should lead to a paid pilot project with a customer.

3. Pilot: This is the stage where one customers is willing to pay to test the solution. At this time, it is important to commit to the delivery of the solution as long as the customer signs a contract or statement of work. The project will be considered a pilot only if there is a paying customer. If no customer is willing to pay, then the co-innovation process continues until the product meet the needs of a customer. Even when one customer is using the solution, it is still not appropriate to call it a product, until the solution has been implemented for at least two customers.

4. Minimum Viable Product: I consider any functionality or solution to be a product, even a minimum viable one, only when it meets the needs of at least two paying customers. Once the solution has been implemented for two customers, the sales team can start selling the product to several customers. This stage will require multiple senior product managers  and multiple engineering teams working full time to build features that meet the needs of the market and strategic customers. It is important to note that a product considered minimum viable for one customer may not be so for another customer. this will be the case until the product reaches about 10 customers.

5. Growth: When a product goes beyond two customers, it is time to think about scaling the product for dozens of customers. This is when a platform product manager, usually a senior leader in the team, and a senior engineering architect join the team to build foundations such as configuration tools, reports, scalable infrastructure and data models. This usually takes about on 1-2 product managers working with multiple engineering teams for 2-3 quarters. The skills required to do this are very different from the skills required to build a minimum viable product.

The key take aways:

• It is possible to incorporate an innovation framework into the quarterly product planning and development process to tap into the talent and creativity of many people in the organization in a systematic and sustainable manner.
• Product organizations that recognize the different stages of product design and development and apply different execution frameworks to each type of work will outperform those who don't.

Executive teams that recognize this and invest in such frameworks and people will outperform those who do not do so.

The Anatomy of an Effective Product Definition Document

Imagine a tool that not only organizes your team's thoughts but also becomes the cornerstone of collaboration and debate in product development. That's the power of a well-crafted product definition document. As a Vice President of Product Management, I've seen firsthand how this crucial artifact serves dual purposes. It's not just about gathering ideas; it's a platform for engaging discussions with customer success managers, professional services teams, and the product team itself.

Empowering Product Managers with a Structured Approach

During each quarterly planning stage, I encourage every product manager, especially the newcomers, to craft a product definition document. For those unfamiliar with this process, I provide a structured framework. This framework is designed to:

1. Develop empathy for the user.
2. Garner inputs from customer success and professional services.
3. Foster collaboration with technical architects during the definition phase.
4. Shift the mindset from 'inventor' to 'custodian' of the solution.
5. Encourage the development and defense of a unique point of view.
6. Build credibility in proposing solutions.

Distinguishing the Product Definition Document

It's crucial to understand that a product definition document is distinct from a vision document. Its primary function is to outline functionalities achievable within a planning period, typically a quarter or 3 months.

Key Characteristics of an Effective Product Definition Document

1. Meaningful Title: The title should be clear and understandable across the organization. A well-chosen title, like "Contact Center Agent Experience for Web Chat - Phase 1," indicates thoughtful consideration of the problem's nature and scope.
2. Date: Including the creation and last updated dates provides valuable context, especially when the document is revisited months later
3. Authors and Contributors: This section should reflect a diverse range of inputs, highlighting the collaborative nature of the project.
4. Document Purpose: Clearly articulate whether the document is for collaboration, development readiness, or requires legal or clinical review. This reflects the product manager's awareness of the document's audience and its multi-stage development process.
5. Key Word Explanation: Define key terms in simple language to avoid confusion and facilitate clear, concise discussions. For example, explaining terms like "Sweepstakes" or "Call Disposition" ensures everyone is on the same page.
6. Executive Summary: This is not just for executives; it's a concise overview for all readers, outlining the user persona, their challenges, and the proposed solutions. A well-written summary is often a hallmark of an experienced product manager. The executive summary should explain the JOB the customer is trying to get done. The PAIN they will experience, if they do not have a solution and the GAIN. The gain is the solution and its value proposition for the user. For example, a tenant in an apartment wants to get a broken faucet fixed quickly by explaining the problem to the customer service person of the property manager. In this case the JOB is to resolve the problem fast. The PAIN is wasted time and a flooded apartment. The GAIN: Attaching a picture of the leaking faucet to her text message to the customer service team will explain the problem better and will lead to faster resolution.
7. User Interaction Diagram: Include a persona description, a simple, abstract diagram showing how the user will interact with the product or feature. This visual element can be more effective than lengthy descriptions. Figure: Sometimes all it takes is a hand drawn diagram.
8. Feature Details: Use sentences and diagrams to describe the feature. This section evolves with each work session, often containing links to detailed use cases, mocks or prototypes.
9. Risks, Assumptions, and Limitations: A transparent discussion of these aspects shows a realistic understanding of the project scope.
10. Acceptance Criteria: Define what criteria will be used to verify the completion of work.

Conclusion: The Art of Conciseness and Clarity

The best product definition documents are concise, typically starting as a 2-page document and potentially extending up to 5 pages. Anything more comprehensive is better suited for a prototype or can be broken down into multiple phases. As a product manager, your goal is to convey your vision clearly and succinctly, setting the stage for successful development and collaboration.

When A Product Says It Uses Machine Learning What Does It Mean?

I was at my employer's annual sales conference recently where many digital health vendors and their CEOs were present. In one of the sessions they all said that they use machine learning to personalize their product for users. The CEOs did not go into any details. I could tell that some of my colleagues and partners in the audience were skeptical, but they did not ask the leaders of those companies to elaborate. I suspect that most people in the audience did not understand what the leaders of those companies meant when they said they use machine learning. Unfortunately that might be the case with many enterprise software buyers.  But it does not have to be that way. While designing machine learning driven products is tough and requires a highly skilled team of data analysts, data scientists data engineers, and product managers, the basic concept of machine learning is quite simple. In this post, I will take a few examples from the real world and my experience at Castlight, building machine learning driven products, to explain what a machine learning model is, what a simple rules based model is, and how those models are used in the real world to benefit employees of companies that buy software driven by such models.

At Castlight Health, we make it simple for employees of American companies to navigate their healthcare, benefits and wellness programs. We provide employees with a web application and a mobile application where they can see health benefits information personalized to their needs. To do this, we use a combination of rules based and machine learning based models. This is a hypothetical example of a rules based model. "If a pregnant woman is over 35 years of age, place her in a segment called high-risk-pregnancy". This model is a not a machine learning driven model. It is driven by clinical rules written by experienced clinicians. Software developers simply listen to clinicians and turn that rule into a software program. This information is then given to our personalization engine. Once our personalization engine understands a woman is in the high-risk-pregnancy segment, benefits programs that are relevant for a woman with high risk pregnancy are promoted to her via our web, mobile and email channels. When she logs into our application, instead of viewing information about all the benefits her employer provides, which could be exhausting, she will see benefits that are relevant for her. The hypothesis is that such personalization makes employees aware of relevant benefits, engages them with the benefits providers and wellness programs in a timely manner, improves their health, and reduces healthcare costs for them and their employers. This actually works. That is why hundreds of employers pay us tens of millions of dollars every year. The example we saw above is rules based prediction and personalization. You may be wondering about an example where machine learning is used to predict and personalize. To understand that we need to first understand what machine learning is.

What Is A Machine Learning Learning Model?

A machine learning model is where software developers do not program the computer (the machine) with an explicit logic. Instead they make the machine learn by training the machine with historical information. For example, if we want to check if an email is spam or not spam we can create a machine learning model. To do this data scientist will take several known spam emails, label them as 'spam' and feed those to a suitable computer program, 'the machine'. Data scientists will not say why the email is spam. They will simply say "My dear machine, this is a spam email. I want you to look at the email and recognize that this email is spam". So the machine learns what a spam email might be like and, after looking at enough spam emails, gets better and better at identifying a spam email. After it gets really good at identifying spam email, the model is deployed and goes to work identifying spam email in the real world and putting them in the junk folder.  It is important to note that data scientists in most product teams don't invent the computer algorithms they use. They simply use an existing computer algorithm and build a model. It is a bit like this. An electric car engineering team does not have to invent the electric motor. They just have to design it for the particular type of car and build it.

Image Courtesy: Europeana Collections

A Machine Learning Model use Case For Benefits Navigation

We just looked at a real world example of a machine learning model. I now want to give you a hypothetical use case of a machine learning model in a health navigation product such as the one Castlight Health provides. Let's say that employers want employees who get an unnecessary back surgery to get a second opinion before they decide on the back surgery. This is because clinicians know from experience that many back surgeries do not improve the condition of a person's back. Instead they cost a lot of money for the employer and the employee and cause a lot of pain and suffering for the employee. In most cases, a surgery also results in weeks of time off from work and, in some cases, lost wages for the employee. So there is a big incentive to identify people who might get a back surgery and make them aware of second opinion programs as well as inform them about the costs and benefits of back surgeries. The problem is there is no simple rule to find out who might be considering a back surgery. This is where machine learning comes handy. Castlight data scientists have access to de-identified information about the medical history of people who had a back surgery. They can feed that information to a computer algorithm (the machine) and tell that machine "My dear machine, this is the medical history of people who had a back surgery in the past. I don't know why they got a back surgery. But they all did. I want you to look at this data and learn to identify people who are likely to get a back surgery." With enough data the Castlight machine learning models gets really good at identifying people who are likely to get a back surgery. The model is then deployed to analyze medical data of employees and predict if someone is likely to get a back surgery. Once it identifies such people, the model informs the Castlight personalization engine about this. The personalization engine then goes to work, promoting second opinion programs and educational information to those identified via web, mobile and email channels. Once again, the hypothesis is that even if the product prevents only a few unnecessary back surgeries a year, the cost savings could be in the hundreds of thousands of dollars and the health improvements could be significant too.

I avoided technical terms on purpose in the above examples. Data scientists reading this post, will recognize that what I described above is supervised learning. There are other types of machine learning, which I did not go into in this post.

If you are an enterprise buyer or a sale person competing against another product claiming to be machine learning driven or artificial intelligence driven, ask the product manager or the sales person to explain the use case. If they are not able to explain in simple terms what they are using machine learning and how it turns into real value, you should be very skeptical about their claims and verify before buying their software. A product does not have to be machine learning driven to be good. Simple rules based engines can do a good job to address many problems. However, it is important to understand the difference.

Please note that for business reasons, I did not use actual use cases. I also did not go into more details about our personalization engine, and our system of intelligence, which are far more sophisticated than what I outlined here. If you would like to learn more, contact me or my colleagues at Castlight Health and we will be glad to share more. If you are in the San Francisco Bay area drop by at our San Francisco or Mountain View offices and I will be glad to give you a demo of our products. If you like this kind of work and want to join Castlight, give me a call. We are always looking for good data scientists, data engineers, clinicians and product managers who understand machine learning and data driven products.