Data Driven Product Design Enabled By Application Usage Tracking

A couple of years back I read a book about how buildings are reshaped by their occupants after the architects who built them are long gone. It is called How Building Learn. What happens after they are built.   It got me thinking about what happens to software products after the designers who build them move on to other assignments. I wrote my thoughts down in a post titled Landlords Vs Building Architects to document my thought on how product managers of cloud products needs to think about product design.

Six months back I joined Castlight Health to build a product called Castlight Action. Action is a tool that helps benefits leaders get the most out of their benefits strategy by improving benefits utilization, reducing costs, and improving benefits satisfaction.

Right from day one I requested my team to think like a land lord rather than think just like an architect. I asked them to think not only about building a great product, but also learn how it is used after it is built and use that learning to improve the product in the long run. I told them that we are a cloud product and we have the responsibility to not only build the product but also operate it flawlessly, at a reasonable cost and improve it continuously based on what we learn.

David Tischler under the guidance of Alka Tandon, took responsibility for putting the underpinnings in the product that help us track usage, conduct experiments and, based on our findings, prioritize our roadmap to reshape the product. We are using some cutting edge tools, chosen by Anumeha Goel Dhanrajani from our strategic analytics team, to track not only pages but also every action and every user flow in the application. Rather than make tracking and internal reporting an after thought, we are putting the necessary instrumentation in place behind every event to do the following.

1. Help our product managers with real time information about how how their user flows are performing.
2. Help our user experience designers with real time information about how their experience design is performing.
3. Help our content strategy managers and clinicians with real time information about how each and every piece of content is performing.

Since every one of them will have this information, we are laying the foundation for a better chance of success from day one.

Predictive Insights, Personalized Recommendations and Evaluation of Engagement

I have not written a post in many months. That is because for the past six months I have been spending all my time building a product called Castlight Action. At Castlight Health I have the privilege and honor of leading a team of world class experts in the fields of data science, clinical research, product design, product management and behavior change. I decided to write this post to explain what Action does and shine light on some of the many talented people building the Action product.

Castlight Action's goal is to make American workers healthier by connecting them with better care in a timely manner, at a lower cost. This is a worthwhile cause because American employers spend over $ 620 Billion on healthcare out of which 30 percent is wasted. Despite having the most expensive health care system, the United States ranks last overall among 11 industrialized countries on measures of health system quality, efficiency, access to care, equity, and healthy lives, according to a Commonwealth Fund report.

Castlight Action helps American workers manage their health better at a lower cost to them and their employers. To accomplish this Alka Tandon  and the Action team worked with many large employers to design a unique product that is very much like three products in one package.

First, it is a identification and prediction product. Our data scientists, led by Arjun Kulothungan, and epidemiologist Dr. Tara Yun Zhang, have built models that identify employees who need care and predict their behavior based on a variety of data.

Second, it is a personalized recommendations product. Our clinicians, led by Dr. Zach Landman, and our content and behavior change experts,  led by Christa Dahlstrom, design recommendations that are delivered via email and our web application, called Castlight Essentials. Our personalization experts led by Vicki Mar have designed a personalization engine to deliver the right recommendation at the right time via the right channel to users.

Third, it is a benefits engagement evaluation product. Our Analytics experts led by Sunil Ravichandran have designed data visualizations that track the utilization and inform benefits managers about the efficacy of their benefits design.

The identification and prediction module

Sample recommendation

Evaluation of benefits utilization and engagement

I have always enjoyed building things. I consider Castlight Action among my best and most meaningful work. We hope to reach and improve the health of millions of American workers, increasing their happiness and productivity and reducing the cost of healthcare for hundreds of employers, making them more competitive.

There are few things that I am particularly proud of.

1. We followed the principle that the best interface is no interface. So we have designed the product in such a way that we don't need the benefits manager to do anything. In fact they can get most of the value associated with the product even if they never login to the product. Since, from my SAP days, I understand how busy they are, I was particularly keen on not burdening them with one more tool to administer.  This unique design was done by a team of designers led by Phuong Tran.

2. Unlike traditional analytics products that analyze a snapshot of data, we run our predictive models every day so that timely and relevant recommendations can be made to employees.

3. Rather than wait for the end of the quarter or year to provide benefits leaders with feedback about how their benefits design is doing, Action provides feedback on employee engagement with benefits every day to benefits leaders.

4. Our plan is also to provide insights as a service to all health care partners who serve our customers. This insights as a service design is led by Alvin Cadman.

5. While our predictive models have performed well in initial tests, we believe that we will learn more in the next few months and fine tune even more. Same is the case with our recommendations. We are building a completely new infrastructure to track usage and engagement on a very granular level and make that information available to all product managers so that they can improve the product every month. We are investing in some cutting edge technology for this. This effort is led by David Tischler. I might be able to share more about tracking and continuous improvement in about 6 months.

You can read more about Action in the Castlight Health web site. Please note that all images are from the public website of my employer Castlight Health.

Human Expertise Vs Data

In the movie  Moneyball, there is a scene where Billy Beane the general manager of the Oakland Athletics baseball team discusses the problem the team is facing with the scouts of the baseball team. It is a fantastic scene that depicts the problem faced by human experts in todays data driven world.



Another interesting scene is the one where the an economist from Yale, who works as a data scientist for the Oakland Athletics explains the medieval approach taken by most baseball scouts. He also talks about how to look at the problem differently, from a data point of view.



This different points of view creates friction. If you are developing data driven products for any industry, you will face the problem of friction between subject matter experts and data scientists. Unfortunately for human experts, almost all technologies eventually becomes faster, better, cheaper and win in most cases.

Human experts do play an important role in the process of build data driven products and artificial intelligence products. They train the machine to interpret data and take right decisions. Experts play the important role of teaching machines, like a parent teaches a child. No machine in the world, not even IBM Watson, has inherent knowledge. A human has to train it.

Product managers who understand this and product teams that appreciate this reality and work together will build the most successful data products of the future.

What is IBM Watson and how can product managers use it?

IBM Watson is IBM's artificial intelligence technology that can be taught to answers questions posed in simple English. IBM Watson learns from documents, the same way a human being learns by reading. The difference is Watson can learn from millions of documents in a short period of time and keep up with the growing body of written knowledge on a topic.

IBM Watson learns a topic very quickly.
Experts I spoke with said that when provided with access to adequate written material, IBM Watson can answers questions as accurately as an expert in the field 60% of the time. When an expert coaches IBM Watson for 8 weeks on a topic, IBM Watson can answers questions as accurately as an expert in the field 90% of the time.

You don't program Watson. You teach it
There is no programming language for IBM Watson. Product managers and domain experts can simple teach Watson by providing written material and training it.

I think there are interesting possibilities with Watson in the areas of education and healthcare.

Prototypes For Data Driven Products and A new Role Called Data Product Managers

For many years I prototyped my products very early in the design process and used it a tool for conversation and design participation. I have even tried it for home improvement projects. Recently I started building predictive analytics products that help enterprises identify a segment of people, take actions to influence the behavior of those people, track the engagement of such interventions and measure the eventual impact of such behavior change.

For such products I learned, by trial and error, that we will have to build multiple prototypes. These fall into two main categories; user experience prototypes and data prototypes.

1. User Experience Prototypes
You may have to build a user experience prototype for every persona that encounters the product. In our case, there is a professional user who looks at segments of people and takes action and an end user who receives such interventions.

We built one prototype to show the experience of the professional user and another prototype to show the experience of the end user.

There are good tools to build such prototypes. My team uses Axure RP. It is the best in the business. Product managers can develop skills to use this tool. User experience designers and visual designers also developed skills to build these prototypes within days.

Prototype could also be a sales demo tool
Our visual designer built a version of the prototype with the final visual design screens. We plan to use this prototype for early sales conversations with customers and prospects.

2. Data Prototypes
Data prototypes may fall into multiple categories. The first prototype might be to visualize insight from existing data. The second prototypes might be to visualize insights from engagement data. The third prototype might be to visualize the impact the action taken has had.

The purpose of building a data prototype is the following.

a. Ensure that the data you want to visualize exists.
b. Verify the quality of the data you want to derive insight from.
c. Check to ensure that the data is ready to tell the story you want the data to tell.

A team of experts built the data prototype and determined if the insight we want to visualize were meaningful, useful and reliable. I anticipate that this data prototype significantly improved the quality of our design, improved the communication of the design to our engineers and significantly reduced possible errors in the product. I called the experts who built the data prototype and arrived at the final design, data product managers.

Data Product Managers
Building data products might require skills that traditional software product managers may not have. So I created a role called data product managers who have technical skills such as querying a data base, manipulating it and visualizing the data in a spreadsheet.

If you would like to learn more, drop me a note and I will be glad to share what I can.