Launching a data driven recommendations product is a bit like launching a rocket

At Castlight Health we recently launched our first data-driven recommendations product called Castlight Action. We launched it for our own employees first before launching it for customers. It has four modules. The first one is the identification and prediction engine that segments people, the second is the application for the benefits leaders to turn on campaigns, the third is the delivery of email and web-based recommendation content. The fourth is the module that tracks the engagement of members with recommendations.

These modules start to execute one after the other. Every module depends on the successful execution of the previous module. Every module has a product manager in-charge of it. I was watching the leader of every module take a deep breath and verify things when their modules go live. Nerve wracking, in a good way.

While we are operating in new territory, my plan is to make the process of desining and developing data driven products a lot more predictable than it is today. I hope to learn from others who have done it and learn by doing when I can. 

Beautiful Evidence

I am in the middle of designing a data driven product for the benefits industry. As part of my prep, I went through the book Beautiful Evidence by Edward Tuft to understand the purpose behind every kind of data visualization. As you probably already know, it is a wonderful book. If you are someone who cares about the intersection of data and design, the book is worthy of keeping your library for a long time.

The most important lesson for me from the book is this. Don't let any digital tool dictate your thinking. Break the rules and disrupt the rules of the tool.

I was also fascinated to learn about the rich possibilities of sparklines. 

Data Driven Product Design Enabled By Application Usage Tracking

A couple of years back I read a book about how buildings are reshaped by their occupants after the architects who built them are long gone. It is called How Building Learn. What happens after they are built.   It got me thinking about what happens to software products after the designers who build them move on to other assignments. I wrote my thoughts down in a post titled Landlords Vs Building Architects to document my thought on how product managers of cloud products needs to think about product design.

Six months back I joined Castlight Health to build a product called Castlight Action. Action is a tool that helps benefits leaders get the most out of their benefits strategy by improving benefits utilization, reducing costs, and improving benefits satisfaction.

Right from day one I requested my team to think like a land lord rather than think just like an architect. I asked them to think not only about building a great product, but also learn how it is used after it is built and use that learning to improve the product in the long run. I told them that we are a cloud product and we have the responsibility to not only build the product but also operate it flawlessly, at a reasonable cost and improve it continuously based on what we learn.

David Tischler under the guidance of Alka Tandon, took responsibility for putting the underpinnings in the product that help us track usage, conduct experiments and, based on our findings, prioritize our roadmap to reshape the product. We are using some cutting edge tools, chosen by Anumeha Goel Dhanrajani from our strategic analytics team, to track not only pages but also every action and every user flow in the application. Rather than make tracking and internal reporting an after thought, we are putting the necessary instrumentation in place behind every event to do the following.

1. Help our product managers with real time information about how how their user flows are performing.
2. Help our user experience designers with real time information about how their experience design is performing.
3. Help our content strategy managers and clinicians with real time information about how each and every piece of content is performing.

Since every one of them will have this information, we are laying the foundation for a better chance of success from day one.

Predictive Insights, Personalized Recommendations and Evaluation of Engagement

I have not written a post in many months. That is because for the past six months I have been spending all my time building a product called Castlight Action. At Castlight Health I have the privilege and honor of leading a team of world class experts in the fields of data science, clinical research, product design, product management and behavior change. I decided to write this post to explain what Action does and shine light on some of the many talented people building the Action product.

Castlight Action's goal is to make American workers healthier by connecting them with better care in a timely manner, at a lower cost. This is a worthwhile cause because American employers spend over $ 620 Billion on healthcare out of which 30 percent is wasted. Despite having the most expensive health care system, the United States ranks last overall among 11 industrialized countries on measures of health system quality, efficiency, access to care, equity, and healthy lives, according to a Commonwealth Fund report.

Castlight Action helps American workers manage their health better at a lower cost to them and their employers. To accomplish this Alka Tandon  and the Action team worked with many large employers to design a unique product that is very much like three products in one package.

First, it is a identification and prediction product. Our data scientists, led by Arjun Kulothungan, and epidemiologist Dr. Tara Yun Zhang, have built models that identify employees who need care and predict their behavior based on a variety of data.

Second, it is a personalized recommendations product. Our clinicians, led by Dr. Zach Landman, and our content and behavior change experts,  led by Christa Dahlstrom, design recommendations that are delivered via email and our web application, called Castlight Essentials. Our personalization experts led by Vicki Mar have designed a personalization engine to deliver the right recommendation at the right time via the right channel to users.

Third, it is a benefits engagement evaluation product. Our Analytics experts led by Sunil Ravichandran have designed data visualizations that track the utilization and inform benefits managers about the efficacy of their benefits design.

The identification and prediction module

Sample recommendation

Evaluation of benefits utilization and engagement

I have always enjoyed building things. I consider Castlight Action among my best and most meaningful work. We hope to reach and improve the health of millions of American workers, increasing their happiness and productivity and reducing the cost of healthcare for hundreds of employers, making them more competitive.

There are few things that I am particularly proud of.

1. We followed the principle that the best interface is no interface. So we have designed the product in such a way that we don't need the benefits manager to do anything. In fact they can get most of the value associated with the product even if they never login to the product. Since, from my SAP days, I understand how busy they are, I was particularly keen on not burdening them with one more tool to administer.  This unique design was done by a team of designers led by Phuong Tran.

2. Unlike traditional analytics products that analyze a snapshot of data, we run our predictive models every day so that timely and relevant recommendations can be made to employees.

3. Rather than wait for the end of the quarter or year to provide benefits leaders with feedback about how their benefits design is doing, Action provides feedback on employee engagement with benefits every day to benefits leaders.

4. Our plan is also to provide insights as a service to all health care partners who serve our customers. This insights as a service design is led by Alvin Cadman.

5. While our predictive models have performed well in initial tests, we believe that we will learn more in the next few months and fine tune even more. Same is the case with our recommendations. We are building a completely new infrastructure to track usage and engagement on a very granular level and make that information available to all product managers so that they can improve the product every month. We are investing in some cutting edge technology for this. This effort is led by David Tischler. I might be able to share more about tracking and continuous improvement in about 6 months.

You can read more about Action in the Castlight Health web site. Please note that all images are from the public website of my employer Castlight Health.

Human Expertise Vs Data

In the movie  Moneyball, there is a scene where Billy Beane the general manager of the Oakland Athletics baseball team discusses the problem the team is facing with the scouts of the baseball team. It is a fantastic scene that depicts the problem faced by human experts in todays data driven world.



Another interesting scene is the one where the an economist from Yale, who works as a data scientist for the Oakland Athletics explains the medieval approach taken by most baseball scouts. He also talks about how to look at the problem differently, from a data point of view.



This different points of view creates friction. If you are developing data driven products for any industry, you will face the problem of friction between subject matter experts and data scientists. Unfortunately for human experts, almost all technologies eventually becomes faster, better, cheaper and win in most cases.

Human experts do play an important role in the process of build data driven products and artificial intelligence products. They train the machine to interpret data and take right decisions. Experts play the important role of teaching machines, like a parent teaches a child. No machine in the world, not even IBM Watson, has inherent knowledge. A human has to train it.

Product managers who understand this and product teams that appreciate this reality and work together will build the most successful data products of the future.