Author Archives: kathifletcher

Building the RIGHT Thing

Product managers are responsible for creating and delivering the right thing to their customers. 

Product Managers are sometimes called product ‘CEO’s’, because of their central position of authority with respect to the product, and because of the need to communicate the vision for the product both to an audience of ‘investors’ (the company leadership) who must provide people and resources, and to their own team who must deliver the product. They are not ‘owners,’ however. Ownership conveys the wrong set of skills. Product Managers aren’t buying and investing as an owner would, and they aren’t deciding where the team invests time and energy based on their own personal preferences as an owner would. Instead, a good Product Manager has to learn the needs of their customers, and figure out how their team can fulfill them to achieve the organization’s strategic goals. 

Great product managers are skilled at determining what the right thing to build is with the consumer in mind, communicating that to leaders, colleagues, and team members, and then working productively and flexibly to deliver the right value to their customers. 

I recently gave a talk to Rice University Computer Science Alumni as part of a panel on Product Management as a career, and you can access my talk and slides below. In this series of posts, I will be going into a lot more depth to explain  more ‘tools’ for the product management toolbox and how to use them effectively to create useful and beneficial products.

Panel: (Minute 36:44)Slides: 

Crowd 'speaking' with labels "Level Up!, 5 stars, A+"

Leveling up crowd-sourced educational content — with a little help from machine learning and lessons from HCOMP19.

More lessons from HCOMP 2019

Crowd-sourcing content: Although publishers (I work for one) create high quality content that is written, curated, and reviewed by subject matter experts (SMEs, pronounced ‘smeez’ in the industry), there are all sorts of reasons that we always need more content. In the area of assessments, learners need many, many practice items to continually test and refine their understanding and skills. Also, faculty and practice tools need a supply of new items to ‘test’ what students know and can do at a certain point in time. Students need really good examples that are clearly explained. (Khan Academy is a great example of a massive collection of clearly explained examples). When students make attempts to solve homework problems, they also need feedback about what NOT to do and why those aren’t the correct approaches. Therefore, we need to know what the core concepts needed for each activity or practice are. 

Faculty and students are already creating content! Because faculty and students are already producing lots of content themselves as part of their normal workflow, and faculty are assembling and relating content and learning activities, it would be great to figure out how to leverage the best of that content and relationship labeling for learning. This paper by Bates, et. al, looked at student generated questions and solutions in the Peer Wise platform ( and found that with proper training, students generate high quality questions and explanations. 

So at HCOMP 2019 I was listening for ways to crowdsource examples and ground truth. In particular, it would be useful to see if machine learning algorithms could find faculty and student work that is highly effective at helping students improve their understanding. The two papers below address both finding content exemplars and training people to get better at producing effective content. 

Some highly-rated examples are better than others. Doroudi et. al wanted to see what effect showing highly rated peer-generated examples to crowd workers would have on the quality of work they submitted. In this study, the workers were writing informative comparison reviews of products to help a consumer decide which product better fits their needs. The researchers started with a small curated ground truth of high quality reviews. Workers that viewed highly rated reviews before writing their own ended up producing better reviews (more highly rated). That isn’t surprising, but interestingly, even among equally highly rated reviews, some reviews were much more effective in helping improve work! They used machine learning to determine the most effective training examples. So that suggests that, while viewing any highly rated examples will improve new contributors’ content,  we can then improve the training process even more by selecting and showing the examples with the best track record of improving workers’ content creations.  

Using training and leveling up:  Madge et. al introduced a crowd-worker training ‘game’ with a concept of game-levels. They showed the method was more effective than standard practice at producing highly effective crowd workers. Furthermore, they showed how machine learning algorithms could determine which tasks belonged in each game level by observing experimentally how difficult tasks were. 

Crowd workers are often used to generate descriptive labels for large datasets. Examples include tagging content in images (‘dog’), identifying topics in twitter feeds (‘restaurant review’), and labeling the difficulty of a particular homework problem (‘easy’). In this particular study, workers were identifying noun phrases in sentences. The typical method of finding good crowd workers is to start out by giving a new worker tasks that have a known “right answer” and then picking workers that are best at those tasks to do the new tasks you actually want completed. The available tasks are then distributed to workers randomly, meaning a worker might get an easy or difficult task at any time. These researchers showed that you could train new workers using a ‘game’, so that they improve over time and are able to do more and more difficult tasks (harder and harder levels of the game), and the overall quality of labeling for the group of workers is better.  

Better education content: Faculty and students could become more effective producers of education content with the help of these two techniques. Motivating, training and selecting contributors via comparison with highly rated examples and leveling up to ‘harder’ or more complex content would be useful to help contributors to create high quality learning content (example solutions, labeling topics and difficulty, giving feedback). These techniques also sound really promising for training students to generate explanations for their peers, and potentially to train them to give more effective peer feedback. 

personified lock (unlocked) fighting personified corona virus

Open Vs. COVID Round 2: Collaborating for the Knockout

Opening up information is one of the keys to a concerted and effective response to the COVID-19 Pandemic. We need to make sure that insights from the world-wide treatment efforts are discovered quickly and shared widely to ensure effective preventions and treatments spread. A case in point on the power of coordinating and sharing medical information comes from the Castleman’s Collaborative, illustrating how open processes and open resources can be an asset to humanity in this global health crisis.

“Doing in a year what often takes a decade.1 I recently caught Terry Gross’ interview with David Fajgenbaum on Fresh Air about ‘crowdsourcing’ a cure for Castleman’s disease, which he suffers from. It’s a great listen. Castleman’s is a deadly disease for which little is known because of its rarity and the dispersal of cases. Luckily for the world, Fejgenbaum was in medical school when he had his first attack and, because he was close to finishing his medical degree, he began researching.  He discovered that little coordinated information and study was available. The Castleman’s Collaborative,, has come up with an eight step research process that is crowdsourced, prioritized, funded, executed, and published (almost all freely available). Much of the research targets off-label use of FDA-approved drugs, since development of specifically targeted new drugs is prohibitively expensive and time consuming, but many, many drugs already exist and might be effective. While, individually, rare diseases impact few people, cumulatively, many many rare conditions impact large numbers of people, and so applying this collaborative method can be tremendously impactful. 

They are applying the same collaborative method to COVID19 treatment: The Castleman Collaborative is now applying the methodology to COVID19, Their first step is compiling a database of off-label use of drugs to treat COVID19 symptoms, aptly named CORONA for COvid19 Registry of Off-label & New Agents. Having that database openly available to those around the world who are researching treatments helps prevent duplication, combine efforts in the medical community, and lets researchers and practitioners build on each other’s knowledge, which has always been the promise and practice of science.  

‘Open’ is a critical tool for solving hard, global problems. Open resources (education, data, software) and collaborative processes are unique in their ability to pivot to address new crises and public concerns, because they remove barriers to building on previous work and disseminating knowledge quickly and widely. In the case of the pandemic, this gives open collaboration a fighting chance against the virus, which also spreads widely and builds on its own infectious success. My day job is helping students learn and achieve by building effective products, which also benefits from and is accelerated by open content, open research, and collaborative team processes. My interest in open software started in graduate school and the more I learn, the more I believe in its potential to help the world. In this global health crisis, open solutions are continuing to establish themselves as an integral part of the thriving open software ecosystem that I am proud to be a part of. 

1 From the Castleman Collaborative website, July 6, 2020

Student sleeping akimbo at desk covered with books.

The Second Shift: Does homework fly in the face of current productivity research?

Students are doing homework after a full day, and may be caring for siblings, working, and helping out at home. Some of them don’t have adequate tech or space to work. Homework is a second or third shift for them and may be increasing educational inequity.

Is ed-tech exacerbating inequity?

I have been thinking a lot about where ed-tech might be exacerbating existing inequity. And that led me to read a colleague’s tweet of “Homework is a Social Justice Issue”, originally published here in 2015. It talks about the underlying assumptions being made when we give homework, especially in K12: that students have the time, background knowledge, and tools to do school work at home. If students are working or taking care of younger siblings, they don’t have the time. If the type of work often lures in parents of affluent students to help, then they probably don’t have the background knowledge yet. And if the homework is on a laptop/phone and requires internet access, or requires space to organize and maintain materials, they may not have the tools. We must take these environmental realities into account when designing and building educational software that will meet the needs of students from all walks of life. 

Long hours don’t work.

I recently realized that the people I meet with after 4pm aren’t getting the same creativity and deep listening as people that talk to me at 9 or 10am. It made me wonder why we are asking students to do a second or often third shift, when the research on the harms of long hours to productivity of overwork are so clear (here’s a summary of the harms) and similarly there are real harms to work quality (see this study on long medical shifts). Do you want someone in their 18th hour doing brain surgery on you? 

When and what to assign?

So, even IF students have the time, background knowledge, and tools, does it really make sense to ask them to work a second shift? Students do need time to grapple with hard problems, and many students need quiet to work. So it isn’t an easy problem to fix. The article suggests that if you are assigning homework in K12, you should ask yourself these questions.

  1. “Does the task sit low on Bloom’s Taxonomy? In other words, are students likely to be able to do it independently?
  2. If not, does the task build primarily on work already performed or begun in class? In other words, have students already had sufficient opportunity to dig deep into the task and work through their difficulties in the presence of peers and/or the teacher?
  3. Does the task require only the technology to which all students have sufficient access outside of school?
  4. Can the task reasonably be accomplished, alongside homework from other classes, by students whose home life includes part-time work, significant household responsibilities, or a heightened level of anxiety at home?”

How could ed-tech help? 

Homework systems and courseware could make it easy and safe for students to provide feedback on their assignments, including individual questions and tasks within their assignments. Rather than focusing so much on giving analytics about students, ed-tech could provide instructors with analytics about the assignments, questions, and tasks they give. Which ones seem to require a lot of prerequisite knowledge that students don’t already have? Which ones seem to help students do well in the course? Which questions behave like “weed-out” questions? Maybe ed-tech should find ways to collect demographic information and measure outcomes to report on inequitable results, while protecting student privacy. 

I am interested in hearing your ideas, too.

See you earlier tomorrow! And by the way, I have started making sure that the people that I mostly speak to later in the day occasionally meet with me at an earlier time, so that they get the benefit of my full listening capacity and creative potential.

personified lock (unlocked) fighting personified corona virus

The Two Pandemics, Part 2, Open Vs. COVID in Education

Several years ago I had a fellowship from the Shuttleworth Foundation, an organization that supports people who are using ‘open’ to improve the world. With COVID19, I think we are seeing now more than ever that opening up information is one of the keys to a concerted and effective response. Over 6 million people world-wide have contracted the virus and in the US alone, over 100,000 have died. Those numbers are staggering in their human cost, and at the very least, we need to make sure that the learning from those cases can ensure effective preventions and treatments are discovered quickly and and shared widely. 

One area I am a part of that showcases the power of open during the pandemic is the tremendous outpouring of support from open education organizations, many of whom already had high quality, free, open-source materials ready for faculty to use in their transition to online teaching due to the need for social distancing. Many of these organizations beefed up their free and open offerings to make them even more full service and continue to do that over the summer so that faculty have even better options for the Fall.  

Help for Moving Learning Online: These links showcase some of the options for moving education online.

  • OpenStax Allies offer free access to learning tech amidst COVID-19 : Thirty different offerings, from clicker services to homework systems to courseware, that cover basic college subjects. 
  • Webinar Series: Effective Online Instruction. The Association of College and University Educators (ACUE) series covers moving a course online, facilitating online discussions, creating and engaging students in readings and microlectures.
  • Remote learning with Khan Academy during school closures This is a great one stop page for getting started or continuing with Khan Academy at any age. It has lots of how-to’s for parents and teachers and has daily plans for covering school subjects. 
  • Free Resources for Schools During COVID-19 Outbreak (Updated in June) This list of free content, services, classes, lectures, and tools contains hundreds of listings that regularly updates to stay up to date. The printable format button in the upper right will get you to a full list that is easy to scroll through. 
  • Remote Learning, EdTech & COVID-19 This is the World Bank’s collection of relevant blog posts and resource listings prepared by the World Bank’s ed-tech team. I was intrigued by Bad practices in mobile learning that compiles a top-9, with a bonus 10th left blank for future mistakes. Although it is from 2014 and likely needs updating, it includes one of my favorite anti-patterns, “don’t spend time with your target user groups – assume you understand their needs.”

But what about disabilities? Learning how to serve students with disabilities well during a transition to online learning is considerably more challenging, but getting more material online itself can be useful.

  • Remote learning shift leaves students with disabilities behind : Gives a succinct summary of types of disabilities, supports that are needed during in-person teaching, and the effects of the shift online on those supports, showcasing voices of college students.  
  • Helping special needs students with remote learning : This article focuses on K12 students, the requirements that schools must continue to meet during a school closure where instruction is being provided to the general population, how to continue IEP (Individualized Education Plan) meetings virtually, and tips for parents with the general theme ‘provide structure with lots of patience‘.
  • ECAR Study of the Technology Needs of Students with Disabilities, 2020 This report collected 2000 open-ended responses from students with disabilities in 2019 pre-COVID19 to understand what they most want from their faculty: make all material available online, well-organized in the LMS (learning management system), with multiple methods of presentation (text, audio, visual, lecture), and make sure assistive technology (captions and speech to text) will work. All of these recommendations will improve access for students with disabilities and make learning better for all students as a side-effect, because these are Universal Designs for Learning.

The Digital Divide: It is critical that we address the racial and socioeconomic divide in education that is further exacerbated by the rush to online. 

  • The COVID-igital Divide African Americans are twice as likely to die from COVID-19, more likely to have lost jobs, twice as likely to be attending higher-ed institutions at risk of closing due to financial pressures, and more likely to be among the 20% of students without needed tech for online learning.  Estimates show African American students maybe 10 months behind, compared to 7 months for white students. 
  • Houston-area schools lose contact with thousands of students during pandemic shutdown In the early days of the pandemic shutdowns districts that were reporting metrics lost track of up to 30% of their students. Most have gotten that down to less than 10%, but that is still a lot of students displaced and missing their education. Texas City and Friendswood were able to do better and contacted all but 1.3% and 2 students respectively. 
  • ‘The need is real’: Houston-area schools scramble for hotspots so students don’t fall behind Even when districts can contact their students, many students lack access to wireless internet and are either having to use paper packets delivered or picked up weekly, or parents’ cell phones. Districts have funds to purchase hotspots, but with a district like HISD needing more than 100,000 devices, the supply just isn’t available. 
  • Research Shows Students Falling Months Behind During Virus Disruptions Using a variety of research methods (data from past disruptions, data from online apps before and after the shutdowns by zipcode) researchers estimate that rural and low-income students could be very far behind when schools reopen.  

I am interested in who is figuring out how to ensure equitable access to education during the COVID19 pandemic for already underserved students. What are they doing? Which things are working?  What are the highest priority needs? How can open education providers do the most good? Where should open educational providers be partnering with other organizations and providers? 

As we transition to online learning for the foreseeable future, I’m excited about the potential technology has to make learning safe and accessible during a worldwide crisis. At the same time, it is important to be aware and intentional about building tools that help all students equitably. The last thing we want in a crisis is to exacerbate the divides that already exist in education. Instead, let’s imagine a future where technology bridges those gaps, and actively work to build resources for students who might otherwise be left behind.