The Second Shift: Does homework fly in the face of current productivity research?

Is ed-tech exacerbating inequity?

I have been thinking a lot about where ed-tech might be exacerbating existing inequity. And that led me to read a colleague’s tweet of “Homework is a Social Justice Issue”, originally published here in 2015. It talks about the underlying assumptions being made when we give homework, especially in K12: that students have the time, background knowledge, and tools to do school work at home. If students are working or taking care of younger siblings, they don’t have the time. If the type of work often lures in parents of affluent students to help, then they probably don’t have the background knowledge yet. And if the homework is on a laptop/phone and requires internet access, or requires space to organize and maintain materials, they may not have the tools. We must take these environmental realities into account when designing and building educational software that will meet the needs of students from all walks of life.

Long hours don’t work.

I recently realized that the people I meet with after 4pm aren’t getting the same creativity and deep listening as people that talk to me at 9 or 10am. It made me wonder why we are asking students to do a second or often third shift, when the research on the harms of long hours to productivity of overwork are so clear (here’s a summary of the harms) and similarly there are real harms to work quality (see this study on long medical shifts). Do you want someone in their 18th hour doing brain surgery on you?

When and what to assign?

So, even IF students have the time, background knowledge, and tools, does it really make sense to ask them to work a second shift? Students do need time to grapple with hard problems, and many students need quiet to work. So it isn’t an easy problem to fix. The article suggests that if you are assigning homework in K12, you should ask yourself these questions.

“Does the task sit low on Bloom’s Taxonomy? In other words, are students likely to be able to do it independently?
If not, does the task build primarily on work already performed or begun in class? In other words, have students already had sufficient opportunity to dig deep into the task and work through their difficulties in the presence of peers and/or the teacher?
Does the task require only the technology to which all students have sufficient access outside of school?
Can the task reasonably be accomplished, alongside homework from other classes, by students whose home life includes part-time work, significant household responsibilities, or a heightened level of anxiety at home?”
https://modernlearners.com/homework-is-a-social-justice-issue/

How could ed-tech help?

Homework systems and courseware could make it easy and safe for students to provide feedback on their assignments, including individual questions and tasks within their assignments. Rather than focusing so much on giving analytics about students, ed-tech could provide instructors with analytics about the assignments, questions, and tasks they give. Which ones seem to require a lot of prerequisite knowledge that students don’t already have? Which ones seem to help students do well in the course? Which questions behave like “weed-out” questions? Maybe ed-tech should find ways to collect demographic information and measure outcomes to report on inequitable results, while protecting student privacy.

I am interested in hearing your ideas, too.

See you earlier tomorrow! And by the way, I have started making sure that the people that I mostly speak to later in the day occasionally meet with me at an earlier time, so that they get the benefit of my full listening capacity and creative potential.

The Two Pandemics, Part 2, Open Vs. COVID in Education

Several years ago I had a fellowship from the Shuttleworth Foundation, an organization that supports people who are using ‘open’ to improve the world. With COVID19, I think we are seeing now more than ever that opening up information is one of the keys to a concerted and effective response. Over 6 million people world-wide have contracted the virus and in the US alone, over 100,000 have died. Those numbers are staggering in their human cost, and at the very least, we need to make sure that the learning from those cases can ensure effective preventions and treatments are discovered quickly and and shared widely.

One area I am a part of that showcases the power of open during the pandemic is the tremendous outpouring of support from open education organizations, many of whom already had high quality, free, open-source materials ready for faculty to use in their transition to online teaching due to the need for social distancing. Many of these organizations beefed up their free and open offerings to make them even more full service and continue to do that over the summer so that faculty have even better options for the Fall.

Help for Moving Learning Online: These links showcase some of the options for moving education online.

OpenStax Allies offer free access to learning tech amidst COVID-19 : Thirty different offerings, from clicker services to homework systems to courseware, that cover basic college subjects.
Webinar Series: Effective Online Instruction. The Association of College and University Educators (ACUE) series covers moving a course online, facilitating online discussions, creating and engaging students in readings and microlectures.
Remote learning with Khan Academy during school closures This is a great one stop page for getting started or continuing with Khan Academy at any age. It has lots of how-to’s for parents and teachers and has daily plans for covering school subjects.
Free Resources for Schools During COVID-19 Outbreak (Updated in June) This list of free content, services, classes, lectures, and tools contains hundreds of listings that regularly updates to stay up to date. The printable format button in the upper right will get you to a full list that is easy to scroll through.
Remote Learning, EdTech & COVID-19 This is the World Bank’s collection of relevant blog posts and resource listings prepared by the World Bank’s ed-tech team. I was intrigued by Bad practices in mobile learning that compiles a top-9, with a bonus 10th left blank for future mistakes. Although it is from 2014 and likely needs updating, it includes one of my favorite anti-patterns, “don’t spend time with your target user groups – assume you understand their needs.”

But what about disabilities? Learning how to serve students with disabilities well during a transition to online learning is considerably more challenging, but getting more material online itself can be useful.

Remote learning shift leaves students with disabilities behind : Gives a succinct summary of types of disabilities, supports that are needed during in-person teaching, and the effects of the shift online on those supports, showcasing voices of college students.
Helping special needs students with remote learning : This article focuses on K12 students, the requirements that schools must continue to meet during a school closure where instruction is being provided to the general population, how to continue IEP (Individualized Education Plan) meetings virtually, and tips for parents with the general theme ‘provide structure with lots of patience‘.
ECAR Study of the Technology Needs of Students with Disabilities, 2020 This report collected 2000 open-ended responses from students with disabilities in 2019 pre-COVID19 to understand what they most want from their faculty: make all material available online, well-organized in the LMS (learning management system), with multiple methods of presentation (text, audio, visual, lecture), and make sure assistive technology (captions and speech to text) will work. All of these recommendations will improve access for students with disabilities and make learning better for all students as a side-effect, because these are Universal Designs for Learning.

The Digital Divide: It is critical that we address the racial and socioeconomic divide in education that is further exacerbated by the rush to online.

The COVID-igital Divide African Americans are twice as likely to die from COVID-19, more likely to have lost jobs, twice as likely to be attending higher-ed institutions at risk of closing due to financial pressures, and more likely to be among the 20% of students without needed tech for online learning. Estimates show African American students maybe 10 months behind, compared to 7 months for white students.
Houston-area schools lose contact with thousands of students during pandemic shutdown In the early days of the pandemic shutdowns districts that were reporting metrics lost track of up to 30% of their students. Most have gotten that down to less than 10%, but that is still a lot of students displaced and missing their education. Texas City and Friendswood were able to do better and contacted all but 1.3% and 2 students respectively.
‘The need is real’: Houston-area schools scramble for hotspots so students don’t fall behind Even when districts can contact their students, many students lack access to wireless internet and are either having to use paper packets delivered or picked up weekly, or parents’ cell phones. Districts have funds to purchase hotspots, but with a district like HISD needing more than 100,000 devices, the supply just isn’t available.
Research Shows Students Falling Months Behind During Virus Disruptions Using a variety of research methods (data from past disruptions, data from online apps before and after the shutdowns by zipcode) researchers estimate that rural and low-income students could be very far behind when schools reopen.

I am interested in who is figuring out how to ensure equitable access to education during the COVID19 pandemic for already underserved students. What are they doing? Which things are working? What are the highest priority needs? How can open education providers do the most good? Where should open educational providers be partnering with other organizations and providers?

As we transition to online learning for the foreseeable future, I’m excited about the potential technology has to make learning safe and accessible during a worldwide crisis. At the same time, it is important to be aware and intentional about building tools that help all students equitably. The last thing we want in a crisis is to exacerbate the divides that already exist in education. Instead, let’s imagine a future where technology bridges those gaps, and actively work to build resources for students who might otherwise be left behind.

The Two Pandemics

What product managers can learn from hospice nurses

I wasn’t expecting to learn anything related to product management and software development in a book about how we age and care for aging family, but I did. I have just finished reading Being Mortal by Atul Gawande, who also wrote and conducted the research and transformational changes in The Checklist Manifesto. (Image credit: Public domain: https://commons.wikimedia.org/wiki/File:Stethoscope-2.png)

From doctor-knows-best to patient-knows-best. Gawande talks about the changing role of doctors over time. In our grandparent’s era, ‘doctors knew best’. That was the age of the authoritarian doctor who made the decisions and was trusted to do so. Now, in contrast, for the most part, doctors are considered technical experts who can share information, but decisions ultimately rest with the patient. The idea is that the patient knows best, when given all the facts. However, when patients face important crossroads in their treatment and there are many uncertainties, neither approach works well and both lead to escalating interventions and, often, miserable people.

Both lead to suffering. The authoritarian path didn’t work well for patients because it didn’t take the patient’s fears and hopes into account at all. Without the patient’s preferences, doctors recommend actions within their own sphere of expertise. Surgeons recommend surgery rather than hospice because surgery is what they know best. But Gawande, who was trained in the current technical-expert-sharing-information model of doctoring, illustrates how information sharing goes just as wrong when it comes to delivering the experience that patients would want. He tells several stories from his own practice, where patients clearly said, I don’t want to suffer, and I don’t want pointless heroics, but then choose to proceed through many, many rounds of painful procedures with very low probability of success. Why is that?

Software development has the same duality. In software development, we also have this same duality. The authoritarian model: Which employee is the ‘owner’ of this product? They should make all the decisions about what to develop, and be responsible for the consequences. Or, alternatively, the ‘Information’ model; let’s have the expert (product manager) gather the facts and present choices to leaders and other stakeholders; or lets develop objective metrics to guide us. And similarly, it can feel like we see-saw between decisions made with too little information and decisions that feel like the information was there, but it was never pulled together into the right decision. So I was very interested in why neither approach is working for doctors and patients, and what might be a better approach.

Why the informed patient model still fails. Gawande’s analysis of the informed patient that still makes the wrong decisions is that they don’t have the experience or the medical model to make the decision, even when they have all the facts. So, assuming you aren’t an astronaut or experienced physicist, think about it this way. If I put you in a space capsule, ask you where you want to go (which is what the doctors-know-best doctors forgot to do), and I tell you a bunch of readouts and their percentage likelihood of being correct, and then ask you whether to launch, you still aren’t going to be able to make a good decision, because you don’t have a model in your head about how all those measurements add up. Gawande specifically talks about how a patient might be imagining that a particular procedure with a high likelihood of success could extend their life by years, when in reality it is likely to be weeks, not years. The patient doesn’t have the experience to put all the information together into a coherent model and make a good choice.

So now what? So, is there a middle ground? Gawande describes a model where doctors gather even MORE information about what a patient WANTS by using four questions that come from the world of hospice. Then, the doctor combines the patient’s answers with their professional experience to guide patients in making decisions that are consistent with the patient’s own desires.

Learning from hospice nurses. The four questions are also interesting. 1. What do you understand about your situation? 2. What do you fear? 3. What do you hope for? 4. What trade offs are you willing and not willing to make?

Ask, tell, or guide? A middle path to product development. In the world of software development, many organizations (including the one I work for) hire Product Managers to lead product development. In specific, Product Managers are responsible for deciding what features should be added to a product and with what priority. So the question is “Are Product Managers owners (the authoritarian model), expert consultants (the information sharing model), or expert guides (the new model Gawande proposes)?”

I would posit that the same insight Gawande has about doctoring is the right insight for product development. Product Managers don’t own their product. The product isn’t FOR them and there are too many critical stakeholders for them to be owners. But they also can’t present information and expect decisions from business leadership, precisely because the business leaders don’t have the full context and understanding of the detailed workings of the product and market. The product managers DO have that context.

My key insight from Being Mortal is those four questions that hospice nurses taught doctor’s to use to help them guide their patients. I am curious about whether those questions can be adapted to gather the right information, especially from business leadership and organizational stakeholders that don’t directly interact with the product, to allow Product Managers to wisely incorporate their requirements into good decisions.

So here are the four questions again

What do you understand about your situation?
What do you fear?
What do you hope for?
What trade offs are you willing and not willing to make?

Frankly, they almost work as is. I would only change the phrase ‘your situation’ in the first one to match the context. It could be ‘what do you understand about our goals’, ‘what do you understand about our revenue position’, ‘what do you understand about our strategy for …’

I am going to give these a try. Let me know if you do too!

Getting to the truth, the ground truth, and nothing but the ground truth.

1 Reply

Takeaways for learning from HCOMP 2019, Part 2

At HCOMP 2019, there was a lot of information about machine learning that I found relevant to building educational technology. Surprisingly to me, I didn’t find other ed-tech companies and organizations at the Fairness, Accountability, and Transparency conference I went to last year in Atlanta or the 2019 HCOMP conference. Maybe ed-tech organizations don’t have research groups that are publishing openly and thus they don’t come to these academic conferences. Maybe readers of this blog will send me pointers to who I missed!

Mini machine learning terminology primer from a novice (skippable if you already know these): To train a machine learning algorithm that is going to decide something or categorize something, you need to start out with a set of things for which you already know the correct decisions or categories. Those are the ‘ground-truths’ that you use to train the algorithm. You can think of the algorithm as a toddler. If you want the algorithm to recognize and distinguish dogs from cats, you need to show it a bunch of dogs and cats and tell it what they are. Mom and Dad say — “look, a kitty”; “see the puppy?” An algorithm can be ‘over-fitted’ to the ground truth you give it. The toddler example is when your toddler knows the animals you showed them (that Fifi is a cat and Fido is a dog), but doesn’t know what new animals are, for example the neighbor’s pet cat. To add a further wrinkle, if you are creating a ground-truth, it is always great if you have Mom and Dad to create the labels, but sometimes all you can get are toddlers (novices) labeling. Using novices to train is related to the idea of wisdom of the crowd, where the opinion of a collection of people is used rather than a single expert. You can also introduce bias into your algorithm by showing it only calico cats in the training phase, causing it to only label calicos as “cats” later on. Recent real world examples of training bias come from facial recognition algorithms that were trained on light-skinned people and therefore have trouble recognizing black and brown faces.

Creating ground truth: A whole chunk of the talks were about different ways of creating ‘ground truths’ using ‘wisdom of the crowd’ techniques. Ed-tech needs quite a bit of ground-truth about the world to train algorithms to help students learn effectively. “How difficult is this task or problem?” “What concepts are needed to do this task/problem?” “What concepts are a part of this text/example/explanation/video?” “Is this solution to this task/problem correct, partially correct, displaying a misconception, or just plain wrong?”

Finding the best-of-the-crowd: Several of the presentations were about finding and motivating the best of the crowd. If you can find and/or train ‘experts’ in the crowd, you can get to the ground-truth at lower cost (in time or money). I am hoping that ed-tech can use these techniques to crowdsource effective practice exercises, examples, solutions, and explanations.

Wisdom of the toddlers. Heinecke, et. al (https://aaai.org/ojs/index.php/HCOMP/article/view/5279) described a three step method for obtaining a ground truth from non-experts. First, they used a large number of people and expensive mathematical methods to obtain a small ground truth. (If we are sticking with the cats and dogs example from the primer above, you have a large number of toddlers tell you whether a few animals are cats and dogs and use math to decide which animals ARE cats and ARE dogs using wisdom of the toddlers.) From there, step 2 is to find a small set of those large numbers of people who were the best at determining a ground-truth, and use them to create more ground-truth. (Find a group of toddlers who together labeled the cats and dogs correctly, and use them to label a whole bunch more cats and dogs). Finally, you use the large set of ground truth to train a machine learning algorithm. I think this is very exciting for learning content because we have students and faculty doing their day to day work and we might be able to find sets of them that can help answer the questions above.
Misconceptions of the herd: One complicating factor in educational technology ground truths is the prominent presence of misconceptions. The Best Paper winner at the conference, Simoiu et. al (https://aaai.org/ojs/index.php/HCOMP/article/view/5271), found an interesting, relevant, and in hindsight unsurprising result. This group did a systematic study of crowds answering 1000 questions from 50 different topical domains. They found that averaging the crowd’s answers almost always yields significantly better results than the average (50th percentile) person. They also wanted to see the effects of social influences on the crowd. When they showed the ‘consensus’ answer (current three most popular answers) to individuals, the crowd would be swayed by early wrong answers and thus did NOT perform on average better than the average unswayed person. Since misconceptions (wrong answers due to faulty understanding) are well known phenomena in learning, and are particularly resistant to change (if you haven’t seen Derek Muller’s wonderful 6 minute TED talk about this, go see it now!) we need to be particularly careful not to aid their contagion when introducing social features.

Are misconceptions like overfitting in machine learning? As an aside, my friend and colleague Sidney Burrus told an interesting story that sheds light on the persistence of misconceptions. Sidney talked about how, during the initial transition point between an earth-centered and sun-centered model of the solar system, the earth-centered model was much better at predicting orbits, because people had spent a lot of time adding detail to the model to help it correctly predict known phenomena. The first sun-centered models, however, used circular orbits and did a poor job of prediction, even though they had more ‘truth’ in them ultimately. Those early earth-centered models were tightly ‘fitted’ to the known orbits. They would not have been good at predicting new orbits, just like an overfitted machine learning model will fail on new data.

Kathi Fletcher

Delivering helpful technology well