While reading recently from Make It Stick, I ran across an excerpt that really got me thinking about formative assessment in the classroom and the emphasis teachers and students place on right and wrong answers. Here’s the quote, which was actually said by former Secretary of State, Donald Rumsfeld***:
“There are known knowns; there are things we know that we know. There are known unknowns; that is to say, there are things that we now know we don’t know. But there are also unknown unknowns — there are things we do not know we don’t know.”
I’m not really a ‘favorite quote’ type of person, as I believe they are often overgeneralized and a bit vacuous…but I do really appreciate how Mr. Rumsfeld’s quote can be applied to the classroom. When I think about it, this covers many of the reasons we assess of/for learning in the classroom. Say you have your students complete an assessment of material covered over the past few lessons or unit (here’s a strategy I use for this). What happens after the formative assessment? What conversations are you having with the class? Through what lens are the students viewing their score and how are they using this material? By breaking this information into known knowns, known unknowns, and unknown unknowns, I believe students can easily compartmentalize all aspects of the assessment and learn how to analyze this information for their own feedback.
Known knowns
I’ve observed students somewhat intuitively first see their known knowns…what they got correct. They know they know this information.* Great. Absolutely nothing wrong with this. However, this is often where we stop with the analysis. Students see their score, they know they got some stuff correct, some stuff wrong, and they move on. We really need to work to get our students past this quite surface level look at their work.
Known unknowns
I think we all know that it is much more pleasant to look at the material we got correct on an assessment (known knowns). But this does not utilize perhaps the most important aspect of the assessment…the known unknowns. By having a look at the things they answered incorrectly, students have now identified what the know they don’t know. Now, they know better what to study. They can focus on the known unknowns instead of using that time studying the known knowns. It sounds easy, but I have not found that students do not practice this. And like I said earlier, it isn’t too fun to look at where you failed on an assessment…graded or ungraded. To this, I tell my students that they would much rather find out what they don’t know now (before the unit test) than find out on the test, when it’s too late to study.
Unknown unknowns
This category is a bit trickier. I see the unknown unknown information as vast holes in the student’s learning. Perhaps they missed a lesson or two of material or maybe the formative assessment involved an essay where the student completely omitted a chunk of information. In my best estimation, a student’s unknown unknowns can only be found by someone else; namely another student or teacher. Only then does the unknown unknown become a known unknown. So, maybe after the assessment and scoring, students are permitted to evaluate their peer’s work in an attempt to identify the unknown unknown. This could also be accomplished via a class discussion on the assessment. You could ask for students to share their answers and point out all of the major topics that should be included for successful completion.
Looking at it now, we should be working to move student’s understanding from unknown unknowns to known unknowns to known knowns. Sometimes we can skip the known unknowns category, but it is still quite important for students to learn to analyze this aspect of formative assessment. Again, recognizing their known unknowns should drive their future practice and show them exactly what information needs to be moved into the known known category.
So, how can identifying these three categories be best accomplished in the classroom? Here are two ways I plan on doing this:
1. Provide the following prompts on the overhead:
- Known knowns
- What topics do you know you know? (What did you answer correctly?)
- Known unknowns
- What topics do you now know you don’t know well enough? (What did you answer incorrectly?)
- How will you organize this information for future practice?
- What do you need to help you study this material later?
- Unknown unknowns
- What did you not even know you needed to know?
- Why did you not know this chunk of material?
- Where can you receive assistance to introduce you to these topics?
My hope in presenting the three categories this way is to get students thinking about and having an internal dialogue about the analysis…trying to create this habit of inspection will only help them with future studies.
2. Another way to assist students with identifying the three categories might be to make a table:
With this table, I just want to provide a basic prompt and really have the students do the thinking…let their brains do the cognitive work. Learning should be effortful. 🙂
Will you incorporate this into your repertoire for assessment of/for learning? If so, how?
What other methods can you think of to adapt and incorporate knowns and unknowns into practice?
Please leave a comment. Let’s have a conversation.
*Be careful with known knowns. If you are assessing material immediately after the lesson, students may falsely believe they know more than they actually do. I definitely recommend utilizing the benefits of spaced practice. Wait at least 24 hours before assessing the material. Maybe begin the next lesson with a review of prior material. Here is a link to three formative assessment strategies I use that attempt to capitalize on spaced retrieval practice.
***Mr. Rumsfeld acquired this knowledge from Dr. Werner Erhard. Access this link to read his words on the topic of unknowns and knowns.
I really enjoyed this post Blake. I had to read it twice to get the known knowns, known unknowns and unknown unknowns straight. I have been looking for better ways to use the results of formative assessments, to help students explore on their own, what they know and identify what they don’t know, and reading this definitely helps to give me a direction to go in. I really like how you explained this and set it up so that others can give it a go in their classrooms. Getting past that initial “this is what I got right/this is my score” is sometimes a challenge,
Thanks for helping me to better understand the learning sciences and for pushing my thinking!
Blake, I have enjoyed your writing for a bit now. Thanks very much. You have come at a fundamental principle of the way we do experience-based transformational learning at CITYterm. Something we have been working on for twenty or so years. But i love the way you put it into formal formative assessment work. For us, it works best when it is organic, but you raise some interesting applications. Thanks again. Here are a couple of old blog posts that try to explain what the DKDK zone is and how to use it. As you can tell from the end of the first one–we were way ahead of Rumsfeld.
The first introduces the concept…http://dkdkzone.blogspot.com/2010/07/dkdk-zone.html
the second goes into what it matters. http://dkdkzone.blogspot.com/2010/07/importance-of-dkdk-zone.html
the rest of the blog (which i am using as the basis for a book) explores from there
love your stuff…thanks again
keep the faith
david
This year I asked my students to submit their work self-corrected, and I found that eye-opening and transformative enough, but I wanted to go further next year and invite reflections as well. Your table will go onto every work sheet! Thank you!
I love this as an exam wrapper strategy because it closes the loop and feeds into the next cycle of learning.
I love this and read it several times. There may be another refinement lurking. Where would you put the unknown unknowns that exist, not because they were never known, but because students thought they knew and didn’t actual know correctly? In physics we present discrepant events to jar them out of incorrect understandings but even when they ‘see it’ and talk through a new understanding they often revert back to how they’ve thought of it for so long. That is, the unknown unknowns can be categorized into two prongs – omission or misunderstanding, each of which would be addressed differently. Thank you for continuing to give me things to think about.