Last week, the TED conference put an MIT researcher on stage to share the “largest home video collection ever made.” It contained over 90,000 hours of video and 140 hours of audio, capturing the first three years of life with his new son.
Deb Roy wanted to understand children learned language. What makes this collection of captured sounds and images revolutionary, however, is the techniques his team has developed to track, visualize and analyze the everyday connections we humans make by living in the world.
By the time his baby-makes-three moment finally arrived, Roy had wired up his house with cameras intending to catch (most) every moment of his son’s life. The nine rooms of his house were tricked out with a birdseye lens looking down on the activity below. Unlike a lab setting, where observations of interactions are artificial and prompted, Roy’s recorders could find thousands of key unsolicited, natural moments of childhood development. After scrubbing the artifacts to make provisions for privacy, he handed the files over to his Cognitive Machines research group at MIT. Together, they turned the detailed home movies into a massive data set of language development.
The little Roy had picked up a vocabulary of 503 words by his second birthday. The TED audience was treated to an audio montage of the evolution of the boy’s understanding of water, moving from “gaga” to a an articulated “water.” In that clip, you can hear the cognitive spurts and experimentation until the happy ending when he owns the correct word.
In his talk, Roy describes the stages of meaning-making the researches progressed through to find some interesting insights about the relationship between context and learning. The first step was to create “space-time worms” using motion analysis to track movement in a room as a function of time. This allowed the researchers to focus on activity in the data the revolved around Roy’s son. The result was about 7 million words of transcripts.
The team then studied the relationship of words and their use in the context of the world. By mapping the individual players in space, Roy found “social hotspots”—where a more activity was spent together—and “solo hotspots” showing the boy interacting alone with his surroundings. These trails were mapped in the house as a function of time, creating wordscapes—hotspots rise like mountains on the map. For “water,” the peaks are in the kitchen. For “bye,” they are near the door.
After examining the caregiver speech in the key moments when a new word was acquired, researchers found that caregiver speech would change. The language the child heard from caregivers was simplified in those critical moments, allowing the learning to take place before gradually building it back up to complex language.
Beyond the boon to language researchers, Roy’s work has two other big impacts.
This same method of working with massive datasets can be applied to how we interact with the mass media—specifically, television—through social media. Applying the language-mapping techniques to media led to the founding of Blue Fin Labs, Roy’s agency that measures engagement with mass media. In near real-time, they can track how tweets and other individual digital footprints are tied to specific shared events on television. They mine about 3 million comments a month on relevant topics, linking them to an event and thus allowing a wordscape to be generated.
The second impact is one of personal history and reflection. Digital artifacts can capture moments in early childhood that the brain can’t possibly remember from a perspective no one has experienced, a kind of God’s eye view of history. To underscore what it means to be able to share your own developmental milestones with your children, Roy rolls back the 3D rendering of the video camera footage when his son took his first few steps. In response, I can only echo the word shared by father and son in that moment: Wow.