“We have translated a pictorial dictionary into speech”
WHAT IF THERE is no real difference between French and English, Hebrew and Malayalam, Sanskrit and Swahili? What if every human language follows the same cognitive algorithm to express thought? And what if the underlying structures that constitute the grammar of a language are not just some immutable rules prescribed centuries ago, but a set of logical relationships between words? The idea is a dazzling one. It has been decades since Noam Chomsky, the American linguist and polymath, proposed the theory of a universal grammar: the idea that all languages are built on the same template and that children’s minds come programmed with this, which is why they can pick up language—any language—fast. A pathbreaking app for the iPhone and the iPad is now exploring how the brain learns language and applying it to teach grammar to autistic children. The man behind the app is no linguist or neuroscientist. Ajit Narayanan is an unassuming engineer with a ready smile and some serious communication skills. An expressive speaker, he can win you over with his passion for the beauty of language—and how it can empower people. His company, Invention Labs, is located at the IIT-Madras Research Park in Taramani. It occupies one corner of a large open-plan office space and comes with a tiny cabin and scribble board for Narayanan. This is where he, and his dozen employees, dive headlong into the complex world of language and learning, and indeed, into the structure of thought itself.
Invention Labs’ latest app, FreeSpeech, is a grammar prediction engine. When you input words, it will prompt you to form logical relationships between them by asking questions like what, when, how and where, and then string them together into coherent, grammatically correct sentences. Since every word is represented by a picture, and every selected word and derived sentence converted from text to speech, FreeSpeech can help children with learning disabilities communicate and grasp the nuances of language. Pick a picture tile denoting ‘dog’ and it asks whose, what kind, how many, and what it does. Drag ‘eat’ into the frame and the chart rapidly expands to ask questions about what the dog eats, and when, where and how it eats it. In a few steps, it can get you to form a sentence such as ‘My new dog ate my friend’s homework again’ by prompting you to enter more information and stitching it together with articles and prepositions. The coldness of grammar is suddenly gone, and with it the enervating dilemmas of how to use gerunds with possessive nouns and how to split your infinitives. By feeding thousands of English sentences into the engine, the makers of the app have ensured that it can paper over the gaps in your grammatical knowledge and the discontinuities in your sentence. “Many autistic children have an expressive imagination but they lack the ability to form complete sentences. In clinical trials conducted ahead of the launch of the app last year, FreeSpeech was shown to be greatly effective in helping them develop language skills,” Narayanan says.
The app is more than a sentence formation tool. It harbours basic learnings about how the brain processes language. “In 2012, I started out by trying to understand how children learn language, and why autistic children have trouble doing the same. The eureka moment came when I found that by representing meaning in a meta-linguistic way, I can equip these children with the skills needed for language development,” says Narayanan. By constructing sentences from first principles, FreeSpeech has validated research that suggests there is a simple, logical syntax at the core of language. Sentences are born equal, from a computer- like programme in our minds that has more to do with translating thought and meaning into language than following the rules of grammar, which are tacitly absorbed. This is how people generate their sentences, by instinctively fitting words where they belong in the web of logical relationships that ties them together. And if this is true, Narayanan realised that his grammar prediction engine should be language agnostic for the most part. FreeSpeech could, with some tinkering, help solve the fundamental problem of learning a new language as an adult. This was huge.
FreeSpeech was shown to be greatly effective in helping autistic children develop language skills
The way children learn a language is different from how adults grapple with the semantics of a foreign tongue. But if one could learn organically, building the language bit by bit from the structure of our thoughts rather than viewing it through the tint of another language, the results should be remarkably different. “We took the FreeSpeech engine and started creating Chinese sentences without having to change much in the app. We are now running clinical trials to study how the app makes learning a new language easier. Preliminary findings have been heartening,” says Narayanan. When you pick a picture tile—transliterated into English—the text-to-speech feature reads it aloud in Chinese with the correct accent. You can form sentences the same way you would in English, by joining words and creating relationships between them. The result is a shiny sentence in Mandarin, spoken the way it is meant to be spoken. Narayanan is building artificial intelligence into the platform, so that the app tapers off grammatical support as your knowledge of Chinese improves. “Imagine what this could do for a multilingual country like India,” he says.
He will, however, test the waters with China first. He has taught himself Chinese and hired a full-time resource in China to figure out how to enter what could be a big market for FreeSpeech. Some large Chinese investors have expressed interest, he says. (Invention Labs last raised $550,000 in funding in 2014 from Inventus Capital, Mumbai Angels, Google India Managing Director Rajan Anandan and others.) Among the hottest startups in China are English learning apps, some of which put students on video calls with English educators in the West. FreeSpeech for Chinese learners of English could provide a fundamentally different approach.
When FreeSpeech was released in Apple’s app store in March this year, it made it to the best new apps list, trumping millions of other apps. Apple CEO Tim Cook was so impressed with how it codified the steps in language learning that he endorsed the app and met Narayanan as part of an interaction with startups during his India visit in May.
NARAYANAN HAS THE exuberance of someone on the verge of a breakthrough. His journey began in 2010 with the launch of Avaz, an Alternative and Augmentative Communication (AAC) app that had graphical representations of words to help children with autism speak. The app was the result of a collaboration with Vidya Sagar, an organisation that works with children with special needs. “Autistic children have a lot of trouble with abstraction: a word is an abstraction while a picture is more concrete. Avaz helps them bridge the real world with the abstractions of language, but it doesn’t teach them the structure of language, or how words can be arranged in certain ways to convey meaning. That is something I wanted to address,” Narayanan says. Available in eight languages, Avaz became a global sensation, and despite its high price—$150-200—has seen over 30,000 downloads, most of them by special learning institutions. FreeSpeech, which costs just $10, has 10,000 users already. The Tamil Nadu government recently bought iPads with the Avaz app for schools participating in a pilot programme—the first such project with an AAC device by an Indian government. Much of the interest in Avaz has been from abroad so far.
In his 2013 TED talk, Narayanan talks about how language is really “the brain’s invention to convert rich, multi-dimensional thought on one hand into speech on the other hand”. Language isn’t a perfect medium, he hints. It flattens thought and forces us to filter it through the data structure of speech. If we could compose an essay on the FreeSpeech ecosystem, people could potentially read it in any language. If we could run a FreeSpeech Google search, it would match thought with thought and not English search with English results. “That’s a provocative idea,” Narayanan says, with a goofy smile. How can a lover of languages not succumb?
The Wealth Issue 2016: For the full list of portraits of the Smart-up Generation, click here
More Columns
‘AIPAC represents the most cynical side of politics where money buys power’ Ullekh NP
The Radical Shoma A Chatterji
PM Modi's Secret Plan Gives Non-Dynasts Political Chance Short Post