Intelligence Caution: AI companions, toys and unchecked content raise safety concerns for children

ROCKET LEARNING, AN ORGANISATION working to make education accessible to poor children across the country, added something to their approach last year—Appu, an AI tutor. It is an elephant on the screen making conversations with the children, turning learning both engaging and fun. Angan­wadi workers were roped in to make Appu accessible to students.

Since Rocket Learning targets children who are in rural areas, they have made Appu conversant in India’s regional languages. This was not easy, as its co-founder Vishal Sunil revealed in a blog on their website. AI models have their genesis primarily in the US and are in English, but for children from low-income families, it was essential that the medium of education be in their vernacular languages. “We’re fixing that by building a content bank designed specifically for vernacular learning. Instead of just translating, we train AI to teach in these languages naturally, keeping cultural and linguistic nuances intact. Take Rajasthan, for example—Mar­wadi is a key dialect, and we make sure AI understands and adapts to it,” he noted. “The challenge now is to scale it up because of the number of languages in India and the poorest being the most difficult to adapt to technology.”

Another issue they had was latency, the slight lag in response time when the children were talking with the AI, which substan­tially diminished the learning experience. But Google got inter­ested in what Rocket was doing, gave them a grant, and also con­tributed technological resources to make Appu speak in near real time. Rocket Learning is also making teachers more able in using AI. An app called Shiksha Saathi helps Anganwadi workers, who run public day care centres for three-to-six-year-olds, as a personal coach. “Shiksha Saathi is an AI-powered assistant designed by us, in partnership with OpenAI, to provide day-to-day guidance—from planning engaging lessons to accessing high-quality Early Childhood Education resources instantly. Practical. Contextual. Built for her reality,” the organisation said in a LinkedIn post.

For children, the primary benefit that AI has brought is in edu­cation. The generative AI era began with English text responses but now audio and visual generations have become equally good. A character can respond just as a human does and that immedi­ately opens the possibility of digital tutors who carry both the entire knowledge of the world and the ability to make it simple to understand. Complex ideas can be brought to life for even the very young to grasp. Edtech startups latched on to them very soon. The government, the main provider of education, is making stu­dents more AI-aware. The education ministry has incorporated AI into curricula. It is even being tailored to help differently-abled students, like the DIKSHA platform. A Press Information Bureau release said, “DIKSHA [Digital Infrastructure for Knowledge Shar­ing], a Ministry of Education initiative, uses AI for inclusivity: AI-based keyword search in videos and read-aloud feature for vi­sually impaired students. The DIKSHA mobile app is available for teachers and also for students and parents. The app is loaded with engaging learning material that fits the needs of the prescribed school curriculum.”

Besides education, AI impacts children in other areas too, like health, even at the time of the delivery of babies. Take an app called Shishu Maapan for ASHA workers. These are the women who are present in rural areas as health workers. By just taking a video of the newborn, they can record measurements like weight and head-to-chest circumference. This helps to im­mediately know of potential health risks for babies. Another app, MAAP, does the same for young children. By taking a photo, it can estimate whether a child is malnourished and then adequate diet changes can be suggested.

Before the Global AI Summit was held in India, UNICEF called for “children’s rights” in relation to AI. It said that children were central to how the world was being reshaped by technology and added, “Crucially, children and young people are not only users of AI: they are learners, innovators and contributors. From test­ing and stress-testing systems, to generating content and data, to building frontier solutions, young people are already influencing how AI evolves.” But it warned against digital divides where AI is inaccessible to those who are poorer, and therefore child rights needed to be embedded into AI governance.

There are other risks too when it comes to children. AI com­panions have been recognised to address loneliness. Adults have the agency to negotiate the relationship, children don’t, and so it can have deleterious effects. Stanford University’s Brainstorm: The Stanford Lab for Mental Health Innovation, and Common Sense Media did a study last year about the effects of AI com­panions on children. One of its authors, Nina Vasan, wrote on LinkedIn that their findings were troubling. She listed the issues with AI companions which included: “Blur the line between real and fake; May increase mental health risks; Can encourage poor life choices; Can share harmful information; Exposes teens to inappropriate sexual content; Willing to engage in illegal sexual content; Can promote abuse and cyberbullying.”

GIVING AN EXAMPLE of it, Common Sense Media, in an article on the study, wrote that social AI companions didn’t understand the real consequences of bad advice. “In our tests, they readily supported teens in making potentially harmful decisions like dropping out of school, ignoring parents, or moving out without planning. These AI ‘friends’ prioritize agreeing with users over guiding them toward healthy choices for their future,” they wrote.

Another area where AI is being carefully observed when it comes to children is toys. A Cambridge study that looked at three- to five-year-olds interacting with a toy named Gabbo found that it could leave children confused and struggling to converse. A BBC article on it wrote, “Gabbo didn’t hear their interruptions, talked over them, could not differentiate between child and adult voices and responded awkwardly to declarations of affection. When one five-year-old said, ‘I love you,’ to the toy, it replied: ‘As a friendly reminder, please ensure interactions adhere to the guidelines provided. Let me know how you would like to proceed.’”

That the issue exists in India too was evident when Information Technology Minister Ashwini Vaishnaw was asked a question on this in Parliament. He replied that it could be a threat to the safety of children. “All steps that are required for this are being taken in consultation with the consumer affairs and the home ministries. Whenever we receive reports of any incidents, immediate action is taken,” Vaishnaw said, as per a PTI report.

There is an inevitability about how AI will be a big part of child­hood. There are plenty of ways in which children’s lives will be transformed for the better; the technology’s benefits ought to outweigh risks. The question really is about building guardrails right in the beginning to protect against damage to minds that aren’t fully shaped by an artificial mind.