In this blog post, we’ll examine the evolution of speech recognition and synthesis technologies and explore the challenges robots must overcome to understand human speech and engage in natural conversation.
“I’m sorry, Dave. Unfortunately, I cannot carry out that command.”
Anyone who has seen Stanley Kubrick’s masterpiece ‘2001: A Space Odyssey’ will likely remember this line. These are the words of HAL 9000—the computer serving as Captain Dave’s crew member—and mark the moment when artificial intelligence, created by humans, turns against them. It’s no exaggeration to say that ‘2001: A Space Odyssey’, released long ago, presented a vision of communication with machines. Since then, films have progressed from HAL 9000 to the ‘Terminator’ and ‘Iron Man’’s JARVIS—who speak much more naturally—and eventually to Samantha from ‘Her’, who develops deep emotions with a human that border on romance. The fact that voice recognition and synthesis technologies—once confined to the science fiction genre—appeared in the film ‘Her’, which was at one point classified as a drama, is because these technologies no longer remain solely within the realm of science fiction. In fact, by the standards of that time, 83% of South Koreans were carrying an exceptionally advanced voice recognition device—the smartphone. According to a presentation by Craig Federighi, Apple’s Senior Vice President of Software Engineering, at the Apple Worldwide Developers Conference at the time, “Siri”—the voice recognition system built into the iPhone—had a misrecognition rate of just 5%. Of course, while humans typically have a misrecognition rate of only about 1%, this demonstrates that the technology had reached a remarkably high level of sophistication. In South Korea, Shinhan Card has long offered a service that allows users to input voice responses via its automated voice response (ARS) system.
Voice recognition technology is based on the fact that human speech is, ultimately, sound. If we set aside the grammatical structure and meaning of the sentences contained in speech, human speech is not significantly different from animal sounds such as the chirping of cicadas or the singing of birds. In other words, the basic process of conversation involves adding vibrations from the vocal cords to exhaled air to make it vibrate, and then decomposing and deciphering those vibrations according to criteria unique to humans. The insight that spoken language is no different from other sounds allows us to leverage the knowledge of waves and acoustics that mathematicians and engineers have accumulated since the time of Pythagoras. Meanwhile, linguists have also studied how humans pronounce consonants and vowels and the rules governing their combination. If these two fields are combined—if engineers can organize sound into information that can be analyzed linguistically, and linguists can use that information to extract consonants and vowels—then machines will be able to understand human speech.
As such, speech recognition remains one of the most active interdisciplinary research fields today. However, let us now examine how mechanical engineers have tackled more fundamental technical challenges through the characteristics of sound discovered by Pythagoras, as mentioned earlier. Even before Pythagoras, people already knew that shorter strings produce higher pitches, while longer strings produce lower pitches. This is easy to understand when considering instruments such as the tuba and trumpet, or the bassoon and piccolo. The great mathematician discovered that if the length of a string is halved, the pitch rises by exactly one octave.
His subsequent reasoning was as follows: Imagine plucking a taut string. The string will vibrate tautly in an elliptical pattern—similar to the motion of a jump rope when spun—and produce a specific pitch. But what if you hold the string at its exact center to prevent it from moving and then pluck it again? You might expect the string not to vibrate, but it actually vibrates in a wave-like pattern, as if two strings of half the original length were connected. The sound produced is exactly one octave higher than when the string was simply plucked. What’s fascinating, however, is that even when the string is simply plucked, it contains the same wave-like vibration as when the center is held down. This holds true not only at the center of the string but also at every point—such as 1/2, 1/3, and 1/4 of its length—and since these vibrations correspond to frequencies that are “multiples” of a specific pitch, they are called overtones. The reason this concept—which might seem specific to stringed instruments—is important in speech recognition technology is that the overtone structure of the human voice helps distinguish between vowels. Let’s try saying “a, ya, eo, yeo, o, yo, u, yu, eui,” just as we did when first learning the vowels. We can pronounce different vowels at the same pitch without changing the pitch of our voice at all. In this case, we can think of our throat as a string. However, while pronouncing vowels, changes in the size of the oral cavity (the space is noticeably smaller when pronouncing “uh” than when pronouncing “ah”) or the position of the tongue (the tongue protrudes much further forward when pronouncing “ee” than when pronouncing “oh”) result in a completely different combination of overtones being superimposed on the fundamental frequency of our voice. Fortunately, engineers who build musical instruments and concert halls have developed many tools capable of analyzing the structure of overtones, and robots have become able to recognize vowels.
While the sound “aaaa” can be produced continuously, “dddd” cannot be produced continuously by any means. This is because vowels are pronounced simply by changing the shape of the mouth and the position of the tongue, whereas consonants are produced by obstructing the sound trying to escape from the mouth in some way. Therefore, to understand consonants, we must examine their characteristics immediately after the sound is produced. There are various ways to obstruct the flow of air when pronouncing consonants. These include briefly closing the lips (“ㅂ”), tapping the tip of the teeth with the tongue (“ㄷ”), or pressing the tongue against the roof of the mouth (“ㅈ”). The form in which the sound begins—that is, the form when the consonant is suddenly pronounced after a moment of silence—differs depending on the method of articulation for each consonant. By analyzing this form, we can decipher consonants, and by analyzing the harmonic structure of the sound that follows, we can decipher vowels.
Even though we have deciphered both consonants and vowels in this way, there is still one hurdle to overcome in conversations with robots. Even if a robot has broken down human speech into consonants and vowels, it must still understand the meaning behind those symbols. This is precisely why, when we speak to Siri, it may perfectly transcribe what we said but give a completely nonsensical response. The fundamental reason why speech synthesis—despite being merely the reverse process of speech recognition—cannot be considered as advanced as speech recognition is that the robot hears human speech but does not understand it. One of the community buses in Mapo-gu uses a text-to-speech system to announce stop names. I recall an instance where the announcement read, “This is Yanghwa, Jinseongji Park,” suggesting the system either didn’t understand what “Seongji” in “Yanghwajin Seongji Park” meant or failed to recognize that “Yanghwa” and “Yanghwajin” are used interchangeably. Ultimately, even though the technical field of converting sound into text and text into sound has reached a considerable level of sophistication, the challenge remains for robots to understand the meaning of the text being converted and the words they are supposed to speak. This has gradually evolved into a research area distinct from acoustics and linguistics. Research that began with linguists and acoustic engineers collaborating has now started to draw in not only these two fields but also computer scientists specializing in artificial intelligence. If we incorporate the emerging field of machine learning (machines that learn on their own) to enable robots to understand the meaning of text before reading it, they will be able to communicate actively with humans based on their already outstanding listening comprehension skills. The day is truly not far off when we won’t need to sit in front of a computer—instead, we’ll simply instruct a program like “JARVIS” from ‘Iron Man’ to search for information, then sit back and receive the results.
