Meta and Google's Advancements in AI Audio

Earlier, I posted a video on Meta's Voice Box that impressed me with its advancements in the AI audio space[^1^]. However, it seems that Google is not far behind. They have responded with their own attempt at AI audio, called Audio Palm[^1^]. In this article, we will take a closer look at Google's Audio Palm and explore Meta's recent development in giving AI a taste of human-like common sense[^2^].

Meta's Vision for Common Sense AI

In order to achieve artificial general intelligence, common sense is crucial. Humans and animals acquire knowledge by observing the world around them, without the need for explicit labeling or training data[^3^]. Meta's Chief AI scientist, Jan Lacun, recognized this and proposed an architecture that addresses the challenge of teaching machines common sense[^3^]. Lacun envisions machines that can learn internal models of how the world works, allowing them to learn quickly, plan tasks, and adapt to unfamiliar situations[^3^].

The First AI Model from Meta: Igepa

Meta's first real-world implementation that aligns with Lacun's vision is Igepa[^4^]. Igepa revolutionizes AI's understanding of the world by changing the way it learns from its surroundings[^4^]. Unlike traditional AI models that rely on labeled data, Igepa learns in a self-supervised manner, similar to how humans and animals acquire common sense[^4^]. By observing and analyzing the world, Igepa can develop abstract representations of important aspects while filtering out irrelevant details[^4^].

The World Model Module

The world model module of Igepa is a complex component of Meta's architecture[^5^]. It is responsible for predicting missing information about the environment and generating plausible future states[^5^]. This module acts as a simulator, representing multiple possible predictions in uncertain situations[^5^].

Introduction to Japan: Joint Embedding Predictive Architecture

Central to Igepa is a prediction architecture known as Japan, which stands for joint embedding predictive architecture[^5^]. Japan creates abstract representations of input and predicts differences between these representations[^5^]. This approach allows for the generation of various plausible predictions based on the big picture concept, rather than getting lost in irrelevant details[^5^]. One significant advantage of Japan is its ability to naturally produce informative abstract representations that eliminate irrelevant details[^5^].

Google's Audio Palm

While Meta focuses on common sense AI, Google has developed Audio Palm, a combination of the Palm 2 language model and audio language modeling[^1^]. Although details about Audio Palm are limited, Google's effort in the AI audio space indicates a growing interest in advancing audio technologies.

AI Focus: Advancements in Image and Speech Understanding

IJAPA: Bridging the Gap Between Generative Approaches and Human-like Predictions

In the pursuit of more accurate and human-like predictions, researchers have developed a novel approach called IJAPA. This approach aims to overcome the limitations of large language models and their generative techniques. By avoiding the creepy and unrealistic outputs often produced by these models, IJAPA strives to predict missing information in a more abstract manner, closely resembling how a human would do it.

To achieve this, IJAPA utilizes the multi-block masking strategy. A single context block is used to predict representations of various target blocks originating from the same image. The context encoder processes the visible information and passes it to the predictor, which predicts the missing parts of the image. These predicted representations are then aligned with the target encoder, resulting in a primitive world model capable of predicting high-level information about unseen regions in an image.

As shown in the figure above, IJAPA successfully predicts the missing portions of an image, such as the top of a dog's head, bird's legs, a wolf's legs, and even the other side of a building. Remarkably, this approach maintains the original position of the image without relying on generative methods. Moreover, IJAPA's training process is computationally efficient, as only one view of the image needs to be processed by the target encoder and only the context blocks need to be processed by the context encoder.

In terms of performance, IJAPA outperforms pixel and token reconstruction methods on ImageNet 1K dataset, showcasing its superior predictive capabilities. Additionally, IJAPA excels in low-level vision tasks, such as object counting and depth prediction, outperforming models that rely on handcrafted data augmentation during pre-training. This emphasizes that IJAPA not only stands out as a cooler alternative but also proves to be more practical in image prediction.

IJAPA has also paved the way for future advancements. The blog suggests extending this architecture to more complex modalities, such as video understanding. By conditioning predictions on audio or text prompts, IJAPA could potentially achieve video understanding similar to what Domo achieved with its video llama AI. This exciting prospect demonstrates the first step towards AI learning a general model of the world, beyond just images.

Audio Palm: The Powerful Fusion of Text and Audio Processing

Meanwhile, at Google, a groundbreaking development called Audio Palm has emerged. This impressive large language model focuses on speech understanding and generation and leverages a unique combination of written and spoken language models. By harnessing the strengths of both models, Audio Palm excels in tasks like speech recognition, speech translation, and more.

The core structure of the Audio Palm model remains similar to traditional language models. It operates by accepting a sequence of both text and audio tokens as input and generates either text or audio tokens as output. Initially trained only on written text, researchers found that incorporating knowledge from a text model significantly improved the model's speech understanding and processing capabilities.

One of the notable features of Audio Palm is its ability to retain important details, such as the speaker's voice and intonation, while also leveraging its deep understanding of language from the text model. This combination enables Audio Palm to not only recognize speech but also translate spoken words into different languages with remarkable accuracy. Additionally, it can even mimic someone's voice in a different language if provided with a short recording of their speech.

The versatility of Audio Palm is further enhanced by its multimodal processing capabilities. By effectively handling both text and audio information, Audio Palm is capable of handling a wide range of tasks, including speech-to-speech translation, speech-to-text translation, and ASR (Automatic Speech Recognition). Its proficiency in multimodal tasks positions Audio Palm as a transformative model in the field of speech understanding.

To process audio data, Audio Palm incorporates additional stages from the audio element model. By converting audio tokens back into raw audio, the model gains the ability to manipulate and process audio data effectively. This technology expands the range of audio tasks Audio Palm can perform, empowering it to handle diverse challenges with ease.

Closing Thoughts

The advancements showcased by IJAPA and Audio Palm push the boundaries of AI capabilities in image and speech understanding. We witness IJAPA's ability to predict missing information in images in a manner that closely resembles human-like reasoning. With its efficient and accurate predictions, IJAPA showcases the potential for learning image representations without excessive training.

On the other hand, Audio Palm's fusion of text and audio processing ushers in a new era of speech understanding. By leveraging its deep knowledge of language from text models and combining it with the complexities of speech recognition and translation, Audio Palm achieves remarkable accuracy and fluency.

These breakthroughs not only highlight the progress made in AI research but also point towards a future where AI models can truly comprehend and navigate the world around us. As we delve further into complex modalities and expand the capabilities of these models, the potential for AI to enhance our lives and achieve feats previously unimaginable becomes increasingly limitless.

Subscribe to our channel to stay updated on the latest AI news and witness the exciting advancements that lie ahead.

If you want to learn more about Meta's Voice Box and its capabilities, check out our video on it and share your thoughts on this remarkable innovation.Both meta and google are making significant advancements in ai audio and common sense ai respectively. while meta's igepa model allows ai to learn the internal models of the world and achieve human-like results, google's audio palm combines language models to enhance audio understanding. these developments provide a glimpse into the future of ai and its potential impact on various industries.

note: additional information about the capabilities and potential applications of igepa and japan have been included in this article to provide a more comprehensive understanding of meta's advancements in common sense ai.