Google Gemini, The Biggest and Most Powerful AI Model, is Now Available.
Presently, we advance one step closer to this aspiration with the introduction of Gemini, our most comprehensive and versatile model to date.
Gemini is the outcome of extensive collaborative endeavors involving teams from various departments within Google, including the research colleagues. It was designed from the foundation to be multimodal, meaning it can comprehend, combine, and operate across various data formats (text, code, audio, image, and video) without difficulty.
Additionally, Gemini is the most adaptable model to date, operating efficiently on mobile devices and data centers. The advanced functionalities of this product will greatly improve the manner in which developers and enterprise clients construct and expand AI systems.
The initial iteration, Gemini 1.0, has been optimized for three distinct sizes:
| Gemini Ultra | The most sizable and powerful model for exceedingly complex duties. |
| Gemini Pro | The most effective model for scaling a diverse array of duties. |
| Gemini Nano | The optimal model for performing duties on the device. |
State-Of-The-Art Performance
The efficacy of our Gemini models has been rigorously evaluated across a vast array of tasks. In the realm of mathematical reasoning, natural image comprehension, audio and video analysis, and thirty of the thirty-two widely-used academic benchmarks utilized in large language model (LLM) research and development, Gemini Ultra outperforms current state-of-the-art results.
Gemini Ultra is the first model to surpass human experts on MMLU (massive multiplex language understanding), an assessment that evaluates problem-solving skills and world knowledge across 57 subjects including physics, mathematics, history, law, medicine, and ethics. It achieved a score of 90.0%.
By implementing our novel benchmark approach to MMLU, Gemini is empowered to employ its reasoning capabilities in a more deliberate manner prior to responding to complex inquiries. This results in substantial advancements compared to relying solely on initial impressions.
| Capability | Benchmark Higher is better |
Description | Gemini Ultra | GPT-4 API numbers were calculated where reported numbers were missing |
| General | MMLU | Representation of questions in 57 subjects (incl. STEM, humanities, and others) | 90.0% CoT@32* |
86.4% 5-shot** (reported) |
| Reasoning | Big-Bench Hard | A diverse set of challenging tasks requiring multi-step reasoning. | 83.6% 3-shot |
83.1% 3-shot (API) |
| DROP | Reading comprehension (F1 Score) |
82.4 Variable Shots |
80.9 3-shot (reported) |
|
| HellaSwag | Commonsense reasoning for everyday tasks | 87.8% 10-shot* |
95.3% 10-shot* (reported) |
|
| Math | GSM8K | Basic arithmetic manipulations (incl. Grade School math problems) | 94.4% maj1@32 |
92.0% 5-shot CoT (reported) |
| MATH | Challenging math problems (incl. Algebra, geometry, pre-calculus, and others) | 53.2% 4-shot |
52.9% 4-shot (API) |
|
| Code | HumanEval | Python code generation | 74.4% 0-shot (IT)* |
67.0% 0-Shot* (reported) |
| Natural2Code | Python code generation. New held out dataset Human Eval-like, not leaked on the web | 74.9% 0-shot |
73.9% 0-shot (API) |
*See the technical report for details on performance with other methodologies.
**GPT-4 scores 87.29% with CoT@32 – see the technical report for full comparison
Gemini outperforms current industry standards across various benchmarks, such as coding and text analysis.
In addition, Gemini Ultra attained an unprecedented score of 59.4% on the novel MMMU benchmark, an assessment comprised of multimodal exercises that span various domains and demand intentional thought.
Gemini Ultra exhibited superior performance compared to prior cutting-edge models when evaluated using image benchmarks. Notably, this triumph was not facilitated by optical character recognition (OCR) systems, which extract text from images for subsequent processing. These benchmarks serve to underscore Gemini’s inherent multimodality and serve as preliminary indicators of the development of Gemini’s rational complexity.
Refer to our Gemini technical report for further information.
| Capability | Benchmark Higher is better |
Description Higher is better unless otherwise noted. |
Gemini | GPT-4V The previous SOTA model listens when the capability is not supported in GPT-4V. |
| Image | MMMU | Multi-discipline college-level reasoning problems | 59.4% 0-shot pass@1 Gemini Ultra (pixel only*) |
56.9% 0-shot pass@1 GPT-4V |
| VQAv2 | Natural image understanding | 77.8%
0-shot |
77.2% 0-shot GPT-4V |
|
| DocVQA | Document understanding | 90.9% 0-shot Gemini Ultra (pixel only*) |
88.4% 0-shot GPT-4V (pixel only) |
|
| Infographic VQA | Infographic understanding | 80.3% 0-shot Gemini Ultra (pixel only*) |
75.1% 0-shot GPT-4V (pixel only) |
|
| MathVista | Mathematical reasoning in visual contexts | 53.0% 0-shot Gemini Ultra (pixel only*) |
49.9% 0-shot GPT-4V |
|
| Video | VATEX | English video captioning (CIDEr) |
62.7 4-shot Gemini Ultra |
56.0 4-shot DeepMind Flamingo |
| Perception Test MCQA |
Video Question Answering | 54.7% 0-shot Gemini Ultra |
46.3% 0-shot SeViLA |
|
| Audio | CoVoST 2 (21 languages) |
Automatic Speech translation (BLEU score) |
40.1 Gemini Pro |
29.1 Whisper v2 |
| FLEURS (62 languages) |
Automatic Speech recognition (based on word error rate, lower is better) |
7.6% Gemini Pro |
17.6% Whisper v3 |
*Gemini image benchmarks are pixel only – no assistance from OCR systems
Gemini exhibits performance that exceeds the current state of the art across various multimodal benchmarks.
Next-generation capabilities
Prior to this development, the prevailing method for constructing multimodal models entailed the training of distinct components for each modality, followed by their assembly in an attempt to approximate a portion of the functionality being modeled. These models may exhibit proficiency in specific tasks, such as image description, but encounter difficulties when confronted with more intricate and conceptual reasoning.
We pre-trained Gemini on various modalities from the outset to ensure that it was inherently multimodal. It was subsequently refined with additional multimodal data in order to increase its efficacy. This enables Gemini to reason and comprehend any type of input from the ground up in a seamless manner, which is significantly superior to existing multimodal models; furthermore, its capabilities are cutting-edge in virtually every domain.
Sophisticated reasoning
The advanced multimodal reasoning capabilities of Gemini 1.0 can assist in comprehending intricate written and visual data. This attribute endows it with an exceptional ability to reveal insights that may be challenging to discern amidst extensive volumes of data.
Its exceptional capability of extracting insights from hundreds of thousands of documents via information comprehension, filtering, and reading will facilitate the delivery of digital-speed innovations in numerous disciplines, including finance and science.
Understanding text, images, audio, and more
Gemini 1.0 was designed to simultaneously recognize and comprehend text, images, audio, and more; as a result, it is more attuned to nuanced information and can respond to inquiries concerning complex subjects. This characteristic renders it particularly proficient in elucidating the logical processes involved in intricate disciplines such as mathematics and physics.
Advanced coding
Our initial iteration of Gemini possesses the capability to comprehend, elucidate, and produce code of exceptional quality in the most widely used programming languages of the world, including Python, Java, C++, and Go. Its capability to reason about complex information and operate across languages positions it as one of the foremost foundation models for coding globally.
HumanEval, a significant industry standard for evaluating performance on coding tasks, and Natural2Code, our internal held-out dataset comprised of author-generated sources rather than web-based information, are two of the coding benchmarks in which Gemini Ultra excels.
Additionally, Gemini can function as the underlying framework for more sophisticated coding systems. AlphaCode, the inaugural artificial intelligence code generation system to achieve a competitive level of performance in programming competitions, was introduced by us two years ago.
AlphaCode 2, a more sophisticated code generation system developed with a specialized version of Gemini, demonstrates exceptional performance in resolving competitive programming challenges that transcend mere coding and encompass intricate mathematical and theoretical computer science concepts.
AlphaCode 2 exhibits significant enhancements when assessed on an identical platform as its predecessor. It solves approximately twice as many problems and, according to our estimation, outperforms more than 85% of competitors (compared to AlphaCode’s nearly 50%). AlphaCode 2 exhibits enhanced performance when developers collaborate with it by defining specific properties that code samples ought to adhere to.
We are enthusiastic about the growing adoption of advanced AI models by programmers as collaborative tools that aid in problem-solving, code design proposal generation, and implementation support. This will enable them to release applications and develop services more efficiently.
More reliable, scalable, and efficient
On our AI-optimized infrastructure, we trained Gemini 1.0 at scale utilizing Tensor Processing Units (TPUs) v4 and v5e developed in-house by Google. Furthermore, we engineered it to be our most dependable, scalable, and effective training and serving model to date.
Gemini operates substantially quicker on TPUs than its predecessors, which were smaller and less capable. These custom-built AI accelerators have been the foundation of billions of users’ favorite Google products fueled by AI, including Android, Search, YouTube, Gmail, Google Maps, and Google Play. They have also facilitated the cost-effective training of large-scale AI models for businesses worldwide.
Cloud TPU v5p, the most potent, scalable, and efficient TPU system to date, is being introduced today. It is specifically engineered for training state-of-the-art AI models. This TPU of the following generation will facilitate the training of large-scale generative AI models by enterprise clients and developers, thereby enabling the introduction of new products and functionalities to consumers in a more timely manner.
Built with responsibility and safety at the core
In all that we do at Google, we are dedicated to the advancement of responsible and audacious AI. Expanding upon Google’s AI Principles and the comprehensive safety protocols that underpin our products, we are incorporating additional safeguards to accommodate the multimodal capabilities of Gemini. We consider potential risks at each stage of development and work diligently to evaluate and mitigate them.
To date, Gemini has undergone the most exhaustive safety assessments of any Google AI model, encompassing toxicity and bias evaluations. Innovative research has been undertaken to explore potential risk domains such as cyber offense, persuasion, and autonomy. Google Research’s preeminent adversarial testing techniques have been utilized to assist in the early detection of critical safety concerns prior to the deployment of Gemini.
In order to detect deficiencies in our internal evaluation methodology, we are conducting stress testing of our models across a variety of challenges in collaboration with a diverse group of external experts and partners.
In order to identify content safety concerns throughout Gemini’s training stages and guarantee adherence to our policies, we employ benchmarks including Real Toxicity Prompts, an anthology of one hundred thousand web-based prompts curated by professionals at the Allen Institute for AI and featuring diverse levels of toxicity. Information regarding this endeavor will be forthcoming.
We developed safety-specific classifiers to identify, label, and categorize content containing negative stereotypes or violent material, for instance, in order to prevent damage. By integrating strong filters, this multi-layered strategy aims to enhance the safety and inclusivity of Gemini for all users. Moreover, we persist in tackling established obstacles that models encounter, including but not limited to factuality, grounding, attribution, and corroboration.
Ensuring accountability and security shall perpetually guide the design and implementation of our models. As this endeavor necessitates sustained collaboration, we are establishing safety and security benchmarks and defining best practices in conjunction with the industry and broader ecosystem via organizations such as MLCommons, the Frontier Model Forum and its AI Safety Fund, and our Secure AI Framework (SAIF). Designed to assist in the mitigation of security risks unique to AI systems in both the public and private sectors, SAIF is a collaborative effort. We will maintain global collaborations with governments, researchers, and civil society organizations as we advance the development of Gemini.
Making Gemini available to the world
Gemini 1.0 is currently being distributed on the following platforms and products:
Gemini Pro in Google products
Gemini is being made available to billions of individuals via Google products.
Bard will begin utilizing a refined version of Gemini Pro today in order to improve his rationale, planning, comprehension, and more. This is the most significant update to Bard since its inception. We anticipate that it will be accessible in English across over 170 countries and territories. Furthermore, we have ambitious plans to extend our support to additional languages, regions, and modalities in the coming times.
Additionally, Gemini is joining Pixel. The first smartphone designed to operate Gemini Nano is the Pixel 8 Pro. This operating system powers new features such as Summarize in the Recorder app and Smart Reply in Gboard, which is launching with WhatsApp, Line, and KakaoTalk1 before expanding to more messaging apps next year.
Gemini will be integrated into an expanding array of our products and services, including Search, Ads, Chrome, and Duet AI, in the coming months.
Already in the process of experimenting with Gemini in Search, it is accelerating our Search Generative Experience (SGE) for users in the United States by 40% in English, in addition to quality enhancements.
Building with Gemini
Gemini Pro will be accessible to developers and enterprise clients via the Gemini API in Google AI Studio or Google Cloud Vertex AI beginning on December 13.
Google AI Studio is a web-based, no-cost development tool that enables rapid app prototyping and activation using an API key. Vertex AI enables Gemini to be customized with complete data control and provides additional Google Cloud features for enterprise security, privacy, safety, and data governance and compliance when the time comes for a fully managed AI platform.
A new system capability introduced in Android 14, beginning with Pixel 8 Pro devices, will enable Android developers to construct with Gemini Nano, our most efficient model for on-device tasks, via AICore. Register to receive an advance look at AICore.
Gemini Ultra coming soon
We are in the process of conducting thorough trust and safety assessments for Gemini Ultra. This includes red-teaming by reputable external entities and additional model refinement through reinforcement learning from human feedback (RLHF) and fine-tuning prior to its general release.
We will provide early feedback and experimentation with Gemini Ultra on a select group of customers, developers, partners, and safety and responsibility experts prior to its release to enterprise customers and developers early the following year.
We will also introduce Bard Advanced, an innovative AI experience that grants access to our finest models and capabilities, beginning with Gemini Ultra, early the following year.
The Gemini Age: Facilitating An Innovative Future
This marks a momentous milestone in the progression of artificial intelligence and signifies the commencement of a fresh era for Google as we persistently innovate and conscientiously enhance the functionalities of our models.
Significant advancements have been achieved thus far with Gemini, and diligent efforts are being put forth to augment its functionalities in forthcoming iterations. These efforts will encompass improvements in memory and planning, as well as an enlargement of the context window to facilitate the processing of even more data in order to deliver more effective responses.
We are enthralled by the extraordinary prospects of a world empowered responsibly by AI — an innovative future that will augment ingenuity, expand understanding, propel scientific progress, and revolutionize the way in which billions of individuals worldwide labor and reside.