Difference between Google PaLM 2 and OpenAI GPT-4

Google announced the next generation of Pathways Language Model (PaLM 2) on May 10, 2023, at Google I/O 2023. PaLM 2's new Large Language Model (LLM) boasts many improvements over its predecessor. with its predecessor (PaLM) and may finally be ready to take on its biggest rival, OpenAI's GPT-4.

But how much has Google improved? Is PaLM 2 what makes the difference that Google expects? How is PaLM 2 different from OpenAI's GPT-4?

Performance overview of PaLM 2 and GPT-4

PaLM 2 is packed with new and improved capabilities compared to its predecessor. One of the unique advantages of PaLM 2 over GPT-4 is that it is available in smaller sizes specifically for some applications without a lot of built-in processing power.

All of these different sizes have their own smaller models called Gecko, Otter, Bison, and Unicorn, where Gecko is the smallest, followed by Otter, Bison and finally Unicorn, the model biggest.

Difference between Google PaLM 2 and OpenAI GPT-4 Picture 1

Google also claims an improvement in inference ability over GPT-4 in WinoGrande and DROP, while GPT-4 previously had a narrow margin in ARC-C. However, there is significant improvement when it comes to PaLM and SOTA.

PaLM 2 is also better at math, according to Google's 91-page PaLM 2 research paper. However, the way Google and OpenAI structure their test results makes it difficult to directly compare the two models. Google also omits some comparisons, possibly because PaLM 2 doesn't perform as well as GPT-4.

In the MMLU, GPT-4 scored 86.4, while PaLM 2 scored 81.2. The same goes for HellaSwag, where GPT-4 scored 95.3, but PaLM 2 could only get 86.8 and ARC-E, where GPT-4 and PaLM 2 scored 96.3, respectively. and 89.7.

Difference between Google PaLM 2 and OpenAI GPT-4 Picture 2

The largest model in the PaLM 2 series is the PaLM 2-L. Although its exact size is unknown, it is certainly considerably smaller than the largest PaLM model but uses more training computers. According to Google, PaLM has 540 billion parameters, so "significantly smaller" should place PaLM 2 anywhere between 10 and 300 billion parameters. Keep in mind that these numbers are just assumptions based on what Google said in the PaLM 2 paper.

If the number is anywhere near 100 billion or less, PaLM 2 is most likely parametrically smaller than GPT-3.5. It is impressive to consider that a capable model under VND 100 billion can take on the GPT-4 and even beat it in some tasks. The original GPT-3.5 blew everything away, including the PaLM, but the PaLM 2 recovered pretty well.

Differences in GPT-4 and PaLM 2 . training data

While Google has not revealed the training dataset size of PaLM 2, the company reported in its research paper that the new LLM training dataset is significantly larger. OpenAI also took a similar approach when it published GPT-4, making no statement about the size of the training dataset.

However, Google wants to focus on deeper understanding of math, logic, reasoning, and science, meaning the majority of PaLM 2 training data focuses on the aforementioned topics. Google says in its paper that PaLM 2's pre-training corpus includes multiple sources, including web documents, books, code, math, and conversational data, giving it improvements on wide area, at least when compared with PaLM.

PaLM 2's conversational skills must also be on another level as the model has been trained in over 100 languages ​​to help it understand context and be able to translate better.

As far as GPT-4 training data goes, OpenAI told us it trained the model using publicly available and licensed data. The GPT-4 research page states: "Data is a web-scale repository of right and wrong solutions to problems, weak and strong arguments, contradictory and consistent statements, at the same time representing many different ideologies and ideas".

Difference between Google PaLM 2 and OpenAI GPT-4 Picture 3

When GPT-4 is asked a question, it can generate many different answers, not all of which are relevant to your query. To match user intent, OpenAI refined the model's behavior using reinforcement learning with human feedback.

While we don't know the exact training data on which either of these two models was trained, we do know that the training purposes are very different. We'll have to wait and see how the training purposes differ between these two models in real-world deployment.

Services and Chatbots PaLM 2 and GPT-4

The first gateway to accessing both LLMs is to use their respective chatbots, PaLM 2's Bard and GPT-4's ChatGPT. However, GPT-4 requires payment via ChatGPT Plus, and free users only have access to GPT-3.5. Bard, on the other hand, is free for everyone and available in over 180 countries.

That doesn't mean you can't access GPT-4 for free. Microsoft's Bing AI Chat uses GPT-4 and is completely free, open to everyone, and available right next to Bing Search, Google's biggest competitor in this area.

Difference between Google PaLM 2 and OpenAI GPT-4 Picture 4

Google I/O 2023 is flooded with announcements about how PaLM 2 and next-generation AI integration will improve the Google Workspace experience with AI features that will be available in Google Docs, Sheets, Slides, Gmail, and nearly every translation services provided by the search giant. Additionally, Google has confirmed that PaLM 2 has been integrated into more than 25 Google products, including Android and YouTube.

For comparison, Microsoft has included AI features in its Microsoft Office suite and many of its services. Currently, you can experience both LLMs in their own versions.

However, since the launch of GPT-4, and careful to avoid many of the silly mistakes Google made with the original Bard, it has become the de facto LLM for third-party developers, startups, and more. industry and anyone else looking to incorporate a capable AI model in their offerings to date.

Difference between Google PaLM 2 and OpenAI GPT-4 Picture 5

That's not to say developers won't switch to or at least try out PaLM 2, but Google still has to catch up with OpenAI in this respect. And the fact that PaLM 2 is open source, rather than locked behind a paid API, means it has the potential to be more widely adopted than GPT-4.

Can PaLM 2 beat GPT-4?

PaLM 2 is still very new, so the answer of whether it can compete with GPT-4 still needs time. However, with all that Google is promising and the aggressive way it has decided to use to spread it, it seems that PaLM 2 will be a formidable opponent to the GPT-4.

However, the GPT-4 is still a pretty capable model and, as mentioned earlier, beats the PaLM 2 in some comparisons. However, the many smaller models of the PaLM 2 give it an indisputable advantage. Gecko itself is so lightweight that it can work on mobile devices, even when offline. This means that PaLM 2 can support a completely different type of product and equipment that may have difficulty using GPT-4.

The AI ​​race is heating up

With the launch of PaLM2, the race for AI dominance has heated up, as it could be the first worthy competitor to go head-to-head with the GPT-4. With a newer multimodal AI model called "Gemini" also being trained, Google doesn't show any signs of slowing down here.

Marvin Fry

Update 27 June 2023