What is Le Chat by Mistral AI? What's the difference compared to ChatGPT?

But does this fledgling AI chatbot really deserve this title? Is Mistral's Le Chat better than ChatGPT?

What is Le Chat by Mistral AI?

Le Chat is a conversational AI chatbot developed by French startup Mistral AI. It is supported by several major language models owned by Mistral, including Mistral Large, Mistral Small and Mistral Next, all of which you can choose to use when interacting with the AI ​​chatbot. Although it is a relatively new entrant in the AI ​​chatbot space, it is highly regarded for the performance of its AI models despite its smaller scale when compared to industry heavyweights such as Gemini and GPT-4.

To understand what this means, imagine you are playing with building blocks. The more blocks you have, the more complex and detailed structures you can build, right? AI Language models are a bit like that. They come in many sizes, usually expressed by the number of parameters. So you may have heard terms like 'parameter 7B' or 'parameter 70B' in AI models. The number of parameters is like the number of building blocks that the model must understand and generate responses. So if a language model has more parameters, it can understand and generate better and more complex responses.

Currently, while GPT-4 is estimated to have around 1.76 trillion parameters, Mistral AI is estimated to have between 7 and 56 billion parameters. Do you see a difference in quantity? This is one of the reasons leading to the hype about the capabilities of this AI chatbot.

Although Le Chat does not enjoy the same level of exposure as ChatGPT nor the brand value of Gemini, it still finds its way into the conversation whenever a potential ChatGPT competition is discussed. But does it deserve a place on the scale?

Compare Le Chat and ChatGPT

Creativity

Creativity is one of the most important metrics to evaluate the performance of conversational AI chatbots. Remember, the purpose of AI chatbots is to replicate or imitate human conversational and creative abilities at scale. This makes creativity a very important strength for any AI chatbot. The world has been testing ChatGPT for more than a year and its creativity is undeniable. But what about Le Chat? Let's put both chatbots through a series of creativity tests!

Start by asking both chatbots: "How would you describe yourself to an artist?" to test your ability to use creative and imaginative words to self-conceptualize.

This is how ChatGPT would describe itself:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 1

And this is also how Le Chat describes itself:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 2

Both answers are relevant in their own ways. ChatGPT is more invested in using vivid imagery and metaphors to describe itself, demonstrating creative flair. On the other hand, Le Chat's response was informative and focused on describing its nature as an AI chatbot. Some might say that it lacks the creative flair and artistic approach that ChatGPT's response represents. However, many people will prefer Le Chat's more visualizable answer to ChatGPT's abstract description.

Then, the author of the article asked ChatGPT and Le Chat to write a rap song about becoming rich by growing cucumbers - a difficult request to test the creativity of other chatbots. How many rap songs about cucumbers can you find online?

Here is ChatGPT's response:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 3

And here is Le Chat's response:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 4

This may be a subjective matter, but ChatGPT's response seems to be the better option here. Le Chat's lyrics seem quite wordy and don't really sound like something a rapper would create. To test how both lyrics would sound if they were turned into music, the author used the Suno AI music generator to create music from the lyrics. All 3 tries, ChatGPT's lyrics sounded much better. Below is a sample from both AI chatbots, you can evaluate which chatbot works better.

Samples created from ChatGPT's lyrics

Sample 1:

Sample 2:

Samples created from the lyrics of Mistral Le Chat

Sample 1:

Sample 2:

The author tried several other creative tasks, like writing poetry, writing articles, and drafting complex work emails using an AI chatbot. Although it shows promise, Le Chat is clearly inferior to ChatGPT in all cases. It's important to point out that one area in which Le Chat is particularly strong is writing, despite some suggestive style complexity. However, in terms of comprehensive creativity, the victory belongs to ChatGPT.

Programming skills

Proficiency in coding has become an important requirement for large AI chatbots. Writing good code is a fundamental skill, but to truly stand out among the elite, an AI chatbot must demonstrate the ability to create code that can effectively solve a variety of complex problems. Previously, we built an entire web application from scratch using ChatGPT, which demonstrated its superior capabilities as a programming tool. But how good is Le Chat at writing code?

The author tasked both chatbots with writing a simple to-do list application using CSS, HTML, and JavaScript. ChatGPT has no trouble producing good results. The author copied the generated code and previewed it in the browser and this is what ChatGPT created:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 5

Each time the reminder repeats, ChatGPT creates a function-enriched to-do list app with different styles. There are no cases where the generated code does not work.

When I tried the same prompt with Le Chat, it produced code that seemed understandable, but when I tried running it in the browser, it didn't work. After repeating the prompt 3 times, no version produced code that could complete the given task. Le Chat failed at one of the most basic coding tasks!

Of course, one wouldn't judge Le Chat by just one failed test. Next, the author asked both chatbots to generate JavaScript and PHP code to encode and decode the text. In this second test, both ChatGPT and Le Chat generated code that could perform the given task. However, Le Chat's version seems like the version an inexperienced novice programmer would write. On the other hand, ChatGPT's code is more complete and appears to be written by an experienced programmer.

The author repeated several other programming experiments related to finding and fixing errors. As a result, ChatGPT always performs better than Le Chat. In some cases, Le Chat didn't even know what to do. It's very possible that Le Chat can match ChatGPT in some areas, but coding skills don't seem to be one of them.

Common sense and logical reasoning

An interesting feature of AI chatbots is their ability to solve complex tasks with ease but fail at simple tasks that require basic common sense to complete. Many chatbots, both established ones like ChatGPT and newer ones like Le Chat, have difficulty solving tasks that require common sense and human-style logical reasoning. So how do Le Chat and ChatGPT compare in this regard?

The author of the article asked both chatbots:  "If you have a 3-litre bottle and a 5-litre bottle. How can you measure 4 liters of water by using the 3-litre and 5-litre bottles?" (If you have a 3 liter bottle and a 5 liter bottle. How to measure 4 liters of water using the 3 liter and 5 liter bottles?)

ChatGPT solved the problem elegantly:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 6

Le Chat tried the same operation and solved the problem, albeit using a different approach.

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 7

Both chatbots showed comparable results in this test.

Next, the author asked both chatbots a trick question: "If a spaceship from Mars breaks into two, with one part crashing into the Atlantic Ocean near Brazil and the other into the Pacific Ocean near Japan, where do you bury the survivors ?" (If a spacecraft from Mars broke in two, with one part crashing into the Atlantic Ocean near Brazil and the other part falling into the Pacific Ocean near Japan, where would you bury the survivors?)

ChatGPT saw through this trick and responded accordingly:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 8

Le Chat also saw through this trick and responded accordingly:

What is Le Chat by Mistral AI? What's the difference compared to ChatGPT? Picture 9

The author tried more trick questions, it seems that both ChatGPT and Le Chat are quite proficient in handling logical and common sense reasoning prompts. However, with more complex logic questions, only ChatGPT can give appropriate answers.

While Le Chat has generated some buzz as a potential competitor to ChatGPT, testing shows it still has a lot of work to do before it can truly take on ChatGPT's heavyweights. world of AI chatbots. Although Le Chat demonstrated impressive abilities in areas such as common sense reasoning, its coding skills and creative output lagged significantly behind ChatGPT.

Like many aspiring contenders before him, Le Chat needs continued screening and training before he is ready for the big tournaments. For now, AI chatbots like ChatGPT are clearly still the undisputed kings of the AI ​​chatbot world. But the playing field of competitors is increasingly crowded, so the leaders cannot rest on their laurels.

Lesley Montoya

Update 28 March 2024