Orca Mini Model: The First Implementation of Orca-like Training Techniques

Microsoft Research recently published a paper called Orca Progressive learning from complex explanation traces of GPT4. The paper discussed how smaller open source models could be trained using explanations to become smarter models overall. This training method teaches models to understand the logic of how to get from a prompt to a solution, not just mimic large foundational models. By applying this technique, Microsoft hypothesized that these models would become much more intelligent, and therefore more useful.

After the publication of this paper, Microsoft did not release any models associated with it. However, we now have the first implementation of an Orca-like model: the Orca Mini Model.

What is Orca Mini Model?

The Orca Mini Model is a small open-source model that was created using the techniques described in the Orca research paper. There are three versions of the model: the 3 billion parameter model, the 7 billion parameter model, and the 13 billion parameter model. These models are relatively small, with the 3 billion parameter model able to fit on most consumer hardware.

To create the Orca Mini Model, the creators took the open Llama 13 billion parameter model and trained it on explain-tuned data sets created using instructions and input from Wizard LM Alpaca and Dolly V2 data sets. They applied the Orca research paper techniques, and the result is the Orca Mini Model.

How to Get Set Up

To get started with the Orca Mini Model, you need to download it from the RunPod website. The website provides a model card page that includes a prompt template - a simple prompt to test the model's ability.

To set up the Orca Mini Model, follow these steps:

1. Download the model from the RunPod website
2. Use the X llama model loader
3. Load the model onto your GPU
4. Test the model using the prompt template

Testing the Orca Mini Model

Unfortunately, the Orca Mini Model did not perform as well as expected during testing. Here are the results:

Test One: Outputting Numbers 1 to 100

The Orca Mini Model failed this simple test.

Test Two: Building a Snake Game

The Orca Mini Model also failed at this task.

Test Three: Writing a Poem about AI using Exactly 50 Words

The Orca Mini Model failed this task as well.

Test Four: Writing an Email to a Boss Informing Them of Your Resignation

The Orca Mini Model passed this task, but only just.

Test Five: Simple Math

The Orca Mini Model failed this task.

Test Six: Providing a Healthy Meal Plan

The Orca Mini Model was able to provide an answer, but it did not meet the requirements of the task. The answer provided was not a meal plan, but rather a single recommendation.

Test Seven: Who was the President of the United States in 1996?

The Orca Mini Model passed this task.

Test Eight: Recognizing Censorship

The Orca Mini Model passed this task.

Test Nine: Solving the Five Shirts Problem

The Orca Mini Model failed this task.

Test Ten: The Killers Problem

The Orca Mini Model passed this task.

Test Eleven: Identifying the Current Year

The Orca Mini Model failed this task.

Test Twelve: Identifying Bias

The Orca Mini Model passed this task.

Test Thirteen: Summarizing a Text

The Orca Mini Model passed this task.

Overall, the Orca Mini Model performed poorly during testing. However, this early implementation is not the official Orca model, and we can hope that the official version will vastly improve upon its current performance. In the meantime, the Orca Mini Model serves as an interesting study in the potential of Orca-like training techniques.