Forem: Celso Jr

Closures: Performance implications

Celso Jr — Thu, 25 Apr 2024 18:55:58 +0000

Performance implications with closures capture

Originally posted on https://www.celsojr.com/post/closures-performance-implications

The default closure capture mechanism used by the compiler is often the most efficient way to handle closure capture in many scenarios. If your struct is large or you're unsure about the implications, it's generally recommended to avoid custom closure captures and instead use an imperative code style or traditional functions. Compositions with closures are not primarily intended to optimize performance, so they may not be the best choice in mission-critical, resource-saving operations.

But, if you want, you can always create your own helper class or struct to do this for you. However, believe me, it will be very hard for you to take care of all your closures alone depending on the size of your application and the APIs you're working with. And also very difficult to beat the efficiency of the default closure capture mechanism done by the compiler. But let's give it a try. Let's look at the following code example:

using System;
using System.Runtime.CompilerServices;

static class ClosureCompare
{
    private static int n = 0;
    private delegate void AddDelegate(int n);

    private readonly static DisplayStruct adder = new DisplayStruct(ref n);
    private readonly static AddDelegate invoke = adder.Add;

    static void Main()
    {
       invoke(1);
       Console.WriteLine(adder.GetValue()); // Output the result: 1
    }
}

readonly unsafe struct DisplayStruct
{
    private readonly int* num;

    public DisplayStruct(ref int initialValue)
    {
        fixed (int* ptr = &initialValue)
        {
            num = ptr;
        }
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public void Add(int n)
    {
        *num += n;
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public int GetValue()
    {
        return *num;
    }
}

Note that I'm working with a low-level pointer so I can change a "read-only" property, and so I'm also running this code in an unsafe environment. I know this code doesn't seem very convenient, but otherwise, believe me, it won't be worth doing your own closure capture because the compiler will do it better than you. If you check the low-level C# code, you will see that the old <>c__DisplayClass0_0 class is now gone.

But, on the other hand, if you do not want to use closures at all, you can still have a specialized struct to do the dirty work for you, like so:

using System;
using System.Runtime.CompilerServices;

static class ClosureCompare
{
    static void Main()
    {
       int num = 0;

       DisplayStruct adder = new DisplayStruct(ref num);

       adder.Add(1); // Perform addition operation

       Console.WriteLine(adder.GetResult()[0]); // Output the result: 1
    }
}

public readonly ref struct DisplayStruct
{
    private readonly Span<int> num;

    public DisplayStruct(ref int initialValue)
    {
        num = new Span<int>(ref initialValue);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public void Add(int n)
    {
        if (num.Length > 0)
            num[0] += n;
        else
            throw new InvalidOperationException("Span is empty.");
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public Span<int> GetResult()
    {
        return num;
    }
}

Note the use of the ref modifier for this new struct. That will both allow you to use the Span<T> as a struct field and also prevent people from using this struct to work with the most common closures because you cannot use a ref local inside an anonymous method, lambda expression, or query expression as per their official documentation.

You can try this code yourself, but at least on my machine it was the fastest one in the last LTS runtime. And of course, it's always recommended to do your own benchmarks when necessary because this result below can vary from one machine to another:

// * Summary *

BenchmarkDotNet v0.13.7, Windows 11 (10.0.22631.3447)
AMD Ryzen 5 1600, 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.204
  [Host]   : .NET 8.0.4 (8.0.424.16909), X64 RyuJIT AVX2
  .NET 7.0 : .NET 7.0.18 (7.0.1824.16914), X64 RyuJIT AVX2
  .NET 8.0 : .NET 8.0.4 (8.0.424.16909), X64 RyuJIT AVX2


|                Method |  Runtime |        Mean |     Error |    StdDev |      Median | Ratio | Rank |   Gen0 | Allocated |
|---------------------- |--------- |------------:|----------:|----------:|------------:|------:|-----:|-------:|----------:|
|  CustomClosureCapture | .NET 7.0 |   3.2424 ns | 0.0994 ns | 0.1104 ns |   3.2463 ns |  0.16 |    1 |      - |         - |
|             NoClosure | .NET 7.0 |  12.3134 ns | 0.2658 ns | 0.2356 ns |  12.2293 ns |  0.60 |    2 |      - |         - |
| DefaultClosureCapture | .NET 7.0 |  20.4729 ns | 0.4359 ns | 0.9840 ns |  20.2206 ns |  1.00 |    3 | 0.0017 |      88 B |
|                       |          |             |           |           |             |       |      |        |           |
|             NoClosure | .NET 8.0 |   0.0454 ns | 0.0299 ns | 0.0456 ns |   0.0337 ns | 0.002 |    1 |      - |         - |
|  CustomClosureCapture | .NET 8.0 |   3.1839 ns | 0.0956 ns | 0.1243 ns |   3.1440 ns | 0.149 |    2 |      - |         - |
| DefaultClosureCapture | .NET 8.0 |  21.5585 ns | 0.4636 ns | 0.4961 ns |  21.5625 ns | 1.000 |    3 | 0.0014 |      88 B |

How to avoid surprises with closures

This is better to understand how the closure capture work in C# to avoid surprises. We already know that closures are something running in a different scope or environment, whether it is a function or an expression. And by different scope or environment, closures can also be running in a different thread. That is when we should start to be more aware of how things work. Let's take a look at this code:

using System;
using System.Threading;

static class ClosureCompare
{
    static void Main()
    {
       int[] arr = [1, 2, 3, 4, 5];

       foreach(int n in arr)
       {
           ThreadPool.QueueUserWorkItem(_ => Console.Write(n));
       }

       // Wait a bit for the Thread Pool threads to do their work
       // as we are not joining the threads together again
       Thread.Sleep(2_000);
    }
}

This code should run smoothly and, on most machines, two seconds should be enough for all threads to be scheduled and perform their work on time. It should output something like 12345 to the console, but not always in the same order because the execution scheduling is not being managed by code and it depends on availability of threads.

So far so good, ah? And what about this next code snippet below, now using a for loop?

using System;
using System.Threading;

static class ClosureCompare
{
    static void Main()
    {
       int[] arr = [1, 2, 3, 4, 5];

       for (int i = 0; i < arr.Length; i++)
       {
           // Without manually capturing the closure here, value is always 5 leading
           // to an unhandled out of range exception that you may never know about
           ThreadPool.QueueUserWorkItem(_ => Console.Write(arr[i]));
       }

       // Wait a bit for the Thread Pool threads to do their work
       // as we are not joining the threads together again
       Thread.Sleep(2_000);
    }
}

But why this error if we have limited the loop for to the same size of the array correctly?

i < arr.Length

Well, we will see that the way these loops work is a little different. Microsoft has changed the way the foreach loop works since version 5.0 of the language. And according to the C# language specification [1], "The placement of v inside the while loop is important for how it is captured by any anonymous function occurring in the embedded_statement."

If you take a look at the generated low-level C# code, you will see that the foreach is still capturing the closure by reference. But now, the compiler is creating a new instance of that helper class with a copy of the array item for each iteration. Can you spot the difference in the code snippet below?

private static void Main()
{
    <>c__DisplayClass0_0 <>c__DisplayClass0_ = new <>c__DisplayClass0_0();
    int[] array = new int[5];
    RuntimeHelpers.InitializeArray(array, (RuntimeFieldHandle));
    <>c__DisplayClass0_.arr = array;
    int[] arr = <>c__DisplayClass0_.arr;
    int num = 0;
    // Foreach loop translated
    while (num < arr.Length)
    {
        <>c__DisplayClass0_1 <>c__DisplayClass0_2 = new <>c__DisplayClass0_1();
        <>c__DisplayClass0_2.n = arr[num]; // Making a copy of the array item here
        ThreadPool.QueueUserWorkItem(new WaitCallback(<>c__DisplayClass0_2.<Main>b__0));
        num++;
    }
    <>c__DisplayClass0_2 <>c__DisplayClass0_3 = new <>c__DisplayClass0_2();
    <>c__DisplayClass0_3.CS$<>8__locals1 = <>c__DisplayClass0_;
    <>c__DisplayClass0_3.i = 0;
    // Loop for translated
    while (<>c__DisplayClass0_3.i < <>c__DisplayClass0_3.CS$<>8__locals1.arr.Length)
    {
        ThreadPool.QueueUserWorkItem(new WaitCallback(<>c__DisplayClass0_3.<Main>b__1));
        <>c__DisplayClass0_3.i++;
    }
    Thread.Sleep(2_000);
}

This is interesting the way these loops are translated into the same while loop and are still different. But the key point is to know how do they work, because this is not a problem as it may look like. And the way the loop for works, can possibly be more performant if not degraded by JIT [2] compilation.

So, what's happening with the way this for loop is written is that when the loop is translated into a while loop, what we get is a reference to the variable i of the helper class. This way, when the variable is incremented for the last time within the loop from 4 to 5, before the last loop check, those threads that have a reference to that same variable will be trying to access index 5 of an array of size 4.

And the value of the variable will almost always be 5, because at this point the loop has already incremented it by 5 times. The loop runs faster, a matter of nano seconds or even less, than the operation necessary to instrument the creation of threads in the thread pool including, but not limited, to the scheduling of execution.

In order to make it work as expected, we just need to copy the current increment by manually capturing the closure by value instead of by reference, as shown in the code snippet below:

using System;
using System.Threading;

static class ClosureCompare
{
    static void Main()
    {
       int[] arr = [1, 2, 3, 4, 5];

       for (int i = 0; i < arr.Length; i++)
       {
           // Making a copy of the current increment
           // and capturing the closure by value, instead of reference
           int closureCapture = i;
           ThreadPool.QueueUserWorkItem(_ => Console.Write(arr[closureCapture]));
       }

       // Wait a bit for the Thread Pool threads to do their work
       // as we are not joining the threads together again
       Thread.Sleep(2_000);
    }
}

But this is not a "problem" reserved to the loops only. This can also happen with Timers. And worse than that, it can happen in the opposite way. Timers have low-level APIs and are not very commonly used because there are higher-level abstractions, such as BackgroudWorker [3], which offer more flexible APIs and a better experience. But let's check this pseudo code example below with a low-level Timer:

using System;
using System.Timers;
using System.Threading.Tasks;

using Timer = System.Timers.Timer;

class Program
{
    static int count = 0;
    static int[] items = [1, 2, 3];
    static TaskCompletionSource tcs = new TaskCompletionSource();

    static async Task Main()
    {
        var timer = new Timer() { Interval = 100 };

        timer.Elapsed += (sender, e) => CronJob(sender, e,
            count); // Capturing count by value

        timer.Enabled = true;
        await tcs.Task;

        timer.Stop();
        timer.Dispose();

        Console.WriteLine("Timer stopped.");
    }

    private static void CronJob(object? source, ElapsedEventArgs e, int count)
    {
        Console.WriteLine("Item: {0}", items[count]);

        count++;

        if (count == items.Length)
        {
            tcs.SetResult(true);
        }
    }
}

In this example, the variable count is being passed to the CronJob function by value and, therefore, will never be incremented more than once inside that function scope, leading to an infinity run.

By default, arguments in C# are passed to functions by value. That means a copy of the variable is passed to the method. [4]

To make this code work as expected, we just need to use a small ref key word that will work the same way the & sign works in PHP, as we saw in the first blog post of this closures series. Please check the updated code below:

using System;
using System.Timers;
using System.Threading.Tasks;

using Timer = System.Timers.Timer;

class Program
{
    static int count = 0;
    static int[] items = [1, 2, 3];
    static TaskCompletionSource tcs = new TaskCompletionSource();

    static async Task Main()
    {
        var timer = new Timer() { Interval = 100 };

        timer.Elapsed += (sender, e) => CronJob(sender, e,
            ref count); // Capturing count now by reference

        timer.Enabled = true;
        await tcs.Task;

        timer.Stop();
        timer.Dispose();

        Console.WriteLine("Timer stopped.");
    }

    private static void CronJob(object? source, ElapsedEventArgs e,
        ref int count) // Capturing count now by reference
    {
        Console.WriteLine("Item: {0}", items[count]);

        count++;

        if (count == items.Length)
        {
            tcs.SetResult(true);
        }
    }
}

Of course, this is not recommended to use low-level APIs in the development of enterprise applications unless it is really necessary. Low-level code is more error prone and, among other things, should also have a negative impact on readability.

The code example previously shown was just simulating the problems that can arise when we don't really know how things work. And with this, I hope to have helped more people understand a little more about scope, closures and compositions. And also, how to take advantage of it. Happy coding!

Disclaimer

It's worth noting that I'm not a Microsoft employee. All opinions in this blog post are my own. The information displayed here is not endorsed by Microsoft, .Net Foundation or any of their partners. This is not a sponsored post. All rights reserved.

C# language specification Learn Microsoft, Retrieved April 25, 2024.
Managed execution process Learn Microsoft, Retrieved April 25, 2024.
BackgroundWorker Class Learn Microsoft, Retrieved April 25, 2024.
Method Parameters, Learn Microsoft, Retrieved April 27, 2024.

Closures: Lifting the hood

Celso Jr — Thu, 25 Apr 2024 18:55:11 +0000

Lifting the hood of the closures

Originally posted on https://www.celsojr.com/post/closures-lifting-the-hood

Closure is not something new. There are a bunch of articles and tutorials out there on the internet talking about closures. In fact, closures have been around for several years since this term was first introduced by Peter J. Landin in the year 1964 [1], the same year my mother was born.

But thanks to the Scheme programming language (a lexically scoped variant of Lisp) which was defined by Sussman and Steele in 1975, closures then became very popular. After that, pretty much all programming languages have closures.

Scheme is a functional-first programming language. Closures are most commonly used in a functional programming paradigm context. But we can see the use of closures in enterprise applications nowadays because most programming languages, like Scheme, are multi-paradigm even though some of them are not functional-first. Only the most simple/primitive programming languages are single-paradigm.

But despite being an old subject, closures are still a hot topic today. For example, in this report from the Rust programming language, asynchronous closures are one of the most desired features that people hope will be stabilized.

If you are a beginner and you want to better understand this programming concept, I suggest that you first try to understand about scope and compositions. A scope refers to the region of code where a variable or function is accessible. It determines where you can reference and use variables or functions within your program. In C# and many other languages, the simplest scope can be delimited between curly braces. Look at this pseudo code example:

var a = "outer scope";

{
    var b = "inner scope";
    var c = a; // Yeah, 'c' can see the 'a' outer scope from here
}

var d = b; // ERROR: But 'd' cannot see 'b'

This is still not a closure because there is no closed operation/expression involved.

I'm just trying to clarify things so that everybody can understand.

Imagine that you are a function and you ate food a few hours ago. Now you are a closure entering the bathroom with the food variables in your stomach to process your stuff — and please, I don't need to know details about how you process your stuff from here, on the outer scope ;)

When a variable in the outer scope is captured by a function to be part of its enclosed processing block in a different scope or environment resulting in a closed operation or expression, we can call this a closure. But this is not the definition, I'm just trying to explain closures with my own words. You can do quick research and find multiple different explanations better than mine.

But why are closures really necessary? Well, as we are talking about scope, they are necessary because those functions or expressions may want to use the variables in its inner scope regardless of what is happening in the outer scope or not. Also, depending on the API you are working with, it should allow you to create powerful compositions by using closures wisely.

In C#, the closures were introduced to help with the anonymous functions and lambdas in the language version 2.0 [2]. Closures make it much simpler to create delegates and design APIs which use delegates.

For example, the entire Linq [3] is built on top of this concept. In my opinion, Linq is a very well-designed API, although it was not initially designed with major concerns in terms of low-level programming and memory consumption. It allows us to make beautiful compositions to better express our intentions without having to create separate functions throughout the application. Also, most code editors and IDEs should provide you with IntelliSense[4] and/or IntelliCode[5] capabilities to empower your developer experience and productivity.

Both approaches, query or method syntax, are composed using closures because query syntax translates into method syntax at runtime anyway. Most applications should use both approaches together.

Without IntelliSense and IntelliCode, compositions with closures wouldn't be the same. Some time ago, my friend at Microsoft, Gustavo, and I had some (external) conversations via email about IntelliCode and he shared with me a paper [6] about the idea behind the IntelliCode. He is Brazilian too. It was a wonderful read and it's always a great experience to be able to work with IntelliCode in Visual Studio. IntelliCode is an amazing tool from Gustavo, Mark and their team.

Closures capture

Programmers working with Linq make use of closures all the time, even if they are not aware of it. Closures are normally captured by reference. And in C#, this happens in an implicit way. Let's look at this pseudo code example:

using System;

int num = 0; // outer scope

Action added = (n) =>
{
    num = num + n; // inner scope
};

added(1);

Console.WriteLine(num); // Output will be 1

As you can see, we are using an Action delegate to sum up two numbers. The num variable, which is in the outer scope, is being captured by reference by the closure and the delegate in the inner scope. This way, when the action is executed, the original variable num is changed.

In C# there are a variety of delegate types with different flavors. There is also the local functions [7] that can work as closures if needed.

The language provides flexible APIs that allow us to work with transformations and compositions as we prefer. But we won't go into detail about delegates or local functions for now.

But how to check how the closure is being captured and whether it's being captured by reference or value? Now let's take a look under the hood, at one lower level, at what the compiler is doing for us:

[CompilerGenerated]
internal class Program
{
    [CompilerGenerated]
    private sealed class <>c__DisplayClass0_0 // <- 😐 Compiler generated class
    {
        public int num;

        internal void <<Main>$>b__0(int n)
        {
            num += n;
        }
    }

    private static void <Main>$(string[] args)
    {
        <>c__DisplayClass0_0 <>c__DisplayClass0_ = new <>c__DisplayClass0_0();
        <>c__DisplayClass0_.num = 0;
        Action<int> action = new Action<int>(<>c__DisplayClass0_.<<Main>$>b__0);
        action(1);
        Console.WriteLine(<>c__DisplayClass0_.num);
    }
}

In this code snippet generated by SharpLab, we can see that the compiler has generated some sort of helper class that contains the closure capture, in this case, the variable num. It's worth remembering that a class is a reference type object in C# and, therefore, will be allocated in the managed heap. And every time this delegate Action is invoked, the num variable of the same instance is changed.

If this behavior is not intended and you want to avoid this by any reason, you have to "manually" make a copy of the value yourself because there is no reserved keyword or any other mechanism that does this for you. See how to do it:

using System;

int num = 0; // outer scope

Action added = (n) =>
{
    int closureCapture = num; // Captured by value, not by reference
    closureCapture = closureCapture + n;

    // Do whatever you want with the closure capture copy ...
};

added(1);

Console.WriteLine(num); // Output now will be 0

Very simple, don't you think? Yeah, but closure capture may be done slightly different in other programming languages. For example, I prefer the way PHP allows us to explicitly capture closures with the help of the use keyword and the & sign, as shown:

<?php
    $num = 0;

    // Closure explicitly captured by reference
    $added = function($n) use (&$num) {
        $num = $num + $n;
    };

    $added(1);

    echo $num; // Output will be 1
?>

To me, the fact that a strongly typed language like C# captures closures implicitly is funny. Whereas PHP, which is a weakly typed language, does this explicitly. I mean, of course we can express ourselves however we want in both languages, but at least in PHP there is a reserved word use for that. And, by the way, in PHP it seems like there is a reserved word and a method for almost everything. I don't know if that's good or not. Judge yourselves!

Please check out the next blog post about closures.

Disclaimer

Closure (computer programming) Wikipedia, Retrieved April 18, 2024.
Richard B., & Andrew C., Pro Asynchronous Programming with .NET, Apress, 2013, p. 23.
Language Integrated Query (LINQ) Learn Microsoft, Retrieved April 21, 2024.
IntelliSense in Visual Studio Learn Microsoft, Retrieved April 21, 2024.
Visual Studio IntelliCode, Retrieved April 21, 2024.
Paper (PDF) Anders Miltner, Sumit Gulwani, Vu Le, Alan Leung, Arjun Radhakrishna, Gustavo Soares, Ashish Tiwari, and Abhishek Udupa. 2019. On the Fly Synthesis of Edit Suggestions. Proc. ACM Program. Lang. 3, OOPSLA, Article 143 (October 2019), 29 pages. https://doi.org/10.1145/3360569
Local functions (C# Programming Guide) Learn Microsoft, Retrieved April 21, 2024.

Building A Python Script For Natural Language Q&A In The Console For Free Using A LLaMA Model

Celso Jr — Mon, 19 Jun 2023 10:07:44 +0000

Motivation

In today's digital age, our dependency on the internet for various tasks has become a norm. However, there are instances when we find ourselves without a stable internet connection while venturing outside. Imagine having access to a powerful language model without relying on the internet, and being able to experiment with different configurations and scenarios.

One of the challenges we face when we step outside is the unreliable or non-existent internet connection. But having a console-based language generation liberates us from this constraint by providing a self-contained environment that doesn't rely on continuous internet access.

Why pay for complex and expensive language generation services in the cloud when you only need simple and low-quality outputs? The console-based approach caters to those seeking quick, straightforward responses without the burden of hefty costs. By leveraging offline capabilities, it eliminates the need for constant connectivity, ensuring cost-effectiveness for those looking for minimalistic language generation solutions.

Every language model has its own unique characteristics and applications. With console-based language generation, you can explore and experience the diverse capabilities of different llama models. Whether you're looking for a specific tone, style, or expertise, the ability to taste and experiment with various llama models opens up a world of creative possibilities.

No two scenarios are the same, and our language generation needs can vary based on context. The console-based approach empowers users to experiment with different configurations and scenarios. By modifying prompts, adjusting token limits, or exploring stop conditions, users can tailor the language generation experience to meet their specific requirements.

Prerequisites

Python and basic skills with pip
llama-cpp bindings for Python
A compatible LLaMA model
A computer with enough CPU, Memory and Storage (GPU is not required)

First of all, you have to install the dependencies

python3 -m pip install llama-cpp-python termcolor

Or if you have previously installed llama-cpp-python through pip and want to upgrade your version, just run this instead:

python3 -m pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

Note: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh

Downloading the LLaMA model

To harness the power of llama-cpp-python, we need to start by downloading a suitable model.

For this blog post, we'll be working with the WizardLM model, specifically the wizardLM-7B.ggmlv3.q4_1.bin variant. This particular model strikes a balance between accuracy and inference speed, making it ideal for CPU usage.

While there are other models available with varying levels of accuracy and speed, feel free to experiment and find the one that best suits your needs. Keep in mind that the selected model may require 2.8 GB to 8 GB of disk space, and up to 10 GB of memory.

Ensure you explore the linked models in the README to discover the full range of supported options. This flexibility allows you to choose models that align with your specific requirements. With the right model in hand, you're ready to embark on a journey of language generation and exploration.

The Python Script

Using your favorite text editor, create a python script and name it whatever you want. e.g., my-script.py

Here’s our script to experiment with:

Note: Don't forget to insert you LLaMA model name in the MODEL_PATH variable.

#!/bin/python
import os
import sys
from termcolor import colored
from llama_cpp import Llama

MODEL_PATH = "models/<here comes the LLaMA model file name>"
MAX_TOKENS = 2048

def run_console_app():
    if not os.path.isfile(MODEL_PATH):
        print(f"Error: Invalid model path '{MODEL_PATH}'")
        sys.exit(1)

    llm = Llama(model_path=MODEL_PATH)

    print(colored("\n## Welcome to the Llama Console App! ##\n", "yellow"))
    print("Enter your prompt (or 'q' to quit):")

    while True:
        user_input = input("> ")
        if user_input.lower() == 'q':
            print("Exiting...")
            break

        user_input = f"Q: {user_input} A:"
        print(colored("Processing...", "yellow"))

        try:
            output = llm(user_input, max_tokens=MAX_TOKENS, stop=["Q:"], echo=True)

            choices_text = output["choices"][0]["text"]
            choices_text = choices_text.replace(user_input, "").strip()
            formatted_text = colored(choices_text, "green")
            print(f"\n{formatted_text}\n")
        except Exception as e:
            print(f"Error: Failed to generate response. {str(e)}")

if __name__ == '__main__':
    run_console_app()

How to use it?

That’s pretty much it! Now you can open your preferred terminal and just run python3 <your .py file name>.py and voilà.

You should see a prompt like this:

Conclusion

With the ability to taste different llama models and experiment with configurations, this approach puts the power of language generation directly into our hands. Whether you're a writer, researcher, or problem-solver, this console-based solution opens up a world of possibilities, even when you're on the go.

Happy chatting's!