Forem: Phantz

Wanna have fun with Lazy Iterators in good ol' C?

Phantz — Fri, 14 May 2021 11:27:05 +0000

I made a demonstration + explanation of implementing a lazy iterator interface in pure C99. As well as implementing strong lazy abstractions with the iterators!

Check it out on github

How you can implement resumable tasks in celery

Phantz — Sun, 20 Dec 2020 12:45:07 +0000

Preface

Celery is highly scalable task system for python. It allows you to send tasks to queues and execute them asynchronously over a worker. Celery tasks on their own are very useful, however a lot of times I've noticed the need to have to pause/cancel a task. Now celery does allow you to shut down a task queue to cancel tasks. But I wanted something scalable, to pause, continue and cancel a task (without disrupting the queue) without any loss of data.

So I decided to use the workflow pattern, aided with a control bus to achieve this. Maybe this guide will help out someone else looking for pausable/resumable celery tasks as well!

You can find the full demonstration to play around with, in my github repo.

Guide

Concept

With celery workflows - you can design your entire operation to be divided into a chain of tasks. It doesn't necessarily have to be purely a chain, but it should follow the general concept of one task after another task (or task group) finishes.

Once you have a workflow like that, you can finally define points to pause at throughout your workflow. At each of these points, you can check whether or not the frontend user has requested the operation to pause and continue accordingly. The concept is this:-

A complex and time consuming operation O is split into 5 celery tasks - T1, T2, T3, T4, and T5 - each of these tasks (except the first one) depend on the return value of the previous task.

Let's assume we define points to pause after every single task, so the workflow looks like-

T1 executes
T1 completes, check if user has requested pause
- If user has not requested pause - continue
- If user has requested pause, serialize the remaining workflow chain and store it somewhere to continue later

... and so on. Since there's a pause point after each task, that check is performed after every one of them (except the last one of course).

But this is only theory, I struggled to find an implementation of this anywhere online so here's what I came up with-

Implementation

from typing import Any, Optional

from celery import shared_task
from celery.canvas import Signature, chain, signature


@shared_task(bind=True)
def pause_or_continue(
    self, retval: Optional[Any] = None, clause: dict = None, callback: dict = None
):
    # Task to use for deciding whether to pause the operation chain
    if signature(clause)(retval):
        # Pause requested, call given callback with retval and remaining chain
        # chain should be reversed as the order of execution follows from end to start
        signature(callback)(retval, self.request.chain[::-1])
        self.request.chain = None
        return "Pausing"
    else:
        # Continue to the next task in chain
        return retval


def tappable(ch: chain, clause: Signature, callback: Signature, nth: Optional[int] = 1):
    """
    Make a operation workflow chain pause-able/resume-able by inserting
    the pause_or_continue task for every nth task in given chain

    ch: chain
        The workflow chain

    clause: Signature
        Signature of a task that takes one argument - return value of
        last executed task in workflow (if any - othewise `None` is passsed)
        - and returns a boolean, indicating whether or not the operation should continue

        Should return True if operation should continue normally, or be paused

    callback: Signature
        Signature of a task that takes 2 arguments - return value of
        last executed task in workflow (if any - othewise `None` is passsed) and
        remaining chain of the operation workflow as a json dict object
        No return value is expected

        This task will be called when `clause` returns `True` (i.e task is pausing)
        The return value and the remaining chain can be handled accordingly by this task

    nth: Int
        Check `clause` after every nth task in the chain
        Default value is 1, i.e check `clause` after every task
        Hence, by default, user given `clause` is called and checked
        after every task

    NOTE: The passed in chain is mutated in place
    Returns the mutated chain
    """
    newch = []
    for n, sig in enumerate(ch.tasks):
        if n != 0 and n % nth == nth - 1:
            newch.append(pause_or_continue.s(clause=clause, callback=callback))
        newch.append(sig)
    ch.tasks = tuple(newch)
    return ch

*The implementation can also be seen in tappable.py

Explanation - `pause_or_continue`

Here pause_or_continue is the aforementioned pause point. It's a task that will be called at specific intervals (intervals as in task intervals, not as in time intervals). This task then calls a user provided function (actually a task) - clause - to check whether or not the task should continue.

If the clause function (actually a task) returns True, the user provided callback function is called, the latest return value (if any - None otherwise) is passed onto this callback, as well as the remaining chain of tasks. The callback does what it needs to do and pause_or_continue sets self.request.chain to None, which tells celery "The task chain is now empty - everything is finished".

If the clause function (actually a task) returns False, the return value from the previous task (if any - None otherwise) is returned back for the next task to receive - and the chain goes on. Hence the workflow continues.

Why are `clause` and `callback` task signatures and not regular functions?

Both clause and callback are being called directly - without delay or apply_async. It is executed in the current process, in the current context. So it behaves exactly like a normal function, then why use signatures?

The answer is serialization. You can't conveniently pass a regular function object to a celery task. But you can pass a task signature. That's exactly what I'm doing here. Both clause and callback should be a regular signature object of a celery task.

What is `self.request.chain`?

self.request.chain stores a list of dicts (representing jsons as the celery task serializer is json by default) - each of them representing a task signature. Each task from this list is executed in reverse order. Which is why, the list is reversed before passing to the user provided callback function (actually a task) - the user probably expects the order of tasks to be left to right.

Quick note: Irrelevant to this discussion, but if you're using the link parameter from apply_async to construct a chain instead of the chain primitive itself. self.request.callback is the property to be modified (i.e set to None to remove callback and stop chain) instead of self.request.chain

Explanation - `tappable`

tappable is just a basic function that takes a chain (which is the only workflow primitive covered here, for brevity) and inserts pause_or_continue after every nth task. You can insert them wherever you want really, it is upto you to define pause points in your operation. This is just an example!

For each chain object, the actual signatures of tasks (in order, going from left to right) is stored in the .tasks property. It's a tuple of task signatures. So all we have to do, is take this tuple, convert into a list, insert the pause points and convert back to a tuple to assign to the chain. Then return the modified chain object.

The clause and callback is also attached to the pause_or_continue signature. Normal celery stuff.

That covers the primary concept, but to showcase a real project using this pattern (and also to showcase the resuming part of a paused task), here's a small demo of all the necessary resources

Usage

This example usage assumes the concept of a basic web server with a database. Whenever an operation (i.e workflow chain) is started, it's assigned an id and stored into the database. The schema of that table looks like-

-- Create operations table
-- Keeps track of operations and the users that started them
CREATE TABLE operations (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  requester_id INTEGER NOT NULL,
  completion TEXT NOT NULL,
  workflow_store TEXT,
  result TEXT,
  FOREIGN KEY (requester_id) REFERENCES user (id)
);

The only field that needs to be known about right now, is completion. It just stores the status of the operation-

When the operation starts and a db entry is created, this is set to IN PROGRESS
When a user requests pause, the route controller (i.e view) modifies this to REQUESTING PAUSE
When the operation actually gets paused and callback (from tappable, inside pause_or_continue) is called, the callback should modify this to PAUSED
When the task is completed, this should be modified to COMPLETED

An example of `clause`

@celery.task()
def should_pause(_, operation_id: int):
    # This is the `clause` to be used for `tappable`
    # i.e it lets celery know whether to pause or continue
    db = get_db()

    # Check the database to see if user has requested pause on the operation
    operation = db.execute(
        "SELECT * FROM operations WHERE id = ?", (operation_id,)
    ).fetchone()
    return operation["completion"] == "REQUESTING PAUSE"

This is the task to call at the pause points, to determine whether or not to pause. It's a function that takes 2 parameters.....well sort of. The first one is mandatory, tappable requires the clause to have one (and exactly one) argument - so it can pass the previous task's return value to it (even if that return value is None). In this example, the return value isn't required to be used - so we can just ignore it.

The second parameter is an operation id. See, all this clause does - is check a database for the operation (the workflow) entry and see if it has the status REQUESTING PAUSE. To do that, it needs to know the operation id. But clause should be a task with one argument, what gives?

Well, good thing signatures can be partial. When the task is first started and a tappable chain is created. The operation id is known and hence we can do should_pause.s(operation_id) to get the signature of a task that takes one parameter, that being the return value of the previous task. That qualifies as a clause!

An example of `callback`

import os
import json
from typing import Any, List

@celery.task()
def save_state(retval: Any, chains: dict, operation_id: int):
    # This is the `callback` to be used for `tappable`
    # i.e this is called when an operation is pausing
    db = get_db()

    # Prepare directories to store the workflow
    operation_dir = os.path.join(app.config["OPERATIONS"], f"{operation_id}")
    workflow_file = os.path.join(operation_dir, "workflow.json")
    if not os.path.isdir(operation_dir):
        os.makedirs(operation_dir, exist_ok=True)

    # Store the remaining workflow chain, serialized into json
    with open(workflow_file, "w") as f:
        json.dump(chains, f)

    # Store the result from the last task and the workflow json path
    db.execute(
        """
        UPDATE operations
        SET completion = ?,
            workflow_store = ?,
            result = ?
        WHERE id = ?
        """,
        ("PAUSED", workflow_file, f"{retval}", operation_id),
    )
    db.commit()

And here's the task to be called when the task is being paused. Remember, this should take the last executed task's return value and the remaining list of signatures (in order, from left to right). There's an extra param - operation_id - once again. The explanation for this is the same as the one for clause.

This function stores the remaining chain in a json file (since it's a list of dicts). Remember, you can use a different serializer - I'm using json since it's the default task serializer used by celery.

After storing the remaining chain, it updates the completion status to PAUSED and also logs the path to the json file into the db.

Now, let's see these in action-

An example of starting the workflow

def start_operation(user_id, *operation_args, **operation_kwargs):
    db = get_db()
    operation_id: int = db.execute(
        "INSERT INTO operations (requester_id, completion) VALUES (?, ?)",
        (user_id, "IN PROGRESS"),
    ).lastrowid
    # Convert a regular workflow chain to a tappable one
    tappable_workflow = tappable(
        (T1.s() | T2.s() | T3.s() | T4.s() | T5.s(operation_id)),
        should_pause.s(operation_id),
        save_state.s(operation_id),
    )
    # Start the chain (i.e send task to celery to run asynchronously)
    tappable_workflow(*operation_args, **operation_kwargs)
    db.commit()
    return operation_id

A function that takes in a user id and starts an operation workflow. This is more or less an impractical dummy function modeled around a view/route controller. But I think it gets the general idea through.

Assume T[1-4] are all unit tasks of the operation, each taking the previous task's return as an argument. Just an example of a regular celery chain, feel free to go wild with your chains.

T5 is a task that saves the final result (result from T4) to the database. So along with the return value from T4 it needs the operation_id. Which is passed into the signature.

An example of pausing the workflow

def pause(operation_id):
    db = get_db()

    operation = db.execute(
        "SELECT * FROM operations WHERE id = ?", (operation_id,)
    ).fetchone()

    if operation and operation["completion"] == "IN PROGRESS":
        # Pause only if the operation is in progress
        db.execute(
            """
            UPDATE operations
            SET completion = ?
            WHERE id = ?
            """,
            ("REQUESTING PAUSE", operation_id),
        )
        db.commit()
        return 'success'

    return 'invalid id'

This employs the previously mentioned concept of modifying the db entry to change completion to REQUESTING PAUSE. Once this is committed, the next time pause_or_continue calls should_pause, it'll know that the user has requested the operation to pause and it'll do so accordingly.

An example of resuming the workflow

def resume(operation_id):
    db = get_db()

    operation = db.execute(
        "SELECT * FROM operations WHERE id = ?", (operation_id,)
    ).fetchone()

    if operation and operation["completion"] == "PAUSED":
        # Resume only if the operation is paused
        with open(operation["workflow_store"]) as f:
            # Load the remaining workflow from the json
            workflow_json = json.load(f)
        # Load the chain from the json (i.e deserialize)
        workflow_chain = chain(signature(x) for x in serialized_ch)
        # Start the chain and feed in the last executed task result
        workflow_chain(operation["result"])

        db.execute(
            """
            UPDATE operations
            SET completion = ?
            WHERE id = ?
            """,
            ("IN PROGRESS", operation_id),
        )
        db.commit()
        return 'success'

    return 'invalid id'

Recall that, when the operation is paused - the remaining workflow is stored in a json. Since we are currently restricting the workflow to a chain object. We know this json is a list of signatures that should be turned into a chain. So, we deserialize it accordingly and send it to the celery worker.

Note that, this remaining workflow still has the pause_or_continue tasks as they were originally - so this workflow itself, is once again pause-able/resume-able. When it pauses, the workflow.json will simply be updated.

And that's it! You can check out a more complex example in the repo itself. The tasks are in tasks.py, these tasks are managed by the flask routes in operations.py

A guide to testing flask applications using unittest!

Phantz — Mon, 02 Nov 2020 14:14:08 +0000

Flask is a backend web framework for python. One that is beloved by many python developers! But to make a great flask webapp, testing is important. I noticed the lack of a good library to integrate with the stdlib unittest for testing flask apps. So I made one myself!

Check out flask-unittest on github!

This tutorial will demonstrate how to test flask applications using the FlaskClient object, i.e in an API centric way, and also demonstrate how to test flask applications live, using a headless browser!

If you're looking to use a FlaskClient to test your app, I recommend you to read through the official testing guide. It's super simple and should only take 5 minutes to read through!

Test using a `FlaskClient` object

The library provides us with the testcase ClientTestCase, which creates a FlaskClient object from a Flask object for each test method, that you can use to test your app. It also provides direct access to flask globals like request, g, and session!

Let's see how you could use ClientTestCase

import flask_unittest

from flask_app import create_app

class TestFoo(flast_unittest.ClientTestCase):
    # Assign the flask app object
    app = create_app()

    def test_foo_with_client(self, client):
        # Use the client here
        # Example request to a route returning "hello world" (on a hypothetical app)
        rv = client.get('/hello')
        self.assertInResponse(rv, 'hello world!')

We have a flask app in a module named flask_app, which has a function to create and return the Flask app object. Just need to assign the returned object to app.

Remember, you don't need a function to create and return the app. As long as you have a correctly configured app object, you can simply assign it!

Now, we define a test method test_foo_with_client with 2 parameters. The mandatory self and a parameter named client. For each test method, ClientTestCase will create a FlaskClient by using .test_client and pass it to the test method.

Now you can freely use this to make API calls to your app! In our example, we make a request to /hello, which is a simple route returning hello world! as a response. You can now use assertInResponse, which is a utility method provided by flask-unittest, to check if hello world! actually exists in the response! (Note: you can also just use assert 'hello world!' in rv.data for the same effect)

Inside this test method, you also have access to flask globals like request, g, and session.

def test_foo_with_client(self, client):
    rv = client.get('/hello?q=paramfoo')
    self.assertEqual(request.args['q'], 'paramFoo')    # Assertion succeeds
    # Do a POST request with valid credentials to login
    client.post('/login', data={'username': 'a', 'password': 'b'})
    # Our flask app sets userId in session on a successful login
    self.assertIn('userId', session)    # Assertion succeeds

This is obviously very useful for testing!

You can also use the setUp method to login to your webapp, and the session will persist in the actual test method! Because setUp, tearDown and the test method are ran together in a set - using the same FlaskClient. The next test method along with its setUp and tearDown methods, however, will use a brand new FlaskClient - and hence a new session.

def setUp(self, client):
    # Login here
    client.post('/login', data={'username': 'a', 'password': 'b'})

def test_foo_with_client(self, client):
    # Check if the session is logged in
    self.assertIn('userId', session)    # Assertion succeeds

def tearDown(self, client):
    # Logout here, though there isn't really a need - since session is cleared for the next test method
    client.get('/logout')

There are also multiple utility methods for common assertions!

assertStatus - Assert the status code of a response returned from a client API call.
assertResponseEqual - Assert the response .data is equal to the given string
assertJsonEqual - Assert the response .json is equal to the given dict
assertInResponse - Assert given string/bytestring exists in response .data
assertLocationHeader - Assert the location header in response is equal to the given string. Useful for redirect requests.

Check out a full example of using this testcase in the repo

Would you like the Flask app to be built per test method too? Instead of having it as a constant property of the class? Check out AppClientTestCase (or even AppTestCase if you don't need the functionality of the FlaskClient)

Test a live flask server using `selenium`

What if you want to just run a regular flask server live and use a headless browser like selenium to test it out? LiveTestCase and LiveTestSuite is for you!

import flask_unittest
from selenium.webdriver import Chrome, ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from flask_app import create_app

class TestFoo(flask_unittest.LiveTestCase):
    driver: Union[Chrome, None] = None
    std_wait: Union[WebDriverWait, None] = None

    @classmethod
    def setUpClass(cls):
        # Initiate the selenium webdriver
        options = ChromeOptions()
        options.add_argument('--headless')
        cls.driver = Chrome(options=options)
        cls.std_wait = WebDriverWait(cls.driver, 5)

    @classmethod
    def tearDownClass(cls):
        # Quit the webdriver
        cls.driver.quit()

    def test_foo_with_driver(self):
        # Use self.driver here
        # You also have access to self.server_url and self.app
        # Example of using selenium to go to index page and try to find some elements (on a hypothetical app)
        self.driver.get(self.server_url)
        self.std_wait.until(EC.presence_of_element_located((By.LINK_TEXT, 'Register')))
        self.std_wait.until(EC.presence_of_element_located((By.LINK_TEXT, 'Log In')))

This is pretty simple, we instantiate the driver in setUpClass, use it as we would normally in the test methods and quit it in tearDownClass.

You'll have access to the flask app as well as the url the app is running on (localhost + port) in the testcase. (only in the instance methods though)

The real juice of this is actually in LiveTestSuite. Unlike the previous testcases, which can be run using the regular unittest testsuite, or simply doing unittest.main() - LiveTestCase requires you to use LiveTestSuite

# Pass the flask app to suite
suite = flask_unittest.LiveTestSuite(create_app())
# Add the testcase
suite.addTest(unittest.makeSuite(TestFoo))
# Run the suite
unittest.TextTestRunner(verbosity=2).run(suite)

We have to pass the flask app object to LiveTestSuite, so it can spawn it using .run - this same app object will be available in the testcase.

The flask server will be spawned as a daemon thread once the suite starts running and it runs for the duration of the program.

Check out a full example of using this testcase in the repo!

For more examples, including a full emulation of the official flask testing example, check out the repo!

Pushing Native Messaging to the limits! - C vs Rust

Phantz — Thu, 04 Jun 2020 14:51:12 +0000

I recently stumbled upon Native Messaging, which is basically the way for a browser extension (chrome or firefox) to communicate with a native application. A friend was implementing this feature in his rust app and I decided to help out, which also was enough of a motivation for me to finally start learning Rust.

Native Messaging - Illustrated

Courtesy of MDN

Libraries used for Native Messaging

For C

made by yours truly - shameless plug

No but seriously, I didn't find another library for native messaging in C on github, though there were 2 for Rust.

For Rust

The story (also performance is love)

I already had a pretty decent base built up on the low level and after reading the very well documented MDN and Google docs, and with major help from this crate's code I had these functions-

pub fn read_input<R: Read>(mut input: R) -> io::Result<serde_json::Value> {
    let length = input.read_u32::<NativeEndian>().unwrap();
    let mut buffer = vec![0; length as usize];
    input.read_exact(&mut buffer)?;
    let json_val: serde_json::Value = serde_json::from_slice(&buffer).unwrap();
    Ok(json_val)
}

pub fn write_output<W: Write>(mut output: W, value: &serde_json::Value) -> io::Result<()> {
    let msg = serde_json::to_string(value)?;
    let len = msg.len();
    // Chrome won't accept a message larger than 1MB
    if len > 1024 * 1024 {
        panic!("Message was too large", length: {}, len)
    }
    output.write_u32::<NativeEndian>(len as u32)?;
    output.write_all(msg.as_bytes())?;
    output.flush()?;
    Ok(())
}

Which facilitated reading and writing the messages respectively. And it all worked like a charm!

Now I'm really obsessed with performance, probably a bit more than I should be, though I usually don't prefer performance over resource efficiency and safety. So I had to dig further.

I decided to write this in C and also make the Rust code just a bit better for performance and also make both codebases as fair as possible. They both had to do exactly the same thing. Nothing more, nothing less.

So I removed the serde_json completely and just worked with strings instead, the user can parse themselves. So instead of-

let json_val: serde_json::Value = serde_json::from_slice(&buffer).unwrap();

I used-

let val: String = match String::from_utf8(buffer) {
        Err(why) => panic!(why.to_string()),
        Ok(val) => val
    };

Same for the output too, I just removed let msg = serde_json::to_string(value)?; completely and changed the parameter to &str instead.

Not much difference syntactically but definitely a great boost to performance! The C code also did the same thing, here's how that looked like-

char* read_input(FILE* stream)
{
    uint32_t length;
    size_t count;
    int err;
    count = fread(&length, sizeof(uint32_t), 1, stream);
    if (count != 1)
    {
        if (feof(stream))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while reading file\n");
        }
        else if ((err = ferror(stream)))
        {
            fprintf(stderr, "An error occured while reading file, err code: %d\n", err);
            clearerr(stream);
        }
        return NULL;
    }
    char* value = malloc((length + 1) * sizeof(char));
    if (value == NULL)
    {
        fprintf(stderr, "An error occured while allocating memory for value");
        return NULL;
    }
    count = fread(value, sizeof(char), length, stream);
    if (count != length)
    {
        if (feof(stream))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while reading file\n");
        }
        else if ((err = ferror(stream)))
        {
            fprintf(stderr, "An error occured while reading file, err code: %d\n", err);
            clearerr(stream);
        }
        free(value);
        return NULL;
    }
    value[length] = '\0';
    return value;
}

size_t write_output(const char* const value, uint32_t length, FILE* stream)
{
    size_t count;
    int err;
    if (length > (1024 * 1024)) {
        fprintf(stderr, "Message too large");
        return -1;
    }
    count = fwrite(&length, sizeof(uint32_t), 1, stream);
    if (count != 1)
    {
        if (feof(stream))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while reading file\n");
        }
        else if ((err = ferror(stream)))
        {
            fprintf(stderr, "An error occured while reading file, err code: %d\n", err);
            clearerr(stream);
        }
        return -1;
    }
    count = fwrite(value, sizeof(char), length, stream);
    if (count != length)
    {
        if (feof(stream))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while writing file\n");
        }
        else if ((err = ferror(stream)))
        {
            fprintf(stderr, "An error occured while writing file, err code: %d\n", err);
            clearerr(stream);
        }
        return -1;
    }
    fflush(stream);
    return length + 4;
}

Yeah I know, a lot of error handling. But! It's time to benchmark! So how do they fare?

Hold up! Not quite yet! Let's check the profile I used for optimization on the rust side

[profile.release]
opt-level = 3
debug = false
debug-assertions = false
overflow-checks = false
lto = true
panic = 'abort'
incremental = false
rpath = false

Looks quite juicy for the benchmark right? I'd say so!

What about C? Well, I didn't have a gcc + Native Messaging setup handy so I had to use clang.....on windows, long story. This was kind of a disadvantage. All I could do was -Ofast. That's it.

Yes, I know. Not very fair on the optimization side. But I did some testing on the non optimized versions of both, and C was absolutely wiping the floor so I figured - "whatever"

Also the benchmark was all done on the javascript side and the following is the connectionless version. So each time a message is sent, the extension invokes the executable, waits for it to read and send a response. This is how it looked like-

var start;

chrome.browserAction.onClicked.addListener(() => {
    console.log('Sending: ping')
    start = performance.now();
    chrome.runtime.sendNativeMessage("pingpong", {text: "ping"}, onResponse);
});

function onResponse(res) {
    let end = performance.now();
    console.log(`Received: ${res.msg}, Took: ${end - start} ms`);
}

Alright benchmark time, here's what rust looked like-

And here's C-

Close, very close! But C still seems to win just a bit over. Impressive results nevertheless, how about some more optimization?

That question lead to many, many code revisions - a lot of rust programmers helped me out so much to optimize the code. So after many, many revisions. This is how the rust code looked like-

pub fn read_input() -> io::Result<Vec<u8>> {
    let mut instream = io::stdin();
    let mut length = [0; 4];
    instream.read(&mut length)?;
    let mut buffer = vec![0; u32::from_ne_bytes(length) as usize];
    instream.read_exact(&mut buffer)?;
    Ok(buffer)
}

pub fn write_output(msg: &str) -> io::Result<()> {
    let mut outstream = io::stdout();
    let len = msg.len();
    if len > 1024 * 1024 {
        panic!("Message was too large, length: {}", len)
    }
    outstream.write(&len.to_ne_bytes())?;
    outstream.write_all(msg.as_bytes())?;
    outstream.flush()?;
    Ok(())
}

Got rid of all dependencies and instead of converting to String, returned the raw bytes. Also make stdin and stdout constants as those are always the streams used for Native Messaging anyway.

The C code also changed accordingly, though not as many revisions-

uint8_t* read_input()
{
    uint32_t length;
    size_t count;
    int err;
    count = fread(&length, sizeof(uint32_t), 1, stdin);
    if (count != 1)
    {
        if (feof(stdin))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while reading file\n");
        }
        else if ((err = ferror(stdin)))
        {
            fprintf(stderr, "An error occured while reading file, err code: %d\n", err);
            clearerr(stdin);
        }
        return NULL;
    }
    uint8_t* value = malloc((length + 1) * sizeof(*value));
    if (value == NULL)
    {
        fprintf(stderr, "An error occured while allocating memory for value");
        return NULL;
    }
    count = fread(value, sizeof(*value), length, stdin);
    if (count != length)
    {
        if (feof(stdin))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while reading file\n");
        }
        else if ((err = ferror(stdin)))
        {
            fprintf(stderr, "An error occured while reading file, err code: %d\n", err);
            clearerr(stdin);
        }
        free(value);
        return NULL;
    }
    return value;
}

size_t write_output(const uint8_t* const value, uint32_t length)
{
    size_t count;
    int err;
    if (length > (1024 * 1024)) {
        fprintf(stderr, "Message too large");
        return 0;
    }
    count = fwrite(&length, sizeof(uint32_t), 1, stdout);
    if (count != 1)
    {
        if (feof(stdout))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while reading file\n");
        }
        else if ((err = ferror(stdout)))
        {
            fprintf(stderr, "An error occured while reading file, err code: %d\n", err);
            clearerr(stdout);
        }
        return 0;
    }
    count = fwrite(value, sizeof(char), length, stdout);
    if (count != length)
    {
        if (feof(stdout))
        {
            fprintf(stderr, "Unexpectedly encountered EOF while writing file\n");
        }
        else if ((err = ferror(stdout)))
        {
            fprintf(stderr, "An error occured while writing file, err code: %d\n", err);
            clearerr(stdout);
        }
        return 0;
    }
    fflush(stdout);
    return length + 4;
}

And thus, it was benchmark time again! The final iteration!

Here's Rust-

And here's C-

Well, would you look at that...not much difference from the last time. Well, I sure as hell was disappointed.

Just as I was about to call it a ~~day~~ benchmark, someone suggested I should use connectionful messaging, because rust executable invokations take a long time, far longer than C due to panics and a whole lot of other setup. I decided, "sure I could try that!"

So I switched my js code to this-

var start;
var port = chrome.runtime.connectNative('pingpong');
port.onMessage.addListener(function(res) {
    let end = performance.now();
    console.log(`Received: ${res.msg}, took: ${(end - start) * 1000} μs`);
});
port.onDisconnect.addListener(function() {
    console.log("Disconnected");
});
chrome.browserAction.onClicked.addListener(() => {
    console.log("Sending: ping")
    start = performance.now();
    port.postMessage({ text: "ping" });
});

And also changed my native app to use an infinite loop - to keep listening for input

This actually brought the accuracy down to microseconds! So I had to change the timer to that instead, how astonishing!

And thus, here be the final results-

Rust-

C-

Incredible! Now that's a feat. Averaging these 2 results, with 1 anomaly from each result discarded (1k+ microseconds for C and 600+ microseconds for Rust), we can see that C beats Rust by just 20-30 microseconds!

And thus, C is the winner! Though Rust put up a very admirable fight.

If you'd like to test out the code yourself, and I insist you do as my benchmarking really isn't very sophisticated, you can find the code in my gists-

This has been very fun! And a super nice start to my rust journey! I think this was a pretty practical demonstration of stdin and stdout IO speeds for Rust vs C. I hope to continue these benchmarks in the future, hopefully focusing on other practical topics.

Once again, if you're looking forward to implementing Native Messaging in your C application, please check out libnativemsg!

My Image/GIF glitching python library is now on pypi!

Phantz — Sat, 22 Feb 2020 16:26:42 +0000

After getting some requests, I finally put glitch-this on pypi

Not interested in the library? Check out the commandline script here!

I made a commandline python script to glitchify images and make GIFs!

Phantz — Fri, 21 Feb 2020 16:03:56 +0000

Make glitched GIFs and images from the comfort of your commandline!

Check it out here!

The README contains the full quickstart guide.

Please consider starring, if you like it!

Looking for contributors with python 3 tensorflow experience for a cool chatbot!

Phantz — Mon, 17 Feb 2020 05:19:01 +0000

What the project is

Facebook-Messenger-bot is a chatbot you can train to talk like you! Check the repo here!

What we need help with

We're looking for someone experienced with python 3 tensorflow and the seq2seq model. We need to upgrade our codebase to python3 but since we used legacy_seq2seq in our python 2 code, we're having trouble replacing it with a python 3 equivalent. More details on the issue here!

I wrote a lightning fast prime number generator in C!

Phantz — Thu, 06 Feb 2020 05:22:37 +0000

I recently found out about this improved and cache optimized version of the segmented sieve of eratosthenes algorithm. Now, me being super interested in primes, I wanted to make it even faster. So I wrote it in C!

Check it out here!

Cool Algorithm: Calculating any index of a Cartesian Product!

Phantz — Sun, 02 Feb 2020 05:54:53 +0000

Ever tried generating the cartesian product of 42 large sets/sequences just to get the 236th index of the product?

Trick Question: You don't have enough memory for that! (probably)

Thankfully, there's an algorithm to get any index you desire, real freakin' fast!

Sometimes, you just need to compute very large cartesian products just so you can use a handful of the indexes from the result. The problem is that generating the whole thing is slow, and often times, too much for your memory to even handle.

That's what prompted me to come up with this algorithm, apparently this is really hard to find on the internet (at least I couldn't find it). So I went ahead and asked some other folks about it and they too confirmed that, although many people know of this algorithm, it's nowhere to be found on the internet. So I'm making a post here for people to hopefully be able to find later :)

What does it do?

Suppose we have 3 sequences -

A : {1, 2, 3}
B : {2, 6, 7, 9}
C : {1, 7}

Simply call get_tuple_by_index and pass in the list/array of sequences (i.e [A, B, C] or {A, B, C}) and the index of the result you want.
For C (the programming language), you'd also need to pass in the number of sequences (3 in this case) and an array containing the lengths of each sequences ({3, 4, 2} in this case).

For example, if we'd call get_tuple_by_index with the 6th index (starting from 0) of the cartesian product A x B x C, we'd get (1, 9, 1)

How does it work?

For this demonstration, we'll be using the sequences mentioned above as examples

The list/array of sequences, that we'd like the cartesian product to, in this case [A, B, C] or {A, B, C} is reverse iterated through.
In each iteration, the result tuple is filled up from backwards.
In each iteration, the current element of the result tuple will be the index % length_of_current_setth index of the current set.
Additionally, in each iteration, the index is set to index // length_of_current_set where // signifies integer division So, in this case, in the first iteration, for index = 6, the current element will be C[6 % 2] or C[0] or 1. That will now be added as the last element of result tuple. It'll now look like -> (1)

index is now set to 6 // 2 or 3

In the second iteration, the current element will be B[3 % 4] or B[3] or 9.
That will now be added as the second to last element of result tuple. It'll now look like -> (9, 1)

index is now set to 3 // 4 or 0

In the third iteration, the current element will be A[0 % 3] or A[0] or 1.
That will now be added as the third to last element (or first element in this case) of result tuple. It'll now look like -> (1, 9, 1)

The Code

Pretty cool! Here's some psuedocode to demonstrate the algorithm-

getNthTuple(sets, pos):
    tuple = ()
    for i in [(|sets|-1)..0]:
        set = sets[i]
        elem = set[pos mod |set|]
        tuple = (elem, tuple)
        pos = pos // |set|
    return tuple

If you'd like to see examples in real programming languages, you can find the java, python and C versions here!

How is the human brain so good at matching delimiters?

Phantz — Sun, 19 Jan 2020 10:15:00 +0000

Just to be clear, by delimiters I mean mathematical delimiters - parentheses, braces, and brackets. Also this post will be discussing code from a java class I wrote to be used as a library. You can check the repo here. Alright, with that out of the way, let's go!

Part 1 - Auto-complete

Introduction

A few months ago I needed to have an auto-complete delimiters function in one of my programs. I never really thought about how to do it because it's really not something that just occurs. I mean like, have you ever manually went step by step thinking how to match that one open brace that you didn't close in your program yet? No, you just know instinctively. Your brain is just incredibly good at detecting and matching patterns. But a program doesn't have instincts. So, what's the algorithm behind this pattern matching? How does the brain do it, step by step?

Before I start the explanation, I highly recommend you check out the code on github and follow along! Feel free to import the class if you ever need it in the future :)

The Algorithm

Well, just check the number of opening delimiters (i.e (, { and [) in an expression and put that number of closing delimiters at the end! So if you were given log(sin(2^(5 you'd just put 3 closing parentheses in the end, right?

Not so fast. What if the expression is log(sin(2^(5)? You can't put 3 closing delimiters now, you have to put 2, because 1 is already closed. So we can't just count the appearance of opening delimiters, we also have to 1) count the number of closing delimiters, 2) Grab the difference between the 2 and then 3) append accordingly.

That's all well and good, but we're not done yet. What about mixed delimiters? Like so log[sin(2^{. Now we can finally discuss the fun part, step by step how your brain does (probably) it.

Simply put, your brain just scans the expression backwards really fast. If it sees an opening delimiter, it immediately appends the corresponding closing delimiter to the expression. Now add the above paragraph about checking the number of appearance to the mix and no matter what expression you feed in, it'll always output a perfectly matched output expression.

You can implement this whole thing in a switch statement that checks each unit expression (i.e log, sin, cos, (, {, )....) like so in Java-

// Inside the Switch statement
case "(":
    ++openingParentheses;
    if (openingParentheses > closingParentheses) {
        resultstr.append(")");
        closingParentheses++;
    }
    break;

Now you just have to put the whole switch statement in a reverse iterator loop!

Here it is in action!

note that although the frontend shows only parentheses, the backend uses different delimiters for different functions, so it still uses the exact same algorithm

Part 2 - Function associated with given closing delimiter

Introduction

Once I was done with the auto-complete, I needed to know exactly which function is associated with which delimiter. Suppose you were given this expression log10[sin(23^{56*cos(loge[2])})] and you were told to find out which function does the second to last parenthesis belongs to. In this case, that belongs to cos. But what's the algorithm behind it?

The Algorithm

Again, I recommend opening the source code if you haven't already

Once again, we reverse iterate the expression, this time we also need a user provided index and the expression must be an array, to separate each unit expression. The expression log10[sin(23^{56*cos(loge[2])})], if turned into a String array, it'd be-

{"log10", "[", "sin", "(", "23", "^", "{", "56", "*", "cos", "(", "loge", "[", "2", "]", ")", "}", ")", "]"}

This time, we start reverse iterating from the given index, but skip the index itself. So, the index of the second to last parenthesis in this array is 15. So we'll start reverse iterating from 14.

Now, the reverse iteration must go on until the number of appearances of opening delimiters is greater than(not equal to) the number of appearances of closing delimiters. The moment the loop stops, wallah!, you magically have the index of the corresponding function, in this case, cos.

Well, not magically, but you know what I mean.

If we wanted to implement this for parentheses ((). We'd do it like so;

while (openParentheses <= closeParentheses) {
    switch (expression[i]) {
    case ")":
        closeParentheses++;
        break;
    case "(":
        openParentheses++;
        break;
    }
    i--;
}

The resulting i is the index of the function.

Here it is in action!

notice that whenever I remove a closing delimiter using backspace, the program is able to tell which function region I just entered

Conclusion

That's it! That's how your brain does it (probably). I found it really fascinating at first and when I figured out the algorithm, it felt really nice. I figured I'd share with everyone!

Once again, if you need to use it in any of your java programs, I personally used it in Android dev stuff, you can just grab the .jar file from the repo!

Python: Using JWT in cookies with a flask app and restful API!

Phantz — Sat, 11 Jan 2020 12:36:06 +0000

This guide aims to provide an in-depth tutorial on how to set up flask-jwt-extended using cookies. There's a LOT of docs online but they are mostly using authentication headers and a frontend framework like react. I wanted to share my experience with using jwt through just the backend (e.g returning redirects instead of json objs). This will also outline some of the guidelines about token freshness, access/refresh token expiry and refreshing expired tokens (through the backend!!).

Note: Usually the backend only returns a jsonify-ed dict with the tokens (when using headers) or simply a status report (when using cookies). Once the frontend framework receives the tokens/status report, it redirects to the desired url. We're just achieving all that without the frontend.

I'll be assuming you know what JWT is and are aware of the python package flask-jwt-extended. You don't need to know how to use the package, I'll show how to do that in the guide :)
You'll also need to know the basics of flask and flask-restful.

Need a working example? Here's a gist

Alright, time to make the flask app itself!

Initializing our app

First let's see what imports we'll be needing. All of these will be explained as we go on, so don't worry too much about them. Just know that these are the functions we are using later if you get confused and ask "wait what does that function do and where did it come from?"

from flask import Flask, redirect, make_response, render_template
from flask_restful import Api
from flask_jwt_extended import (JWTManager, jwt_required, 
                                jwt_refresh_token_required, 
                                jwt_optional, fresh_jwt_required, 
                                get_raw_jwt, get_jwt_identity,
                                create_access_token, create_refresh_token, 
                                set_access_cookies, set_refresh_cookies, 
                                unset_jwt_cookies,unset_access_cookies)

For the purposes of this guide, we're declaring all functions in the same file __init__.py. NEVER DO THIS, always separate your JWT handlers on a different file.

let's make a __init__.py and initialize all the required stuff

app = Flask(__name__)
app.config['BASE_URL'] = 'http://127.0.0.1:5000'  #Running on localhost
app.config['JWT_SECRET_KEY'] = 'super-secret'  # Change this!
app.config['JWT_TOKEN_LOCATION'] = ['cookies']
app.config['JWT_COOKIE_CSRF_PROTECT'] = True
app.config['JWT_CSRF_CHECK_FORM'] = True
jwt = JWTManager(app)

Don't worry too much about the imports yet, in the above snipper we are only really concerned about Flask, redirect, make_response, and JWTManager. All we're doing here is creating a Flask app instance and then setting up the JWTManager.

As for the configs:-

We need a JWT_SECRET_KEY for the JWT encryption. Try to go for a completely random string of alphanumeric characters. You can generate them using this.
We have to set JWT_TOKEN_LOCATION to ['cookies'] if we want to use cookies as our preferred token storage. The default is always auth header.
We also set JWT_COOKIE_CSRF_PROTECT to True for cookie security, this is True by default so you don't have to do this.
Lastly, we set JWT_CSRF_CHECK_FORM to True. This will come in handy when we have to pass tokens to api resources. Don't worry about it much yet.

Setting cookies and returning redirects

Let's make a function to create and assign one access token and one refresh token to the cookies. Remember, the tokens are always assigned to a flask response object which must then be returned.

A response object can be anything from a jsonify-ed dict to a redirect. There's a LOT of docs on returning jsonify-ed dicts so I'll be showing how to return a redirect instead.

def assign_access_refresh_tokens(user_id, url):
    access_token = create_access_token(identity=str(user_id))
    refresh_token = create_refresh_token(identity=str(user_id))
    resp = make_response(redirect(url, 302))
    set_access_cookies(resp, access_token)
    set_refresh_cookies(resp, refresh_token)
    return resp

We take in the url to redirect to as an argument. We also take a unique identifer (user_id), which is essential to create a token

Explanation : create_access_token(identity) and create_refresh_token(identity) will create and return an access and refresh token based on the identity provided. Remember the identity should be unique for each user. We then have to make a flask response object that will redirect us to the desired url. We need to attach the cookies to this response object.
make_response(redirect(url, 302)) will create and return a response object that redirects to url with code 302.
Now we just need to assign the access and refresh tokens. We do that using set_access_cookies(response, access_token) and set_refresh_cookies(response, refresh_token).
Finally we return the response object that now has the cookies attached. This will redirect the user to the give url and the user session now has the desired tokens, Awesome!

Token Freshness

From the docs

[Fresh pattern] is useful for allowing fresh tokens to do some critical things (such as update an email address or complete an online purchase), but to deny those features to non-fresh tokens.

Let's think of a situation, as long the refresh token is valid and non-expired, we can keep generating access tokens, right? If an access token goes into the wrong hands, the attacker will only have a short period of time to use them as expire quite fast (15 mins by default). But what if the attacker gets their hands on a refresh token, which usually have a longer expiry (1 day by default)?

This is why we use the fresh/non-fresh pattern. When creating access tokens using create_access_token we can set fresh = True to create a fresh access token. By default when an access token is created without a fresh value supplied, it will be non-fresh.

We can create a fresh access token by create_access_token(identity = id, fresh = True).

Now we can only let refresh tokens create non-fresh access tokens. non-fresh tokens cannot access endpoints protected with @fresh_jwt_required, only @jwt_required. Fresh tokens can access endpoints protected with both @fresh_jwt_required and @jwt_required. Make sure fresh tokens are only created when the user actually logs in using their password.

Read!

Usage

Now we can finally put the jwt handlers to the test! To use these handlers we have to decorate our functions using jwt required decorators. Here's an overview on them!

jwt_required will block access to everyone unless they have a valid, non-expired non-fresh or fresh access token.
fresh_jwt_required will block access to everyone unless they have a valid, non-expired fresh access token.
jwt_optional won't block access but it'll still allow the jwt to be used inside the function if it is present. Otherwise get_jwt_identity() will return None.
@jwt_refresh_token_required will block access to everyone unless they have a valid non-expired refresh token

Read more about jwt endpoint decorators!

Let us first assign the tokens first, we'll ideally want to assign them when the user signs up or logs in. Like so

@app.route('/login', methods=['POST'])
def login():
    // Verify username and password //
    return assign_access_refresh_tokens(username , app.config['BASE_URL'] + '/')

Great now we have tokens in the session! Let's declare some functions using the decorators!

@app.route('/account')
@fresh_jwt_required
def account():
     username = get_jwt_identity()
    // very important account settings //

@app.route('/service', methods=['GET'])
@jwt_required
def services():
    username = get_jwt_identity()
    // Not important stuff but still needs to be logged in //

@app.route('/')
@jwt_optional
def index():
    username = get_jwt_identity()   # None if not logged in
    // Accessible to everyone but maybe different to logged in users //

The first endpoint, account can only be accessed by a fresh jwt. Someone who recently logged in will be able to access this as long as their fresh jwt does not expire. When it does expire, the refresh token won't create a fresh jwt and hence to access this, the user will have to log in again.

The second endpoint, services can be accessed by either a fresh or non-fresh jwt. A logged in user can access this for the entirety of their refresh token lifetime without logging in again.

The third endpoint, index can be accessed by anyone. However it may offer a different look/functionality if the user is logged in, i.e get_jwt_identity is not None.

Once the user is done looking around they'd wanna logout, we need to unset their cookies if they do so.

@app.route('/logout')
@jwt_required
def logout():
    # Revoke Fresh/Non-fresh Access and Refresh tokens
    return unset_jwt(), 302

That's it! Now let's learn how to pass the tokens correctly into api resources

Passing JWT to RESTful API resources

Let's assume we have a Class in another module like so

from flask_restful import Resource
from flask_jwt_extended import fresh_jwt_required, get_jwt_identity
class AccountSettings(Resource):
    @fresh_jwt_required
    def post(self):
        username = get_jwt_identity()
        // important setting change commits //

and we add this to our api in __init__.py using api.add_resource(AccountSettings, '/account/change_settings')

So now whenever the url http://127.0.0.1:5000/account/change_settings receives a POST, it'll execute the post function in AccountSettings. However a POST function in an api resource` **cannot* access the CSRF token in our JWT cookies directly. Without that token, the jwt is useless. The reason this happens, along with it's solution is written in the docs-

Typically JWT is used with API servers using JSON payloads, often via AJAX. However you may have an endpoint that receives POST requests directly from an HTML form. Without AJAX, you can’t set the CSRF headers to pass your token to the server. In this scenario you can send the token in a hidden form field. To accomplish this, first configure JWT to check the form for CSRF tokens. Now it’s not necessary to send the csrf in a separate cookie, you can render it directly into your HTML template.

All we have to do is turn the JWT_CSRF_CHECK_FORM option on (which we already did earlier) and put the token in a hidden input field. You can do this by putting <input hidden name="csrf_token" value="{{ csrf_token }}"> INSIDE your form that will POST to the api resource. Now you can use render_template to pass the csrf_token to the template.

Get the csrf_token using csrf_token = (get_raw_jwt() or {}).get("csrf") and pass it to the DOM like so
return render_template("account.html", csrf_token=csrf_token), 200

Note : You can only extract the csrf_token (for access tokens) in endpoints decorated with @fresh_jwt_required or @jwt_required. @jwt_optional will also work however, if there is no JWT present, it'll return None.

Note : You only have to do this on RESTful API resources that use POST methods. GET method can access the csrf protected jwt just fine.

JWT Error handling in API resources

You may notice the jwt loaders (e.g @jwt.unauthorized_loader, @jwt.invalid_token_loader) not getting hit when the API resource URLs are directly accessed. This is because of flask's default exception handling behaviour, to fix this, simply set PROPAGATE_EXCEPTIONS to True.
app.config['PROPAGATE_EXCEPTIONS'] = True

Advices, Standards and Discussions

Expiry : The default expiry for access and refresh tokens is 15 minutes and 1 day respectively. You can change this using app.config['JWT_ACCESS_TOKEN_EXPIRES'] or app.config['JWT_REFRESH_TOKEN_EXPIRES'] and assigning a datetime.timedelta() value. It is recommended to keep the access token duration low as it can simply be refreshed.
Cookie path : It's always nice to only send the tokens to the routes where they are needed, instead of sending them to every route by default. If I wanted to send the access tokens to only /accounts and /services I'd use app.config['JWT_ACCESS_COOKIE_PATH'] = ['/account', '/services']. As you can see it's simply a list of the urls you want the access tokens to be accessible from.

As for refresh tokens, I always send them only to the refresh url.
app.config['JWT_REFRESH_COOKIE_PATH'] = '/token/refresh'

Verify jwt identity : Even if get_jwt_identity returns an actual id, check whether the id exists in your database before assuming them to be an user outright.
Fresh/Non-Fresh pattern : Make sure the important routes are protected using fresh tokens and never let fresh tokens be refreshed using refresh tokens. They should only be created when the user actually uses their password/2FA to log in.
Change up the tokens on password change : It's a good idea to recreate both access and refresh tokens if a user changes their password.

And That's it! I hope this guide helps you in your JWT adventures!

Porting C structs into Python, because speed!

Phantz — Wed, 01 Jan 2020 14:15:10 +0000

Why use C with Python?

Python is a really fun and friendly language, it offers so many cool features, has a great supportive community and has modules of pretty much everything you can think of. Programs that would be written in 10-20 lines in other languages, can sometimes be written in just a single line in Python.

But all that comes at a major cost, Performance.

It's no coincident that so many libraries/frameworks and even compilers for python exist that focus on improving python's performance. PyPy, Numba being only a few of them. Performance is crucial for pushing the limits of a program. Luckily python has an inbuilt module called ctypes, which allows us to port/wrap C code with python. This is incredibly useful in any advanced tasks as, by nature, C is very fast.

Now, ctypes is a vast and advanced topic of python. So I'll only be covering one of the more useful features, using C structs in Python. I'll be using a C program to calculate the Cartesian product of a large order and port it to python!

Sounds cool, but how?

Ok so first, we're gonna need some C code. You'll be able to find a lot examples using basic C programs but I want to give a practical example. Infact, I actually used this very program as I was learning ctypes.

Ideally, you'll want to use a C program that performs tasks that'd otherwise be extremely slow or downright impossible in python. Cartesian product is one such example. The cartesian product using itertools that would take you multiple seconds in Python will only take you a few hundred milliseconds in C.

So, let's represent this piece of code ls = list(itertools.product(range(0, 256), repeat = 3)) in C.

The C code!

First we need to figure out what kind of datatype we would want. Ofcourse the above python code returns a list of tuples of integers ranging from 0 - 255. Now we could use multi-dimension arrays but I think the most efficient way of representing this in C is by using struct. So let's declare a struct containing 3 int values, each corresponding to Red, Green, and Blue. Let's name it PixelTuple, because this was originally intended to represent pixel values.



struct PixelTuple
{
    int red;
    int green;
    int blue;
};

(optional) We might as well name it something easier to type...
typedef struct PixelTuple pxtup

Now we can allocate an array of struct PixelTuple variables in our main function. A Cartesian product of 3 sets of length 256 yields a total number of 256^3 elements. So the array will contain that number of struct PixelTuple type variables. Now such a large variable should be stored in the data segment of your system memory. To do that, you can either make the array global or declare the array with the static keyword.

Note: I highly recommend reading up on C's memory layout as well as static variables to get a clear idea. But TL;DR: the static keyword allocates memory for a variable in the data segment which allows for larger variables to be stored. They are also kept in memory as long as the program runs as opposed to other variables which are destroyed outside of the scope/function.

So let's declare our array in main():
static pxtup pxarray[256*256*256];

Now we will need some other variables, specifically one array of integers from 0 to 255 (valuearray), 3 variables (i, j, and k) to iterate through the 0 to 255 3 times, 1 variable (pxindex) to keep track of the index of the pxarray variable and a FILE pointer variable to open the file where we'll store the struct array in the end. Let's declare these first too:



int i, j, k, l, pxindex = 0;
FILE* pxF;
for (i = 0; i < 256; i++)
{
    valuearray[i] = i;
}

Now we need to write a function to actually calculate the Cartesian Product itself, let's do that!



for (i = 0; i < 256; i++)
{
    for (j = 0; j < 256; j++)
    {
        for (k = 0; k < 256; k++)
        {
            pxarray[pxindex].red = valuearray[i];
            pxarray[pxindex].green = valuearray[j];
            pxarray[pxindex++].blue = valuearray[k];
        }
    }
}

So yeah... all that code could literally be represented by a single line in python. But! on the bright side, this code is significantly faster than the python code. In my opinion, speed is what really matters.

Now we have to save our pxarray in a file so python can read from it later. Let's use our previously declared pxF pointer to create a .bin file, then we can simply use fwrite(const void * ptr, size_t size, size_t count, FILE * stream) to write the struct array inside.



pxF = fopen("PixelTuples.bin", "wb");
if (allcF == NULL)
{
    printf("File cannot be created\n");
    return;
}
fwrite(&pxarray, sizeof(pxtup), 256*256*256, pxF);
fclose(pxF);

The first size_t argument in fwrite() i.e size_t size should be the size of the data type and the next size_t argument i.e size_t count should be the number of datatypes present in the first argument i.e const void * ptr. Obviously, there are exactly 256^3 number of pxtup or struct PixelTuple elements in pxarray.

To recap, here's the full code

There we go! The .bin file is ready to be used by Python! But before we move on, I'd just like to mention that if you'd like to import this .bin file back to a C struct array, you can simply open the File (PixelTuples.bin) in rb mode and use fread(const void * ptr, size_t size, size_t count, FILE * stream), which works much like fwrite(). So to read the data into the variable pxarray_read of size 256^3 you'd use:
fread(&pxarray_read, sizeof(pxtup), 256*256*256, pxF);

Ok! Now for ctypes!

The Python code!

The python code is actually pretty small and straightforward, as always...

First we have to make sure Python knows the structure/arrangement of the C struct we are porting, to do that, we make a Python class and pass the Structure parameter for ctypes into it. Then we have to declare the __fields__. Since our C struct had 3 int values we will refer to these with c_int in python. I name them 'red', 'green', and 'blue' but it doesn't really matter what you name them. Just make sure the datatypes are correct and there are exactly the same number of variables in __fields__ list as there were in our actual C struct.



from ctypes import *

class pxtup(Structure):
    _fields_ = [('red', c_int),
                ('green', c_int),
                ('blue', c_int)]

Now all we have to do is open the .bin file we saved earlier, open it in rb mode and keep importing all the struct variables that are of the same size as pxtup() until we reach EOF.

We use the python function file.readinto() to read the file stream and append all the structures into a list. The end result is a list of tuples of 3 integers, exactly what we wanted!



with open('allc.bin', 'rb') as file:
    result = []
    allc = pxtup()
    while file.readinto(allc) == sizeof(allc):
        result.append((allc.red, allc.green, allc.blue))

This is what the result will look like-

Aaand that's it! That's how you port your C structs into Python. With performance improvements and efficiency guaranteed! I hope you learned something from this guide, thanks for reading!

Special thanks to the pixcryption project on GitHub to motivate me to find a way to improve Cartesian Product performance which eventually led me to learn about ctypes!

Forem: Phantz

Wanna have fun with Lazy Iterators in good ol' C?

How you can implement resumable tasks in celery

Preface

Guide

Concept

Implementation

Explanation - pause_or_continue

Why are clause and callback task signatures and not regular functions?

What is self.request.chain?

Explanation - tappable

Usage

An example of clause

An example of callback

An example of starting the workflow

An example of pausing the workflow

An example of resuming the workflow

A guide to testing flask applications using unittest!

Test using a FlaskClient object

Test a live flask server using selenium

Pushing Native Messaging to the limits! - C vs Rust

Native Messaging - Illustrated

Libraries used for Native Messaging

The story (also performance is love)

My Image/GIF glitching python library is now on pypi!

I made a commandline python script to glitchify images and make GIFs!

Looking for contributors with python 3 tensorflow experience for a cool chatbot!

What the project is

What we need help with

I wrote a lightning fast prime number generator in C!

Cool Algorithm: Calculating any index of a Cartesian Product!

What does it do?

How does it work?

The Code

How is the human brain so good at matching delimiters?

Part 1 - Auto-complete

Introduction

The Algorithm

Part 2 - Function associated with given closing delimiter

Introduction

The Algorithm

Conclusion

Python: Using JWT in cookies with a flask app and restful API!

Initializing our app

Setting cookies and returning redirects

Token Freshness

Usage

Passing JWT to RESTful API resources

JWT Error handling in API resources

Advices, Standards and Discussions

Porting C structs into Python, because speed!

Why use C with Python?

Sounds cool, but how?

The C code!

The Python code!

Explanation - `pause_or_continue`

Why are `clause` and `callback` task signatures and not regular functions?

What is `self.request.chain`?

Explanation - `tappable`

An example of `clause`

An example of `callback`

Test using a `FlaskClient` object

Test a live flask server using `selenium`