Forem: Caleb Winston

A language for data flow and analysis

Caleb Winston — Tue, 02 Jul 2019 15:29:15 +0000

For many people, Python is their go-to language for data analysis. This in large part because the Python language makes it very easy to think about and write functions that operate on units of data. For example, this function determines a number segment a person of a certain age belongs to.

def age_segment(age):
    if age < 12: return 0
    elif age < 18: return 1
    elif age < 28: return 2
    elif age < 48: return 3
    elif age < 68: return 4
    elif age < 88: return 5
    else: return 6

Here's another function. It takes in an age segment number and returns whether or not the person is legally a minor.

def is_under_age(age_segment):
    if age_segment <= 1:
        return True
    return False

And here's one last one. This function's job is to pretty print an age. That's all it's job is.

def print_age(age_segment):
    if age_segment == 0: print("0-12")
    elif age_segment == 1: print("12-18")
    elif age_segment == 2: print("18-18")
    elif age_segment == 3: print("28-48")
    elif age_segment == 4: print("48-68")
    elif age_segment == 5: print("68-88")
    elif age_segment == 6: print("88+")

All of these functions operate on "single units of data". But a very common thing to do in data analysis is have a bunch of "units of data" and then have a sort-of "pipeline" of functions that each unit passes through with each function doing an operation that progressively transforms the data.

As an example, let's say we have such a list of data. And you want to take each element of data and pass it through this sort-of "pipeline" of functions that you could construct that passes data first to age_segment then to is_under_age and finally to print_age.

ages = [9, 19, 37, 28, 48, 13]

Now you might realize that you could just use Python's for loop construct to loop over this and call the functions in order on the ages in the list. And that is what many people will do. But there are a few issues you will run into.

You will have to know Python to change the structure of the pipeline (e.g. - you want to remove age_segment because your data is already age segments)
You will have to write (in my opinion) messy code to make functions run in parallel (Python provides the multiprocessing library to make functions run at the same time - instead of one after the other - and easily pass data between them; but this introduces quite a bit of extra code)

In fact, this is what that will look like.

from multiprocessing import Process, Queue
from ages_utils import age_segment as age_segment
from ages_utils import is_under_age as is_under_age
from ages_utils import ages as ages
from ages_utils import print_age as print_age
class PLPipeSentinel: pass
def run_ages(stream, out_queue):
    for data in stream:
        out_queue.put(data)
    out_queue.put(PLPipeSentinel())

def run_age_segment(in_queue, out_queue):
    while 1:
        inp = in_queue.get()
        if isinstance(inp, PLPipeSentinel):
            outp = PLPipeSentinel()
        if not isinstance(inp, PLPipeSentinel):
            outp = age_segment(inp)
        if out_queue is not None:
            out_queue.put(outp)
        if isinstance(inp, PLPipeSentinel):
            break

def run_is_under_age(in_queue, out_queue):
    while 1:
        inp = in_queue.get()
        if isinstance(inp, PLPipeSentinel):
            outp = PLPipeSentinel()
        if not isinstance(inp, PLPipeSentinel):
            result = is_under_age(inp)
            if result:
                outp = inp
            else:
                continue
        if out_queue is not None:
            out_queue.put(outp)
        if isinstance(inp, PLPipeSentinel):

            break
def run_print_age(in_queue, out_queue):
    while 1:
        inp = in_queue.get()
        if isinstance(inp, PLPipeSentinel):
            outp = PLPipeSentinel()
        if not isinstance(inp, PLPipeSentinel):
            outp = print_age(inp)
        if out_queue is not None:
            out_queue.put(outp)
        if isinstance(inp, PLPipeSentinel):
            break

if __name__ == "__main__":
    data = ages()
    in_age_segment = Queue()
    in_is_under_age = Queue()
    in_print_age = Queue()
    ages_process = Process(target=run_ages, args=(data, in_age_segment))
    age_segment_process = Process(target=run_age_segment, args=(in_age_segment,in_is_under_age,))
    is_under_age_process = Process(target=run_is_under_age, args=(in_is_under_age,in_print_age,))
    print_age_process = Process(target=run_print_age, args=(in_print_age,None,))
    ages_process.start()
    age_segment_process.start()
    is_under_age_process.start()
    print_age_process.start()
    age_segment_process.join()
    is_under_age_process.join()
    print_age_process.join()

Maybe this is legible to you but it certainly isn't for me.

What if there was a simple high-level language to describe the structure of a pipeline with a compiler that compiles the language code to Python. This language would be a language that anybody can use - a scientist, a business executive, ...literally anyone.

You would work with components of a pipeline having a basic understanding of what the components do (e.g. - is_under_age tells me if the age is considered legally underage) but no required knowledge of how they are implemented. Somebody who does know Python would write the implementations of the components but then the pipeline can be written and re-written and re-structured without needing to consult a person who knows Python because the pipeline is written in a simple high-level language that abstracts away the functions that make up any pipeline.

This high-level language exists! It's called Pipelines. With the Pipelines language, the above 63 lines of Python code is reduced to the below 5 line description.

from ages_utils import ages
from ages_utils import age_segment
from ages_utils import is_under_age
from ages_utils import print_age

ages |> age_segment /> is_under_age |> print_age

The |> indicates that each element passed into age_segment and print_age is transformed by passing into the age_segment() and print_age() functions with the results then passed onto the next function in the pipeline if one exists.

The \> indicates that each element passed into is_under_age is filtered. The element is passed into is_under_age and if the result is True it is passed on. If the result is False it does not get passed on.

There's much more to the language than just pipes and filters and features designed for more complex data flow and everything can be found in the "README" on the GitHub repository.

calebwin / pipelines

An experimental programming language for data flow

Pipelines is a language and runtime for crafting massively parallel pipelines. Unlike other languages for defining data flow, the Pipeline language requires implementation of components to be defined separately in the Python scripting language. This allows the details of implementations to be separated from the structure of the pipeline, while providing access to thousands of active libraries for machine learning, data analysis and processing. Skip to Getting Started to install the Pipeline compiler.

An example

As an introductory example, a simple pipeline for Fizz Buzz on even numbers could be written as follows -

from fizzbuzz import numbers
from fizzbuzz import even
from fizzbuzz import fizzbuzz
from fizzbuzz import printer

numbers
/> even 
|> fizzbuzz where (number=*, fizz="Fizz", buzz="Buzz")
|> printer

Meanwhile, the implementation of the components would be written in Python -

def numbers()
    for

…

View on GitHub

If you do decide to use in your own data analysis work, please let me know - I would love to see what you make! If not, I would appreciate it if you could star the GitHub repository for later if you find this interesting and potentially useful.

A language for programming GPUs

Caleb Winston — Sat, 01 Jun 2019 18:32:39 +0000

Most languages are for programming CPUs (that stands for Central Processing Unit). If you write a program in Python or Rust, for example, and run it - it gets run on your computer's CPU(s). But the CPU isn't the only part of your computer that can do computing. Your computer might also come with a GPU. GPU stands for Graphics Processing Unit and while GPUs are often used for processing graphics they are in general best at processing lots of numbers. If you have a list of a million numbers and want to scale them by a factor of 2. Your program will likely run faster on a GPU than a CPU.

But if languages like Python or Rust don't usually run on GPUs, what languages do? There are several. OpenCL, CUDA, and Halide are all languages for writing programs that can be run on GPUs. But all of these are designed first and foremost for C or C++. Also, there syntax can be a bit confusing...

static const char *kernelSource =
"#pragma OPENCL EXTENSION cl_khr_fp64 : enable  \n" \
"__kernel void mult(__global double *v) {       \n" \
"    int id, v_re, v_im;       \n" \
"    id   = get_global_id(0);  \n" \
"    v_re = 2*id;              \n" \
"    v_im = v_re + 1;          \n" \
"                              \n" \
"    v[v_re] = 2*v[v_re];      \n" \
"    v[v_im] = 4*v[v_im];      \n" \
"}                             \n" \
"\n" ;

There has to be a better way of programming GPUs. Well, there is a little something I've been excitedly working at for the past few months. It's a tiny programming language called Emu...

emu! {
    // multiplies 2 matrices
    // n is the dimension of the matrices
    // a and b are the matrices to be multiplied, c is the result
    multiply_matrices(n i32, global_a [f32], global_b [f32], global_c [f32]) {
        // indices of cells to multiply
        let i: i32 = get_global_id(0);
        let j: i32 = get_global_id(1);

        // execute step of multiplication
        for k in 0..n {
            global_c[i * n + j] += global_a[i * n + k] * global_b[k * n + j];
        }
    }
}

There are a few things you might notice right off the bat - (1) it's designed for use in Rust through a procedural macro instead of in C or C++, (2) it's syntax is Rust-like with Go-like parameter declarations. But wait - there's more!

You can annotate numbers with units. With the following syntax, the prefix of the unit specifies how much different numbers should be scaled and everything gets automatically normalized to be in terms of basic SI units. Here's an example...

emu! {
    move_particles(global_particles_x [f32], global_particles_y [f32], global_particles_z [f32]) {
        global_particles_x[get_global_id(0)] += 1 as cm;
        global_particles_y[get_global_id(0)] += 1 as cm;
        global_particles_z[get_global_id(0)] += 8.92 as nm;
    }
}

You can also use certain special numbers that are built in. These special numbers include a number of my favorite mathematical constants as well as physical constants. You can even use PAU!

emu! {
    add_particles(num_particles u32, num_moles u32) u32 {
        return num_particles + num_moles * L;
    }
}

I really hope this small programming language will help you program GPUs more easily from Rust. Please check out the repository - it's at https://github.com/calebwin/emu (there's even a little book in there!) and let me know if you have any comments, ideas, or constructive crticism. 🙌

calebwin / emu

The write-once-run-anywhere GPGPU library for Rust

The old version of Emu (which used macros) is here.

Overview

Emu is a GPGPU library for Rust with a focus on portability, modularity, and performance.

It's a CUDA-esque compute-specific abstraction over WebGPU providing specific functionality to make WebGPU feel more like CUDA. Here's a quick run-down of highlight features...

Emu can run anywhere - Emu uses WebGPU to support DirectX, Metal, Vulkan (and also OpenGL and browser eventually) as compile targets. This allows Emu to run on pretty much any user interface including desktop, mobile, and browser. By moving heavy computations to the user's device, you can reduce system latency and improve privacy.
Emu makes compute easier - Emu makes WebGPU feel like CUDA. It does this by providing...
- DeviceBox<T> as a wrapper for data that lives on the GPU (thereby ensuring type-safe data movement)
- DevicePool as a no-config auto-managed pool of devices (similar to CUDA)
- trait Cache…

View on GitHub

You can try it out right now by adding [dependencies] em = "0.1.2" your Cargo.toml or you can star repository for later.