Forem: gaborschulz

Time Series Analysis with Polars

gaborschulz — Sun, 10 Dec 2023 14:13:04 +0000

Polars seems to be one of the most exciting developments recently in the field of data analysis. It promises to be a fast and easy to use tool that can help overcome some of the trickiest challenges analysts face when using Pandas. E.g. it can process datasets that are larger than the computer's memory by using lazy processing, and offers a friendlier learnings curve than PySpark.

Pandas offers many great features to simplify the analysis of time series. In this post I'm going to try to prepare and analyze a temporal dataset to see how far I can go with Polars.

Why use Polars?

Pandas has played a major role in making the data revolution possible and bringing data scientists to Python. It's still a great library that has a lot of power, but it has been around for almost 16 years. It has seen a lot of development and it still goes a long way if you need a robust dataframe library, but it has also some shortcomings.

One is related to the heritage of being built around the NumPy library, which is great for processing numerical data, but becomes an issue as soon as the data is anything else. Pandas 2.0 has started to bring in Arrow, but it's not yet the standard (you have to opt-in and according to the developers it's going to stay that way for the foreseeable future). Also, pandas's Arrow-based features are not yet entirely on par with its NumPy-based features. Polars was built around Arrow from the get go. This makes it very powerful when it comes to exchanging data with other languages and reducing the number of in-memory copying operations, thus leading to better performance.

The 2nd point is that pandas is not able to use all computing power of your computer without external help. No matter how many CPU cores you've got, Pandas computation will always use only one of them. There are tools like Dask that can help you get around that limitation, but that also adds another dependency and another tool you've got to learn. Polars has the ability to use the full computing power available to you without having to do anything in addition.

The 3rd point I love is the ability to use lazy APIs. These offer you the possibility to build your data pipeline first and work on datasets that are larger than your computer's memory. This is very powerful if you have to analyze huge datasets or if you are working with streaming data. It also comes with query optimization, which can be a huge performance blessing in its own right.

The API is very friendly, even though if you're coming from pandas, you might have to unlearn a couple of things and learn a few new concepts, but overall, you'll find that the learning curve stays moderate.

If you're curious to read more about the performance benefits, check out this benchmark.

The dataset

For this tutorial we're going to use the Supply of electricity dataset provided by Eurostat. This dataset contains the electricity supply of European countries per month between January 2008 and December 2019. You can download the dataset as a TSV file.

The dataset offers a few interesting challenges. First, it's pivoted, i.e. the months are on the columns and non-temporal identifiers are on the rows. Second, these identifiers are contained in a single column, separated by commas. Third, the dataset is a TSV file, i.e. we'll have to use tabs instead of commas to read the dataset.

We'll have to deal with missing values, data type issues, etc. as well.

What are we going to do?

We'll start by reading the dataset and checking if it's good for our purpose.

As usual, it's not, so we'll do some data wrangling to bring it into a shape that makes it easier for us to find answers to our questions.

Finally, we're going to ask a couple of questions and try to answer them using Polars.

What are we not going to do (yet)?

We're not going to go into visualization yet, but I'm planning to add some charts to the notebook as well at a later stage.

We're not going to take advantage of the lazy API as the dataset is quite small. I'm planning to do a separate post about this topic, so stay tuned.

Let's see the code

Feel free to download the dataset and the Jupyter notebook and run it in your own environment if you'd like to follow along. The notebook is available here:
https://github.com/gaborschulz/learning-polars/blob/main/01-time-series-analysis/time-series-analysis.ipynb

You'll find a detailed description of each step in the notebook.

Conclusions

If you've followed the notebook, you've seen that the Polars API is quite easy to use and offers you very powerful capabilities when it comes to working with time series. Feel free to continue experimenting with the dataset and the code, and let me know if you've got questions or comments.

I'm also happy to see your feedback on the topics you'd like me to write about around Polars.

References

Data source:
Supply of electricity - monthly data. EuroStat Data Browser

Polars user guide:
User Guide

Running Python 3.12 on AWS Lambda

gaborschulz — Sun, 11 Dec 2022 16:54:06 +0000

AWS Lambda functions have completely revolutionized the way I work with (and think about) compute. It's just amazingly convenient to implement a quick function, push it to Lambda, schedule it with EventBridge and let it run for free or for almost free.

However, if you're like me, you want to keep pace with the latest and greatest in the Python world, which, at the time of this writing, is Python 3.12. I find the new error messages unbelievably useful and I could hardly wait for the ability to use the same quotation mark inside the curly braces in f-strings.

And, this passion to keep up with the latest and greatest interferes with using AWS Lambda, which only offers Python 3.7, 3., 3.9, 3.10 and 3.11.

Luckily, there's a way around this, and it's fairly easy to implement: create your own Lambda runtime.

There's ample documentation and a wide array of tutorials that show you how to do this (e.g. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html). However, I also like to follow the D.R.Y. (Don't Repeat Yourself) principle, so why not package it as a reusable Docker image and use it in all my Lambda apps?

Step 1. Setting things up

Let's start by creating a new directory for our entire build setup:

mkdir -p python-lambda-runtimes 
cd python-lambda-runtimes

Let's create a Dockerfile in the directory with the following content (we're going to look at what each line does after the code).

# Define custom function directory
ARG FUNCTION_DIR="/var/task/"

FROM python:3.12-slim-bookworm as build-image

# Include global arg in this stage of the build
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev

# Install the function's dependencies
RUN pip install --target ${FUNCTION_DIR} awslambdaric


FROM python:3.12-slim-bookworm

# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}

# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]

Let's see what's happening here. We initiate a multi-phase build to reduce the size of our final image. On the FROM lines we choose which Python version we want to use for our runtime. If you wanted to work with Python 3.11 then you could simply replace the 3.12 part with 3.11.
As a last step of the first build phase we install awslambdaric, which is the AWS Lambda Runtime Interface Client which makes sure that the Lambda environment is able to communicate with our own code.
In the 2nd stage we simply copy the runtime interface client into our final image.

Step 2: Create a repository in your favorite Docker Container Registry

Since Docker has changed pricing model some time ago I started using AWS Elastic Container Registry (ECR) for my custom, private images. So, let's create a new repo. You can either use the AWS Console if you prefer the GUI, or if you like the CLI, you can simply run this command:

aws ecr create-repository --repository-name python-lambda-runtime

This will create a repository called python-lambda-runtimes. Feel free to replace the name with anything you prefer.

Step 3: Log in to your preferred repo

For example, if you are using ECR you can use this to log in:

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com

Please, replace with the ID of your AWS account.

Step 4: Build and push your image

To build your image and push it to ECR, you can do the following:

docker build -t python-lambda-runtime . && \
docker tag python-lambda-runtime:latest <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/python-lambda-runtime:latest && \
docker push <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/python-lambda-runtime:3.12

After this, you have everything in place to start using your pre-built image for your Lambda functions. In the next section, we'll do through all the steps you need to take to use it in your SAM template.

Step 5: Use the image in your SAM template

Create a Dockerfile in the directory of your Lambda function. Add the following content:

FROM <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/python-lambda-runtime:3.12
COPY . /var/task/
RUN chmod -R 0755 .
RUN pip install -r requirements.txt
CMD ["app.lambda_handler"]

Look for the Type: AWS::Serverless::Function part of your template.yaml file. Remove the CodeURI, Handler, and the Runtime lines, and add PackageType: Image instead.
Add a new section with the same indentation as the Type: AWS::Serverless::Function with the following content:

Metadata:
    DockerTag: <NAME OF YOUR FUNCTION>
    DockerContext: ./<DIRECTORY OF YOUR FUNCTION>
    Dockerfile: Dockerfile

For example, if your function was called TestFunction and lived in the directory test_function of your project root, your Metadata would look like this:

Metadata:
    DockerTag: TestFunction
    DockerContext: ./test_function
    Dockerfile: Dockerfile

If you run sam build, it will create the new container image for you.

If you've already deployed your Lambda function before, you'll have to delete it or deploy the containerized one under a new name. Also, because your samconfig.toml file already contains settings for your previous deployment, it makes sense to rename it to something else, and run sam deploy --guided to re-populate it with your new settings.

Building An Authorization Engine In Python...

gaborschulz — Mon, 05 Apr 2021 08:33:18 +0000

Some time ago I developed a competitive intelligence platform. Not much later the simple authorization model used in the tool turned out to be way too simple for all the demands users came up with.

Where we came from

Before the project started, we had a solution in place that was based solely on sending Excel files back and forth. Permission management was easy: you got the file, so you're authorized to view and edit it.
These files were split by country, so it was obvious that permissions in the tool would also be based on countries. If you were assigned to a country, so you could go in to view and edit all entries in that country.
Otherwise you couldn't even see any entries. Later, management decided that we could gain more benefits from opening up the tool a bit, so everybody should be able to see everything
but only edit items in the countries they were assigned to. But, since there is also an option to download data in bulk, there was also some risk.
What if someone who might be leaving the company on bad terms would go and download the entire database. So, let's limit that to only own countries.

Of course, there are some key users who are authorized to edit master data in the tool. And they are the only people who can delete customer entries from the database.
Key users were approaching me for a way to give other users the permission to delete stuff but with limitations. And, as always, there are some edge cases that need to have more
limitations than the tool could currently offer. Or have more rights and less limitations.

This whole authorization and permission thing started to get veeery complex, to say the least.

Where we wanted to be

I've always been a big fan of how AWS handles authorization to access resources. IAM policies are great: they give you a way to grant any permission on any level to any user and make sure that they
can do everything they need to, nothing more, nothing less.

Before, I'd already created a prototype that helped me understand how such a model works. Not that sophisticated, of course, but an authorization system that had at least the following properties:

it should be based on policies written in JSON
it should be able to deny or allow any action on any resource in the platform
you should be able to grant access based on groups and individual policies and these should be effective together
it should not put too big a performance burden on the end user, i.e. they should not wait more than 10% in addition to what they would spend waiting without the authorization framework

How we got there

Let me give you a quick rundown of the process I used to develop a solution that ended up in the productive application. It is based on my authorization experiment I mentioned above.

Step 1: The concept of my experiment

The concept is surprisingly simple.

Each user and group (country assignment) has a policy document, which is a JSON string. This policy document consists of a policy name, an action (what the users wants to do), a resource
(what the users wants to do it with) and an effect (allow or deny). Each of these policy elements uses regex notation to allow fine tuning. At runtime, these permission regexes are compiled and cached
so they can be reused (this helps to achieve the performance goal).

Each resource publishes its own resource document as a JSON-like object as well. JSON-like, because all quotation marks and white spaces after colons are stripped away, so it's not a valid JSON,
but it's very similar.

Authorization of a user on an individual item goes like this:

Check if there is an explicit DENY policy for the resource and action by matching all Deny policy regexes of the user. If there is, the request is denied.
Check if there is an explicit ALLOW policy for the resource and action by matching all Allow policy regexes of the user. If there is, the request is granted.
If there is no explicit ALLOW for the resource the request is implicitly denied.

Step 2: The prototype

I had created a prototype in Python for my experiment (https://github.com/gaborschulz/authorization-prototype) to understand how policy-based
authorization could work in a language-agnostic framework. The resource and the policy document processing were nice and smooth, so my idea was to recreate the whole thing in C# for this project.

Step 3: Putting it into action

Since the platform is developed in C# I had to convert the prototype into C#. I've built the authorization resource object into each model and added a central method for checking authorization
on individual objects. I also wanted to get lists filtered by authorizations.

Step 4: Learnings

The solution has proven to be amazingly effective as we are now able to write policies with almost any desired effect. We can hide items generally, from certain users only, hide only their details, etc.
The full power of regular expressions is making this tool very flexible and still fairly easy to use.

It uses only standard library stuff that is available in almost any programming language.

It's very important to use proper caching of the compiled regex statements to maintain performance.
It's enough to purge the user's authorization from the cache whenever a user policy gets changed or a predefined amount of time has passed.