Forem: Nicolas Galler

Managing Python Monorepos with uv Workspaces and AWS Lambda

Nicolas Galler — Fri, 16 Jan 2026 17:31:52 +0000

uv workspaces are a super tool when developing interconnected Python packages, especially in mono-repo setups. uv will set it up pretty automatically if you have a pyproject.toml at the root of the repo and you run uv init inside a subfolder.

What this means is:

only one venv will be set up, with all dependencies for all projects (this is pretty nice for the IDE because you don't need to keep switching venvs, but note the important caveat below)
you can reference a local project by adding it in [tool.uv.sources], and uv will automatically install it as editable (so you won't need to build each time you make a change and keep reinstalling your project):

[tool.uv.sources]
common_logging = { workspace = true }

one caveat: if your projects have conflicting dependencies, uv will fail the install! For microservices, it is probably best to keep dependencies on compatible versions anyway
another caveat: since there is only one venv, it's possible the IDE will detect a package that is not really in the project dependencies, and you accidentally import it! This can easily go unnoticed during local development.
for that, you might be better off forgoing workspaces and using a path dependency:

[tool.uv.sources]
common_logging = {
    path = "../common/logging", editable = true
}

I have decided to keep using uv workspaces, but we have a CI check to catch that type of problem before it reaches production.

Running uv in a workspace

From the root of a workspace, uv sync will install only the workspace-level packages. Likewise uv run will not install all the packages. This is not a way to specify common dependencies, because they will only be installed if the workspace root is installed (so for example uv export inside of the package will not include those "shared" deps).

Now, as you go into a package and run uv sync, uv will add the dependencies from that package to the venv... so eventually you will have everything installed. But you can't be sure that is always the case. This is also nice if you want to install deps for only one package, before running tests in CI, to make sure it does not accidentally depend on the other packages' dependencies.

To run and include all package dependencies (needed for example for a type checker!), use

uv run --all-packages

Use in Docker build

The following examples assume AWS Lambda container images, which explains some of the paths (/var/task, /var/lang/lib) and the base images used. The general approach still applies to non-Lambda containers, but paths and base images would need to be adjusted.

There are two catches:

you don't want to install all the workspace dependencies necessarily — some are not shared among all your microservices
you don't want to install the local dependencies in the same layer as the core dependencies, because the core dependencies are less likely to change (and, more importantly, are much bigger)

This is especially important for Lambda images, where image size and layer caching directly impact cold start performance.

I struggled a bit with the latter, but in fact uv makes it rather easy. It's not super well documented though — there are many different ways to get there that will sort of work but leave you with some cruft in the final image.

First attempt using uv sync

you do need to copy all the pyproject.toml files and the single uv.lock file that you are going to need. This includes all your local dependencies, and that makes sense — if you changed a requirement in a local dependency, it's now part of your core deps, and you can include it in the base layer

COPY pyproject.toml uv.lock /build
COPY services/common/logging/pyproject.toml /build/services/common/logging/
COPY services/${SERVICE_NAME}/pyproject.toml ./

then, install the core deps, making sure you specify the --package argument to install only for that part of the workspace, and the --no-install-local argument to prevent including other workspace members:

RUN uv sync --frozen --no-install-local --no-dev --package ${SERVICE_NAME}

next, the local dependencies… uv will figure out which ones of those are needed for the project based on the workspace graph:

COPY services/common /build/services/common
RUN uv sync --frozen --no-editable --no-install-project --no-dev --package ${SERVICE_NAME}

and finally, the code for this service:

COPY services/${SERVICE_NAME} ./
RUN uv sync --frozen --no-editable --no-dev --package ${SERVICE_NAME}

I refined this a little bit, because the above will leave two copies of the source code (one under /build/services and one under the venv), which is undesirable in a Lambda image where size matters:

# Copy dependency files: workspace, and service project
COPY pyproject.toml uv.lock /build
COPY services/${SERVICE_NAME}/pyproject.toml ./

# Install core dependencies
RUN uv sync --frozen --no-install-local --no-dev --package ${SERVICE_NAME}
ENV PYTHONPATH=/build/.venv/lib/python3.14/site-packages

# Install local dependencies
RUN --mount=type=bind,source=services/common,target=/build/services/common \
    uv sync --frozen --no-editable --no-install-project --no-dev --package ${SERVICE_NAME}

# Finally, copy the project source
COPY services/${SERVICE_NAME}/src /var/task

Now, the downside is that you are left with a few unneeded things in this final image: there are some leftovers from the venv and the uv command itself. I was also worried that running the uv command directly in the final image could leave me with some surprises, especially in a production Lambda environment.

Second attempt using pip

To work around the above problem, I install the dependencies in two steps into separate folders. Then, I do another stage in the Docker build to collect all these dependencies into a clean final image.

Despite the title, this still relies on uv for dependency resolution — pip is only used for the final installation step, which keeps the runtime image simpler.

# --- Build Stage ---
FROM public.ecr.aws/lambda/python:3.14-arm64 AS builder

# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

ARG SERVICE_NAME

# Set up the working directory
WORKDIR /build

# Copy dependency files: workspace, and service project
COPY pyproject.toml uv.lock /build
COPY services/${SERVICE_NAME}/pyproject.toml /build/services/${SERVICE_NAME}/pyproject.toml

# Install core dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-install-local --no-dev --package ${SERVICE_NAME}

# Install common dependencies
RUN uv export --package ${SERVICE_NAME} --no-editable --no-dev --frozen --format requirements.txt | \
    grep '^./services/common' > common.txt
RUN --mount=type=bind,source=services/common,target=/build/services/common \
    uv pip install --no-deps -r common.txt --target /build/common

# Application code (we could use uv pip install here too, but this is simpler)
COPY services/${SERVICE_NAME}/src /build/app

FROM public.ecr.aws/lambda/python:3.14-arm64 AS final

# Copy core dependencies from venv
COPY --from=builder /build/.venv/lib /var/lang/lib

# Copy common
COPY --from=builder /build/common /var/task

# Application code
COPY --from=builder /build/app /var/task

# Set the CMD to your handler
CMD [ "ingestion_worker.main.handler" ]

This results in a smaller, cleaner final image that contains only what Lambda needs at runtime.

Exploring Modern Python Type Checkers

Nicolas Galler — Mon, 12 Jan 2026 14:23:41 +0000

I’ve been playing around with alternative Python type checkers and language servers, looking beyond the more legacy options to find a setup that’s powerful without getting in the way. Along the way, I compared ty and Pyrefly, tweaked extensions and inlay hints, and ran into a few interesting differences in behavior, ergonomics, and typing semantics. The result isn’t a definitive winner yet, but it did surface some useful insights—and a couple of gotchas about Python typing that are easy to miss.

Both extensions were very easy to install from the Extension Marketplace in Cursor.

I disabled the “Python” extension from Anysphere (keeping the ms-python one). Anysphere’s extension is essentially pyright, which was doing double duty with ty.

For ty, I also disabled the language server, as suggested in the documentation:

{
  "python.languageServer": "None"
}

After that, there were still some logs being output:

ty
ty language server

There were also inlay hints showing inferred types. They’re somewhat useful, but overall a bit too much visual clutter, so I disabled them. I can still see variable types via tooltips when needed.

I also tried Pyrefly. The setup was similar, but I didn’t need to manually disable the language server—Pyrefly automatically disables Pylance. It offers a few more customization options. One thing to note: by default, it will not display type errors unless a configuration file is found in the project. This can be forced via the “Display Type Errors” setting.

Where Pyrefly’s language server was better:

In one instance, it was able to automatically locate missing imports when I used Cmd + .. This involved a type alias, where ty took a few more shortcuts.

Inlay hints comparison

Ty was a bit more obnoxious with inlay hints out of the box, while Pyrefly tended to include them only when they were genuinely useful. For example:

def merge_data(self, source: str, dest: str):
    source = check_sql_id(source)
    self._validate_source_schema(source)
    stmt = build_update_statement(self.con, source, dest, namespace, scope)

Both tools correctly inferred the type of source as SqlIdString (rather than the original str), but:

ty added an inlay hint
pyrefly did not, unless I was also renaming the variable

Ty also tended to union types with | Unknown when using list comprehensions or list literals, which was pretty weird and annoying. Thankfully, there’s already an open issue for this, so I’ll keep tracking it.

Pyrefly had better support for Pydantic models, correctly reporting type errors when using field aliases for example.

Pyrefly had one case where it refused to redefine a variable’s type via annotation. That’s valid Python, though arguably questionable style.

Finally, I use Protocol for my interfaces on the project, and Pyrefly had generally more helpful messages when an interface was not correctly implemented.

Both of those are easy to integrate in the CI workflow, and they play nicer than pyright with Python's package manager. They both have LSP support, though the one of Pyrefly is a little more polished.

Overall, I’m more aligned with Pyrefly. I want to keep evaluating ty because Astral's tools have been so great, I have no doubt they will catch up! I’ll revisit that choice if ty starts to get ahead.

One final typing takeaway: a type alias is not the same as NewType. A type alias doesn’t provide real type safety—it’s just a nickname for an existing type.

Stop Fighting Circular Imports

Nicolas Galler — Sun, 05 Oct 2025 16:09:01 +0000

The code is good, but you are missing the type hints!

Oh, I know, but as soon as I add them, I get these circular import errors!

Mmm, that's often a smell, let's dig a little

Sounds familiar? Yet sometimes even when digging in, the design is fine, and the modules really need to import each other, especially if you want to keep things simple and not introduce a Java-style interface hierarchy.

Turns out, modern Python has a solution for this, in fact, several of them. Let's check it out :)

Here a simple example:

# post.py
from models.user import User

class Post:
    def __init__(self, author: User):
        self.author = author

# user.py
from models.post import Post

class User:
    def __init__(self, name: str, posts: list[Post]):
        self.name = name
        self.posts = posts

This gives:

ImportError: cannot import name 'Post' from partially initialized module 'models.post' (most likely due to a circular import) (/home/nico/scratch/models/post.py)

So, first solution, from PEP-563, was to be able to do the annotations as strings.

class Post:
    def __init__(self, author: "User"):
        self.author = author

But that's not enough for most tools, you also need to tell it where to find User... yet you can't just import it as it would be back to a circular import, so you use a special "TYPE_CHECKING" guard:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from models.user import User 

class Post:
    def __init__(self, author: "User"):
        self.author = author

And finally you can turn on these string annotations for the whole module using a future import:

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from models.user import User 

class Post:
    def __init__(self, author: User):
        self.author = author

Now, as often in Python, there are a few layers added over time. For the full details, you'll want to read the PEPs:

start with PEP-563, which is rather simple, and is available today (and as early as Python 3.7)
then move on PEP-649 and its little sister PEP-749. They change the way this is done, but this won't be available until Python 3.14.

Then, under the hood things will be quite different and it won't use the "string" annotations anymore. For most usage (when annotating code for static type hints), you will just be able to drop the __future__ import.

Your database is not elastic!

Nicolas Galler — Fri, 12 Sep 2025 08:44:59 +0000

This post stems from a recent experience with a web application that was not scaling as expected. The application was designed to be able to scale horizontally, but the database was not. This is a common problem in web development, and one that is often overlooked until it is too late. The problem is that databases, especially relational databases, are not as elastic as the rest of the application stack. You can spin a new web server up in seconds, add some queue workers even faster, but the database is a different beast. Because of the strict principles it has to adhere to in order to ensure data integrity and consistency through a convenient API, scaling it horizontally (adding more servers to share the load) is quite complex - read-only replicas are possible, but not particularly cheap or easy to set up (cloud vendors will make that easier of course). Sharding is another option, but it is not trivial to implement and maintain (and can be quite costly too). And then there is the problem of transactions that span multiple shards, which is a whole other can of worms. You can scale it up (vertically) but that is not a sustainable solution. Eventually you will hit the limits of the biggest server you can afford, and then what?

It can be counter intuitive. In the earlier days of distributed application development (Windows-based CRM applications were my bread and butter in the 2000's), before the cloud, the database was often the only thing that could scale, and the application was limited by the performance of the client machines, which were essentially not upgradeable (or at least not easily). Now it is the other way around.

This reality hit home for me last week: a deployment which had been carefully planned and tested for weeks (though as it turned out, not well enough) quickly went south when it became obvious that too much of a burden was shifted to the database. Unfortunately our load testing did not accurately reflect the production usage and the new API was called a lot more than expected. The web workers happily scaled out to try and answer the requests faster, which actually compounded the problem since it added to the load on the one thing that would not, could not scale: the database.

Eventually we were able to resolve this with not too much broken bones by scaling up the database, until the application could be corrected. The cloud managed database service made that easier but it was still not a seamless process (and is also rather costly since it can't automatically scale back down either). And the feature had to be re-architected to be less database intensive - thankfully the change was rather isolated so we could turn that around quickly.

Reading partitioned Delta table with Polars

Nicolas Galler — Wed, 10 Sep 2025 10:46:35 +0000

I recently had the occasion to revisit some findings about the best practice for reading partitioned Deltalake data from Python. We use Polars for processing data, so something that integrates well with Polars is more convenient. Polars has a scan_delta method, but this used to not be optimal for reading partitioned data. The library has progressed so much recently though that it was due for a retest.

I had 2 datasets, each partitioned in 3 roughly equal chunks - 1 very small (700kb), and a somewhat bigger one (79mb), and wrote a script (or rather I had cursor generate a script) to analyze the memory utilization and running time for 3 possible approaches: use lazyframe with scan_delta (which is the most convenient), use dataframe with read_delta, or use the deltatable API directly with pyarrow options to specify the partition to read.

Pleasantly I was surprised to see that the LazyFrame + filter option, which is the most natural and convenient to use, is also the fastest and the most memory efficient. The script gave me this cute results table:

📈 FILTER + AGGREGATION STATISTICAL ANALYSIS

Approach	Time (s)	Memory (MB)	Records
2. DataFrame + Filter + GroupBy	0.653±0.134	967.9±1.8	1330378±0
1. LazyFrame + Filter + GroupBy	0.282±0.033	26.8±0.6	1330378±0
3. DeltaTable + Filter + GroupBy	0.417±0.010	696.5±8.6	1330378±0

🏆 BEST PERFORMERS (Based on Averages):

⚡ Fastest: 1. LazyFrame + Filter + GroupBy Average time: 0.282s ± 0.033s
💾 Most Memory Efficient: 1. LazyFrame + Filter + GroupBy Average memory: 26.8MB ± 0.6MB

Find the code on Github