Forem: Leandro Damascena

Storing large files with DynamoDB and S3 - the easy way

Leandro Damascena — Thu, 08 Jan 2026 10:20:15 +0000

DynamoDB has a 400KB item limit. That's fine for most data. But sometimes you need to store something bigger - a PDF, an image, a JSON blob that grew too large.

The usual solution? Store the file in S3, save the metadata in DynamoDB. It's a common pattern. But it takes work.

The manual way

Here's what you normally do:

Upload the file to S3
Get the bucket and key back
Save those in DynamoDB
When reading, fetch the S3 metadata
Download from S3 if you need the content
When deleting, remove from both places
Handle errors if one succeeds and the other fails

That's a lot of code for something that should be simple. And you have to do it every time.

# The manual way - lots of code
import boto3

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('documents')

# Upload to S3
s3.put_object(
    Bucket='my-bucket',
    Key=f'documents/{doc_id}/file.pdf',
    Body=file_content,
    ContentType='application/pdf'
)

# Save metadata to DynamoDB
table.put_item(Item={
    'pk': f'DOC#{doc_id}',
    'sk': 'METADATA',
    's3_bucket': 'my-bucket',
    's3_key': f'documents/{doc_id}/file.pdf',
    'content_type': 'application/pdf',
    'size': len(file_content),
})

# Later, to read...
response = table.get_item(Key={'pk': f'DOC#{doc_id}', 'sk': 'METADATA'})
item = response['Item']
s3_response = s3.get_object(Bucket=item['s3_bucket'], Key=item['s3_key'])
content = s3_response['Body'].read()

# And to delete...
table.delete_item(Key={'pk': f'DOC#{doc_id}', 'sk': 'METADATA'})
s3.delete_object(Bucket='my-bucket', Key=f'documents/{doc_id}/file.pdf')

It works. But it's a lot of code for something simple. And in pure Python, all that serialization and network handling adds up - especially in Lambda where you pay for every millisecond.

A better way

pydynox is a DynamoDB library with a Rust core. If you haven't heard of it, check out my intro post.

It has an S3Attribute that handles all of this. You define it once, and the library takes care of uploads, downloads, and cleanup.

from pydynox import Model, ModelConfig
from pydynox.attributes import StringAttribute, S3Attribute

class Document(Model):
    model_config = ModelConfig(table="documents")

    pk = StringAttribute(hash_key=True)
    sk = StringAttribute(range_key=True)
    title = StringAttribute()
    file = S3Attribute(bucket="my-bucket")

Now saving a file is one line:

from pydynox import S3File

doc = Document(
    pk="DOC#123",
    sk="METADATA",
    title="Contract",
    file=S3File(data=pdf_bytes, content_type="application/pdf"),
)
doc.save()  # Uploads to S3, saves metadata to DynamoDB

Reading is just as simple:

doc = Document.get(pk="DOC#123", sk="METADATA")

# Get the S3 metadata
print(doc.file.bucket)  # my-bucket
print(doc.file.key)     # documents/DOC#123/file
print(doc.file.size)    # 1234567

# Download the content when you need it
content = doc.file.download()

# Or get a presigned URL for direct access
url = doc.file.presigned_url(expires_in=3600)

Deleting cleans up both places:

doc.delete()  # Removes from DynamoDB AND S3

How it works

When you call save():

pydynox uploads the file to S3
Stores the S3 metadata (bucket, key, size, etag) in DynamoDB
If the upload fails, nothing is saved

When you call delete():

Deletes from DynamoDB first
Then deletes from S3
If S3 delete fails, the DynamoDB record is already gone (orphaned S3 objects can be cleaned up with lifecycle rules)

The S3 key is built from the partition key + the filename you pass in S3File. You can also set a prefix:

# Key will be: uploads/documents/{pk}/{sk}/report.pdf
file = S3Attribute(bucket="my-bucket", prefix="uploads/documents/")

# When saving:
doc.file = S3File(data=pdf_bytes, name="report.pdf")

Async works too

If you're using async:

doc = Document(
    pk="DOC#123",
    sk="METADATA", 
    title="Contract",
    file=S3File(data=pdf_bytes),
)
await doc.async_save()

# Later
doc = await Document.async_get(pk="DOC#123", sk="METADATA")
content = await doc.file.async_download()

When to use this

Use S3Attribute when:

Your data might exceed 400KB
You're storing files (PDFs, images, JSON blobs)
You want presigned URLs for direct downloads
You don't want to manage S3 uploads manually

Don't use it when:

Your data is always small (just use a regular attribute)
You need complex S3 features (versioning, lifecycle rules on specific objects)

The pattern is common

This isn't a new idea. boto3 lets you do this since forever - upload to S3, save metadata to DynamoDB. The difference is doing it automatically instead of writing the same code over and over.

One attribute. One line to save. One line to delete. The library handles the rest.

Links

pydynox: https://github.com/leandrodamascena/pydynox
S3Attribute docs: https://leandrodamascena.github.io/pydynox/guides/s3-attribute/
Install: pip install pydynox

pydynox: A fast DynamoDB ORM for Python with a Rust core.

Leandro Damascena — Thu, 01 Jan 2026 12:28:45 +0000

I've been building serverless apps for years. Lambda, DynamoDB, API Gateway - the usual stack. It's fast, it scales, and once you understand the access patterns, it's a great database. But I always felt like the Python libraries could be better.

The problem

If you use Python with DynamoDB, you have a few options:

boto3 - Works great, but you write a lot of code for simple things
PynamoDB - Nice ORM, though it can be slow with big datasets
Other libraries - Each has its own trade-offs

I wanted something that felt like a modern ORM but didn't sacrifice speed. Especially for Lambda functions where cold starts and execution time matter.

Learning Rust

About a year ago, I started learning Rust. It's a promising language - fast, safe, and growing fast. The syntax and concepts (ownership, borrowing, lifetimes) were hard at first. But once it clicks, it makes sense.

I already knew about PyO3 and always liked how it creates Python bindings. It handles the hard stuff - OS-level optimizations, serialization between Rust and Python, memory management. That's when I thought: why not use this for DynamoDB?

What is pydynox?

pydynox is a DynamoDB library for Python. It has a Rust core that handles the heavy work - serialization, compression, encryption - while keeping a friendly Python API on top.

I wanted something that feels nice to use, runs fast, and works well in environments like Lambda where every millisecond counts.

Getting started

Install with pip:

pip install pydynox

Define a model:

from pydynox import Model, ModelConfig, StringAttribute, NumberAttribute

class User(Model):
    model_config = ModelConfig(table="users")

    pk = StringAttribute(hash_key=True)
    sk = StringAttribute(range_key=True)
    name = StringAttribute()
    age = NumberAttribute(default=0)

Use it:

# Create and save
user = User(pk="USER#123", sk="PROFILE", name="Alice", age=30)
user.save()

# Get by key
user = User.get(pk="USER#123", sk="PROFILE")

# Update
user.update(age=31)

# Delete
user.delete()

If you use Pydantic, there's an integration for that too:

from pydynox.integrations.pydantic import dynamodb_model

@dynamodb_model(table="users", hash_key="pk", range_key="sk")
class User(BaseModel):
    pk: str
    sk: str
    name: str
    age: int = 0

What runs in Rust?

The parts that matter for speed:

Serialization - converting Python dicts to DynamoDB format and back
Compression - zstd, lz4, gzip for large attributes
Encryption - AES-GCM with KMS for sensitive fields
Rate limiting - token bucket algorithm
Batch operations - chunking, retries, error handling

These run on every request. Rust makes them fast.

Other things you can do

Beyond basic CRUD, pydynox supports the DynamoDB features you'd expect.

You can query with conditions and filters. Batch writes work without worrying about the 25-item limit - pydynox splits and retries for you. Same for batch gets with the 100-item limit.

Transactions are there when you need atomic operations across items. Async support if you're using asyncio. Lifecycle hooks let you run code before save, after get, or on delete - good for validation or logging.

Global Secondary Indexes work too. Define them in your model and query by non-key attributes.

There's also built-in observability. Every operation returns metrics - duration, consumed RCU/WCU, item count. You can enable logging to see what's happening. When something is slow or costs more than expected, you'll know. I have plans to add OpenTelemetry and Prometheus exporters too.

I have a big backlog of ideas I want to add - things like query caching, schema migrations, and better debugging tools. But first I want to make sure the core is solid.

Current status

pydynox is in alpha. I use it in personal projects and it works well, but before going to beta I want more people to try it.

If you work with DynamoDB, I'd love for you to try it out. Test it, break it, tell me what's confusing. Open an issue if something doesn't work. Start a discussion if you have ideas.

Contributions are welcome too - whether it's code, docs, or just feedback. The repo has a backlog of features if you want to pick something up.

Links

GitHub: https://github.com/leandrodamascena/pydynox
Docs: https://leandrodamascena.github.io/pydynox
Install: pip install pydynox