Forem: Goce

Working with EXIF, IPTC and XMP in AWS Lambda

Goce — Fri, 15 Jan 2021 08:46:07 +0000

Reading / writing EXIF, IPTC and XMP data is a common operation when doing image processing. Almost all programming languages have some built in functionality for doing this, but there is a tool called ExifTool, which is something like a Swiss army knife when it comes to EXIF, IPTC and XMP. The only downside is the added complexity of setting it up for your environment.

The reason why we ended up using it over the built in PHP functionality, was the limited and somewhat unreliable output of the built in exif_read_data() function which did not properly detect the EXIF data on a number of different photos.

This setup uses Bref / PHP for the examples, but the same process and logic can be used for Node, Go and other languages. We use this approach regularly when building PHP projects at ThinkWeb that need reliable image handling on AWS Lambda.

ExifTool requires Perl to run so, we will need to add a Perl layer to our language of choice setup. Next we will create the ExifTool layer and put it all together. The final step is to use an ExifTool wrapper for your language of choice to do the ops.

I will not be showing the actual EXIF, IPTC and XMP reading / writing. This is quite language specific plus there are many different libraries and examples available online for each language, just Google it.

The Perl layer

Head over to MetaCPAN and scroll to the prebuild public Lambda layers section. Select the ARN for the AWS region and Perl version (5.32 in this case) you want to use. The ARN should look something like arn:aws:lambda:eu-west-1:445285296882:layer:perl-5-32-runtime-al2:2.

Creating the ExifTool layer

Here is a small script that creates the ExifTool Lambda layer. It is a bit verbose but it does the job. The important thing here is changing the shebang line.

#!/bin/sh
export EXIFTOOL_VERSION=12.14
rm -rf layer
curl https://exiftool.org/Image-ExifTool-${EXIFTOOL_VERSION}.tar.gz  --output Image-ExifTool-${EXIFTOOL_VERSION}.tar.gz
tar -xf Image-ExifTool-${EXIFTOOL_VERSION}.tar.gz
rm -rf Image-ExifTool-${EXIFTOOL_VERSION}.tar.gz
mkdir -p layer/bin
cp Image-ExifTool-${EXIFTOOL_VERSION}/exiftool layer/bin/.
sed -i "1 s/^.*$/#\!\/opt\/bin\/perl -w/" ./layer/bin/exiftool
cp -r Image-ExifTool-${EXIFTOOL_VERSION}/lib layer/bin/.
rm -rf Image-ExifTool-${EXIFTOOL_VERSION}
cd layer
zip -r exiftool.zip ./*
cd ..
mv layer/exiftool.zip .
rm -rf layer

Run this script, you will get an ZIP archive that you can use to create your Lambda layer in the AWS console.

After you have created the layer, you will get an ARN that looks like arn:aws:lambda:region:123456789012:layer:exiftool:1.

Putting it all together

The last thing is the proper ordering of the layers. It is important to put the Perl and ExifTool layers first, so that the PHP (or another language of choice) layer bootstrap is not overwritten. The layer setup for Bref (PHP) in the serverless.yml file should look something like:

layers:
  - arn:aws:lambda:eu-west-1:445285296882:layer:perl-5-32-runtime-al2:2
  - arn:aws:lambda:eu-west-1:123456789012:layer:exiftool:1
  - ${bref:layer.php-74}
  - ${bref:extra.gmp-php-74}
  - ${bref:extra.imagick-php-74}

ExifTool can be used by executing:

/opt/bin/perl /opt/bin/exiftool example.jpg

Since you need an actual file on the filesystem, you can use the tmp directory and create a temporary file to use with ExifTool.
There are many PHP / Node / Go libraries that provide a wrapper on top of ExifTool for reading and writing EXIF, IPTC and XMP data. When using these libraries, make sure that you change the ExifTool path with this one /opt/bin/perl /opt/bin/exiftool.

Preventing object replacement in Amazon S3

Goce — Sat, 11 Jul 2020 08:48:04 +0000

S3 bucket permissions do not distinguish between creating new object and replacing / overwriting an existing object and there are cases where this behavior is conflicting some business requirements for the system.

The business requirements

Let's imagine we have a document management service where users can upload documents;
Each document is uploaded to S3 with a unique key (file name) created by the application;
When an already uploaded document is replaced in the app, a new object with a new key is uploaded to the bucket and the old key is scheduled for deletion;
Uploaded object are available publicly via URL without the specific version passed as a parameter;
Once a object is uploaded it should not be overwritten, just deleted;
Object keys are generated in a way that prevents collisions so any attempts to overwrite an existing file is considered an exception and should be reported;

Object locking can not be applied since it prevents the deletion of the object. We could use Governance mode and a separate user with s3:BypassGovernanceRetention permission to preform deletes, but that is another story.

Flow

The general idea is for the bucket to dispatch a "ObjectCreated:Put" event notification on each upload that is piped to a lambda function that listens to all "ObjectCreated" events from the bucket. Once the function is triggered, it fetches all the object versions from the S3 API and if there is more than one version, removes all but the oldest.
In order for this to work, versioning must be enabled on the bucket so all changes to the file are preserved. See how to enable it here;

First time upload

An object is uploaded with a key "123456789.txt", the event is sent to the lambda function and since the uploaded version is the only one, the function does nothing.

Second, third, fourth... upload

Overwriting the "123456789.txt" object once it exists would create the same event and send it to the lambda function. Once executed, the number of file versions that will be fetched from the API will be grater than one. This will trigger the cleanup part of the function where all versions of the object except the oldest (the first upload) will be removed.

Creating the Lambda function

For the lambda function we are going to use Bref which is marketed as "Everything you need to easily deploy and run serverless PHP applications". Bref relies on the Serverless framework for the deployment part. Install instructions are available here. If you have AWS CLI installed and configured, you can skip the Serverless config credentials part;

Lets, start with the serverless.yml file, which defines the lambda function along with the permissions it needs to function properly. Replace the example-bucket-name with the name of your bucket in the custom section and the region under the provider section. If you need more info here is a list of all available properties for AWS.

The serverless.yml file:

service: app

custom:
  bucket: example-bucket-name

provider:
  name: aws
  region: eu-west-1
  runtime: provided
  memorySize: 128
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:ListBucketVersions
        - s3:GetObjectVersion
        - s3:DeleteObjectVersion
      Resource:
      - "arn:aws:s3:::${self:custom.bucket}/*"
      - "arn:aws:s3:::${self:custom.bucket}"

plugins:
  - ./vendor/bref/bref

functions:
  function:
    handler: index.php
    environment:
      TARGET_BUCKET: ${self:custom.bucket}
    events:
      - s3:
          bucket: ${self:custom.bucket}
          event: s3:ObjectCreated:*
          existing: true
    timeout: 5
    layers:
      - ${bref:layer.php-73}

# Exclude files from deployment
package:
  exclude:
    - 'tests/**'

Next comes the actual PHP function that will handle the events.
The index.php file:

<?php
declare(strict_types=1);

use Aws\S3\S3Client;

require __DIR__ . '/vendor/autoload.php';

return function ($event) {
    $bucket = getenv('TARGET_BUCKET');
    $targetObjectKey = $event['Records']['0']['s3']['object']['key'];

    $client = new S3Client(
        [
            'region'  => getenv('AWS_REGION'),
            'version' => 'latest',
        ]
    );

    $objectVersions = [];

    $results = $client->getPaginator(
        'ListObjectVersions',
        [
            'Bucket' => $bucket,
            'Prefix' => $targetObjectKey
        ]
    );

    foreach ($results as $result) {
        foreach ($result['Versions'] as $object) {
            if (strcmp($object['Key'], $targetObjectKey) === 0) {
                /** @var Aws\Api\DateTimeResult $lm */
                $lm = $object['LastModified'];
                $objectVersions[] = [
                    'Key'       => $object['Key'],
                    'VersionId' => $object['VersionId'],
                    'Timestamp' => $lm->getTimestamp(),
                ];
            }
        }
    }

    /**
     * If there is more than one verion of the object something is not right
     */
    $objectVersionsCount = count($objectVersions);

    if ($objectVersionsCount > 1) {
        usort(
            $objectVersions,
            function ($a, $b) {
                return $a['Timestamp'] <=> $b['Timestamp'];
            }
        );

        /**
         * Remove the oldest version (original upload) from the array
         */
        array_shift($objectVersions);

        $deleteQueue = [];
        foreach ($objectVersions as $version) {
            $deleteQueue[] = [
                'Key'       => $targetObjectKey,
                'VersionId' => $version['VersionId']
            ];
        }

        /*
         * Delete the object versions.
         */
        $result = $client->deleteObjects(
            [
                'Bucket' => $bucket,
                'Delete' => [
                    'Objects' => $deleteQueue
                ],
            ]
        );

        if ($result->hasKey('Errors')) {
            throw new Exception('Failed to delete old versions!');
        }
    }
};

To deploy the lambda function, open your terminal and type serverless deploy.

Enable events on the bucket

The last thing that remains is to enable the events on the bucket and link them with the lambda function. Go to the "Properties" tab in the bucket and click the "Events" version. Click "Add notification". Give the event a name, select "PUT" / "All object create events" and in the "Send to" dropdown select "Lambda" and enter the Lambda function ARN.
That's it. Time to test it with real uploads. If everything works as it should, no matter how many times you try to upload / replace / overwrite the same key the lambda function will remove all but the first version of the file.

Conclusion

While the approach shown above works, it is far from perfect and you might want to tweak it to your specific use case.
One improvement that can be made is instead of sending the events directly from the bucket to the lambda function, to pipe them through a SQS queue that is configured as the lambda function trigger. Once the lambda function processes the event it can send a message via a SNS topic to all interested parties. An optional SQS dead letter queue can be attached for better monitoring of the system.

Handing file uploads via commands

Goce — Tue, 19 Feb 2019 08:24:38 +0000

To give a bit of context to the problem at hand, we will imagine that we are building a simple app where users can upload an avatar for their profile. The code shown will not be tied down to any framework.

To avoid any thinking in the wrong direction we will ditch the "uploaded" part from the discussion right away. Should the app really care where the file comes from? In this particular case, I don't think so.

The command

We will name the command SetAvatar. Just a simple immutable class with some getters and no behavior / no business logic, a DTO. It needs the user Id and the uploaded file in order to set the avatar of the user in question.

When it comes to the file part, we have couple of options. We can pass the contents of the file as string to the command and this will be perfectly OK. The string is a primitive type, the command is immutable and serializable, but since it is likely that it will be persisted, I do not like the idea of dumping the full file in the database.

We can go for a resource, but resources cannot be serialized. Same goes for the UploadedFileInterface instance (which contains a resource + can read the file contents via a stream) but we will be messing up the immutability and will always be constrained to objects of this interface. Not a good idea at all.

So, we are left with the file path as the best candidate. Each uploaded file has a tmp name which is actually the file path where the uploaded image is temporarily stored and the app can read it without problems. The PSR-7 UploadedFileInterface exposes methods for getting the stream and the meta data including the path of the file. Since the command accepts a file path, we can use it, not only for uploaded images, but setting avatars using any image on the system.

final class SetAvatar {
    private $userId;
    private $filePath;

    public function __construct(UserId $userId, string $filePath){
        $this->userId = $userId;
        $this->filePath = $filePath;
    }

    public function getUserId(): UserId{
        return $this->userId;
    }

    public function getFilePath(): string{
        return $this->filePath;
    }
}

The implementation for the command handler will not be shown but you have the file path so you can go from there.

final class SetAvatarHandler {
    public function handle(SetAvatar $command): void {
        /**
         * TODO Process the file here, make thumbnails, save to storage, etc...
         */
    }
}

It is important that the handler validates the file, we cannot rely on the validations that are performed in the Presentation layer and there is a chance that the command will be triggered from somewhere else in our app by specifying an arbitrary file.

Functions like mime_content_type and finfo_open can be used to get the mime type and determine the extension of the file.

The presentation layer

The implementation of the controller is pretty basic but it does the job. The form factory creates the form from the request, validates it and if successful, the command is handled via the command bus.

final class AvatarController {
    public function upload(ServerRequestInterface $request): ResponseInterface {
        $response = new RedirectResponse('/');

        $form = $this->avatarFormFactory->createFromRequest($request);
        if(!$form->isValid()){
            $this->session->getFlashBag()->add('errors', implode(' ', $form->getValidationErrors()));
            return $response;
        }

        $this->commandBus->handle($form->toCommand());
        $this->session->getFlashBag()->add('success', 'Avatar uploaded');

        return $response;
    }
}

The UploadAvatarFormFactory creates the form with the uploaded file (UploadedFileInterface) and Id of the logged in user.

final class UploadAvatarFormFactory {
        private $session;

    public function __construct(Session $session){
        $this->session = $session;
    }

    public function createFromRequest(ServerRequestInterface $request){
        /**
         * @var UploadedFileInterface[] $uploadedFiles
         */
        $uploadedFiles = $request->getUploadedFiles();
        $uploadedFile = null;

        if(array_key_exists('avatar', $uploadedFiles) && $uploadedFiles['avatar']->getError() != UPLOAD_ERR_NO_FILE){
            $uploadedFile = $uploadedFiles['avatar'];
        }

        return new UploadAvatarForm(
            $this->session->get('userId', null),
            $uploadedFile
        );
    }
}

The UploadAvatarForm is the place that is best suited for validating the input and available parameters like the file size, media type, extension, etc. and returning a message to the user. Doing the validations here, ensures that we send a most likely valid command, but we should not rely solely on these (for obvious reasons) and implement the real in-depth validation in the command handler.

In this example the client file name is irrelevant, but if you need it, it can be passed to the command constructor as an optional argument.

final class UploadAvatarForm {
    use UploadedFileValidatorTrait;
    private $userId;
    private $avatar;

    public function __construct(?int $userId, ?UploadedFileInterface $avatar){
        $this->userId = $userId;
        $this->avatar = $avatar;
    }

    public function toCommand(){
        return new SetAvatar(
            new UserId($this->userId),
            $this->avatar->getStream()->getMetadata('uri')
        );
    }

    public function isValid(): bool {
        return empty($this->getValidationErrors());
    }

    public function getValidationErrors(): array{
        $errors = [];
        if($this->userId == null){
            $errors[] = 'Invalid user id!';
        }
        if(!$this->isFileUploaded($this->avatar)){
            $errors[] = $this->getFileUploadError();
        }
        return $errors;
    }
}

Replaying commands

The above code assumes that once the command is handled, successfully or unsuccessfully it cannot be replayed, due to the tmp file not being available after the request has been served.

If you want to reply the commands, one approach you can take is, to first save the file to some intermediary storage and pass that path to the command. This way the file will be available as long as you need it.