Forem: Daniel Wellington technology

DWCFLint - The tool supporting our cost control & improves our security roles

Lezgin Zilan — Tue, 19 Jan 2021 08:56:24 +0000

At Daniel Wellington, one of our core principles since 2016 has been serverless first, and almost everything we have developed over the last four years is based upon AWS’ serverless products.

We are heavily built upon CloudFormation and have been since five years and all resources need to be defined in a CloudFormation to be able to be created in any environment outside development. At the time of writing this article, we have 4 310 stacks across all our accounts & regions where 14.45% of the stacks are located in AWS China. At this stage, we started to discover various corner cases and undocumented behavior, the internal saying was CloudFrustation for the service. We also identified a need to align our way of working and follow best practices.

It started with the problem of strings with leading zeros in CloudFormation. Putting a string with a leading zero in a YAML-based CloudFormation file will lead to the aws-cli checking if the string contains only numeric characters. If it does, an implicit cast of the string to a numerical data type is made, which automatically strips away any leading zeros. We noticed this behavior the hard way, as an ID with leading zeros embedded in a CloudFormation file led to very confusing errors in our test environment. After this discovery, we decided it was time to come up with a way of detecting these kinds of errors early and in an automated fashion.

We started to build our own CloudFormation YAML parser lib but found the awesome cfn-lint project with a great set of built-in linting rules and a plugin API.

The way the plugin API works is that custom rules are implemented as Python classes and when running the cfn-lint binary, a file path to the a folder containing the class files or an import path to a module containing the rules must be supplied. For ease of packaging, installation and distribution of our custom rules, we decided to create a wrapper around cfn-lint containing the rules we created.

Over time, more rules were developed in response to discoveries of new corner cases, common mistakes, cost savings and aligning teams. Below is a timeline where we want to highlight some rules that have supported us well.

During 2018, we had a lot going on, and over two years of increasing lambda usage, we noticed that our CloudWatch logs were consuming massive amounts of storage space as the default retention period is infinite, even though we rarely needed to look at logs older than a month or a quarter. This led to the creation of a rule warning if the retention period is not explicitly set on log groups.

At one point, we hit the maximum quota of API Gateways of the endpoint type “edge”. Not many of the allocated API Gateways by that time actually needed edge capacity (same infrastructure as CloudFront), it was a legacy as we started with serverless in 2016 and the regional support of API Gateway was released by end of 2017. We could also see that the move was slow due to reuse of outdated CloudFormation templates from other projects whenever we built something new. To help with the migration and reduce costs as regional type endpoints are cheaper, we created a rule warning if the endpoint configuration was not explicitly defined.

By 2019, we had another period of cost-efficiency improvements, it was decided that all DynamoDB tables should use pay-per-request billing, so we made a rule warning if configuration files containing provisioned throughput were detected. At the end of 2019, the node.js 8.x lambda runtime support was dropped by AWS. To help with our migration effort, we made a rule causing an error status if any deprecated lambda runtime were detected in the CloudFormation files of a project.

In July 2020, during a period of security improvement work, we added a rule warning if the usage of AWS managed “FullAccess”-policies or misusage of “*” for Action or Resource.

if the use of different AWS resources’ built-in full access (*) IAM policies were detected.

At the time of writing this article, we support the following rules

No mismatched log groups and subscription filters
No missing endpoint types
No missing log retention period
No use of deprecated lambda runtime
No use of full access policies
No use of leading zeroes in numbers or strings
No missing/implicit log groups for lambdas
No use of old style subscription filters
No use of provisioned throughput
No use of reserved environment variable names
No use of reserved words for Dynamodb column names
No malformed subscription filters

Now we want to open source this (yet another) lint’ing tool that can be found here. We believe its stable and mature, although we expect to find new cases that can be transformed to rules for automatic detection and mitigation in the future. If this tool sounds useful to you, please check out the repository and feel free to report issues or submit pull requests.

Do you want to know more about how it is to work with technology at Daniel Wellington? Take a few minutes to watch our 3 min video story up to the right, if you are open to new challenges, check out our open tech positions.

Boosting the assembly process with AWS serverless & Axis camera

Lezgin Zilan — Thu, 14 Jan 2021 12:52:14 +0000

Daniel Wellington is a Swedish fashion brand founded in 2011. Since its inception, it has sold over 11 million watches and established itself as one of the fastest-growing and most beloved brands in the industry.

In 2012 we launched our Instagram account, and we became pioneers within influencer marketing. Influencers became our primary marketing channel.

To scale with our challenges and support the developing business, we started using Amazon Web Services (AWS) in 2014. Today we are heavily invested in AWS-platform with a broad range of products such as Amazon Elastic Container Service, AWS Lambda, Amazon DynamoDB, Amazon Sagemaker, Amazon Rekognition, and many more.

Our design principle has been serverless first since 2016. If serverless is not available or practical, we use containers as we consider EC2 as legacy. JavaScript and Go are some of our languages of choice.

The mobile app to optimize our return process that we built with AWS Rekognition service.

Focus on growth and perfection also leads to high standards and requirements for the factories producing our products. To make our pre-quality check work as smoothly as possible to minimize any potential human mistakes, we built an internal mobile app to optimize the return process.

After extensive testing, we realized that this would introduce other problems if we use the same solution but for another challenge. Using the simple optics in a mobile camera on the shining, polished watch wouldn’t give us the level of perfection we needed and not enable other use cases.

We needed efficiency at scale without making this a burden for our assembly line.

Early tests with the Canon EOS 5D Mark IV & Raspberry pi high quality camera connected to a raspberry pi running a bash script (yes, bash!) wrapping aws-cli in the glorious innovation lab at DW

In initial tests, we did we used a DSLR-(canon EOS 5D Mark IV), raspberry HQ- & a USB- camera. We realized that a DSLR-camera would be a better fit with the optics, detail adjustments, and pre-processing on a raspberry pi.

We also searched the market for DSLR cameras running android to deploy our app, but they disappeared as fast as they entered the market. This made us realize that we would need to maintain several layers ourselves, such as hardware, OS, and security (patching, certificate, disk encryption & patching). AWS Greengras would solve parts of the problems, but the ownership was still too much. For us, that meant a step backward, a lot of moving parts, an increase of the TCO (total cost of ownership), and less focus on what is most important, the code.

We started to look for a solution where we could remove such ownership and, at the same time, speed up the transfer time on the edge by doing the pre-processing of the image before it securely touched our code.

We quickly identified Axis Communications cameras to fulfill our pre-processing needs on the edge, and we now started to work together to get it integrated with AWS.

We had learned from our earlier OCR tests that AWS Rekognition service would be the most cost-efficient and powerful setup to use and that it would enable us to do automated quality checks and process-optimization at scale.

We do this by only capturing a specific part (we crop an area) of the picture, then transforming it into black/white and optimizing it. We then have a button connected to the IO port of the camera to trigger the event to send over the image to AWS over secure MQTT.

The benefit of secure MQTT (besides its secured with client certificate) is that it is built for IoT (low bandwidth and low compute power), long-lived and re-usable sessions. You would be amazed how rare the implementation is in off the shelf IoT products.

By presenting our use case, we quickly got the Axis integration team onboard; they utilize their plugin architecture (ACAP) to build secure MQTT support so we could utilize AWS IoT core. Their application is now being released to the public on GitHub with some beta testing from us.

On the AWS side, you can see the diagram below where we consume the data through secure MQTT, then process it via AWS Rekognition, store the result and the original picture on S3 for debugging reasons to be deleted with object expiration automatically.

More in depth about the different parts can be found in the GitHub repository.

For us, this is a big step towards bringing AWS into our assembly process to enable other use cases with the same setup, at the same time, gain quick wins for us. To support the community and the speed of transformation at manufacturers, we will also release our POC (proof of concept) code at GitHub that brings the powerful OCR function from Rekognition and the result delivered over MQTT together with powerful cameras. With some simple code changes, you could quickly get the object Rekognition with thousands of supported objects/scenes and custom labels to train it for your own object detection with a few images.

What about the operational cost? Excluding electricity on the camera we only have a AWS cost of 0,00106371$ USD per capture in the POC without optimizing any code or cost saving plans.

You may ask why? The right platforms and components are key to enable incremental innovation. You may think, to do what? Just To get your creativity started, using custom labels to detect tiny defects or that the clock hands are set to Swedish time when assembled. Yes, we are a Swedish brand and all our customers across the world starts the experience with Swedish daylight-saving time ;-)

This is the POC mounted inside a light tent always to get the same reflections and lighting regardless of whether it is in the lab, test, or the assembly process. A button is also mounted inside of the tent to trigger the event to capture and send the data over MQTT.

The code can be found here, and we would love to see your creative takes on this, even outside the manufacturing area ;-)

Do you want to know more about how it is to work with technology at Daniel Wellington? Take a few minutes to watch the video and if you are open to new challenges, check out our open tech positions.

How we made the return process more efficient thanks to an IPA beer notification hack

Lezgin Zilan — Mon, 04 Jan 2021 11:10:24 +0000

Most people know we were the pioneer with influencer marketing. It is impossible to archive without advanced technology. To make it possible, we started using Amazon Web Services (AWS) since 2014. We use services like Amazon Elastic Container Service, AWS Lambda, Amazon DynamoDB, Amazon Sagemaker and Amazon Rekognition.

We develop our services and applications on the AWS cloud platform and make extensive use of all its amazing serverless features, neatly matched with microservice based architectures.

Our design principle is serverless first. If serverless is not available or practical, containers are recommended, EC2 is legacy. JavaScript and Go are some of our languages of choice.

As a global fashion company that has used technology to disrupt an entire industry, we are continually learning and challenging our way of doing business. Our latest venture is to understand how machine learning can help us develop business value.

Since a year back, we started by looking into how we could do machine learning awareness to the business, since then we started internally a machine learning project where we are building a model to predict and detect certain events. Besides that, one of our employees, Anders, have also been playing around with the recognition service on his spare time, more specific the text in image feature.

The actual beer menu that is automatically captured and published several times per hour by the hipster bar.

Why?

Well, what Anders tried to do is to get a notification when a hipster bar at south side of Stockholm added a new IPA to the beer menu.

Why over engineer the problem?

Now the problem Is that they update the menu so often that they write the menu by hand and publish a picture of it to their website instead of having the menu written on the website

With Anders marvellous personal experience using the Rekognition service, he wondered if we could employ it to solve any actual business problem. What we came up with was to to use it to scan and read the back plate of our watches to help our warehouse workers that take care of returns.

During high peak season, each of them may have to handle several hundreds of returns per day. Now, that is a very tedious process, and we understood that it took some while to look up the details in different systems and relabel the return. The relabelling part alone required minimum 40 but up to 60 seconds per item. At the same time, we realized that the business had been looking for a third party solution for almost 18 months.

So we decided to solve the problem with ML technology. The outcome is a React Native app that communicates with an Amazon API Gateway and several Lambda functions, making use of SQS, Rekognition, and finally a serverless printer server (yes, you read it right).

The architecture diagram

Now, you may think that OCR is not a hard thing to do, and wonder why are we using a service when there are plenty of open source projects available. Well, we did a comparison between several frameworks and the AWS service and we found out that Rekognition was the one with the highest precision, while the other frameworks were just not good enough. Besides the best accuracy, we also found the cost of running Rekognition is low.

Our warehouse workers can save up to 56 seconds of the price $0.0013 per scan and lambda.

As a result, we brought the relabelling process down to four seconds and sometimes even faster, with higher quality and accuracy. The best part is it only took us less than two weeks to build and deploy it as MVP. The consumption of coffee dropped in the warehouse because they no longer need to spend time on tedious, repetitive task.

Here is a demo of it

What’s next? We have now taken this project out of the proof of concept phase and we are building a proper implementation of the app with all the additional requirements needed in a production environment, such as authentication, the ability to choose the AWS region closest to the warehouse, running automated builds, etc.

We will stay curious and keep experimenting until we set our eyes on the next problem to solve with the power of machine learning.

Do you want to know more about how it is to work with technology at Daniel Wellington? Take a few minutes to watch the video