Forem: Dan McKinney

Caching and Upstream Proxying for RedHat Packages

Dan McKinney — Wed, 25 Nov 2020 13:54:14 +0000

In keeping with our vision of offering a universal feature set across all the package formats we support, we are delighted to announce that we are now offering configurable upstream proxying and caching support for RedHat packages.

As we touched upon when announcing the same for Debian and Maven packages, there are a lot of reasons why this is a really good thing, so instead of going over those again, let’s jump straight into how you can set this up in you Cloudsmith repository.

Getting Started.

We don’t think that we could have made this much easier to configure (but, of course, we would love to hear your thoughts on this!)

In your Cloudsmith repository, you’ll see a menu item called “Upstream Proxying”. This is where we will configure our upstreams. Simply click the “Create Upstreams” button and select “RedHat” to create a new RedHat upstream:

You will then see the “Create RedHat Upstream” form:

You just need to add a Name for the upstream, the upstream URL, and a priority weighting – If you have multiple upstreams this will determine the order in which they are used.

We can choose to fetch and cache any requested package (instead of just fetching them), and to verify the SSL certificates provided by the upstream. Then, we select the individual distribution for this upstream.

Optionally - we can also add a GPG key, if required, for package signing, authentication headers for private repositories that require authentication and also some arbitrary headers if you wish to send something custom along with your request.

And that’s it, we have now added a new Redhat upstream to our Cloudsmith repository.

So now, the next request for a package that isn’t present in this repository will be passed through to the upstream and the package fetched it if it is available there. If you also enabled “fetch and cache”, the package will be cached in your Cloudsmith repo for any future requests.

Summary

By adding Redhat upstream proxying and caching, we are moving forwards towards our goal of being the most universal package management solution for your development, deployment and distribution workflows. We would love you to join us on this journey, and we are very excited about the platform we have built, the service that we provide, and where we are going to next. Sign up for free today, and experience Cloudsmith for yourself!

Did We Lose Something With The Adoption Of Containers? And Can We Get It Back?

Dan McKinney — Tue, 13 Oct 2020 10:36:05 +0000

The subject of containers probably doesn’t need much of an introduction. Since the launch of Docker in 2013 containers have become almost ubiquitous, with 89% of IT Professionals confirming their organizations used containers in some way during 2019.

That is no surprise. Docker itself calls the container “a standardized unit of software”, which is a nice way of saying that a typical container packages up all the relevant code, system tools, dependencies and libraries within an application, and effectively isolates that application from the rest of the environment and infrastructure.

As a result, containers have one great merit: they will reliably run, and run in the same way, wherever they are deployed. At a stroke, they eliminate the “well it’s working on my machine” problem that has caused endless heartache for both development and operations teams over the decades.

That benefit should not be under-estimated. For developers, in particular, it is rather wonderful to be able to put everything into a container and throw it over the wall knowing that it is going to deploy, and in turn knowing that it isn’t going to come back over the same wall for round 2.

It’s also - at least in theory - good for the operations team as well, for all of the same sorts of reasons.

But it isn’t all good news.

What Is A “Unit” Of Software, Really?

Prompted by the Docker definition of container I mentioned above, it might be instructive to ask ourselves whether the container is either the only, or indeed the best, ‘unit’ of software.

If we take a 10,000 ft view of the history of software development it probably looks something like this:

The entirety of the source code is the unit
The package is the unit
The container is the unit

In the old days of source code, you really had to be on top of things. The whole thing stood up or fell over as a whole, and things were pretty painful and expensive if (when) it did the latter.

When the package became the unit, things improved. Certainly, if we were following good package management and distribution best practices, then in most cases things fell over less and were easier to fix when they did. I’ll talk more about this later, but let’s also remember that packages also help speed up development full stop as they enable us to more easily re-use code and services.

In the move to containerization, the third stage in our evolution, the ‘unit’ of software has got bigger. As it has done so, it has concealed complexity in order to enable that crucial isolation from the environment it runs.

Essentially, the container runs as a kind of black box. The operations team just need to know what it does. They don’t need more detailed metadata and they don’t need to know what is inside it. After all, it’s all one self-contained package. So the unit of software has not only got bigger, it has also become less transparent.

The Problem With Being Big And Opaque

The problem with a large unit size is that within it are likely to be hundreds if not thousands of dependencies.

And the problem with being opaque is that we don’t know what they are.

What happens when something goes wrong? Let’s imagine (and this won’t take too much effort for veterans of LeftPad, HeartBleed, Event-Stream or one of any number of dependency related screw-ups) that a vulnerability is discovered within a certain package.

Where is that package currently in use in our ecosystem? We don’t know. Can we quickly roll that package back to a known ‘safe’ version wherever necessary? No.

Containers - as a ‘unit’ of software - live or die as a whole. If something is wrong or might be wrong, we have no option other than to spin up a new version and redeploy: not necessarily a simple task. And as we don’t necessarily know which containers are affected, we find ourselves struggling to get on top of an emerging security crisis based on limited data. Not a nice place to be.

The Lost Art Of Package Management

My last point is this: that containerization has led to a general decline in diligence and observance of best practices when it comes to handling packages and dependencies. After all, if I am going to throw them all into one container and check if it runs does it really matter where I get packages from?

The short answer is yes.

The longer answer is yes because if we aren’t going to be sure what is in any given container further down the line, we should be as careful as we possibly can be that nothing we can’t stand over sneaks in now.

Unfortunately, that doesn’t always happen. The path of least resistance is to get dependencies from pretty much anywhere and live with the consequences later. And as the developer cares about speed, and the ops team will be the ones dealing with those consequences, there is all the more reason to cut corners.

As developers, we need to get back to being diligent - and consistently diligent - about questions of security, provenance, reliability and availability when it comes to packages and dependencies. We need to be as sure as we can be that what we integrate into our projects (whether destined for containers or not) is what it says it is. And we need to know what we used where and when - so that we can fall back to a previous version quickly and easily if necessary.

It’s the least our colleagues in operations, and the users of our end product, deserve.

Universal Package Tagging

Dan McKinney — Tue, 22 Sep 2020 10:55:53 +0000

A key factor in good package management is organization. How you organize and structure your repositories will help you to unlock the efficiencies and promises of modern DevOps processes.

There are, of course, a multitude of ways you can organize packages. You can group by version numbers, formats, architectures, filetype and more. At Cloudsmith we have supported this by extracting as much of this metadata as possible when you upload a package and making this metadata available for searching/filtering.

Sometimes, however, the metadata just wasn’t granular enough – or it didn’t provide quite the nomenclature that you may have wanted. You wanted more. And we are extremely pleased to say that we have now added a new feature that greatly expands upon this functionality. Presenting:

UNIVERSAL PACKAGE TAGGING

So, what is new about this? And why does it matter?

Well, in short, we now give you the ability to add ANY custom tags to ANY package or container, either during package upload or after the fact. And you can do this via the Cloudsmith CLI or the Cloudsmith API.

Let’s say for example that you were using Cloudsmith to distribute your packages to end-users, and you have “Free” and “Premium” editions of your package. Well now you can simply add “Free” and “Premium” tags to the respective packages, and then create Entitlement Tokens to give your users access to just the packages they should be allowed – without resorting to adding the edition into the filename or creating two separate repositories. You can manage it all from one central location. And of course, as Cloudsmith repositories are fully multi-tenant for package formats, you can apply these tags across all the package formats you use, all in one repository.

Or perhaps you would like to tag a package based on where it is deployed. You could tag a package as “rest-api”, or similar, to differentiate where in your production application the package is used.

Let’s look at an example

Let’s start off by listing the packages in our demo repo. The CLI command for this is:

cloudsmith list packages OWNER/REPOSITORY

So, we can see that we have a collection of RPM packages in this repo, with various different versions. Let’s check one of those packages to see if it has any custom tags attached. The CLI command for this is:

cloudsmith tags list OWNER/REPOSITORY/PACKAGE-IDENTIFIER

Example:

OK, this package does not have any custom tags. Let’s now add one, and the CLI command for adding a tag is:

cloudsmith tags add OWNER/REPOSITORY/PACKAGE-IDENTIFIER tag1, tag2, tag3

Example:

Great, we have now added a custom tag “free” to this package. We can repeat this for the other packages in the repository, varying the tags to suit our purpose. In addition, we can also specify that tags are “Immutable”, which means they can only be removed or altered by someone with Administrator permissions on the repository, or by the package owner. When we are finished adding tags, we can look at the repository on the Cloudsmith website:

You can see the new custom tags listed for each package alongside the tags that were automatically created from the metadata when the packages were processed after upload. We can now use these custom tags in any searching/filtering we need to do, or add them as a restriction on an Entitlement Tokens that we create for the repository:

The syntax for searching/filtering is the same syntax you use when creating a search-based restriction for an Entitlement Token, so it really is easy to create a set of access tokens that divide the repository up into subsets of packages.

Summary

Visibility of the attributes of your packages is important, it will help you not only structure your repositories but can also help you gain insight on what package is where or what package a group of customers can access. And it’s not just the basic attributes of a package that you need visibility on, it’s anything about a package that matters to your organization and workflows. With universal package tagging, you now have the ability to add your own searchable attributes to your packages, so you can define what is of importance.

Caching and Upstream Proxying for Debian Packages

Dan McKinney — Tue, 15 Sep 2020 19:57:32 +0000

At Cloudsmith, we want to be your “one central source of truth” for your dependencies and package management needs. And in keeping with this ideal, we are extremely pleased to announce that we have added fully configurable transparent Proxying and Caching support for Debian packages.

Why does this matter?

Well, in short, it means that you can now use your private Cloudsmith repository for all of your Debian package needs – whether that is your own private packages or packages that you need from public upstream sources. Your private Cloudsmith repository is all that you need to handle both.

If you request a package from your Cloudsmith repository, and that package isn’t present in the repo, then Cloudsmith will automatically check any upstream repos that you have configured and will then fetch (and optionally cache the package for future requests) from the upstream.

This brings you several important benefits:

Easier setup. Your Cloudsmith repository is the only repository you need to configure on your clients. No more need to configure multiple repos, and handle multiple authentication credentials etc. Configure the upstreams once in Cloudsmith, and that’s it done.
Isolation. If you have cached packages and dependencies that you require in your Cloudsmith repository, then if the upstream repo goes down, is otherwise unavailable, or if the packages are removed then you can still access your cached versions. No more breaking of build or deployment process due to an unreliable upstream.
Visibility. You can view details on what specific packages were requested from the upstreams. Gain insights into what you have, and what’s missing – or who and what else you are currently relying on.
Performance. Cloudsmith repositories are backed by a performant, global CDN. This means that your own packages and those cached from an upstream are delivered with the same low latencies and speeds. Going further than this, with edge nodes in almost all geographic regions, your users will experience this performance wherever they are located. Distributed teams all benefit equally.
Security and Control. All of your packages and dependencies in one place means it’s easier for you to implement the controls and security policies that you need. Multiple sources mean multiple management tasks. Keep everything in one place and keep a tighter hold on what you have.

Sounds Great!. How do we set this up?.

Well, it’s easy. In your Cloudsmith repository, you’ll see a menu item called “Upstream Proxying”. This is where we will configure our upstreams. Simply click the “Create Upstreams” button and select “Debian” to create a new Debian upstream:

You are then presented with the “Edit Debian Upstream” form. This is where we enter the details of the Debian upstream we wish to use.

We add a Name for the upstream, it’s URL and a priority weighting- in cases of multiple upstreams this will determine the order in which they are checked for a package.

We can then choose to fetch and cache any requested package (instead of just fetching them), and to verify the SSL certificates provided by the upstream. In addition, we can choose to enable this upstream for source packages too.

Next, we select the distributions and architectures that we wish to use this upstream for, and finally, we can add optional authentication headers (for private repositories that require authentication) and also optional arbitrary headers, if you wish to send something custom along with your request.

And that’s it, we have now added a new Debian upstream to our Cloudsmith repository.

Behind the scenes, Cloudsmith will now start to index the packages available in the upstream repository. The upstream will be ready for use as soon as the indexing is complete.

So now, for the next request for a package that isn’t present in this repository, Cloudsmith will check the upstream and fetch it if it is available there and also cache it in the Cloudsmith repo (if you enabled caching) for future requests. It’s that simple.

Summary

Debian upstream proxying and caching support is just another step on our path to providing you with the centralized controls, security, management and visibility that you need to enable a modern, high-velocity and DevOps-first workflow for your package management needs. It means fewer things to worry about, less exposure to change and most importantly....

Well, what's most important to you? Let us know!

Integrating a Cloudsmith Repository with a GitLab CI/CD Pipeline

Dan McKinney — Thu, 10 Sep 2020 08:44:38 +0000

What is GitLab CI/CD

GitLab CI/CD is a tool that is built into GitLab. It allows you to create automated tasks that you can use to form a Continuous Integration and Continuous Delivery / Deployment process.

You configure GitLab CI/CD by adding a yaml file (called .gitlab-ci.yml) to your source repository. This file creates a pipeline, which will then run when a code change is pushed to the repository. Pipelines are made up of a series of stages, and each stage can each contain a number of jobs or scripts. The GitLab Runner agent will then run these jobs.

For an on-premise instance of GitLab, you can install the GitLab runner agent on your own instances, and it supports many different operating systems (thereby creating your own fleet of instances to run your pipelines). But to keep things simple in the following examples, we will use gitlab.com and the default hosted GitLab runner environments provided.

Why use Cloudsmith with Gitlab CI/CD

So why would you want to use Cloudsmith with your Gitlab CI/CD pipelines? Well, there are a few reasons:

Universality – Cloudsmith supports over 20 package formats, so whatever artifacts your project produces, you can find a home for them in a private Cloudsmith repository
Control – Cloudsmith private repositories provide extensive security and access controls, that have been designed to accommodate workflows such as internal development, deployment or even distribution to external customers. The fine-grained permissions system available enables you to craft bespoke access control, and lock down or open up your repository as much or as little as you need.
Automation and Integration – Thanks to the Cloudsmith CLI and also native support for format-specific package managers, a Cloudsmith private repository can fit in seamlessly with other tools in your development or distribution processes. It provides you with a single source of truth across your packages and dependencies.
Performance and Reliability – As a cloud-native platform, Cloudsmith manages the availability and performance of your repositories. You don’t need to worry about managing a fleet of servers, containers or virtual machines. Global performance, backed by our ultra-fast CDN and multi-region infrastructure, ensures that your packages are delivered worldwide. Reliably, quickly and securely.

That’s not all. With features like custom domain support, configurable upstream proxying and caching, configurable edge caching rules, download logs/statistics and more, Cloudsmith aims to provide the best solution for all your package management needs. It really is the ideal tool in a high-velocity CI/CD workflow – precisely the type of workflow that GitLab CI/CD is intended to enable you to create.

Let’s work through an example.

OK, so let’s get started with a worked example. The very first thing you will need is some source code in a GitLab repository that you want to build. For this example, we will build Ruby source code into a Ruby Gem package.

Our project on GitLab has the following structure:

The important thing here, as previously mentioned, is the .gitlab-ci.yml file. This is where we define the GitLab CI/CD pipeline that will run when we push a change to the GitLab repo.

Let’s take a look at the .gitlab-ci.yml file:

image: "ruby:2.5"

variables:
  RUBYGEMS_HOST: "https://ruby.cloudsmith.io/demo/examples-repo"


stages:
  - build
  - push

build:
  stage: build
  script:
    - gem build cloudsmith-ruby-example.gemspec
  artifacts:
    paths:
      - cloudsmith-ruby-example-*

push:
  stage: push
  script:
    - mkdir -p ~/.gem
    - mv $CLOUDSMITH_API_KEY ~/.gem/credentials && chmod 0600 ~/.gem/credentials
    - gem push cloudsmith-ruby-example-1.0.1.gem

The first thing we have specified is the image that will be used when creating the Docker container for the GitLab runner that will execute this pipeline, in this case, Ruby 2.5.

Next, we add an environment variable, RUBYGEMS_HOST. This is where we define the URL of the Cloudsmith package repository that we will push the result of the build to.

We then define two stages in this pipeline, the build stage and the push stage.

The Build Stage

build:
  stage: build
  script:
    - gem build cloudsmith-ruby-example.gemspec
  artifacts:
    paths:
      - cloudsmith-ruby-example-*

This stage is pretty straightforward. We just run the gem build command to build our Ruby source as defined in our cloudsmith-ruby-example.gemspec file. Following this, we have defined an artifact job, as we need to temporarily store the output of the build so that the push stage next can use it.

This is because different stages in a pipeline will run on a new runner instance, so a subsequent stage wouldn’t have the access to the package built in a previous stage. You could also perform any jobs required to build and push within the same stage, and therefore on the same runner to avoid this, but for more complex pipelines you’ll likely have many stages.

The Push Stage

push:
  stage: push
  script:
    - mkdir -p ~/.gem
    - mv $CLOUDSMITH_API_KEY ~/.gem/credentials && chmod 0600 ~/.gem/credentials
    - gem push cloudsmith-ruby-example-1.0.1.gem

The push stage is a little bit more complex. This is due to the fact that we are going to push our package to a private Cloudsmith repo, which requires authentication. The Ruby package manager, gem, allows you to store your authentication credentials (in our case, the Cloudsmith API Key) in a credentials file, located at ~/.gem/credentials – But of course, we don’t want to check this credentials file into our GitLab repository along with our source!

So we can make use of GitLab's ability to add variables to the source code repository. We can create a file variable called CLOUDSMITH_API_KEY, and then as part of the push step, we add a job to move this variable into the required location before we run the gem push command.

You add this file variable in your GitLab repository settings:

Now, when our push step runs, the mkdir and mv jobs will create the required ~/.gem/credentials file (with our Cloudsmith API Key in it), and all without exposing our API Key in any logs on the runner instance, nor checked in with our source code.

The final job in our push step simply runs gem push to upload the package we have built to our private Cloudsmith repo, as Cloudsmith repositories offer full native support for the gem package manager.

Triggering the pipeline

All that is left to do is for us to make a change to our source, and then commit and push the change to our Gitlab repo. Let’s see what happens when we do that.

We make a change, commit it and push:

The Gitlab CI/CD pipeline starts, and the build stage executes:

The build stage runs gem build to build the package and then stores it in GitLab’s temporary artifact storage:

The Push stage then starts to execute once the Build stage completes:

The push stage downloads the package from the temporary artifact storage, creates the required ~/gem/credentials file, and runs gem push which uploads the package to our private Cloudsmith repository:

And that’s it, the pipeline is now complete and reports as "Passed":

If we now go and login to Cloudsmith, and check our "examples-repo" repository, we can see that the Ruby gem we just built is present:

Final Thoughts

Integrating a private Cloudsmith repository with GitLab CI/CD pipeline is easy, whether you are building a package that Cloudsmith provides native tooling support for (like Ruby) or even if you are using the Cloudsmith CLI to push a raw file, or other binary artifacts. You can add your own private Cloudsmith repository with just a couple of lines of configuration, and it just works.

Package management can be complex and difficult, and doubly so if you are trying to manage your own package management solution and infrastructure at the same time. Try it for yourself, and see the productivity and efficiency gains that you can get from a centralized, hosted, secure package management service.

Integrating a Cloudsmith Repository and a Buildkite pipeline

Dan McKinney — Tue, 01 Sep 2020 11:45:02 +0000

Cloudsmith and Buildkite

At Cloudsmith, you will often hear us refer to our mantra of “Automate Everything”. It's a quest that we never deviate from, and we believe that anything that can be automated, should be automated.

With that in mind, we would like to show you how simple it is to integrate a Cloudsmith repository with your Buildkite pipeline, and automate the pushing of your build artifacts into your own private repository for further CI/CD steps or even as a source for your global distribution needs.

What is Buildkite?

Buildkite is a platform for running fast, secure, and scalable continuous integration pipelines on your own infrastructure. That means you can use Buildkite to orchestrate and manage your own fleet of build hosts, and these can be anything from containers, to cloud instances, to bare metal servers.

Why would I want to integrate Cloudsmith with Buildkite?
Well, in short, a continuous integration pipeline is going to have an output, and you are going to need and want to put that output somewhere that is secure, controlled and integrates with all the other tooling that will form the next parts of your DevOps workflow - whether that is continuous deployment to production, distribution to an end-user or customer, or even consumption by another internal team as part of their development process.

This is where Cloudsmith fits in, and this is another phrase you might hear us use quite a bit - Cloudsmith offers a central source of truth for your packages and build artifacts. We provide a global platform that gives you the performance, scalability, security and visibility that you need to control and manage your software assets.

Using Buildkite and Cloudsmith – An Example:

So, you’re let’s assume that you’re new to Buildkite, how do you get started?

Step 1- Install the buildkite-agent

The first thing that you need to do is install the buildkite-agent so that you have a machine (remember, this can be a container, a VM, or even a real server) that will act as your build host. Buildkite provides install instructions for all major platforms and operating systems:

Once you have installed the agent, you will see a new build host in the Buildkite UI:

For this example, I have installed the buildkite-agent on a Debian instance.

Step 2(a) – Create a pipeline

The next thing you need to do is create your pipeline. A Buildkite pipeline is a series of steps that your build host(s) will execute in order to build your assets/artifacts. For this example, we will create a pipeline that will compile a simple C source program, package it into a deb package and then push that deb package to a Cloudsmith repository.

To get started, you give your pipeline a name, and then you specify a source repository for the pipeline. In this case, it is a GitHub repository (although Buildkite will also work with many other platforms as a source):

If using a private GitHub repository, you also need to remember to add an SSH key to your GitHub profile so that the host you have installed the buildkite-agent on can access the source repository.

2(b) – Add your pipeline steps.

Pipeline steps are where you specify the commands or scripts that you need to run in order to build your source / project / application. In Buildkite, you can define these steps via the Buildkite UI or as a pipeline.yaml file. To keep things simple, we will define two steps via the Buildkite UI.

PreBuild Step

In this step, we install the tools we need to build our source and package it into a deb package. We also install the Cloudsmith CLI:

BuildAndPushPackage Step

In this step, we use make to compile our source and then we use fpm to build the deb package. Finally, we use the cloudsmith push command to push the package to our Cloudsmith repository:

The equivalent pipeline.yaml file for these steps would look like:

steps:

  - label: "PreBuild"
  commands:
      - sudo apt update
      - sudo apt-get install ruby ruby-dev rubygems build-essential python-pip -y
      - sudo gem install --no-document fpm
      - pip install cloudsmith-cli 

  - label: "BuildAndPushPackage"
    commands:
      - make
      - fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-buildkite-test .
      - cloudsmith push deb demo/buildkite-demo/debian/buster cloudsmith-buildkite-test_1.0.1_amd64.deb

For more complex build pipelines, you’ll likely have a lot more steps and the advantage of using a pipeline.yaml file is that you can version it and check it in right alongside your source.

One thing to note is that pipeline steps in Buildkite are stateless. As a result, if you have a fleet of agents then each step is not guaranteed to run on the same agent. This means that if a subsequent step need / relies on the output from a previous step, the output from one step will need to be stored and then retrieved. Buildkite provides temporary artifact storage that you can use for this purpose (see here for more details), but to keep things simple in this example we performed the build and push in a single step.

Step 2(c) – Add a Github Webhook.

We want this pipeline to start when we push a new commit of our source to our GitHub repository and for that, we can configure a GitHub Webhook. Conveniently, Buildkite provide a webhook URL that we just need to copy and paste into our GitHub webhook configuration, and then select the event type we wish to fire this webhook on (in this case, all pushes):

That’s it, our pipeline is now built and will be ready to run.

Step 3 – Environment hooks

Buildkite supports several types of hooks that can run on a build host during a pipeline run and a final piece of configuration that we need to do is use an environment hook to configure our environment.

As we are using the Cloudsmith CLI to push the package to our Cloudsmith repository, we need to set up our Cloudsmith API Key as an environment variable. We do this because we don’t want to store a sensitive secret like an API-Key in our pipeline source, or within a build step as a plain text environment variable where it could be exposed in logs. There are other alternative methods of managing secrets in Buildkite, see here for more details.

Our environment hook is pretty simple:

set -euo pipefail

if [[ "$BUILDKITE_PIPELINE_SLUG" == "cloudsmith-buildkite-demo" ]]; then
    export CLOUDSMITH_API_KEY="abcdefghijklmnop1234567890"
    export PATH="$HOME/.local/bin:$PATH"

fi

It does two things:

Sets the CLOUDSMITH_API_KEY environment variable
Adds the location that the Cloudsmith CLI is installed to our PATH Additionally, it only runs when executed by a pipeline with the name “cloudsmith-buildkite-demo”

OK, at this stage our pipeline is built and ready, and the environment on our build host will be set up when the pipeline executes. It’s time to test it out.

Step 4 – Push a change to our source repository.

Now if we make a change our source, commit it and then push the change to our GitHub repository, our build will start:

And once the BuildAndPushPackage step has completed, we can view the output in the logs:

We can see that our deb package was successfully built and pushed to our Cloudsmith repository.

If we now go to the repository on the Cloudsmith website, we can see the built package:

To sum up

Buildkite is a very powerful and flexible CI management platform, and integrating your Cloudsmith repositories with your Buildkite pipelines is as simple as installing the Cloudsmith CLI and then using the Cloudsmith push command in your build steps. We would really encourage you to give it a go yourself, our trial is fully-featured and we are here to help you get started. No question is too big or too small.

Happy Build(kite)ing!

Deploy packages from a Cloudsmith repository with Ansible

Dan McKinney — Tue, 18 Aug 2020 15:31:58 +0000

What is Ansible?

Ansible is an open source continuous configuration automation (CCA) tool. You can use it to automate the management of the configuration of host systems. For example: installing and configuring applications, services, security policies; or to perform a wide variety of other administration and configuration tasks.

You also can use Ansible with a provisioning tool (such as the excellent Terraform, from the awesome folks over at Hashicorp) to automate the entire build and deployment of your infrastructure, and take steps towards true Infrastructure-as-code DevOps practices.

Ansible And Cloudsmith

Ansible is also a perfect partner for your artifacts and assets stored in your private Cloudsmith repositories. As your Cloudsmith repositories can be your single source of truth, with all the access control and permission control that they provide, they give you another secure layer to your infrastructure build and management. By providing you with control of the sources of your packages that you use with your Ansible configuration automation, you insulate yourself further from any upstream changes.

Getting started – The Ansible Playbook.

An Ansible playbook is where you define the series of tasks that you wish to perform to ensure the configuration of your hosts is in the desired state. As an example, we have created an Ansible playbook that will install and configure a Cloudsmith repository for Debian packages, and then install a package from this repository

Here is our example Ansible playbook:

- hosts: DebianHosts
  remote_user: dmckinney

  tasks:
    - name: add Cloudsmith Repository GPG key
      apt_key:
        url: https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/cfg/gpg/gpg.7D4D4CE49534374A.key
        state: present
      become: yes

    - name: add Cloudsmith Repository
      apt_repository:
        repo: 'deb https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/deb/debian buster main'
        state: present
        update_cache: yes
      become: yes

    - name: install Cloudsmith Example package
      apt:
        name: cloudsmith-debian-example
        state: present
        update_cache: yes
      become: yes

Let’s break down what this playbook will do.

Hosts:

This is where we specify which of our hosts these tasks will run against. The hosts and their addresses are defined in a separate Ansible Inventory file and for this example, our inventory file is just a single host.

These hosts can be bare metal servers, cloud instances or virtual machines etc. You can define different groups of hosts within a playbook for different tasks and in this way a single playbook can manage a single machine or several hundred or even thousands of hosts.

Remote User:

This is where you specify the user on the host machine that the tasks will run as

Tasks:

This is where we specify the tasks themselves.

The first task is the “add Cloudsmith Repository GPG key” task. This task uses the apt_key ansible module to retrieve the GPG key from our Cloudsmith Repository and install in on our host system. We specify the URL for the GPG key and as this is a private Cloudsmith repository the URL contains an embedded entitlement token to authenticate for read only access:

- name: add Cloudsmith Repository GPG key
      apt_key:
        url: https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/cfg/gpg/gpg.7D4D4CE49534374A.key
        state: present
      become: yes

We also add the state: present – which means Ansible will install the key if it is not already installed, and if it is already installed it will do nothing. This is important, as it means our task will be idempotent, meaning no matter how many times we run this task the outcome, and therefore the configuration will always be the same. We will see this state: present in all our ansible tasks in this playbook. Finally, as all apt operations on our host system require sudo permissions, we add become: yes which tells ansible to run this task with elevated permissions. Again, we will see this on all tasks in this playbook.

The second task is the “add Cloudsmith Repository” task. This task uses the apt_repository Ansible module to install our Cloudsmith repository on our host system. We specify the URL for the repository, again containing our embedded entitlement token for authentication:

    - name: add Cloudsmith Repository
      apt_repository:
        repo: 'deb https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/deb/debian buster main'
        state: present
        update_cache: yes
      become: yes

The final task is the “install Cloudsmith Example package” task, and this task uses the apt Ansible module to install a deb package. We just need to specify the name of the package we wish to install:

- name: install Cloudsmith Example package
      apt:
        name: cloudsmith-debian-example
        state: present
        update_cache: yes
      become: yes

And that’s it, that’s all we need in this playbook to get this repository set up and our package installed.

Running the playbook.

To run a playbook, we use the ansible-playbook command:

ansible-playbook -i cloudsmith-demo-inventory cloudsmith-demo-playbook -K

the -K flag tells ansible to ask for our sudo password, but you can also store secrets and password encrypted using Ansible Vault – which means you could check the encrypted vault file into your version control system, alongside your playbooks themselves. But to keep things simple for this example, we will just have ansible ask us for the password:

When we run this playbook, the output was as follows:

You can see that the three tasks ran successfully, and report that they made a total of three changes. If we now check what packages we have installed our Debian host, we can see our cloudsmith-debian-example package:

And finally, if we were to run this playbook a second time (or any subsequent number of times), it reports that all tasks are OK, no changes need to be made:

In Closing

While this is a very simple example, we hope that it demonstrates the possibilities and the power of using a configuration automation tool like Ansible in association with a private Cloudsmith repository. It gives you the ability to automate not only common system configuration tasks, but deployment of your own packages, your own custom configurations and even the software build of your entire infrastructure. You can version these builds in your version control systems, track changes and provide an audit-trail-of-truth all the way back to your packages, stored securely in your own private repositories.

We hope you have very happy continuous configuration automation!

Caching and Upstream Proxying For Maven Packages

Dan McKinney — Tue, 11 Aug 2020 14:51:43 +0000

Managing dependencies is a fact of life in modern software development. But at Cloudsmith, we’re focused on ensuring that the process is as painless as possible.

To that end, we’re delighted to announce both upstream proxying and caching for Maven packages. Together they mean simpler, more reliable integration of third party packages into the development process. Better software, faster.

Upstream Proxying

In the simplest terms, upstream proxying means Cloudsmith is now your single point of contact for all Maven packages or dependencies. By proxying upstream dependencies located in Maven Central (the ‘accepted’ central repository for Java packages), Cloudsmith enables your organization, and your build systems, to deal with a single point of contact (us) rather than having to build and manage multiple integrations.

To put that another way, if your build requires any specific dependency, and informs Cloudsmith of that requirement, we will find that dependency upstream and proxy within our platform. This ensures that the dependency is made available to your organization in the same way that every other package, dependency or asset within Cloudsmith is.

This process is completely transparent and controllable to the Cloudsmith customer. You determine ahead of time which repositories you want Cloudsmith to check, and the precedence or priority between them. We also allow our users to specify what to do in the event of upstream failure - whether to retry, and after what period of time.

All in, upstream proxying is one more step in ensuring that dependencies are available when you need them, and available through Cloudsmith: meaning simpler integration and faster builds.

Caching

A step further, if you like, is caching. This involves Cloudsmith locating, downloading and storing dependencies within the Cloudsmith environment. So in other words, rather than act as a go-between Cloudsmith does the whole job.

This can have many benefits, but primary among them are:

Guaranteed great performance. Cloudsmith ensures lightning-fast delivery to any location on the globe - not something you can take for granted when integrating packages from public repositories.
Control: storing packages and dependencies within Cloudsmith gives you a greater ability to scan for vulnerabilities, check licensing implications, and monitor where and how packages are used. These things are not possible (or are at least difficult) when integrating straight from public repositories.

Caching gives you all these advantages, but you can still define when Cloudsmith goes looking for a new version of any given package before bringing it into your own private repository, so you lose no control when it comes to precisely how Cloudsmith and Maven Central interact.

Together, these new features bring Cloudsmith closer than ever to our goal: becoming a single source of truth for all the software assets an organization uses. By providing a single integration for all packages and dependencies we greatly improve the reliability and simplicity of development and deployment processes. And by bringing all assets together within the Cloudsmith environment we allow for greater levels of control and security than ever before.

If you want more information on getting started with proxying and caching with Cloudsmith, check out our documentation here.

Integrating AWS CodeBuild with Cloudsmith Repositories

Dan McKinney — Mon, 29 Jun 2020 16:07:47 +0000

AWS Codebuild and Cloudsmith

At Cloudsmith, we pride ourselves in being the central source of truth for your packages and build artifacts. We provide one global platform to centralize your assets and give you the control and visibility that you need as part of a modern DevOps process.

As part of that effort, we’re delighted to announce that if you’re using AWS CodeBuild as part of your CI/CD workflow and pipelines, we can offer you a simple way to get the output of your build processes into your Cloudsmith repository

What is AWS CodeBuild?

AWS CodeBuild is a fully managed continuous integration service that allows you to automate your source compilation, test running and packaging of your software. There is nothing to install or maintain, and you can be up and running in minutes with your own build pipeline.

Why would I want to integrate Cloudsmith with CodeBuild?
Much like CodeBuild, Cloudsmith is a fully managed repository service. And just like CodeBuild, there is nothing to install or maintain, we take care of all of that for you.

When you automate your build process with CodeBuild, you’ll need somewhere to store the resulting build artifact(s). Of course, you could store these somewhere basic, like an S3 bucket or maybe even an on-premise storage system of some kind. But doing that means you’d miss out on some of the distinct advantages Cloudsmith provides, namely:

Performance / Reliability. Cloudsmith will deliver your artifacts and packages to customers or distributed teams at the lowest latency possible. Configurable global storage regions, configurable edge caching rules and a fast Global CDN mean users never have to wait to consume or use the output of your builds - no matter where they are.
Access Controls / Security. We provide fine-grained access controls, enabling you to grant teams and end users the precise level of access that they need, and no more - including SAML/SSO integration so that you can plug into your existing enterprise auth systems. We also allow you to create read-only access tokens with additional configurable restrictions - if you are vendoring / selling software to end users, for example.
Universality. We support a wide variety of package formats, including raw files, binaries or disk images. So whatever the output of your CodeBuild process is, we can support it - including integration with the native package management tools for most formats. The goal with Cloudsmith is simple: that it just works. This list is far from exhaustive - there are many, many reasons you’d want to use Cloudsmith as a platform to host and store your build artifact(s). We are happy to talk about this at great length, far longer than we can fit in here 😊 Just contact us if you’d like to chat more about it!

The bottom line: in much the same way you probably wouldn’t entertain the overheads involved in standing up your own server farm to run your CI/CD process, you don’t need to do that either for your package repositories.

Using AWS CodeBuild with Cloudsmith - A worked example.
OK, so how do we go about integrating our Cloudsmith repository with our CodeBuild projects? Let's work through an example.

This example will be pretty straightforward - we will build a Debian package from source stored in a GitHub repository and then push the resulting build to our private Cloudsmith repository.

Step 1. Set up our CodeBuild Project

You use the AWS Console to set up a new CodeBuild Project. Just select “Create Build Project” to get started:

Next, you give your project a name, an optional description and any additional optional tags:

You then select a source for your project, in this case it’s a GitHub repository, but other source repository types are supported:

As shown above, I have configured a GitHub personal access token to allow CodeBuild to access my private GitHub repository.

Then we configure the webhook that we want to use to kick start this build. In this example, we will just start the build on any push to the specified GitHub repository:

Now we select the container environment that we need to run this build, and in this case we will use the latest available Ubuntu image, and we will create a new AWS IAM service role at the same time:

And finally, we will specify that this project is going to use a buildspec file:

A buildspec file is where we define the phases of the build and what commands / actions we wish to perform as part of the build process. We will go into this in more detail in the next step!

You then just click “Create Build Project” to finish.

Step 2 - The buildspec file

Our buildspec file contains all the information required to define our build process. We store this buildspec file in our GitHub repository along with the source code that we are building. A buildspec file can be as complex or as simple as your individual project requires, but for this example it’s pretty simple.

You’ll need to set up any environment variables that your build process requires. In our example, we only need a single environment variable; our Cloudsmith API Key. We need this to use the Cloudsmith CLI to push our build artifact to our Cloudsmith repository, and we don’t want to store our API-Key as plain text in the buildspec file as it would then be revealed in any build logs - not good!

To achieve this, we use AWS Secrets Manager to securely store our API-Key, and then we can reference it in our buildspec file as follows:

env:
  secrets-manager:
    CLOUDSMITH_API_KEY:CodeBuild/CloudsmithAPI:CLOUDSMITH_API_KEY

At a minimum you then, you’ll need 3 phases in your build:

An install phase
A build phase
A post-build phase

The Install phase:

The install phase is where we install any prerequisites that our build needs, and this of course will vary depending on what you are building and what your individual requirements are. In our case that means the required tooling to build our Debian package, and of course the correct Python runtime for the Cloudsmith CLI and the Cloudsmith CLI itself:

install:
  runtime-versions:
    python: 3.x
  commands:
    - apt update
    - apt-get install ruby ruby-dev rubygems build-essential -y
    - gem install --no-document fpm
    - pip install cloudsmith-cli

The Build phase:

The build phase is where we execute the commands required to actually build our package. For our Debian package, it’s pretty straightforward:

build:
  commands:
    - make
    - fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-codebuild-test .

The Post-build phase

This is where the magic happens! In the post build phase, we only require a single command - cloudsmith push to upload the result of the build to our Cloudsmith repository

post_build:
  commands:
    - cloudsmith push deb demo/codebuild-demo/debian/buster cloudsmith-codebuild-test_1.0.1_amd64.deb

That’s it, that’s all we need to build this package, and store it in our Cloudsmith repository. It really is that simple. Now of course, your build process itself may be a lot more complex, but to add Cloudsmith to your build process really is just two commands:

1. pip install cloudsmith-cli
2. cloudsmith push FORMAT OWNER/REPOSITORY PACKAGE_FILENAME

We don’t get in the way, you don’t need to totally reconfigure your entire build and we try to make it as easy as possible for you to get up and running!

Step 3 - Running the Build

As we configured this CodeBuild project to start on a push to our GitHub repository, If we just make a change to our source and then commit the change the build will start:

The job has been Submitted, Queued and CloudBuild is now provisioning the container to run the build. Once provisioned, we can tail the logs to watch the progress of the build:

Installing the Cloudsmith CLI:

Pushing build artifact to Cloudsmith:

When the build finishes, our package has been built successfully and uploaded to our Cloudsmith Repository. If we now login to Cloudsmith and view the repository, we can see our package:

To sum up

We hope that this demonstrates how simple it is to integrate Cloudsmith with your existing (or new) AWS CodeBuild projects, and gives you an idea of what you can achieve with your own CI/CD pipelines.

If you’d like to try it yourself, you can sign up for a fully-featured free trial at cloudsmith.com, and we are here to help you every step of the way.

We wish you very happy and successful builds!

Add Cloudsmith to your DevOps Toolkit

Dan McKinney — Wed, 29 Apr 2020 20:19:17 +0000

At Cloudsmith, we are all about automation and we believe that first-class command line tooling makes up a significant part of that. With that in mind, we are continually improving the Cloudsmith Command Line Interface (CLI). This short post will guide you through installing / setting up the CLI and show you some examples of the things you can achieve with it!

Setup

The Cloudsmith CLI is built in Python atop the Cloudsmith API and as well as the API powering the CLI, we have designed the API so that it is fully compatible with Swagger and OpenAPI 2.0, allowing us to generate bindings in more or less any programming language.

If you have Python set up already, then it is super simple to install the CLI from pip using:

pip install cloudsmith-cli

This will install the Cloudsmith CLI itself and any required dependencies.
Once installed you can then configure the CLI it with:

cloudsmith login

This command will prompt you for your Cloudsmith username and password, and it will then retrieve your Cloudsmith API key and set up the required configuration and credential files automatically:

To check that it was all installed and configured correctly, use the command:

cloudsmith whoami

This will return the current user that the CLI is authenticating as:

Let's try some commands!

Once you have the CLI installed and configured you can start to try out some of the operations. To get a list of the supported commands, just do:

cloudsmith

There is a LOT you (or the systems you are automating!) can do with the Cloudsmith CLI, but rather than explore every single command here how about we start with what is sure to be a very common use-case - uploading (pushing) a package.

We can see above that the command will be cloudsmith push, and you can get additional help and information on any command by using the -h flag:

OK, that makes more sense now! So to push an npm package we need the push command, the repository owner (namespace), repository identifier, and the package file name! Easy:

cloudsmith push npm OWNER/REPO PACKAGE_FILE

Right, that works! The CLI has now uploaded the files, created the package and waited for synchronisation to complete (synchronisation is where we extract the metadata and files associated with the package, and make the package available for download).

Remembering that all Cloudsmith repositories are fully multi-format, i.e you can upload any of the package types that we support into the same repository, we can then push a Python package using the command:

cloudsmith push python OWNER/REPO PACKAGE_FILE

And to push a Cargo crate it would be:

cloudsmith push cargo OWNER/REPO PACKAGE_FILE

I'm sure you're sensing a theme here... :-) . Some other formats (such as Debian or Maven) have additional requirements that change the command used, but help is always available using the -h flag.

The point is, it's simple, quick and easy! The whole idea of the Cloudsmith CLI is to enable your automation, not get in your way. It supposed to make your life using Cloudsmith easier!

Some other commands!

So what else can we do with the Cloudsmith CLI. Well, lots of things! Let's start by seeing how we can list the contents of a repository. To do this we use the command:

cloudsmith list pkgs OWNER/REPO

Like so:

Again, it's Easy! It will probably come as no surprise at all at this point that the command to get a list of all repositories for an account is just:
cloudsmith list repos OWNER

And similarily, the command to retreive a list of all the Entitlement Tokens created in a repository is simply:
cloudsmith list ents OWNER/REPO

We've also got the following resources to help you:

CLI Documentation - for additional help.
CLI GitHub Repository - to see how it is built.
CLI Cloudsmith Repository - for pre-releases of the CLI.

Finally, we wish you well on your path to better software and better DevOps. Remember - Be Awesome, Automate Everything.

Vendors Connect. Powering digital distribution with Zapier & Cloudsmith.

Dan McKinney — Tue, 14 Apr 2020 09:09:04 +0000

At Cloudsmith, we are always looking for new ways to make our customers’ lives easier, more efficient and more highly automated. With this in mind we are extremely excited to announce that we have developed an official Cloudsmith Zapier integration!

What is Zapier?

Zapier is an automation platform that allows you to connect your favourite apps and services together (such as Stripe, Slack, Gmail and more). You can connect multiple apps together to automate tasks and you can do all of this without needing developers to build or code custom integrations - the simple interface is entirely point-n-click driven and is extremely easy to set up.

For example - maybe you have packages on Cloudsmith that you would like to sell commercially. Every time you get payment from a customer through your payment provider (Stripe, Shopify, Paypal etc), you could log into Cloudsmith, create an Entitlement Token in the relevant repository and then share that Entitlement Token with your customer. (For more information about Cloudsmith Entitlement Tokens, see here)

OR - you could automate this entire process with the Cloudsmith Zapier integration!

Let’s work through an example!

Get started by logging into your Zapier account and creating a new “Zap”. A Zap is the name for the automated series of steps between your chosen apps that Zapier will perform.

Open the menu on the left and click “Make a Zap:

Step 1

Choose the app that you want to use to “Trigger” this Zap, in this example we are going to use Shopify:

NOTE - you will need to authorize Zapier to access your Shopify account after clicking “Continue”. This process varies depending on which payment provider / platform you are using, but for Shopify you are presented with a window to log into the Shopify platform and authorize Zapier. Other payment providers work similarly, or offer you the option to add an API Key.

Once you have authorized Zapier, you continue to set up the trigger event, in this case - specifying exactly what order conditions will fire this trigger:

You then test the trigger and it will pull in any orders from Shopify (or sample order data if your Shopify account does not have any paid orders yet):

Once you have this all completed, click “Done Editing” to move to the next step.

Step 2

Click the "+" icon under your Trigger step, chose the Cloudsmith app from the dropdown list, and then Chose the “Action” that you wish to perform - in this case it is “Create Entitlement Token”:

You will need to authorize Zapier to access your Cloudsmith account, and you do this by adding your Cloudsmith API Key, which you can retrieve from here

Once you have entered your Cloudsmith API Key you then specify the details of the Entitlement Token that you would like to create. The only required fields are:

Token Name
Repository Owner (namespace)
Repository

You can enter a Token Name manually or you can pull in output from your initial trigger step (such as an email address, customer ID or other identifying information from your Shopify order). For the Repository Owner and Repository, you can select from the dropdown menus. In addition, you can specify a custom string for the Token itself (although if you don't then one is generated for you!)

You can optionally specify any restrictions you would like associated with this Entitlement Token, such as an allowed number of downloads or a valid to/from date.

In the example above, we will use the customer email address from the Shopify order as the Token Name, the order placement date as the Token Valid From date, and we will limit the number of downloads to 10.

You can then click “Continue” to test and review your selections, and then click “Done Editing” to complete this step.

At this point, your Zap is now set up to create an Entitlement Token in your specified Cloudsmith Repository for every complete Shopify order that you receive.

If we now create an order in Shopify, and then check the repository that we specified in the “Create Entitlement Token” action, we can confirm that the Entitlement Token has been created, with the details from the Shopify order:

You may wish to add an additional output step to the Zap that could, for example, email the Entitlement Token to your customer directly or maybe use an additional Zapier app to integrate with your chosen mail or CRM platform - This would enable you to keep a track of Entitlement Tokens so that you can automatically disable them using the “Disable Entitlement Token” action in the Cloudsmith Zapier integration (perhaps if a customer unsubscribes or cancels)

We hope that this demonstrates how simple it is to automate a payment and token creation workflow for a Cloudsmith repository. There are many integrations available on the Zapier platform and we are pleased to be able to offer this official Cloudsmith integration as an option for our customers that use the Cloudsmith platform for distribution. As always, if you need any help, guidance or additional information please just contact us and we will be happy to work through it with you!

Happy order processing! :-)

Cloudsmith + CocoaPods

Dan McKinney — Tue, 24 Mar 2020 09:45:16 +0000

We are proud to announce the availability of Cloudsmith for CocoaPods.

One of the key attributes of the Cloudsmith platform is universality. We think it’s important that you can use Cloudsmith to manage packages and dependencies no matter what format they are in.

Taking this approach means distributed development teams have a single source of truth for all their software assets, and can store, share and secure them all in exactly the same way and in the same place.

It also means that other processes within the DevOps stack only need to speak to one partner rather than many, with a consequent reduced risk of things going wrong.

Result: faster delivery pipelines, less broken builds, happy customers.

Introducing CocoaPods

To ensure that Cloudsmith is and remains a ‘universal’ package management solution, we continue to add formats and extend the scope of the platform. Hence the launch today of our support for CocoaPods.

CocoaPods is the dependency manager for Swift or ObjectiveC packages (or Pods in this context) used within Cocoa projects. And Cocoa is the framework within which MacOS and iOS applications are developed.

As a result, Cocoa and CocoaPods are popular. In 2019 Swift was the 6th most loved programming language in the Stack Overflow developer survey and the 10th most active language on GitHub.

It is also commonly used alongside other formats and languages (no surprise for a framework focused solely on MacOS and iOS) so being just one of many popular formats supported by Cloudsmith really matters.

It’s easy to get control of your Pods. In fact it only takes a minute to set up a private repository with Cloudsmith. The short demo video below should help you get started:

Then your entire team can enjoy cloud-native, secure and universal package management - with none of the hassles associated with managing an on-premises solutions.

Taking the Cloudsmith approach allows development teams to:

Develop Pods internally and share them privately to other teams
Distribute and deploy your own Pods in a pipeline at your org
Distribute Pods as commercial software
Make modifications to public Pods, without republishing publicly
Mirror public Pods and isolate from uncontrolled upstream issues
Capture the exact state of your dependencies at a particular version/release
Control (whitelist/blacklist) the exact Pods allowed for your org
Keep track of the exact versions/releases of the Pods you have/use

In short: deliver all the benefits of Cloudsmith that are already enjoyed by development teams all over the world today.

If you want to join them, sign up today and start hosting and distributing your Pods with Cloudsmith!