Forem: Alan Carson

Cloudsmith + Bitbucket

Alan Carson — Tue, 13 Aug 2019 15:57:13 +0000

At Cloudsmith, our mission is to make it as easy and straightforward as possible to get your packages into our system from where you can manage and distribute them to your heart's content.

To this end, we recently spent some time with Atlassian's Bitbucket team at AppWeek in Berlin to build a Bitbucket Pipe that makes it simple for Bitbucket users to easily turn their source code into distributable packages and deploy to Cloudsmith in just a few lines of YAML.

How it works

Pipes build on top of Bitbucket's excellent Pipelines functionality which allows users to automate their build and deploy processes, whatever form they may take.

Bitbucket Pipes are, at their simplest, the combination of a Docker container and some YAML-formatted configuration. Pipes are designed to remove what would otherwise be repeated or verbose configuration from a user's pipeline. Pipes can be shared across repositories or organisations and make many common tasks almost trivial.

Using Pipelines

The Cloudsmith pipe is included in Bitbucket's collection of officially maintained pipes and is available by default for any user that wishes to include it in their pipeline.

To use the pipe you'll first need an application or library that can be packaged into one of the formats that Cloudsmith support. You can see examples of Python and Javascript libraries in Cloudsmith's Bitbucket account.

For the purposes of this example we'll assume you're using the Python example from the link above.

First, ensure that you can package your code using a Bitbucket pipeline by editing bitbucket-pipelines.yml in your repository. You can do so either via Bitbucket's web UI or using your normal editor and Git workflow.

For the example library, a basic pipeline that builds a package when a new tag is created looks like so:

image:
  name: atlassian/default-image:2

build: &build
  step:
    name: Build Python Package
    image: python:3.7
    script:
    - python setup.py sdist
    artifacts:
    - dist/**

pipelines:
  tags:
    release-*:
    - <<: *build

This pipeline will build a new package each time you push an appropriate tag you your Bitbucket repository. The built package is then discarded as we have not yet added further instructions to tell Bitbucket what to do with it.

Configuring the Pipe

Next we'll add publish the package to Cloudsmith using our official pipe. We'll need to provide an API Key that the pipe can use to authenticate with Cloudsmith. You can find the official documentation for pipeline variables in the Bitbucket documentation.

The API Key should be added as a "secure" variable (so it doesn't leak into logs) with the name CLOUDSMITH_API_KEY.

Once authentication is configured, you can add the pipe configuration to your pipeline.

If using the web UI to configure your pipeline, you can select the Cloudsmith pipe right from your browser, from within the list of supported pipes as in the image below:

If using your local editor, you can follow the instructions in the official README on Bitbucket.

Once added, your pipeline YAML should look something like so:

image:
  name: atlassian/default-image:2

build: &build
  step:
    name: Build Python Package
    image: python:3.7
    script:
    - python setup.py sdist
    artifacts:
    - dist/**

publish: &publish
  step:
    name: Publish Python Package
    script:
    - pipe: cloudsmith-io/publish:0.1.1
      variables:
        CLOUDSMITH_REPOSITORY: 'cloudsmith/examples'
        CLOUDSMITH_API_KEY: $CLOUDSMITH_API_KEY
        PACKAGE_FORMAT: 'python'
        PACKAGE_PATH: 'dist/*.tar.gz'

pipelines:
  tags:
    release-*:
    - <<: *build
    - <<: *publish

This configuration instructs the pipe to push artifacts to the cloudsmith/examples repository, with the API key we stored earlier. We're uploading a python package which is located at dist/*.tar.gz.

Using the Pipe

And that's it! Your pipe is now configured and ready to run. You should be able to push a new tag to Bitbucket and see your pipe run.

Bitbucket's pipeline UI provides a great overview of the status of your jobs:

Once pushed, you should see your shiny new package in the Cloudsmith UI, ready for download and install using your preferred tools:

Conclusion

Cloudsmith's Bitbucket pipe provides the easiest and simplest way for Bitbucket users to push their assets to Cloudsmith. Our official pipe is maintained by the Cloudsmith team and you can be sure it'll be kept up to date with all changes and features as we release them.

As always, if you have any questions about the pipe (or anything else), we're available at support@cloudsmith.io.

Cloudsmith + Rust = The World's First Private Cargo Registry Service

Alan Carson — Fri, 17 May 2019 14:57:12 +0000

What is Rust? What is Cargo? What are Crates?

Rust lang, or Rust, is a systems programming language, designed at Mozilla research, and intended to be a safer alternative to existing systems programming languages like C and C++, but still just as fast. Rust was originally started as a personal side-project by Mozilla engineer Graydon Hoare, but the company quickly and rightfully recognised its promise and began sponsorship of the project in 2009.

Rust quickly became a critical part of Mozilla's efforts to improve the performance and reliability of their browser technology as part of the Servo project. Since then, Rust has only grown in popularity and appeal, year on year, and for the last four years has achieved the coveted top spot for "most loved programming language" on Stack Overflow's Developer Survey (also a great read).

Whilst Rust is maybe best known as a low-level systems language, the same properties that make it appealing for working close to the metal also make it appealing for many other use cases. Rust's memory-safety, ownership semantics, performance, and comprehensive type system (among others) combine to make Rust a very compelling choice in many fields of engineering.

Whether you're writing a device driver for an embedded System-on-Chip (SoC), an operating system, a layout engine for a web browser, a web application, a package management service (heh), or a library for use in the WebAssembly ecosystem, Rust probably has you covered.

Early on, the Rust community realised that for a modern programming language to be successful, it needs modern tooling to accompany it. To this end, various teams in the Rust community have concentrated on tooling, such that Rust is now widely regarded as having best-in-class tools in areas like build and package management; which is something that we (Cloudsmith) recognise, respect and adore.

Rust therefore ships with its own package manager called Cargo. Cargo is an all-in-one frontend for building, packaging and configuring your Rust projects. It can build and run your code (providing a developer-friendly interface to rustc) and manages resolving and fetching any required dependencies (also known as "crates" in the Rust world).

Cargo can install dependencies from a remote registry into your local project for building, using a command like cargo install library. Until April of 2019 the only choice you really had was the official, public registry at crates.io. Crates.io is maintained by the Rust team and is the default public registry for the Rust ecosystem. With the release of Rust 1.34, Cargo now officially supports the use of registries other than crates.io, making it possible to run an alternative registry for the hosting of private or internal crates.

There are several equally fantastic reasons for running/using your own registry:

To develop Cargo packages internally and share them privately to other teams.
To distribute and deploy your own Cargo packages in a pipeline at your org.
To distribute Cargo packages as vendored software (i.e. maybe commercially).
To make modifications to public Cargo packages, without republishing publicly.
To mirror public Cargo packages, to isolate from uncontrolled registry events.
To capture the exact state of your dependencies at a particular version/release.
To control (whitelist/blacklist) the exact Cargo packages allowed for your org.
To keep track of the exact versions/releases of Cargo packages you have/use.

As such, if you're interested in the possibilities that hosting your own private or internal crates brings, then this is incredibly good news for you: Cloudsmith are proud to provide the World's first commercially available public and private Cargo registry hosting, with ultra-fast and secure delivery of your Rust packages, alongside all of the usual Enterprise-grade features that we provide.

Setting up a private Cargo registry at Cloudsmith

Getting started with Cloudsmith and Cargo couldn't be simpler. First, you'll need a Cloudsmith account and a repository to which you can upload crates. Secondly, you'll need to ensure you're running at least version 1.34 (or later) of Rust.

If you need to install Rust you can use rustup, the Rust toolchain installer. Rust is available on most commonly used development platforms.

You can check your currently installed Rust and Cargo versions like so:

$ rustc --version
rustc 1.34.1 (fc50f328b 2019-04-24)
$ cargo --version
cargo 1.34.0 (6789d8a0a 2019-04-01)

If you see something like the above, you're ready to go!

Publishing your Cargo crates to Cloudsmith

For the purposes of this demonstration, we'll use the minimal crate created by cargo init. You can initialise your new Cargo crate like so:

$ cargo init my-crate
     Created binary (application) package

If you take a look inside the new crate you'll see a few files:

$ tree my-crate/
my-crate/
├── Cargo.toml
└── src
    └── main.rs

1 directory, 2 files

main.rs contains a simple "Hello World" program, and Cargo.toml contains all configuration and setup data for your crate. For our purposes we can leave both files as-is.

We can use cargo to build our crate and prepare it for upload to Cloudsmith:

$ cd my-crate/
$ cargo package --allow-dirty
warning: manifest has no description, license, license-file, documentation, homepage or repository.
See <http://doc.crates.io/manifest.html#package-metadata> for more info.
   Packaging my-crate v0.1.0 (/Users/cloudsmith/my-crate)
   Verifying my-crate v0.1.0 (/Users/cloudsmith/my-crate)
   Compiling my-crate v0.1.0 (/Users/cloudsmith/my-crate/target/package/my-crate-0.1.0)
    Finished dev [unoptimized + debuginfo] target(s) in 1.45s

NOTE: We use --allow-dirty in the cargo package command as we haven't committed the new crate to git. Under normal circumstances you'll just call cargo package without the extra flag.

Cargo, by default, spits out your packaged crate under the target/package/ directory.

$ ls -l target/package/
total 4
drwxr-xr-x 6 cloudsmith cloudsmith 192 Apr 30 16:09 my-crate-0.1.0
-rw-r--r-- 1 cloudsmith cloudsmith 673 Apr 30 16:09 my-crate-0.1.0.crate

Once built you can push the crate to Cloudsmith using the Cloudsmith CLI. We don't yet support pushing crates with cargo publish, but we'll be adding support in a future release (if you want it, please let us know). For now, use the CLI.

First, ensure you've installed the Cloudsmith CLI and configured authentication:

$ pip install cloudsmith-cli
$ export CLOUDSMITH_API_KEY=xxxxx

Then, use the CLI to push the crate, which operates like cargo publish but is specific to Cloudsmith:

$ cloudsmith push cargo my-org/my-repo target/package/my-crate-0.1.0.crate

Checking cargo package upload parameters ... OK
Checking my-crate-0.1.0.crate file upload parameters ... OK
Requesting file upload for my-crate-0.1.0.crate ... OK
Creating a new cargo package ... OK
Created: my-org/my-repo/my-crate-010crate (ffb4n20QxSYM)

Synchronising my-crate-010crate:  [####################################]  100%  Sync Completed / Fully Synchronised
Package synchronised successfully!

Your repository should now contain the uploaded crates. An example of which is our official Cloudsmith examples repository (this is where we upload our examples from), as shown below:

Once pushed, your crate is ready for installation with Cargo.

Installing your Cargo crates from Cloudsmith

First, we need to tell Cargo how to access the registry. To do so you need to add the following entry in your .cargo/configfile (replace TOKEN with an entitlement token, which is a type of read-only authentication token that we support):

[registries]
cloudsmith = { index = "https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git" }

And then install the crate:

$ cargo install my-crate --registry cloudsmith
    Updating `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git` index
  Downloaded my-crate v0.1.0 (registry `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git`)
  Downloaded 1 crates (673 B) in 0.54s
  Installing my-crate v0.1.0 (registry `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git`)
   Compiling my-crate v0.1.0 (registry `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git`)
    Finished release [optimized] target(s) in 2.60s
  Installing /Users/cloudsmith/.cargo/bin/my-crate

Assuming ~/.cargo/bin/my-crate is on your $PATH then you should be able to run the installed program:

$ my-crate
Hello, world!

It really is that simple, and now you've sourced a crate privately from your very own private Cargo registry on Cloudsmith. Well done!

Conclusion

Rust is an exciting new programming language, with many interesting properties that make it perfect for use in a wide range of scenarios. Cargo provides a robust suite of tools for installing, configuring and managing dependencies for your Rust projects.

Cloudsmith provides fully featured Cargo registries on all plans, flexible enough for use whether you’re hosting public crates for a public or open source project, or private crates for your company’s internal needs. We're extremely proud to be able to support the Rust ecosystem with this first (non-official) implementation of public and private Cargo registries.

You can find further, context-specific information, including detailed setup and integration instructions inside each Cloudsmith repository. You can see an example of this documentation in our public examples repository.

Why wait? Get your public and private Cargo registry hosting at Cloudsmith now.

Cloudsmith + Helm

Alan Carson — Wed, 17 Apr 2019 14:31:26 +0000

TL;DR: Cloudsmith provides public and private Helm repository hosting, with ultra-fast and secure delivery of your Helm charts; for the ultimate in integration with the Kubernetes (k8s) ecosystem.

Helm - What Is It?

Kubernetes has quickly become one of the most popular container orchestration platforms used by development and operations teams all over the world. It was open-sourced by Google in 2014, built based on years of learning deploying containers at scale inside the search giant.

Despite Kubernetes’ rapid rise to popularity, deploying and managing applications can be a complex and daunting task, often requiring the operator to write and configure multiple inter-dependent configuration files for various parts of the systems being deployed. As our applications become more complex, more distributed, the difficulty of managing them grows even further.

Originally built by Deis (later acquired by Microsoft), Helm is a package manager for Kubernetes that allows development and operations teams to easily manage and deploy these increasingly complex cloud native applications to their Kubernetes clusters. Helm allows you to manage applications on your Kubernetes cluster in much the same way as you’d manage applications on your Linux server with apt or yum.

Helm works by packaging up a set of YAML definitions along with the necessary configuration to quickly stand up all components of an application in a repeatable way. A single chart can be as simple or complex as necessary, deploying anything from a single container to a full distributed application. Helm combines these application definitions with user-provided configuration to allow simple overriding of configuration where needed, allowing users to concentrate on shipping software and not on the nitty-gritty of configuring every application they need to run.

Helm packages are known as “Charts” and are stored in a “Chart Repository”. By default, Helm comes bundled with the “stable” chart repository, hosted for free by Google. Most public charts are hosted here, mostly provided by vendors packaging their own software for use by others.

Getting Started With Helm

The main requirement for using Helm is a working Kubernetes installation. Setup of Kubernetes is outside the scope of this post, but you can get started locally an with installation of minikube.

Once installed you can initialise Helm on your cluster:

$ helm init

This will install Tiller, Helm’s server-side component, responsible for interacting with the Kubernetes API and managing state.

You can then run other Helm commands as needed. To view all available charts you can use the following command:

$ helm search
NAME                                            CHART VERSION   APP VERSION                     DESCRIPTION
stable/acs-engine-autoscaler                    2.2.2           2.1.1                           DEPRECATED Scales worker nodes within agent pools
stable/aerospike                                0.2.3           v4.5.0.5                        A Helm chart for Aerospike in Kubernetes
stable/airflow                                  2.4.0           1.10.0                          Airflow is a platform to programmatically author, schedul...
stable/ambassador                               2.0.1           0.52.0                          A Helm chart for Datawire Ambassador
stable/anchore-engine                           0.12.1          0.3.3                           Anchore container analysis and policy evaluation engine s...
stable/apm-server                               1.0.0           6.6.2                           The server receives data from the Elastic APM agents and ...
stable/ark                                      4.1.0           0.10.1                          A Helm chart for ark
...

You should see a full listing of the built-in “stable” repository which contains all supported public charts. You can install any of the charts from the helm CLI also:

$ helm install stable/dokuwiki

This instructs Tiller to pull the packaged chart from the stable repository, render all the templates contained within, and apply those to the running cluster. You can inspect the status of the application with regular Kubernetes CLI tooling:

$ kubectl get pods
NAME                                                 READY     STATUS    RESTARTS   AGE
lopsided-cheetah-dokuwiki-85c96f777d-gskjn           1/1       Running   0          13d

$ kubectl get deployments
NAME                                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
lopsided-cheetah-dokuwiki            1         1         1            1           13d

Or check the status of the chart installation itself using the helm CLI:

$ helm list
NAME                    REVISION        UPDATED                         STATUS          CHART                           APP VERSION             NAMESPACE
lopsided-cheetah        1               Mon Mar 25 17:16:56 2019        DEPLOYED        dokuwiki-4.2.0                  0.20180422.201901061035 default

Managing Helm With Cloudsmith

If using only public charts to install third-party applications, then the above instructions are enough to get started in most cases. But if you want to install software for which there are no public charts, whether your own or third-party, you can create your own chart and host it using Cloudsmith.

If you’re creating your own chart, you can use one of Helm’s starter templates to quickly scaffold out the directory for you:

$ helm create my-chart-name

This will create a my-chart-name folder containing all the files and folders necessary to build your chart. You can adjust the templates as needed for your application, The Chart Template Developer’s Guide is a really useful resource when building your own chart.

Once you’ve finished adjusting your templates, you can create a Chart package with the CLI:

$ helm package my-chart-name

Successfully packaged chart and saved it to: /home/me/my-chart-name-1.0.tgz
Once built you can push the chart to Cloudsmith. First, ensure you’ve installed the Cloudsmith CLI and configured authentication:

$ pip install cloudsmith-cli
$ export CLOUDSMITH_API_KEY=xxxxx

Then, use the Cloudsmith CLI to push the package:

$ cloudsmith push helm my-org/my-repo my-chart-name-1.0.tgz
Checking helm package upload parameters ... OK
Checking my-chart-name-1.0.tgz file upload parameters ... OK
Requesting file upload for my-chart-name-1.0.tgz ... OK
Creating a new helm package ... OK
Created: my-org/my-repo/my-chart-name-10tgz (...)

Synchronising my-chart-name-10tgz:  [####################################]  100%  Sync Completed / Fully Synchronised
Package synchronised successfully!

Once pushed, you can configure Helm to install packages from your Cloudsmith repository:

$ helm repo add cloudsmith 'https://dl.cloudsmith.io/TOKEN/my-org/my-repo/helm/charts/'
$ helm repo update

And then install your application to Kubernetes:

$ helm install cloudsmith/my-chart-name

After pushing Helm charts to Cloudsmith, you'll see something like this in your repository.

Conclusion

Helm is a powerful tool to help manage the ever-growing complexity of deploying and running cloud native applications on Kubernetes. Like other package managers Helm provides a robust suite of tools for installing, configuring and managing applications on its target platform, Kubernetes.

Cloudsmith provides fully featured Helm repositories on all plans, flexible enough for use whether you’re hosting public charts for an open source project, or private charts for your company’s internal applications.

Why wait? Get your public and private Helm repository hosting at Cloudsmith now.