Forem: Edmundo Sanchez

How to install a package from a private repository on Poetry for CI

Edmundo Sanchez — Fri, 30 Jun 2023 21:22:28 +0000

Why?

One of the methods to install a package via poetry is to use deploy tokens directly on the package URL, and that used to work on poetry too before version 1.0.3.

What changed? The git URL parsing has changed significant to be more consistent.

Are they planning on fixing this within poetry? No.
They don't want to expose credentials on the pyproject.toml file, which makes sense.

source: This issue on gitlab

How to do this then?

There are multiple ways, using git credential helpers, setting credentials globally for a particular URL.

The one thing that was easy enough to me was to tell git to replace the URL when calling gitlab via HTTPS.

After obtaining your deploy tokens from gitlab, this is how you tell git to use them

deploy_user=YOUR_DEPLOY_TOKEN_USER
deploy_token=YOU_DEPLOY_TOKEN
git config --global url."https://$deploy_user:$deploy_token@gitlab.com".insteadOf https://gitlab.com

Note this is intended to run in a docker container via CI, so it does not matter that much if you rewrite that URL, I do not recommend you to do this on your local machine, for that use SSH keys.

Which takes me to the next section

Poetry install Locally vs Gitlab CI

I do use SSH to clone mu libraries from gitlab, and the deploy token requires cloning via HTTPS, so now what?

I replace the urls on my before_script, hacky but effective.

For example, let's say you have this library on your pyproject.toml file

testrail-api-wrapper = {git = "git@gitlab.com:mundo03/sample_library.git", rev = "main"}

You can rewrite it using sed like this:

export SSH_URL=git@gitlab\.com:mundo03\/sample_library.git
export TOKEN_URL=https:\\/\\/gitlab.com\\/mundo03\/sample_library.git
sed -i "s|$SSH_URL|$TOKEN_URL|g" pyproject.toml

your .gitlab-ci.yml file would look like this, given you want to install you project and run pytest

stages:
  - test

.build:
  before_script:
      - pip install poetry
      - poetry config virtualenvs.in-project false #Ignore venv
      - export SSH_URL=git@gitlab\.com:mundo03\/sample_library.git
      - export TOKEN_URL=https:\\/\\/gitlab.com\\/mundo03\/sample_library.git
      - sed -i "s|$SSH_URL|$TOKEN_URL|g" pyproject.toml
      - git config --global url."https://$deploy_user:$deploy_token@gitlab.com".insteadOf https://gitlab.com
      - poetry update
      - poetry install -v

pytest_311:
  image: python:3.11
  extends: .build
  stage: test
  script:
    - poetry run pytest --cov
  retry: 2
  coverage: '/TOTAL.*\s+(\d+%)$/'

Conclusion

Poetry did me dirty and I had to find another way to install dependencies in CI that come from private git repositories.

The Ultimate Guide You Need To Publish Your Python Package In Just 3 Easy Steps

Edmundo Sanchez — Wed, 29 Dec 2021 20:34:04 +0000

A reply to The Ultimate Guide You Need To Publish Your Python Package In Just 9 Easy Steps, using poetry instead of a setup.py script.

I will focus on the steps only, you can read a better and fuller article here.

Install all the things

I am assuming you have python and poetry installed, if you do not have it, install it.

1. Create a package

$ poetry new sample-package
Created package sample-package in sample-package

This will create the project layout:

sample-package/
├── sample-package/
│   └── __init__.py
├── tests/
│   ├── __init__.py
│   └── test_sample-package.py
├── pyproject.toml
└── README.rst

2. Describe your package in `pyproject.toml`

Here is where you say what your package does and define dependencies.

For example:

[tool.poetry]
name = "sample-package"
version = "0.0.1"
description = "Sample Description for your sample file"
authors = ["John Doe <john@doe.com>"]

[tool.poetry.dependencies]
# Updated Python version
python = "^3.6"

[tool.poetry.dev-dependencies]
pytest = "^3.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

X. Write Your Package

There isn't a reason to publish an empty package, make sure it does something.

3. Publish it!

After your package works, all test pass, etc., you want to make it available to the world, lets build:

$ poetry build

Make sure you have pypi credentials

Get your token from pypi, add it

poetry config pypi-token.pypi your-api-token

Testing Publish via TestPyPi

This is optional, highly recommended.

Add Test PyPi as a package repository

$ poetry config repositories.testpypi https://test.pypi.org/legacy/

Publish it to Test PyPi:

$ poetry publish -r testpypi

Install it from Test PyPi

pip install --index-url https://test.pypi.org/simple/ flake8-markdown

Really Publish it

If all is good, and ready to make it public:

poetry publish

NOTES

If you are new to Poetry, read the docs
You need an account in Test PyPi and PyPi, register if you haven't

Gitlab's Deploy Keys and Deploy Tokens for CI/CD

Edmundo Sanchez — Mon, 19 Jul 2021 03:13:46 +0000

UPDATE: Poetry no longer supports deploy tokens on the package URLs due to a change on how they parse git urls, go read this other post for a solution

Scenario: Need to install a requirement that is only available on a private repository, from my CI pipeline.

There are two ways (I found) to do this, and the one you use depends on what you need to do:

I have a simple library with no dependencies or with only public dependencies
I have a dependency that depends on other private repositories

For context and illustration I used gitlab, python and poetry.

Simple Dependencies: Use a Deploy Token

A deploy token allows you to access a repository without your user name and password or ssh keys, and it is setup per repository or group.

How to:

Add the dependency on your requirements file as you would normally
Setup the deploy token on the private repository you want to access
Store the user and token on CI Variables on the repository you will access the private repository from
On the CI Script, fill in the deploy token using the CI variables.

Example:

Dependency on pyproject.toml

[tool.poetry.dependencies]
python = "^3.9"
private-project = {git = "ssh://git@gitlab.com/mundo03/sample-private-repo.git", rev = "main"}

Note the SSH URL is kept so you don;t have to use a Personal Access Token every time you install your dependencies locally.

gitlab-ci.yml

image: python:3.9-buster

poetry_token:
  script:
    # Install Poetry
    - pip install poetry
    # setup string replacement
    - export SSH_URL=ssh:\\/\\/git@gitlab.com\\/mundo03
    - export TOKEN_URL=https:\\/\\/$deploy_user:$deploy_token@gitlab.com\\/mundo03
    - sed -i "s/$SSH_URL/$TOKEN_URL/g" pyproject.toml
    # Install repo
    - poetry install
    # Use Clonned Repo
    - poetry run print_something "LOOK AT ME!!"

Notice sed is being used to replace the SSH URL with an HTTP URL that has the Deploy Token in it.

Complex Dependencies: Use a deploy key

Deploy keys also give you access to a repository with out your user name and password or SSH key, it is an SSH key you generate and can be used on multiple repositories, on multiple groups if necessary.

There are also Public deploy keys setup at Gitlab Instance level and can be given access to any of the projects in that instance by an admin.

How to:

Add the dependency on your requirements file as you would normally
Generate an ssh key
Deploy the generated public key on any repo you need
Set up ssh-agent on the CI script

Example

Dependency on pyproject.toml

[tool.poetry.dependencies]
python = "^3.9"
private-project = {git = "ssh://git@gitlab.com/mundo03/sample-private-repo.git", rev = "main"}

gitlab-ci.yml

image: python:3.9-buster

poetry_key:
  script:
      # Install OS dependencies
    - apt-get update -y -qq && apt-get install -y -qq make openssh-client python3-pip
    # Start ssh-agent
    - eval $(ssh-agent -s)
    # Add Private key to SSH Agent
    - ssh-add <(echo "$SSH_PK")
    # Add gitlab as known host
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
    # Install Poetry
    - pip install poetry
    # Install repo
    - poetry install
    # Use Clonned Repo
    - poetry run print_something "LOOK AT ME!!"

Enter the Vault

CI Variables are not the best way available to store a secret, for that gitlab has Secrets, which is meant to store things like SSH keys, Passwords Etc.

Read up: Secrets
Spoilers: You need a Hashicorp's Vault account

TL;DR

I have a dependency in my project that is hosted on a private repository, I need to install it during my CI/CD, I have two options:

Deploy Token: Are setup by project or group, I can add the token info in a CI variable and manipulate my requirements file to insert the token.
Deploy Keys: These are SSH keys that can be added to any project I need or in any group. I need to setup the SSH-Agent on the CI/CD script, the Public Key goes in the repository settings, the Private key goes in a CI variable or Vault, and use the ssh URL when requiring the dependency.

Have a look at the Complete Sample here:
Use a Deploy Token
Use a Deploy Key

Selenium 4 and chrome driver, take full page screenshots

Edmundo Sanchez — Wed, 10 Mar 2021 07:52:51 +0000

I spent some time playing with Selenium 4 trying to get a full page screenshot, this is what I found.

We will use the devtools protocol of Chrome Driver
this is done by driver.execute_cdp_cmd(command, parameters)
The command is Page.captureScreenshot
Parameters are documented here
the tricky param is clip

# I am assuming we already have a driver object.

# We need the dimensions of the content
page_rect = driver.execute_cdp_cmd('Page.getLayoutMetrics', {})

# parameters needed for ful page screenshot
# note we are setting the width and height of the viewport to screenshot, same as the site's content size
screenshot_config = {'captureBeyondViewport': True,
                             'fromSurface': True,
                             'clip': {'width': page_rect['contentSize']['width'],
                                      'height': page_rect['contentSize']['height'],
                                      'x': 0,
                                      'y': 0,
                                      'scale': 1},
                             }
# Dictionary with 1 key: data
base_64_png = driver.execute_cdp_cmd('Page.captureScreenshot', screenshot_config)

# Write img to file
with open("imageToSave.png", "wb") as fh:
    fh.write(base64.urlsafe_b64decode(base_64_png['data')])

Why am I writing this?

There is no good documentation related to the DevTools protocol and its integration to selenium is still in the works in Selenium 4 at the time I wrote this.

I Hope this helps any lost soul.

Ruby, config this

Edmundo Sanchez — Fri, 05 Mar 2021 21:50:58 +0000

Config_this

Have you ever wondered where is the right way to store sensitive information like passwords for your ruby scripts and gems?

I have, doing some research I found a few ways to do that, like the Simplest configuration block implementation for a ruby gem ever or Creating a configurable Ruby gem.

I also found using a file is the best way to store sensitive information but there are use cases to have Environmental variables or just hard-coding values.

However following that for every script I write is a bit cumbersome for me, I like easy things and automation so as a learning exercise I created a gem called config_this.

With Config_this you can load your configuration parameters from a YAML or json file, a block or a hash.

Installation

Add this line to your application's Gemfile:

gem 'config_this'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install config_this

Usage

The Config Module is the main interface for this gem, as you know you can not instantiate a Module so be careful with your values

require "config"

Get the Configuration object

This will get the latest object you created.

Config.get
 => #<Config::Configuration:0x000055d538ef2dc0>

Setting the configuration Object

If you plan on having only one configuration object you don't need to assign it, it will always return

Config.set(id:42, name: "Arthur Dent")
 => #<Config::Configuration:0x000055d538f3c9c0 @id=42, @name="Arthur Dent">

If you do plan to have multiple configurations, handle them properly

Config.set(id:42, name: "Arthur Dent")
 => #<Config::Configuration:0x000055d538f3c9c0 @id=42, @name="Arthur Dent">
number1 = Config.set(id:1, name: "Stefán Karl Stefánsson")
 => #<Config::Configuration:0x000055d53866ea00 @id=1, @name="Stefán Karl Stefánsson">

Setting it With a Hash

As you noticed, this is how you set it with a hash:

Config.set(id:42, name: "Arthur Dent")
 => #<Config::Configuration:0x000055d538f3c9c0 @id=42, @name="Arthur Dent">

Setting it with a YAML file

Config.yaml_file("/absolute/path/file.yaml")
 => #<Config::Configuration:0x000055d538ed7d90 @id=42, @name="Arthur Dent", @weapon="Towel">

Setting it with a JSON file

Config.json_file("/absolute/path/file.json")
 => #<Config::Configuration:0x000055d538a8f210 @id=42, @name="Arthur Dent", @weapon="Towel">

Setting it with a Environmental Variables

Be sure the variables are set and accesible

ENV["id"] = "42"
ENV["name"] = "Arthur Dent"
Config.env_variables("id","name")
 => #<Config::Configuration:0x000055d538ef8b58 @id="42", @name="Arthur Dent">

Setting it with a Block

To do this, you have to use the Config::Configuration class directly

config = Config::Configuration.new do |c|
  c.id = 1
  c.name = "Stefán Karl Stefánsson"
end
p config
 => #<Config::Configuration:0x00005630c8eda630 @id=1, @name="Stefán Karl Stefánsson">

Using the Configuration

Config.get
 => #<Config::Configuration:0x000055d538f3c9c0 @id=42, @name="Arthur Dent">
Config.get.id
 => 42
config = Config.get
config.id
 => 42
config.name
 => "Arthur Dent"

You can also just set new values to the Configuration object

config = Config.get
config.location = "Earth"
p config
 => #<Config::Configuration:0x00005630c87f0e00 @id=42, @name="Arthur Dent", @location="Earth">

Forem: Edmundo Sanchez

How to install a package from a private repository on Poetry for CI

Why?

How to do this then?

Poetry install Locally vs Gitlab CI

Conclusion

The Ultimate Guide You Need To Publish Your Python Package In Just 3 Easy Steps

Install all the things

1. Create a package

2. Describe your package in pyproject.toml

X. Write Your Package

3. Publish it!

Make sure you have pypi credentials

Testing Publish via TestPyPi

Really Publish it

NOTES

Gitlab's Deploy Keys and Deploy Tokens for CI/CD

Simple Dependencies: Use a Deploy Token

How to:

Example:

Complex Dependencies: Use a deploy key

How to:

Example

Enter the Vault

TL;DR

Selenium 4 and chrome driver, take full page screenshots

Why am I writing this?

Ruby, config this

Config_this

Installation

Usage

Get the Configuration object

Setting the configuration Object

Setting it With a Hash

Setting it with a YAML file

Setting it with a JSON file

Setting it with a Environmental Variables

Setting it with a Block

Using the Configuration

2. Describe your package in `pyproject.toml`