Forem: Robin Shen

TOD - command line tool to run OneDev CI/CD jobs against local changes

Robin Shen — Tue, 27 Feb 2024 07:30:11 +0000

OneDev 10.2 comes with TOD (The*OneD*ev), a command line tool to test/debug CI/CD jobs against local changes to avoid the modify/commit/push/run/check cycles.

Under the hood, tod stashes local changes to a commit and pushes to server under a temporal ref to run specified job. Log of running job is streamed back to tod so that you can check job progress and status without leaving terminal.

Since job runs on server, it does not have many quirks and limitations of other tools (gitlab-runner exec or nektos/act for instance), such as requiring you to set up job environments, unable to handle job dependencies, etc. And you can still get fast feedback via shallow clone and cache.

For details, please check tod project

Open Source DevOps Server to Get Most Of Your .NET Projects

Robin Shen — Sat, 09 Sep 2023 03:51:35 +0000

OneDev is an open source git hosting and CI/CD server. Unlike traditional code hosting platforms, it parses C# code to enable symbol search and navigation, both in source view and diff view:

The latest version also adds the ability to suggest CI/CD job templates for .net projects, as well as analyzing TRX formatted unit test report, coverlet coverage report and roslynator code analysis report to annotate source and pull request diff to facilitate code review.

Let’s test with a real project to demonstrate it:

First start up OneDev server with a one-liner on a Linux machine with docker installed:

docker run -it - rm -v $(pwd)/onedev:/opt/onedev -v /var/run/docker.sock:/var/run/docker.sock -p 6610:6610 1dev/server:9.1.5

Access http:localhost:6610 to set up the server, and then import a repository like below:

Specify url as https:github.com/bitwarden/server, and proceed with import with all others remain as default. After a while, we will see two projects get imported, respectively bitwarden, and bitwarden/server
Navigate to project bitwarden/server and click the link adding .onedev-buildspec.yml to set up CI/CD like below:

OneDev under the hood checks if the project root contains dotnet solution file, and suggest a dotnet ci job if it exists. You may further tune the job to meet your requirement. For this demo project, we simply commit the change without any modification and the CI job will be running.
We can open the running build, and check its log like below:
The job takes some time. After it finishes, we will see three new tabs summarizing unit tests, code coverages, and code problems:

From the reports, you may drill down problems to source view directly
When open source files in OneDev, coverage info and code problems will be displayed, and you can even comment the code problems to discuss with author:

This also works for diff view in pull request, and is especially useful when review the change. That’s all. Thanks for watching.

Code Review with OneDev

Robin Shen — Tue, 28 Jun 2022 04:52:16 +0000

OneDev understands importance of code review, and provides many facilities to make this process smooth and efficient. This article introduces how to perform code review in OneDev with these facilities.

Review Rule Definition

Review rule specifies who should review a certain change, and how many approvals should get to accept the change into a branch. Review rule is defined in branch protection settings like below:

In this example we define rules for changes to be accepted into main branch:

The change must pass the CI build
Changes touching models must get at least one approval from core team

With this rule specified, one will not be able to push changes touching models to main branch directly even with repository write permission. Instead a pull request must be submitted to go through the review process.

Reviewer Auto-Suggestion

When reviewers is specified as a group, OneDev will suggest the most appropriate members to review the change, based on commit history of touched files in the change. If for some reason, suggested reviewer can not review the change, you may simply remove and OneDev will pick up other appropriate reviewers. And you can add other reviewers as necessary even if review rule does not require it.

Pull Request Build to Verify Changes

In above screenshot, the pull request also needs to be verified by CI build. We need to disable branch protection rule temporarily to edit build spec of main branch directly to add a pull request trigger for CI job like below:

After committing change, do not forget to re-enable the branch protection setting. Check the pull request again, a CI job will be running to verify merged commit of pull request head and target branch head.

Select Changes to Review

Code review can be performed from file changes tab. File changes can be shown commit by commit, for multiple commits, or since last review (last time approve or request for changes).

Further, files can be filtered to only show interested changes. For instance, below screenshot shows all changed java files under model package:

Jump to Symbol Definition

While reviewing the change, one can jump to symbol definitions, or show all occurrences of the symbol in current commit:

Test Coverage and Code Problems

OneDev can analyze test coverage and code analysis report generated by CI job to help code review, for instance:

For an example set up of diff annotation, please check https://robinshen.medium.com/annotate-source-with-jest-eslint-information-in-onedev-c622641caa45

Code Comment to Discuss Changes

To raise concerns or suggest changes for a code snippet, just select with mouse to add a code comment:

After reviewing the changes, reviewer can give opinion by approving or requesting for changes. The pull request can only be merged after getting approvals from all reviewers.

Pull request submitter can check added code comments in code comments tab. For each suggested change, submitter can commit the change individually, or add to batch to commit together:

Free-Style Review on Commit and File

Besides pull request based review, you can open arbitrary file or commit to add comment to initiate free-style review. User changing selected code will automatically be mentioned and notified to participate in the discussion. Further, source file will be annotated with history comments to help code understanding even if file is modified or renamed.

In this mode, you can still suggest changes. When apply suggested changes, user has to tell which branch to apply the change.

That’s all. Hope these facilities helps with your next code review. Thanks for watching!

Performance Comparison of OneDev and GitLab

Robin Shen — Wed, 01 Jun 2022 01:07:09 +0000

TLDR;

Git Push: OneDev is 40% faster than GitLab
Git Clone: OneDev is 20% slower than GitLab
Web UI: OneDev is 10x~30x faster than GitLab
Memory: OneDev uses 70% less memory than GitLab

For every feature built into OneDev, I take performance seriously. It is very fast with moderate resource usage. In this article I compare it with GitLab to show some numbers.

Environment Setup

OneDev and GitLab runs on same machine. When testing OneDev, GitLab is stopped, and vice versa.

Hardware: AWS EC2 c5a.xlarge (4 core, 8G mem, GP2 SSD EBS storage)
OS: Ubuntu 18.04
OneDev: 7.3.2, installs using the zip file, running on openjdk 8
GitLab: 14.10.2-ee, installs from official repository with apt-get
Test repository: Kubernetes (100K+ commits)

Since OneDev by default uses 50% of available memory, and this is too deluxe for a 8G machine, I modified its conf/wrapper.conf to change max memory to be 20%.

Repository Push/Clone

Command used for a fresh push (repository is empty initially):

$ time git push <repository url> master:master

And command used for clone:

$ time git clone <repository url>

Tests done from another c5a.xlarge linux machine in same subnet. Result is as following:

Conclusion:

Git Push: OneDev is 40% faster than GitLab
Git Clone: OneDev is 20% slower than GitLab

This is measured for full push/clone of a large repository. For daily increment push/pull , this is almost negligible.

I did not measure concurrent push/clone, as both rely on native git to do the heavy lift, and the numbers should be on par.

Web UI

Web UI test is performed with JMeter. The selenium web drive plugin is installed running chrome in headless mode to do end-to-end test. The idea is to visit a url, then wait for certain elements to appear. Taking GitLab pull request changes page for instance, I am using below groovy script for web drive sampler:

import org.openqa.selenium.*
import org.openqa.selenium.support.ui.*
import java.time.Duration
def wait = new WebDriverWait(WDS.browser, 60)
WDS.sampleResult.sampleStart()
WDS.browser.get('http://testserver/root/kubernetes/-/merge_requests/1/diffs')
wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector('.diff-content')))
WDS.sampleResult.sampleEnd()

I started four c5a.2xlarge EC2 machine (8 cores, 16G mem) in same subnet as test server, each machine running 8 browsers via JMeter, and start another c5a.xlarge machine to aggregate results. This effectively simulates 32 concurrent users, and can drive server CPU to nearly 100%. Note that running 32 browsers on a 32 core VM can not drive OneDev to use 100% CPU, due to browser resource contention on same machine.

Six similar pages in OneDev and GitLab are selected for test, displaying same set of data. For each round of test, the page is warmed up for 1 minute, then run 10 minutes to get the performance metrics. Gravatar of both products are disabled to avoid out-of-server resource loading. GitLab auto devops is also disabled to save cpu cycles.

Repository Root Page

Both products show root of the repository including the readme content:

Test result:

Note that for OneDev test in this case, I have to launch five c5a.2xlarge machine to simulate 40 concurrent users to drive it to use nearly 100% cpu.

Recent Commits Page

Both products show recent commits of the repository. OneDev shows 50 commits initially, while GitLab shows 40 commits. OneDev shows commits as verified as it trust GitHub GPG signing key by default.

Test result:

Commit Detail Page

Both products show detail of same commit, including diff of all changed files

Test result:

File Content Page

Both products show content of same repository file.

Test result:

Pull Request Commits Page

Both products show commit list of same pull request. OneDev shows earliest commit first, while GitLab shows latest commit first

Test result:

Pull Request Changes Page

Both products show diffs of changed files of same pull request.

Test result:

For all tests above, OneDev consumes 2.2G mem, while GitLab consumes 4G~4.5G mem.

Conclusion

OneDev web UI performs orders of magnitude faster than GitLab, often 10x~30x faster, with much less memory usage. The UI performance difference is very obvious when navigating through different pages of both products: OneDev loads page instantly after warm up, while GitLab has noticeable delays loading pages all the time.

Thanks for watching. If you would like to run the tests yourself, check the setup detail here.

OneDev 7.1

Robin Shen — Fri, 06 May 2022 03:24:56 +0000

OneDev is a self-hosted git service with built-in CI/CD and
Kanban. The 7.1 release comes with below major improvements:

Command palette to access features quickly

Press cmd/ctrl-k to bring up command palette from anywhere. Search everything and
jump to it.

Easy project navigation via tree breadbrumb

Project navigation is much easier now with tree view in project list and breadcrumb

Two-way repository sync

Push to remote repository upon changes, or pull from remote repository upon remote change. Check this tutorial for details

Display similar issues when create new issue

Similar issues will be displayed in real-time when typing title of new issue, which is very convenient to prevent duplication issues being filed

Full-text search with stop word filtering and word stemming

When do fuzzy search against issues and pull requests, stop words are filtered (for major languages), and word stemming will be performed (for English currently)

Able to change target branch of existing pull request

Occasionally we need to change target branch of existing pull request without losing discussion context

Repository Mirror with OneDev

Robin Shen — Wed, 04 May 2022 07:27:50 +0000

Introduction

It is often desirable to mirror repositories between OneDev and other hosting services. For instance you may want to push changes to a copy at GitHub for public access, or want to pull back any changes contributed by others at GitHub.

OneDev 7.1 made this task trivial with the new repository sync steps. Let’s see how to achieve this.

Preparation

First of all, start OneDev by running below command:

$ docker run -t --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/onedev:/opt/onedev -p 6610:6610 -p 6611:6611 1dev/server

Point your browser to localhost:6610 to setup OneDev, and create a test project.

Then create an empty repository at GitHub side to be used as mirror of our test repository, assume the url is: https://github.com/youraccount/test

Push Changes to Mirror

Now we configure OneDev to push changes to GitHub mirror as long as there is a change in branch or tag:

Login to your GitHub account and generate a personal access token able to push to the mirror repository
At OneDev side, create a job secret in the test project and use access token above as its value:
Define build spec to add a job say Push to GitHub, and add a step of type Repository Sync / Push to Remote. For password / access token field, choose the job token defined above:
Add triggers to run job automatically when branch is updated or tag is created:

Now save the build spec and the job will run automatically to push changes to GitHub mirror. Check at GitHub side, and you will see it contains newly created file .onedev-buildspec.yml. Experiment with other changes such as creating new branch/tag to see the result.

Pull Changes from Mirror

Our GitHub mirror may accept pull requests from contributors, so let’s set up OneDev to pull changes from it:

Continue to edit build spec above, add a job say Pull from GitHub, and add a step of type Repository Sync / Pull from Remote like below:

Here we use the same job token defined above to access GitHub mirror (access token can be left empty if the mirror is publicly accessible).

Add a job trigger of cron type to run job on scheduled time, for instance, on 1:00AM every night:
If your OneDev instance can be accessed publicly, you can configure GitHub to trigger this pull job upon mirror changes. This can be done by creating a webhook at GitHub side like below:

The payload url is defined as:

http://onedev.example.com/api/trigger-job?project=test&job=Pull%20from%20GitHub&branch=master&access-token=<OneDev access token>

Here onedev.example.com is dns name accessing your OneDev instance. Job name is url encoded as it contains space. should be replaced by access token of a OneDev user with permission to run the job like below::

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rkjcdnaopfq3oych4xea.png)

For production use, it is important to enable https protocol for payload url to avoid leaking of access token.

Save the webhook and OneDev should be pulling from GitHub mirror whenever it is changed.

Conclusion

We’ve completed setup of a two-way sync of OneDev repository and its GitHub mirror. Since it is just an ordinary CI job, we can subscribe certain build query to be notified of sync failures, check build log to investigate sync failures, or even configure job resource setting to control resource usage of multiple sync jobs, etc.

Thanks for reading!

Understanding OneDev Pipeline

Robin Shen — Mon, 11 Apr 2022 09:57:41 +0000

OneDev introduced pipeline concept since 7.0. In short, a pipeline is an execution of job dependency graph. It can be created by running any job in the graph, either manually or automatically via job triggers.

Create a Pipeline

Login to OneDev demo site with account demo/demo, switch to CI spec definition page of the demo project, you will see job definitions like below:

Here we define a test job to compile and test our app, and then publish compiled app as artifacts so that it can be deployed later. The deploy job retrieves published app via dependency to the test job, and gets it deployed as shown below:

Now run the test job, you will be brought to the page showing generated build. A pipeline will also be created as the job dependency graph is executed as result of running the test job. The pipeline tab shows the created pipeline current build is part of:

NOTE: pipeline tab is only visible to users with code read permission

Pipeline Characteristics

Pipeline has two characteristics:

Builds fired manually from pipeline view, or by other builds in the pipeline via job triggers, dependencies and post-build actions belong to same pipeline.
For a particular pipeline, only one build will be generated from same job and parameter. If no parameter is defined for the job, it can only be triggered once then.

To demonstrate, run the deploy job from above pipeline, from the build log you will see that it is actually deploying the app published by existing test build in that pipeline:

Also pipeline tab of the deploy build shows that it belongs to same pipeline as test build above:

If we run deploy job again in this pipeline, OneDev will indicate that the build is already fired:

However if you run deploy job from outside of the pipeline, a new pipeline will be created and new builds for test and deploy job will be generated for the new pipeline. Again, let’s check this in action. Switch to commits page to run deploy job for some commit with existing test build:

A new deploy build will be generated, as well as a new pipeline. Switch to pipeline tab of the new build, you will see that new test build also gets generated in regardless of existing test builds for the commit.

Pipeline Automation

Pipeline can be created automatically if a job is set to fire upon certain event. For instance, in CI spec of this demo project, we set job test to trigger automatically when there are new commits in master branch:

Make some changes to app.txt and save the file, you will see that test job is running:

Click the running test build and you will be brought to build detail page, from which created pipeline can be accessed.

Now assume you want to deploy the app automatically when test job is successful. To do it, add a trigger to fire the deploy job when dependency jobs finished like below:

To test it in action, make some changes to app.txt in this project, save the file, open created pipeline and wait to see deploy job running automatically after test job is successful:

Alternatively, you may also configure a branch update trigger for deploy job. In that case, both deploy job and test job will be fired upon changing file, but deploy job will wait for test job to complete before running as shown in the pipeline:

Now you should get an idea how pipeline works in OneDev. Any questions, please post at our issue tracker.

Thanks for watching!

Set Up OneDev Issue Labels

Robin Shen — Sun, 10 Apr 2022 07:48:45 +0000

OneDev is designed to be flexible, and this is no different for issues. In this tutorial, I will show how to customize issue labels.

OneDev issues by default do not have label field, as custom fields work much better, for instance, you may have Type field with values Feature/Bug/Task, or Subsystem field with values Client/Server/Api, etc. However if you do want to have labels, here is how to do it:

Login as administrator, and switch to custom issue fields page
Add a custom issue field like below:
Continue to add available choices to include all possible label values, for instance:
Now create an issue and you will see that we can specify labels:
To specify labels for existing issues, batch edit them and choose Labels field to edit:
By default newly created field is not displayed in issue list or board. This can be changed by configuring display fields option in issue list and board respectively:

That’s all. Thanks for watching!

OneDev 7 - self-hosted git service with CI/CD and Kanban

Robin Shen — Mon, 04 Apr 2022 12:02:31 +0000

The self-hosted open source git service OneDev reaches 7.0. Some feature highlights:

Show job dependencies as pipeline

Visualize job dependencies as pipeline to get instant understanding of job relationships and their execution statuses.

Two-factor authentication

TOTP based two-factor authentication can be enabled by user. Site administrator can also enforce 2FA for all users or certain groups

Commit/Tag GPG signature verification and generation

Commits and tags with GPG signature can be verified. Branch/tag protection rule can be set up to enforce signed commits/tags. OneDev can use its own signing key to sign commits/tags generated by itself (pull request merge commits etc). GPG public key of other git hosting services can be imported so that their signed commits/tags can be trusted

Able to cache cloned repository to improve CI/CD speed

Repository can now be cloned into a sub directory under job workspace, and can be cached to improve speed. For instance a fresh clone of Linux repository takes 5 minutes, while updating a cached one takes 5 seconds

Other improvements

For a complete list of changes, please check the release notes

Effortless OneDev CI/CD Farm with Agents

Robin Shen — Thu, 10 Feb 2022 08:17:35 +0000

Introduction

From day 1, OneDev is able to run CI/CD jobs as pods on Kubernetes clusters. In case you do not have a Kubernetes cluster, it is now possible to install OneDev agents on other machines to get a CI/CD farm. Agent is designed to be maintenance free: the server will push updates to agents automatically when it is upgraded.

How to Set Up

Let's go through an example to see how to set up the CI/CD farm

Start OneDev server with below command on Linux or Mac (this command is for demonstration purpose, follow installation guide for production use):
```
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/onedev:/opt/onedev -p 6610:6610 -p 6611:6611 1dev/server
```
Open browser to visit http://localhost:6610 to complete the setup
Visit agent management page, click the add agent button, and show the docker command to run agent:
Copy the command to your terminal (Linux/Mac), change serverUrl environment to http://:6610 (do not use localhost here as otherwise OneDev agent running inside container can not access OneDev server), and run the command to start agent. If connected, the agent will be online in agent management page like below:
Now let's create a job executor to use the agent. Switch to job executors page, delete the default auto-discover executor, and create an executor of type Remote Docker Executor, give it a name and leave others as default. Test and save the setting:
Create a test project, define CI spec with a simple job, and run the job. You will see that the job is running on the agent:
When OneDev dispatches jobs to agents, it choses the best one based on agent cpu/memory capacity and job cpu/memory requirement. If no agents satisfies the requirement, the job will be in waiting state.

Thanks for reading!

Connect Customer to Developer with OneDev Service Desk

Robin Shen — Fri, 06 Aug 2021 05:14:24 +0000

Introduction

OneDev is a performant open source git server, with customizable issue workflow and built-in CI/CD.

Version 4.9 comes with the service desk feature, which allows your customers to submit tickets via email without the need to have a OneDev account. These tickets can then be created in desired projects, and assigned to appropriate staffs in your team. All further discussions over the tickets can take place completely in email, with discussion contents posted to ticket as comments. Customer will also get notifications when there are any ticket activities, for instance, when a relevant release is created or deployed.

Feature Highlights

Before diving into service desk setup, let’s check what the service desk can do for you:

Customer can either send ticket to a uniformed email address, for instance support@yourcomany.com, or send to project specific one like support+yourproject@yourcompany.com. Tickets sent to uniformed email address will be created in default project of the customer, and you can move them to other projects later if desired.
Customer can only create tickets in authorized projects. Authorization is done by checking customer email address.
All recipients of the email will be added to watch list of created ticket, so that they can be notified of ticket activities and join the discussions.
Html format and attachments of the email are carefully handled to achieve same experience at Email side and web UI side.
For different customers, you may configure different ticket properties such as priorities and assignees.
All emails relating to a particular ticket will be organized into a single thread to track discussion context, as well as reduce notification noises. Users can also unsubscribe from the ticket.
Content of the ticket notification email can be customized.

How to Set Up

First of all, you need to enable mail setting and its check incoming email option:

Also subaddressing needs to be enabled at your mail server for system email address. Test mail sending and receiving at bottom of the page to make sure it works.

Now OneDev is capable to send ticket notifications via email. Users receiving notification email can also add comments to the ticket by replying the email. Content of the notification email can be customized via notification templates below the mail setting.

However to be able to create tickets via email, you should also enable the service desk setting like below:

All done! Now authorized users can send email to system email address to create tickets!

CI/CD configuration reuse in OneDev

Robin Shen — Sat, 24 Apr 2021 07:42:50 +0000

Maintaining CI/CD configuration for many projects can be complex and challenging. To reduce the overhead, OneDev 4.3 adds the mechanism to define common CI/CD building blocks in separate projects and use/customize them in other projects.

CI/CD configuration is maintained in OneDev via build spec. When define build spec for a project, it is now possible to import build specs from other projects. Objects (jobs, services, step templates and properties) contained in imported build specs are treated as if they are defined locally, except that imported objects will be overridden by local objects with same name. In case of multiple imports, latter imported objects will override prior imported objects with same name.

With this import and overridden mechanism, you may maintain common jobs/services/step templates in a separate build spec, and use property placeholders in places where customization might be necessary. When this build spec is imported into other build specs, user may selectively override these properties to get different behaviors for different projects.

Let’s practice with an example to see how it works.

Start OneDev with below command in Linux or Mac:

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/onedev:/opt/onedev -p 6610:6610 -p 6611:6611 1dev/server:4.3.3

Open browser to visit http://localhost:6610 to complete the setup. Add a project say commons, and set up its build spec as below:

Here we define a common CI job running a step with image specified as value of property nodeImage. Other projects can import this build spec and override the property to use different node image if desired. This job also defines a trigger to fire build automatically when there are new commits and property applicable projects is defined to exclude the commons project.
Now we have a common CI job. In order to reuse it in other projects, we need to add a tag. We also configure default role of the project so that everyone in the system has permission to read code for simplicity reason:
Let’s continue to add a test project, and set up its build spec to import and use the job:

Here we override property nodeImage to use node:14.16-buster. Save the build spec and the imported job will be running automatically to print the expected node version.

Besides overriding property to customize imported objects, we can also use parameters to customize common step templates. Now let’s practice with this approach:

Continue to edit build spec of our commons project as below:

Here we delete objects defined previously, and add a step template instead to do the job. The template defines a parameter to get image name dynamically from caller
Add a new tag say v2 for our new version of build spec
Edit build spec of the test project to import the new version, add a CI job to use imported step template and specify parameter Node Image as node:14.16-buster

Save the build spec to see the job running automatically to print expected node version.

Thanks for reading!