Forem: JetBrains TeamCity

How We Taught AI Agents to See the Bigger Picture

JetBrains TeamCity — Wed, 25 Mar 2026 20:36:03 +0000

The problem: Good code is not always accepted code

AI agents can write working code. They can make changes that compile, pass tests, and finish tasks. And yet the pull request might still be rejected.

That happens because every codebase has its own internal logic. Teams have naming conventions, preferred APIs, and patterns they trust. Some of those rules are documented, but many are not.

Pavel Sher (Technical Lead for TeamCity) has reviewed thousands of commits over the years. That experience helps him spot when a change technically works but still does not fit the project. He can suggest better naming, a more appropriate approach, or a solution that stays consistent with the rest of the codebase.

This led us to a practical question: how can we help AI agents see the bigger picture, too?

The trap: Agents learn from what they see

When you ask an AI agent to write code, it usually looks through the codebase for examples. It tries to infer conventions from existing code and follow the patterns it finds.

That sounds reasonable, but it becomes a problem in large, long-lived projects.

If an agent sees the same pattern repeated 100 times, it will often assume that this is the right way to do things. It repeats the pattern because it is trying to stay consistent with the project.

But in legacy codebases, the most common pattern is not always the one the team wants to keep using.

Image generated with the help of AI

An agent cannot tell whether those 100 examples represent the current standard or an old approach the team is gradually replacing. It sees frequency and mistakes it for correctness.

That is the core issue. Without guidance, agents tend to inherit legacy approaches simply because those approaches appear more often.

The experiment: Refactoring legacy database code

Like many mature products, TeamCity has a fair amount of legacy code. In one of our experiments, we wanted to migrate several classes from an old database API to a newer one.

This was not a mechanical refactoring. The new API did not map directly to the old one, so a simple method-by-method replacement was not enough. The migration required understanding how to reshape the code, which patterns to use, and and how to handle special edge-cases.

We asked an AI agent to perform the migration.

The result looked good at first. The code worked, tests passed, and the new API was in place. But during review, Pavel noticed a major issue.

The agent had updated the API calls, but everything around them still followed the old design. It reused legacy error handling, legacy data flow, and legacy structure because that was what it found in the surrounding code.

So the result was technically correct, but conceptually inconsistent: new API calls implemented using the old design.

First step: Show examples of accepted work

Our first improvement was simple. Instead of relying only on the current codebase, we gave the agent examples of migrations that had already been reviewed and accepted.

In effect, the instruction became: “Here is how we migrated Class A, now apply the same approach to Class B.”

That worked noticeably better. The agent no longer focused only on replacing API calls. It started following the direction of the migration more accurately, and the resulting code fit the project better.

However, this approach did not scale well. We could not keep feeding accepted examples to the agent one by one for every new task.

Second step: Turn accepted changes into explicit rules

We continued by looking at accepted migration commits and studying what they had in common. From those commits, we extracted patterns the team had already approved and turned them into a practical set of rules.

The document was straightforward and simply told the agents: “When migrating from API X to API Y, follow these patterns, avoid these mistakes, and use these naming conventions.”

Using this document led to further improvements in the quality of changes. Merge requests began to pass review with far fewer comments.

The key insight: History is documentation

This is where we came to a realization.

We had been writing rules by hand, but the underlying knowledge already existed in the repository. Accepted commits showed what good changes looked like. Review comments reflected the concerns of the team. The project history already contained what we were trying to teach to AI.

So instead of documenting everything manually, we had another idea: we’d extract that knowledge directly from Git history.

CommitAtlas: Learning from repository history

That idea served as the basis for CommitAtlas, an internal tool that reads Git history and answers questions about how things should be done in the project.

At a high level, it works like this:

Import commits into a database
Use an LLM to describe each record based on the commit message and patch
Answer questions such as "How do I migrate from API A to API B?" or "What naming convention do we use for services?"

Unlike ordinary code search, CommitAtlas does not simply show what appears most often in the repository. It analyzes recent accepted changes to identify which patterns the project is actively adopting.

For example, it may recognize that a newer API is replacing an older one, even if the older API still appears more frequently in the codebase.

Documentation on demand for AI agents

Now, before writing code, our AI agents can query CommitAtlas and get a short, task-specific guide assembled from real project history.

That guide can include:

Recommended implementation patterns
Code examples from accepted commits
Links to the source commits
Naming and style guidance

In practice, this gives the agent something close to what an experienced teammate would provide: context before implementation.

What we saw in practice

Our original hypothesis turned out to be correct. Git history contains a large part of the knowledge that makes code not just functional, but acceptable to the team.

When agents use CommitAtlas, they tend to:

Understand project context faster
Produce code that follows local conventions more closely
Get fewer pull request rejections

The knowledge base also improves over time. Each new commit adds more real examples of how the codebase evolves, and human feedback helps refine the generated guidance.

Why this matters for legacy projects

If your project has been around for years, your repository contains a lot of institutional knowledge.

Commits, reviews, and accepted refactorings show how the codebase has evolved and what the team has learned along the way. Traditional documentation rarely keeps pace with that kind of change, especially in large systems. But repository history does keep pace, because it records what was actually changed and what was actually accepted.

For teams working with AI agents, this matters a lot. It is not enough to give the agent access to the code. You also need to give it access to the context that explains which patterns still belong to the project and which ones are being left behind.

That context is what helps agents produce code that fits naturally into an existing codebase and meets the standards experienced reviewers expect.

A practical takeaway

Many teams are starting to experiment with AI coding agents. In practice, the biggest challenge is not getting the agent to write code that compiles. The real challenge is getting it to write code that fits the project.

Every repository contains a large amount of implicit knowledge. Commit history, refactorings, and accepted pull requests all reflect decisions the team has made over time. Together they show which patterns are encouraged, which ones are being phased out, and how the codebase is evolving.

If AI agents can access and interpret that history, they can start making decisions that better match the intent of the project instead of simply copying what appears most often in the code.

For teams working with large or long-lived codebases, repository history may be one of the most useful sources of guidance you can provide to an AI agent. It captures how the system actually evolved and what changes were accepted along the way.

Helping agents learn from that history can make the difference between code that technically works and code that genuinely belongs in the project.

$ teamcity From the Command Line

JetBrains TeamCity — Thu, 19 Mar 2026 10:01:27 +0000

Start builds, tail logs, and manage agents and queues – without switching windows.

Whether you are debugging a failed build, managing build agents, or scripting your deployment pipeline (or your AI coding agent is doing so) – there are times when opening a browser is one step too many.

TeamCity CLI is a lightweight, open-source tool that puts TeamCity at your fingertips.

A CLI for TeamCity has been a long-standing request from the community. Here’s how it works and what you can use it for.

TeamCity CLI includes over 60 commands, and for anything it doesn't wrap yet, teamcity api gives you direct access to the full TeamCity REST API.

Source code (Apache 2.0): https://github.com/JetBrains/teamcity-cli

Getting started

Install and authenticate in seconds:

# macOS / Linux
brew install jetbrains/utils/teamcity

# via a bash script
curl -fsSL https://jb.gg/tc/install | bash

# Windows
winget install JetBrains.tc

# via a powershell script
irm https://jb.gg/tc/install.ps1 | iex

# Connect to your server
teamcity auth login https://example.teamcity.com/

That's all! Now let’s see it in action.

Investigating a failed build

Let’s say something breaks. Here’s how to go from wondering what happened to having it fixed without switching windows.

See what's happening

teamcity run list --all

Build 1389 failed. Here’s how we can drill down:

teamcity run view 1389

Need to jump to the TeamCity UI instead? Add --web.

Find the root cause

You don't need to scroll through the entire log. Show only the failed step, then the failed tests:

teamcity run log 1389 --failed

Only one test failed, so you know exactly what to fix.

Fix and re-trigger

After you push the fix, restart and watch it in real time:

teamcity run restart 1389
teamcity run watch 1408 --logs

You see all the updates happening live in your terminal – build state changes, step progress, and streaming log output.

Test before you push

Want to validate your changes with CI before committing to a push? Run a personal build with your local changes:

teamcity run start Sandbox_Test --local-changes

TeamCity runs the build with your uncommitted work applied as a patch – no branch, no force-push, no noise in the repo.

Remote agent access

You can open an interactive shell session on any build agent – or execute commands remotely:

teamcity agent list
teamcity agent term 813

No SSH keys to manage, no VPN to connect – if your TeamCity server can reach the agent, so can you.

Need to just run a quick command? Here’s how:

teamcity agent exec 813 "docker ps -a"

Built for scripting

teamcity covers queues, agents, artifacts, and project configuration – the commands look exactly as you’d expect:

teamcity queue list                                          # what's queued
teamcity queue top 461                                       # move a build to the front
teamcity queue approve 462                                   # approve a waiting build

teamcity agent list                                          # all agents with status
teamcity agent disable "Linux Agent 03" --comment "Maint"    # take one offline

teamcity run artifacts 453                                   # browse artifacts
teamcity run download 453 "reports/**/*.html" --output ./out # grab what you need

# Restart all failed builds on main
teamcity run list --failed --branch main --plain=id | xargs -I{} tc run restart {}

# Structured data for dashboards or scripts
teamcity run list --failed --json=id,status,buildType.name

# Hit any REST API endpoint directly
teamcity api /app/rest/server

Every list command supports --json with field selection and --plain for tab-delimited text – making teamcity a building block for automation.

teamcity api is the escape hatch – anything the CLI doesn't wrap yet, you can call directly.

Compatible with AI coding agents

TeamCity CLI ships with an Agent Skill that teaches AI coding assistants how to use the tool. Your AI agent can then check the build status, investigate failures, download artifacts, and manage your pipeline on your behalf.

Users should limit the token rights granted to the agent, be aware of security implications, and always check the actions the agent intends to execute.

What's next

TeamCity CLI is open-source under Apache 2.0. There's more we want to build – for example, pipeline visualization, richer TUI interfaces, and deeper integration with version control workflows. Your feedback will shape what comes next.

Try it

brew install jetbrains/utils/teamcity       # macOS
winget install JetBrains.TeamCityCLI        # Windows
curl -fsSL https://jb.gg/tc/install | bash  # Linux/macOS setup script
irm https://jb.gg/tc/install.ps1 | iex      # Windows setup script

Source code and docs: github.com/JetBrains/teamcity-cli

Report issues or request features: GitHub Issues

Case Study: How Junie Uses TeamCity to Evaluate Coding Agents

JetBrains TeamCity — Tue, 03 Jun 2025 15:34:14 +0000

Introduction

Junie is an intelligent coding agent developed by JetBrains. It automates the full development loop: reading project files, editing code, running tests, and applying fixes, going far beyond simple code generation.

Similar to how developers use tools like ChatGPT to solve coding problems, Junie takes it a step further by automating the entire process.

As the agent’s architecture evolved, the team needed a secure, robust way to measure progress. They wanted to build a scalable, reproducible evaluation pipeline that would be able to track changes across hundreds of tasks.

That’s where TeamCity came in. Junie’s development team uses TeamCity to orchestrate large-scale evaluations, coordinate Dockerized environments, and track important metrics that guide Junie’s improvements.

The challenge

Validating agent improvements at scale

As Junie’s agents became more capable, with new commands and smarter decision-making, every change needed to be tested for real impact. Evaluation had to be systematic, repeatable, and grounded in data.

Did it get better or not?’ is a very poor way to evaluate. If I just try three examples from memory and see if it got better, that leads nowhere. That’s not how you achieve stable, consistent improvements. You need a benchmark with a large and diverse enough set of tasks to actually measure anything. - Danila Savenkov, Team Lead, JetBrains Junie

The team identified five core requirements for this process:

Scale: Evaluations had to cover at least 100 tasks per run to minimize statistical noise. Running fewer tasks made it hard to draw meaningful conclusions.
Parallel execution: Tasks needed to be evaluated in parallel, as running them sequentially would take over 24 hours and delay feedback loops.
Reproducibility: It had to be possible to trace every evaluation back to the exact version of the agent, datasets, and environment used. Local experiments or inconsistent setups were not acceptable.
Cost control: Each evaluation involved significant LLM API usage, typically costing USD 100+ per run. Tracking and managing these costs was essential.
Data preservation: Results, logs, and artifacts needed to be stored reliably for analysis, debugging, and long-term tracking.

Benchmarking with SWE-bench

For a reliable signal, Junie adopted SWE-bench, a benchmark built from real GitHub issues and PRs. They also used SWE-bench Verified, a curated 500-task subset validated by OpenAI for clarity and feasibility.

In parallel, Junie created in-house benchmarks for their internal monorepo (Java/Kotlin), Web stack, and Go codebases, continuously covering more languages and technologies by the benchmarks.

The operational challenge

Running these large-scale evaluations posed operational challenges:

Spinning up consistent, isolated environments for each task.
Managing dependencies and project setups.
Applying patches generated by agents and running validations automatically.
Collecting structured logs and metrics for deep analysis.
Manual workflows wouldn’t scale. Junie needed automation that was fast, repeatable, and deeply integrated into their engineering stack.

TeamCity enabled that orchestration. With it, the Junie team built an evaluation pipeline that is scalable, traceable, and deeply integrated into their development loop.

The solution

To support reliable, large-scale evaluation of its coding agents, Junie implemented an evaluation pipeline powered by TeamCity, a CI/CD solution developed by JetBrains.

TeamCity orchestrates the execution of hundreds of tasks in parallel, manages isolated environments for each benchmark case, and coordinates patch validation and result collection.

If we tried running this locally, it just wouldn’t be realistic. A single evaluation would take a full day. That’s why we use TeamCity: to do everything in parallel, isolated environments, and to ensure the results are reproducible. - Danila Savenkov, Team Lead, JetBrains Junie

The setup enables the team to trace outcomes to specific agent versions, gather detailed logs for analysis, and run evaluations efficiently, while keeping infrastructure complexity and LLM usage costs under control.

Execution pipeline design

At the heart of the system is a composite build configuration defined using Kotlin DSL, which gives Junie full control over task orchestration. Each top-level evaluation run includes multiple build steps.

Environment setup

Each coding task is paired with a dedicated environment, typically a pre-built Docker container with the necessary dependencies already installed. This guarantees consistency across runs and eliminates local setup variability.

Agent execution

Junie’s agent is launched against the task. It receives a full prompt, including the issue description, code structure, system commands, and guidelines. It then autonomously works through the problem, issuing actions such as file edits, replacements, and test runs.

The final output is a code patch meant to resolve the issue.

Patch evaluation

The generated patch is passed to the next build step, where TeamCity applies it to the project and runs the validation suite. This mimics the GitHub pull request flow – if the original tests were failing and now pass, the task is marked as successfully completed.

Metric logging

Execution metadata, including logs, command traces, and success/failure flags, is exported to an open-source distributed storage and processing system. Junie uses it to store evaluation artifacts and perform large-scale analysis.

With the solution’s support for SQL-like querying and scalable data processing, the team can efficiently aggregate insights across hundreds of tasks and track agent performance over time.

Developers rely on this data to:

Track the percentage of solved tasks (their “North Star” metric).
Analyze the average cost per task for LLM API usage.
Break down agent behavior ( like the most frequent commands or typical failure points).
Compare performance between agent versions.

Scalability through automation

By using Kotlin DSL and TeamCity’s composable build model, Junie scales evaluations to hundreds of tasks per session – far beyond what could be managed manually. For larger datasets (typically 300-2000 tasks), each execution is spun up in parallel, minimizing runtime and allowing the team to test changes frequently.

We use Kotlin DSL to configure everything. When you have 13 builds, you can still manage them manually, but when it’s 399, or 500, or 280, it starts getting tricky. - Danila Savenkov, Team Lead, JetBrains Junie

Results: reproducible, scalable, insight-driven agent development

TeamCity has enabled Junie to measure agent performance efficiently and at scale, making their development process faster, more reliable, and data-driven.

Key outcomes

Scalable evaluation: Tasks are evaluated in parallel, reducing runtime from 24+ hours to a manageable window.
Full reproducibility: Every run is traceable to a specific agent version and dataset snapshot, which is crucial for tracking progress and debugging regressions.
Reliable metrics: Logs and outcomes from each task are automatically stored and analyzed. This allows deep introspection without relying on TeamCity for data retention.
Cost awareness: LLM usage is monitored per run and per-task, helping the team optimize both internal development and future product behavior.
Cleaner feedback loops: The ability to evaluate 100+ tasks per change helps the team distinguish real improvements from statistical noise.

Automate Your React App Deployment With JetBrains TeamCity

JetBrains TeamCity — Thu, 20 Feb 2025 12:34:02 +0000

This tutorial was brought to you by Kumar Harsh, a software developer and technical author.

If you regularly work with React projects, you probably already know how tricky it can be to deploy them smoothly. Issues like manual errors, inconsistent deployment practices, and slow iteration cycles often turn React app deployment into a challenging process and a source of frequent headaches. Effective use of automation can help reduce those headaches and significantly improve the transition from code to users.

TeamCity, a continuous integration and continuous deployment (CI/CD) platform from JetBrains, is designed to simplify deployment pipelines. It offers features like build automation and optimization, detailed test reporting, and secure credential management.

By securely automating your deployment workflow, TeamCity helps reduce errors, improve consistency, and accelerate delivery, allowing your team to focus on building great applications.

In this tutorial, you’ll learn how to set up an automated deployment pipeline with TeamCity Pipelines. You’ll learn how to configure TeamCity to deploy a React app from a GitHub repository to an Amazon S3 bucket, making it ready to serve as a static website.

Prerequisites

Before you get started, here’s what you’ll need to follow along:

A GitHub account.
An AWS account. The pipeline you’ll create in this tutorial will deploy the final app to an Amazon S3 bucket. Since you only need to use S3, a free AWS account will work.
A TeamCity Pipelines account. If you don’t have one, you can sign up for a 14-day free trial.

Once you have these ready, start by forking this GitHub repo to your own GitHub account:

https://github.com/krharsh17/teamcity-react-test-app

The above repo contains a React 18 app created using Vite. The app’s home page displays framework logos and includes a button that tracks how many times it’s clicked:

Output of the React app

The repo also contains two integration tests in the src/App.test.jsx file that check whether the App component is rendered and if the button and the counter are working.

You can enter the following commands in a terminal window at the root of the repo to run the tests for yourself:

yarn && yarn test

Here’s what the output should look like:

RUN  v0.34.6 /Users/kumarharsh/Work/Draft/teamcity-react/teamcity-react-test-app

 ✓ src/App.test.jsx (2)

   ✓ App (2)

     ✓ renders

     ✓ should increase count by 1

 Test Files  1 passed (1)

      Tests  2 passed (2)

   Start at  23:05:51

   Duration  642ms (transform 31ms, setup 0ms, collect 146ms, tests 26ms, environment 284ms, prepare 51ms)

✨  Done in 1.61s.

This indicates that the tests ran successfully on your local system. At this point, you’re ready to start building the CI/CD pipeline.

Setting up TeamCity

Let’s start by creating a new pipeline in TeamCity. Once you’ve signed up for TeamCity Cloud, you should see a loading screen:

TeamCity Cloud server starting

Once the server is up, here’s what the dashboard should look like:

TeamCity Pipelines dashboard

Click the blue Create new pipeline button in the middle of the page, then authenticate with your Git platform:

Logging in to a VCS provider

Authenticate with GitHub once again. TeamCity will then ask you to choose the repo you’d like to deploy in your first pipeline. Type “teamcity-react-test-app” in the search bar:

Choosing your forked repo

Select the forked repo and click Create:

Creating the pipeline

This will set up your new pipeline. Here’s what its Edit page should look like:

Newly created pipeline

At this point, you’re ready to start developing the steps and jobs in the pipeline.

Creating a simple, sequential pipeline

You’ll first create a straightforward pipeline that installs dependencies, runs tests, creates a build, sets up AWS credentials, and finally pushes the built artifacts to the S3 bucket. Then, you’ll optimize it by splitting it into multiple jobs that can reuse steps and run in parallel as needed.

Hover over the name of the job (Job 1) in the right pane and click the pencil icon that pops up to edit the name:

Editing the job name

Rename the job “Deploy to S3.” Then, click the NODE.JS icon under the Steps section to add a Node.js-based pipeline step:

Adding a new step

This will open another sidebar that asks for the details of the steps, including the step name, the working directory (which is the repo root by default), and the shell script you’d like to run as part of this step:

Editing the step details

Enter “Install dependencies” as the step name and yarn for the shell script. Click the blue Done button in the bottom left-hand corner of the sidebar to add the step to the job. Here’s what the job should look like now:

Updated job

Adding the test and build steps

Now, add two more steps to the job with the following details:

“Run test”, with the shell script yarn test.
“Build”, with the shell script yarn build.

Here’s what the job should look like when you’re done adding the two steps:

Updated job with three steps

At this point, you can try out the pipeline to see if it runs the tests and builds the app correctly by clicking the blue Save and run button in the top right-hand corner of the page. Here’s what the output should look like:

Results of the first run

You can expand the sections for each of the steps 1 through 3 to learn more about how they ran. Try expanding step 2 to view the test results. You’ll notice results similar to what you received when you ran the tests locally:

Test execution logs

This indicates that the test and build steps are working correctly.

Preparing the S3 bucket for deployment

As mentioned earlier, you’ll be deploying the built React app to an Amazon S3 bucket. But before you do that, you’ll need to create a new S3 bucket and configure it to serve a static website. You can find detailed instructions on how to do that in this AWS tutorial.

Here’s an overview of the steps:

Create a new S3 bucket with a name (for example, myawsbucket-kh) and default settings.
On the bucket details page, go to the Properties tab. Scroll down to find the Static website hosting section and click the Edit button:Enabling static website hosting
Choose Enable under Static website hosting and specify the Index document as index.html. Click the Save changes button at the bottom of the page when done. Here’s what the Static website hosting section should look like now:
Static website hosting enabledYou’ll also find the website endpoint at the bottom of this page. This is where you’ll be able to access the website once it’s deployed.
You also need to enable public access to the bucket. To do that, head over to the Permissions tab from the same page and click the Edit button in the Block public access (bucket settings) section:Editing public access settingsOn the resulting page, you need to uncheck the Block all public access option and click the Save changes button in the bottom right-hand corner:Enabling public accessOnce you’ve enabled public access, you can start uploading files to this bucket and access the website at the URL you retrieved earlier.

Copy and store the S3 bucket name somewhere safe. You also need to generate an AWS access key ID and secret. Before you do that, it’s best to create a dedicated AWS IAM user for the CI/CD pipeline to avoid exposing your admin user’s privileges unnecessarily.

To do that, navigate to the AWS IAM Users page via your AWS console:

Users page via the AWS IAM console

On this page, click the orange Create user button to create a new user. Feel free to give it any name you like. For permissions, choose Attach policies directly under Permission options and search for and add the AmazonS3FullAccess policy to the user. Here’s what the user should look like on the review page:

Reviewing the newly proposed user

Click the Create user button in the bottom right-hand corner of the page. Once the user is created, click its name on the Users page to view its details:

Viewing the newly created user’s details

Here’s what the user details page will look like:

User details

Click the Security credentials tab and scroll down to find the Access keys section:

Finding the access keys section

Click the Create access key button. On the page that opens, choose Command line interface (CLI) under the Use case and check the confirmation checkbox at the bottom of the page before clicking the Next button:

Creating the access key

You can skip adding tags to the access key and click the Create access key button:

Skipping tags and creating the access key

The next page will display the access key and the secret access key values. Copy them and store them safely. You’ll use them in the next step of the pipeline to upload built artifacts to the S3 bucket through the AWS CLI.

You can safely provide your pipeline with the AWS secrets through TeamCity’s secrets. However, the recommended approach to handle infrastructure integration with CI/CD pipelines is to allow administrators to connect and manage cloud platforms while only providing pipelines (and related users) the ability to use those connections as part of the pipeline steps only, with no access to the secret keys or values at all. To learn more about implementing this approach, check out TeamCity’s AWS Connection.

Adding the deploy steps

You now have everything you need to add the deploy steps. To do that, you’ll need to add two more steps to your TeamCity pipeline as you did before.

First, add a step named “Configure AWS credentials.” Set the step type to Script and use the following custom script value:

aws configure set aws_access_key_id "%AWS_ACCESS_KEY_ID%"

aws configure set aws_secret_access_key "%AWS_SECRET_ACCESS_KEY%"

This step uses the AWS CLI to configure your AWS account credentials by accessing the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY TeamCity secrets.

Next, add a step named “Push to S3.” Set its type as Script and add the following custom script value:

aws s3 cp dist s3://%AWS_S3_BUCKET_NAME%/ --recursive

With the AWS CLI properly configured, this step will upload the contents of the dist directory (generated by the build step) to the S3 bucket, using the bucket name stored in the TeamCity secret AWS_S3_BUCKET_NAME.

You can use the AWS CLI directly inside the pipeline because it’s preinstalled on default JetBrains-hosted agents. Here’s a list of all the software resources that you get out of the box when using JetBrains-hosted runners on TeamCity Cloud.

This is what the job should look like when you’re done adding the steps:

Deploy steps added to the pipeline

You’ll notice that the “Deploy to S3” job now shows up in yellow with a warning that says “No compatible agent”. This is because some of the steps in the job rely on pipeline secrets that aren’t defined yet. You can fix that by clicking the Pipeline settings link right above the job and clicking the plus (+) icon to the right of the No Secrets section in the right pane:

Adding secrets to the pipeline

This will show some text input fields in that section that will allow you to add secret names and values. You need to add the following secrets:

AWS_ACCESS_KEY_ID: Your AWS access key.
AWS_SECRET_ACCESS_KEY: Your AWS access key secret.
AWS_S3_BUCKET_NAME: The name of your S3 bucket.

Once you’ve defined and saved these values, you’ll notice the yellow highlight and warning text are gone:

Pipeline ready to run

You can now click the Save and run button in the top right-hand corner to save and run the updated pipeline. Here’s what the output should look like:

Test run of the updated pipeline

This means that your app was successfully built and deployed to your S3 bucket. You can check the bucket to see if the files were actually added there:

S3 bucket with build artifacts

You can also visit the public website URL you received earlier to view the deployed website:

Deployed website as a result of the pipeline

Since the pipeline was created with a GitHub VCS repository, it’s automatically configured to monitor the main branch for new commits and trigger whenever it detects changes.

Optimizing the pipeline

You can make some tweaks to optimize this pipeline further. Currently, there are only a small number of tests, which shouldn’t cause a performance bottleneck. However, if you end up having a large number of tests, especially integration tests that might require extensive setup and teardown in some cases, the test step could become quite time-consuming.

The pipeline is configured to only proceed to the build and the deploy steps once the tests have been completed successfully.

If you’re setting up a pipeline on a production branch that only receives commits after they’ve been approved as part of a PR or another review process earlier, you can consider running the tests and the build in parallel to save some time.

If either of the two fails, the deploy step can be skipped since you wouldn’t want to deploy a version that doesn’t satisfy internal tests or if you’re not able to generate a successful build for it.

You can use TeamCity’s parallel builds to achieve this. Since this will require you to split the pipeline into multiple jobs, TeamCity can reuse job results to avoid rerunning jobs if the last run was successful and no new changes have been made since then.

This approach saves CI/CD minutes when testing the pipeline with the same commit multiple times. Alternatively, you can configure the pipeline to reupload (without rebuilding) the website to the S3 bucket if needed.

To implement these optimizations, you’ll first need to split the steps from the “Deploy to S3” job into three jobs.

Create build

The Create Build job will just install project dependencies and create the build. For that, you need to add two steps to it: a Node.js step called “Install dependencies” with the command yarn and another Node.js step called “Create build” with the command yarn build. Here’s what the job will look like when you’ve added the steps:

New build job with two steps

You also need to scroll down and configure the artifacts for this job. Since the output of this job will be used by the “Deploy to S3” job to deploy the app to the S3 bucket, you need to click the plus sign icon to the right of the Artifacts section and enter ./dist/* in the input field that pops up:

Updated artifacts path in the build job

Run tests

Now, create another job named “Run Tests” and add two steps: “Install dependencies” (same as before) and another Node step called “Run tests” with the command yarn test. Here’s what the job will look like when you’ve added the steps:

New tests job

You don’t need to configure any artifacts for this job since it doesn’t generate any output files after running the tests. You could, however, choose to export test results in a file and add another job to the pipeline to upload the results to a test results analysis platform.

Deploy to S3

Finally, you need to update the “Deploy to S3” job to run only after the other two jobs have finished running successfully. Since the other steps are now being run as part of the other two jobs, this job only needs to run the “Configure AWS credentials” and “Push to S3” steps.

Here’s what the steps list will look like when you’ve deleted the first three steps:

New deploy job

You need to check both Create Build and Run Tests under Dependencies since the deploy job depends on those two jobs running successfully. Here’s how the job’s details (and the pipeline) will look when you do that:

Job dependencies configured

Click the dropdown that says Ignore artifacts next to the Create Build checkbox under Dependencies and change its status to Use artifacts:

Artifact usage configured for the job

Scroll down to the Optimizations section, where you’ll see that the Reuse Job Results toggle is on by default:

Reuse job results toggle

Switch the toggle to off. Disabling this ensures that, regardless of whether the repository receives a new commit, each pipeline run updates the last successful build artifacts on the S3 bucket. This can help in situations where the contents of the S3 bucket have been manually modified and you need to restore them to the last successful build state.

Now, you can click the Save and run button and watch as the pipeline tests and builds your code before deploying it to S3.

You can watch as the various steps of the pipeline run in order to build, test, and deploy your React app:

Optimized pipeline run in progress

You can explore the execution logs of each of the jobs and their steps by clicking the job and then navigating through the logs in the right pane. Once the build completes running, here’s what the page will look like:

Successful test results of the optimized pipeline

You can now try pushing a new commit to your forked GitHub repo and watch the pipeline get triggered automatically. Within minutes, it will test, build, and deploy your updated React website to the S3 bucket!

Conclusion

This tutorial explained how to set up an automated deployment pipeline with TeamCity to deploy a React app from GitHub to an Amazon S3 bucket. This process eliminates manual intervention, ensuring your app is deployed faster, more reliably, and with consistent results.

Automating your React app deployment not only saves time but also minimizes errors and improves overall development workflows, allowing your team to focus on delivering value to users.

With tools like Build Approval, TeamCity allows you to add an extra layer of security by ensuring that only authorized builds are deployed to production. Meanwhile, as you saw in the tutorial, build chains (or pipelines) enable you to manage complex workflows by breaking them into smaller, interdependent tasks, making large projects easier to handle.

These features, combined with TeamCity’s comprehensive reporting and automation capabilities, make it a powerful choice for your deployment needs. Make sure to give TeamCity a try for your next React app project!

What Is DAST? A Guide to Dynamic Application Security Testing

JetBrains TeamCity — Thu, 06 Feb 2025 12:14:15 +0000

This post was brought to you by Matt Keib, draft.dev.

Dynamic application security testing (DAST) is a security testing method designed to identify vulnerabilities in applications while running. Unlike static testing methods, which analyze code at rest, DAST interacts with live applications and mimics real-world attacks to uncover security flaws. This makes DAST particularly effective for detecting issues that only occur when an application is running.

Within the bigger picture of security testing methodologies, DAST stands out for targeting the operational state of applications and APIs. Unlike static application security testing (SAST), which analyzes source code, or interactive application security testing (IAST), which combines runtime testing with code analysis, DAST focuses on identifying vulnerabilities that arise only during application execution.

Thanks to its ability to simulate real-world attacks on live web applications and APIs, DAST can help uncover issues like authentication flaws, configuration errors, and runtime vulnerabilities, which are weaknesses often missed by static methods. This makes DAST an essential component for securing applications in production environments and protecting them against external threats.

In this article, you’ll learn about DAST fundamentals, when it’s appropriate to use it, and how TeamCity can help you maximize its effectiveness.

What DAST is and how it works

DAST’s primary function is to assess live applications from an attacker’s perspective to identify runtime vulnerabilities. DAST uses an outside-in (or black box) testing approach, which evaluates applications in their runtime environment without resorting to accessing source code or internal details.

Unlike other application security tools that analyze internal structures, DAST mimics external inputs by malicious actors and examines how the application responds, providing unique insights into potential vulnerabilities. In other words, the tool attempts to perform the very same exploits an attacker would in a real attack situation.

One of the defining characteristics of DAST tools is their ability to provide actionable insights, including comprehensive reports detailing vulnerabilities, potential impacts, and remediation recommendations.

These tools not only identify flaws, misconfigurations, and issues in a running application but also provide recommendations on how to properly address them. This is especially important today, as developers must balance tight deadlines with meeting compliance standards and implementing guidelines that safeguard customer data from breaches and unauthorized access.

DAST vs. other types of security testing

While DAST excels at runtime testing, other methods like SAST and IAST address different stages of the development lifecycle. SAST analyzes source code before execution, helping developers catch issues early during coding (this approach is also known as shift-left).

On the other hand, IAST combines elements of both static and dynamic testing, offering insights by monitoring the application during runtime while also linking findings to the specific code that raised concerns. This helps developers reduce overhead while focusing on the faulty segments they must fix to remediate active vulnerabilities.

Unlike SAST, which focuses on internal code structure, DAST evaluates the application from an external perspective, making it indispensable for detecting vulnerabilities that arise only during execution.

For example, a runtime vulnerability like exposing sensitive data in API responses or failing to invalidate a session after logout could go unnoticed in static analysis but would be detected by DAST.

Compared to IAST, DAST is less invasive, as it doesn’t require integration into the application’s codebase or runtime environment, nor does it involve deploying sensors into the source code.

This simplicity makes DAST easier to implement and allows teams to switch between DAST tools with minimal friction if needed.

To enhance security, DAST should be combined with SAST for broader visibility into risks. Modern DAST tools can seamlessly integrate into DevOps and CI/CD pipelines, enabling early-stage vulnerability mitigation through a shift-left approach.

Key features of DAST tools

DAST tools provide a range of features designed to streamline and enhance the security testing process, including:

Scanning and mapping: DAST tools initiate the testing process that simulates user interactions with the application. They send HTTP requests to map out all pages, links, functions, and entry points, including those defined in API documents. This step ensures thorough coverage of the application by uncovering all potential flaws that may open a door to attackers.
Analysis of application responses: The tool then examines the application’s responses, looking for anomalies, error messages, and unexpected behavior that could indicate vulnerabilities. Identified issues, such as misconfigurations or runtime errors, are recorded for further review.
Attack simulation: At this stage, DAST simulates common attacks like SQL injection, cross-site scripting (XSS), and cross-site request forgery (CSRF) to detect vulnerabilities. This process highlights issues such as input validation flaws, authentication errors, and data exposure risks that malicious actors could exploit.
Comprehensive reporting: After scanning and analysis, DAST tools generate detailed reports that outline detected vulnerabilities together with their severity, as well as suggested remediation steps. These reports guide development and security teams in prioritizing fixes timely and effectively.This is very useful for complying with the respective SLAs of security teams, ensuring a swift resolution of incidents and improving the organization’s overall security posture. Additionally, granular reports are useful for internal communications such as those with the board or stakeholders, where security teams provide updates about the application’s security posture.
Potential false positives: To ensure accuracy, modern DAST tools include mechanisms to minimize false positives. However, human validation and prioritization are often necessary to confirm the relevance and impact of flagged vulnerabilities.

These features enable teams to proactively identify and address potential security flaws, minimizing risks and enhancing overall application security.

Best practices for effective DAST

Implementing DAST effectively requires a strategy that prioritizes regular and comprehensive scanning to ensure consistent identification of risks and vulnerabilities.

Regular scans help you stay ahead of emerging threats and verify that updates or new features of the application don’t introduce security gaps.

In-depth coverage should include all aspects of the application, including APIs and single-page functionalities, to ensure no critical vulnerabilities remain hidden.

Scheduling scans at regular intervals or after significant updates will contribute to consistent monitoring and reduce the chances of overlooking new vulnerabilities or entry points for attackers.

Combining DAST with other application security testing tools

Integrating DAST with other security testing efforts enhances an organization’s security posture by addressing vulnerabilities at different stages of the software development lifecycle.

Combining DAST with SAST allows teams to identify both static and runtime vulnerabilities, ensuring a more detailed risk assessment. While SAST analyzes the application’s source code for structural weaknesses, DAST focuses on vulnerabilities that emerge during runtime, offering visibility on vulnerabilities that may have slipped into production unnoticed.

Including IAST in the mix bridges the gap between static and dynamic analysis by providing insights during application execution and linking vulnerabilities to specific sections of the code.

Manual testing adds further depth in identifying context-specific risks or logic flaws that automated tools might overlook. Together, these methods create a layered security approach that reduces the likelihood of missed vulnerabilities.

For example, consider an e-commerce application that processes user payments. SAST identifies a hard-coded API key in the source code, which is flagged as a potential security risk early in development.

Later, during runtime testing, DAST detects that the same API key is being transmitted in plaintext over an insecure connection, exposing it to interception. Meanwhile, IAST links the issue to the specific code responsible for establishing the insecure connection, enabling the team to pinpoint and fix the root cause quickly.

Manual testing then verifies the fix and ensures no other related vulnerabilities exist, providing confidence in the resolution.

A layered strategy not only maximizes coverage but also simplifies remediation efforts. For example, early-stage issues detected by SAST can be resolved before reaching runtime, reducing the scope of vulnerabilities flagged during DAST scans.

Similarly, manual testing can validate and prioritize findings from automated tools and narrow down findings through a human-based dismissal of false positives, ensuring that resources are focused on the most critical risks.

Handling false positives and false negatives

First-generation DAST tools were often criticized for generating a significant number of false positives, creating extra workload for developers and security teams. In addition, they lacked automation capabilities and complete vulnerability validation.

Current DAST tools can integrate with others like SAST, which helps confirm findings and reduce noise through the combination of different testing methods. Modern versions of DAST tools can cross-reference results, minimizing both false positives and false negatives, reducing overhead, and ensuring issues are detected.

Additionally, modern DAST tools use advanced techniques like machine learning to improve accuracy. For instance, machine learning algorithms can analyze patterns in past scans to better distinguish legitimate vulnerabilities from false positives and continuously refine their detection capabilities.

Other techniques, such as context-aware analysis and correlation with real-world attack patterns, further enhance their ability to detect meaningful issues while reducing overhead.

Other techniques for managing false positives include tuning DAST configurations to match application-specific requirements and using predefined rule sets to exclude known, safe, and expected behaviors. Regularly updating vulnerability libraries ensures the tool accurately identifies real threats while reducing the likelihood of false positives caused by outdated patterns.

False negatives can be resolved using manual reviews with DAST or combining it with IAST, which links detected vulnerabilities to specific code components for deeper insights.

This way, organizations can ensure that DAST results are more accurate and actionable, and security teams have better data to prioritize and remediate vulnerabilities effectively.

How TeamCity can help

TeamCity is a CI/CD solution that integrates DAST into CI/CD pipelines, embedding security testing directly into the development workflow. With its robust support for a wide range of languages and technologies, TeamCity ensures that projects of any size – whether they involve small teams or enterprise-scale applications – can seamlessly integrate DAST tools into their workflows.

Automating builds, tests, and deployments allows TeamCity to embed DAST scans at key stages of the software development lifecycle. Its integration capabilities extend to popular DAST tools, ensuring security testing runs alongside other CI/CD activities without disrupting productivity.

The platform’s detailed build logs and visual build chains provide insights into test results, making it easier for developers to pinpoint and address security vulnerabilities early.

TeamCity reinforces a shift-left approach to security, ensuring testing is integrated earlier in the development cycle. Automated testing features, such as test parallelization and flaky test detection, help developers maintain high-quality code while uncovering potential runtime vulnerabilities. With its emphasis on automation, scalability, and user-friendly integrations, TeamCity empowers teams to build secure and reliable software with confidence.

Conclusion

Identifying vulnerabilities during runtime allows DAST to address gaps left by other testing methods for thorough coverage of potential security flaws. Combining DAST with tools like SAST and IAST creates a layered security approach that maximizes visibility and optimizes remediation efforts.

Before adopting DAST, evaluate your current security practices to identify areas where it can make the most significant impact, such as detecting runtime vulnerabilities in environments with third-party dependencies, rapidly changing codebases, and tight release schedules tracked through DevOps metrics.

Once gaps are identified, the next step is to select a DAST solution that aligns with your team’s workflows and objectives. Choose a tool that integrates seamlessly into your development pipeline, allowing for regular scans and effective analysis of results.

Incorporating these findings into your workflow ensures vulnerabilities are prioritized and resolved efficiently, reducing risks, enhancing productivity, and maintaining the speed and agility demanded by modern development practices.

TeamCity is a reliable solution that can support these efforts by offering seamless integration of DAST tools into CI/CD pipelines. Its complete set of features, such as automated testing and detailed build insights, empowers teams to detect and remediate vulnerabilities efficiently. With TeamCity, organizations can strengthen their security posture without compromising development speed or agility.

Introducing the New TeamCity Plugin for IntelliJ IDEA

JetBrains TeamCity — Thu, 12 Sep 2024 11:57:49 +0000

We’re excited to announce the release of the updated TeamCity plugin for IntelliJ IDEA! 🎉 You can now download it directly from JetBrains Marketplace.

Using the plugin, you can trigger TeamCity builds directly from within your IDE and test any changes before committing them to the version control system.

Why get the new plugin?

This plugin has been built from the ground up to ensure it will eventually be able to replace the existing TeamCity plugin once support for the most frequently used and requested features has been added.

Here’s what’s new in the plugin:

We’ve added functionality enabling you to link TeamCity projects and build configurations to your IDE project so that you only see build configurations related to your IDE project.

With the help of the remote run feature, you can run build configurations on your local changes without committing them to the VCS.
The plugin’s tool window now contains a new Personal Builds tab where past personal builds are listed. It also shows live updates of all builds executed using the remote run feature.

Now it’s possible to select a build configuration and watch its build status for each commit in the VCS Log tool window.

Key benefits of this updated plugin include:

The ability to manually configure which TeamCity projects relate to your code, giving you more control over your builds.
Enhanced performance that significantly reduces lag between your actions in the IDE and the TeamCity server's response.

We're actively developing this plugin and planning to add even more features in upcoming releases. Your feedback is critical in shaping the tool to better meet the needs of IntelliJ IDEA developers.

You can install both the old and new plugin versions side by side, so feel free to compare and explore!

How to get started with the TeamCity plugin for IntelliJ IDEA

Initial setup

1. Download the plugin from Marketplace.

2. Once the plugin is installed, open your project in IntelliJ IDEA and invoke the plugin’s settings using the Tools | TeamCity (Experimental) | Settings… menu.

3. Click Log In and enter the following values:

Server URL – the HTTP(S) address of your TeamCity server.
Access token – your user access token that can be generated on the Your Profile | Access Tokens page in TeamCity.

With the new plugin, you can link build configurations from TeamCity directly to the project you have open. In the old plugin, this had to be configured through VCS roots, which wasn’t an easy process.

Now, users only need to create a given configuration once, and it will be saved in the source code. Everyone who downloads the project will then have it automatically configured and available without the need to set it up themselves.

Testing your local changes

One of the key benefits of the TeamCity IDEA plugins (both old and new) is the ability to run builds with your local changes before they are pushed to a remote branch, also known as a remote run. This allows you to spot issues without breaking the build for everyone else on your team.

Here’s how you can initiate a remote run from your IDE.

1. Make some changes to your code.

2. Go to Tools | TeamCity (Experimental) | Remote Run….

3. Then, under Remote Run… | Settings…, click the target build configurations that you want to run with your local changes. The plugin will then remember your choice and run builds for the same configuration(s) on subsequent remote runs. You can configure these project-configuration relations in the plugin settings.

Link your projects to TeamCity build configurations

Setting up project-configuration relations allows you to explicitly choose which configurations should be triggered depending on the introduced changes.

TeamCity’s IntelliJ IDEA integration enables you to choose the linking scope, selecting whether you want to link the whole project or only individual project modules to your TeamCity build configurations.

1. Click Tools | TeamCity (Experimental) | Settings… to open the plugin’s settings.

2. Choose the required Linking scope value:

PROJECT – allows you to link the entire IntelliJ IDEA project to the target build configuration(s). This option works best when you need to trigger builds of the same configuration(s) regardless of which part of your code changed.
MODULE – allows you to link individual modules to corresponding build configurations. For example, you can run both Build and Test configurations if the main module of your application changes, and only the Test configuration if you edit a separate module with unit and functional tests. This mode also benefits mono repositories where each module is a separate project with its own target build configuration(s).

Share your feedback

We’re still working on making the new plugin ready to replace the old one. For the time being, you can download both plugins – they won’t interfere with each other.

Is there any functionality that you’d like us to add to the new plugin? Let us know in the comments below! We want to make the plugin as useful as possible, and your feedback can help us do exactly that.

Happy building!

Introducing the Unreal Engine Plugin for JetBrains TeamCity

JetBrains TeamCity — Tue, 14 May 2024 14:40:51 +0000

TeamCity is a popular choice in game development for several reasons, including its support for Perforce, its ability to be installed on premises, and its numerous configuration options.

In addition to its main application as a general-purpose CI/CD solution, we’re also striving to bring dedicated support for various build tools, too. Having already provided a Unity plugin for several years, we're now excited to officially introduce our Unreal Engine Support plugin!

We’ve been in close contact with several of our game development customers throughout the development of this new plugin to ensure it meets the needs of DevOps teams working on real-world Unreal Engine projects.

Setting up a proper build pipeline for an Unreal Engine game can be a daunting task, especially for those with limited experience of the platform’s unique peculiarities. In this blog post, we'll walk you through a basic setup, while showcasing the plugin's features and demonstrating advanced capabilities such as distributed BuildGraph support.

A sample build pipeline generated by the new Unreal Engine plugin

Key features

Here's a quick overview of what you can expect from this version of the plugin:

A dedicated runner tailored for most common use cases (BuildCookRun, BuildGraph, and automation tests).
Build distribution based on your BuildGraph description.
On-the-fly automation test reporting.
Automatic discovery of Unreal Engine installations on the build machines.
Compatibility with the latest 5.x versions of Unreal Engine, plus support for 4.x.

The demo

In this demo, we’ll be setting up a pipeline for two starter games shipped with Unreal: Cropout and Lyra. We’ll use AWS infrastructure (EC2 and FSx for OpenZFS) for build agents and TeamCity Cloud with the Unreal Engine plugin installed.

We’ll look at two scenarios: when the engine is already installed on an agent and when it’s built from source along with the game. At the moment, TeamCity Cloud doesn’t offer agents with the preinstalled Unreal Engine, but you can always add your own self-hosted agent with all the required SDKs. This is the approach we’ll stick to in this blog post.

Building Cropout with BuildCookRun

Building an Unreal project usually involves many steps, such as:

Compilation for the target platforms in the specified configurations.
Asset cooking (converting all of the assets into ones that can be read on the target platforms).
Packaging of the project into a proper distribution format.
And many more.

Then, of course, there’s the testing!

To begin with, let’s start with a simple scenario and run all of these stages sequentially on a single machine utilizing the standard BuildCookRun command.

For this scenario, we’ll use the latest vanilla Unreal Engine version available at the moment of writing, installed from EGL (Epic Games Launcher).

After successfully connecting the agent to our TeamCity Cloud server, we can see it in the Agents tab.

At this point, I’d like to take a moment to briefly explore some of the agent’s properties.

Here, looking at the picture above, we see that the agent discovered the engine and its version (we have more than one on our machine, as you can see). Keep a note of this information, as we’ll need it later for a build step configuration.

Nowadays, the common approach is to store all your configs as code under source control (i.e. configuration as code). In TeamCity, you can do this using Kotlin DSL. Of course, you can also go with the traditional UI configuration, but today, we’ll use the first approach (due to the popularity of YAML and the fact that it has become a de-facto standard, we’ve added it as a part of our recent release of TeamCity Pipelines – check it out if you haven’t already).

The Kotlin DSL configuration for building Cropout in TeamCity looks like this:

unrealEngine {
    engineDetectionMode = automatic {
        identifier = "5.4"
    }
    command = buildCookRun {
        project = "cropout/CropoutSampleProject.uproject"
        buildConfiguration = standaloneGame {
            configurations = "Development+Shipping"
            platforms = "Mac"
        }
        cook = cookConfiguration {
            maps = "Village+MainMenu"
            cultures = "en"
            unversionedContent = true
        }
        stage = stageConfiguration {
            directory = "./staged"
        }
        archive = archiveConfiguration {
            directory = "./archived"
        }
        pak = true
        compressed = true
        prerequisites = true
    }
    additionalArguments = "-utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign"
}

All of the settings are pretty self-descriptive, but there are a couple worth paying closer attention to:

engine detection mode

Here, we can specify either “automatic” or “manual”. The former assumes you already have the engine installed on your agent and registered in the system.

Let me take this opportunity to clarify what “registered” means: Whenever you install the Unreal Engine from EGL or build it from source, it writes to certain files on the target machine (or registry in the case of Windows). More information on that can be found here.

This is essentially what the automatic detection mode is for. We read those files and publish identifiers as agent properties, so you can use them later to select the proper agent for your build.

The “manual” mode allows you to set the exact path to the Unreal Engine root folder, which might be useful when you build from source. We'll touch on that later.

build configuration

By adjusting this parameter, we can specify the type of build we're creating, whether it's a standalone game, client, server, or both client and server components.

Depending on the selected value, the plugin will apply -client, -server, -noserver flags and manage -clientconfig, -serverconfig, -config, -targetplatform, and -servertargetplatform parameters accordingly.

So far, so good. Now, it’s time to add some automation tests and run the final pipeline.

unrealEngine {
    engineDetectionMode = automatic {
        identifier = "5.4"
    }
    command = runAutomation {
        project = "cropout/CropoutSampleProject.uproject"
        execCommand = runTests {
            tests = """
                StartsWith:JsonConfig
                Input.Triggers.Released
            """.trimIndent()
        }
        nullRHI = true
    }
    additionalArguments = "-utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign"
}

As you may know, when working with the Unreal Engine Automation Framework, tests are usually executed using one of the following automation “subcommands”:

RunAll – this is a pretty straightforward command that has the ability to run all required tests.
RunFilter – with this command, tests tagged with one of the specified filters (which includes such values as Engine, Stress, Smoke, etc.) can be executed.
RunTests – this is the most versatile command, as it allows you to specify the list of tests to run (including prefix filters with StartsWith, run group filters, and simple substring matches).

We tried to preserve the same wording in Kotlin DSL to keep it familiar for Unreal folks.

It's also worth mentioning that the plugin parses the results of the tests on the fly and presents them in a nicely formatted manner native to TeamCity.

Here’s what the result of the build looks like:

Below, we can see the game binaries published to the Artifacts Storage. By default, artifacts are published to built-in storage; however, TeamCity also supports various external storage options. In the context of the cloud, it makes sense to configure publishing to S3.

The final configuration in the UI would then look like this:

And here’s the step with the test run:

All the settings are disabled since we've enabled Kotlin DSL configuration and disabled editing it in the UI.

Building Lyra with BuildGraph

We're slowly moving towards a more complex example with Lyra. Since it's a multiplayer game, let's build a Linux server (as we know that everything high-performance should be run on Linux!), along with Windows and Linux game clients.

In the previous example, we ran all the steps sequentially on a single machine. Usually, this takes a significant amount of time. But there’s a better way, and it's called BuildGraph. This is Unreal's method for describing your build declaratively (or almost) as a set of tasks with dependencies among them. The different parts of the graph can then be executed together or split across different machines. The latter approach allows you to dramatically speed up the entire build process, especially for a big and complex project.

The nice thing about BuildGraph is it manages all of the intermediate artifacts between jobs automatically. All you need is shared storage. While building Lyra, we’ll be using this shared storage for two purposes: sharing data between nodes within a single build and maintaining a shared derived data cache (DDC).

As we mentioned earlier, we’ll also build the engine from the source. There are multiple reasons why you might want to do that, but in our specific case, it’s because there’s no way to build a Linux server from the engine installed from EGL (at least at the time this post was written). This is due to the lack of files and SDKs provided with this particular engine version.

Okay, that’s enough preamble – let’s get to grips with the code.

The structure of the project and corresponding streams in Perforce look like this (we assume that you already have the Unreal Engine source code from Epic’s Perforce):

Here’s the excerpt of the BuildGraph XML script that we’ll use for our build. This particular part of the script shows how we’ll build the editor along with the tools required for the compilation. It also contains a few tests, too. In our simple example, these are hardcoded. In a real-world scenario, you’d likely pass a list of tests to the script and employ more complicated logic.

...
<!-- Editors -->
  <ForEach Name="Platform" Values="$(EditorPlatforms)" Separator="+">
    <Property Name="Compiler" Value="$(AgentPrefixCompile)$(Platform)" />

    <Agent Name="Build Editor and tools $(Platform)" Type="$(Compiler)">
            
            ...

      <Property Name="ExtraToolCompileArguments" Value="$(ExtraToolCompileArguments) -architecture=arm64" If="'$(Platform)' == 'Mac'" />
      
      <Node Name="$(ToolsNodeName) $(Platform)" Requires="Setup Toolchain $(Platform)" Produces="#$(Platform)ToolBinaries">
        <Compile Target="CrashReportClient" Platform="$(Platform)" Configuration="Shipping" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="CrashReportClientEditor" Platform="$(Platform)" Configuration="Shipping" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="ShaderCompileWorker" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="UnrealLightmass" Platform="$(Platform)" Configuration="Development" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="InterchangeWorker" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="UnrealPak" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries"/>
        <Compile Target="BootstrapPackagedGame" Platform="$(Platform)" Configuration="Shipping" Arguments="$(ExtraToolCompileArguments)" Tag="#$(Platform)ToolBinaries" If="'$(Platform)' == 'Win64'"/>
      </Node>
      
      ...
      
      <Node Name="$(EditorNodeName) $(Platform)" Requires="$(ToolsNodeName) $(Platform)" Produces="#$(Platform)EditorBinaries">
        <Compile Target="$(ProjectName)Editor" Project="$(UProject)" Platform="$(Platform)" Configuration="Development" Arguments="$(ExtraEditorCompileArguments)" Tag="#$(Platform)EditorBinaries"/>
      </Node>

            <Property Name="AutomationTestsNodeName" Value="Run Tests $(Platform)" />
      <Node Name="$(AutomationTestsNodeName)" Requires="$(EditorNodeName) $(Platform)">
        <Property Name="TestArgs" Value="-Project=$(UProject) -NullRHI -Deterministic" />
        <Property Name="TestArgs" Value="$(TestArgs) -Test=UE.EditorAutomation -RunTest=&quot;StartsWith:Input&quot;" />
        <Property Name="TestArgs" Value="$(TestArgs) -Build=Editor -UseEditor" />
        <Command Name="RunUnreal" Arguments="$(TestArgs)" />
      </Node>
    </Agent>

    <Property Name="BuildNodes" Value="$(BuildNodes);$(EditorNodeName) $(Platform);$(AutomationTestsNodeName);" />
  </ForEach>
  
  ...

The full syntax is described in full on Epic’s website. Here, what we’re essentially doing is iterating over the list of platforms passed to the script via an option EditorPlatforms and building the Editor for each. One interesting feature of this part of the code is the Type property of the agent node.

<Agent ... Type="">

See this table from the documentation:

So, in order to run a set of tasks on a particular agent, you can manipulate this field. At the time of writing, to make everything work, this part requires a little configuration of your build agents. Namely, you should define two properties in the agent’s configuration file:

unreal-engine.build-graph.agent.type: This could be any value (or a list of values separated by ;). The corresponding set of tasks will then be run only on an agent with at least one match.
unreal-engine.build-graph.agent.shared-dir: As discussed earlier, in order to run distributed BuildGraph builds, we need a shared storage location. With this property, we set a path to this storage location on a specific agent.

This is what our setup for our EC2 build agents looks like:

For Linux:

unreal-engine.build-graph.agent.type = CompileLinux;CookLinux;Linux
unreal-engine.build-graph.agent.shared-dir = /mnt/agent-shared-dir/intermediate

And for Windows (for some reason, mounting the folder as a separate drive caused one of the Windows system calls to fail, so we opted for a network share in the configuration):

unreal-engine.build-graph.agent.type = CompileWin64;CookWin64;Win64
unreal-engine.build-graph.agent.shared-dir = \\\\fs-040b8d6dab476baf1.fsx.eu-west-1.amazonaws.com\\fsx\\

For real-world scenarios, it might make sense to differentiate between agents that perform cooking and those that perform compilation and ensure they have the appropriate specs.

Let’s quickly take a look at what the processes of compilation, cooking and packaging of the game look like. In our script, we’ve separated the logic for building a client and building a server (as they differ in terms of passed flags):

<ForEach Name="Platform" Values="$(ClientPlatforms)" Separator="+">
    <!-- COMPILATION -->
    <Property Name="Compiler" Value="$(AgentPrefixCompile)$(Platform)" />
    <Agent Name="Compile $(Platform) Client" Type="$(Compiler)">

      <Property Name="CompileNodeName" Value="Compile $(Platform) Client" />  
      <Node Name="$(CompileNodeName)" Requires="$(ToolsNodeName) $(Platform)" Produces="#$(Platform) Client Binaries">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">    
          <Compile Target="$(ProjectName)Client" Project="$(UProject)" Platform="$(Platform)" Configuration="$(TargetConfiguration)" Arguments="$(ExtraProjectCompileArguments)" />
        </ForEach>
      </Node>

      <Property Name="BuildNodes" Value="$(BuildNodes);$(CompileNodeName);" />
    </Agent>

    <!-- COOKING -->
    <Property Name="Cooker" Value="$(AgentPrefixCook)$(Platform)" />

    <Property Name="CookPlatformNodeName" Value="Cook $(Platform) Client" />
    <Agent Name="Cook $(Platform) Client" Type="$(Cooker)">
      <Property Name="CookPlatform" Value="$(Platform)" />
      <Property Name="CookPlatform" Value="Windows" If="'$(Platform)' == 'Win64'" />

      <Node Name="$(CookPlatformNodeName)" Requires="$(EditorNodeName) $(Platform)" Produces="#Cook $(Platform) Client Complete">
        <Cook Project="$(UProject)" Platform="$(CookPlatform)Client"/>
      </Node>
    </Agent>

    <Property Name="BuildNodes" Value="$(BuildNodes);$(CookPlatformNodeName);" />

    <!-- PACKAGING -->
    <Agent Name="Package $(Platform) Client" Type="$(Platform)">
      <Property Name="BCRArgs" Value="-Project='$(UProject)' -Platform=$(Platform) -NoCodeSign -Client" />

        <!-- Stage -->
      <Node Name="Stage $(Platform) Client" Requires="Compile $(Platform) Client;Cook $(Platform) Client">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">
          <Command Name="BuildCookRun" Arguments="$(BCRArgs) -Configuration=$(TargetConfiguration) -SkipBuild -SkipCook -Stage -Pak" />
        </ForEach>
      </Node>

      <!-- Package -->
      <Node Name="Package $(Platform) Client" Requires="Stage $(Platform) Client">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">    
          <Command Name="BuildCookRun" Arguments="$(BCRArgs) -Configuration=$(TargetConfiguration) -SkipBuild -SkipCook -SkipStage -Package" />
        </ForEach>
      </Node>

      <!-- Publish (Packages) -->
      <Node Name="Archive $(Platform) Client" Requires="Package $(Platform) Client">
        <ForEach Name="TargetConfiguration" Values="$(TargetConfigurations)" Separator="+">    
          <Command Name="BuildCookRun" Arguments="$(BCRArgs) -Configuration=$(TargetConfiguration) -SkipBuild -SkipCook -SkipStage -SkipPak -SkipPackage -Archive" />
        </ForEach>
      </Node>

      <Node Name="Publish $(Platform) Client" Requires="Archive $(Platform) Client">
        <Property Name="PublishPlatform" Value="$(Platform)" />
        <Property Name="PublishPlatform" Value="Windows" If="'$(Platform)' == 'Win64'" />
        <Log Message="##teamcity[publishArtifacts 'game/ArchivedBuilds/$(PublishPlatform)Client=>$(PublishPlatform)Client.zip']" />
      </Node>
    </Agent>

    <Property Name="BuildNodes" Value="$(BuildNodes);Archive $(Platform) Client;Publish $(Platform) Client" />
  </ForEach>

The script is pretty self-explanatory. We iterate over the list of platforms on which we want our client to run, which is again passed as an option. We compile for each of the platforms, cook the assets, and finally package everything. As the last step of the packaging process, we use a service message to publish the built client as a TeamCity build artifact.

As you might have noticed, in this part of the script, we have three agents representing three stages of the build process (compilation, cooking, and packaging). The compilation requires tools belonging to the editor, but not the editor itself:

<Node Name="$(CompileNodeName)" Requires="$(ToolsNodeName) $(Platform)" Produces="#$(Platform) Client Binaries">

Meanwhile, the cooking process requires the actual editor:

<Node Name="$(CookPlatformNodeName)" Requires="$(EditorNodeName) $(Platform)" Produces="#Cook $(Platform) Client Complete">

And since there is no dependency between these two processes, they can run in parallel on different machines (if you have enough machines available). This simple example demonstrates the power of using BuildGraph: You simply describe pieces of work that share dependencies, and then they run simultaneously.

In the interest of brevity, we won’t show the rest of the script that includes the server part, as it looks very similar to what we just described, with only the set of flags differing.

Finally, we've reached the part with the plugin configuration in TeamCity. Since most of the work is described in the BuildGraph XML script, the DSL configuration is fairly brief:

params {
    param("env.UE_SharedDataCachePath", "/mnt/agent-shared-dir/ddc")
    param("env.UE-SharedDataCachePath", "\\\\fs-040b8d6dab476baf1.fsx.eu-west-1.amazonaws.com\\fsx\\ddc")
}

steps {
    unrealEngine {
        id = "Unreal_Engine"
        name = "Build"
        engineDetectionMode = manual {
            rootDir = "engine"
        }
        command = buildGraph {
            script = "game/BuildProject.xml"
            targetNode = "BuildProject"
            options = """
                ProjectPath=%teamcity.build.checkoutDir%/game
                ProjectName=Lyra
                ClientPlatforms=Linux+Win64
                ServerPlatforms=Linux
                EditorPlatforms=Linux+Win64
                TargetConfigurations=Shipping
            """.trimIndent()
            mode = UnrealEngine.BuildGraphMode.Distributed
        }
        additionalArguments = "-utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign"
    }
}

Here, we specify that we’d like to run the given BuildGraph script in distributed mode. Nevertheless, there’s always an option to run all the nodes sequentially on a single machine if we want to. We’ve also specified a couple of environment variables that are required for the engine to enable the usage of a shared DDC. Those folders should already be mounted to the agents connected to TeamCity.

Now, we can pass the following list of three platform options for our game: ClientPlatforms, ServerPlatforms, and EditorPlatforms.

In the UI, it looks like this:

When we launch the build, we get this build chain:

So, as we can see, the plugin has transformed the graph into a proper TeamCity build chain.

We can check the published artifacts on the corresponding tab of the specific build. Here's what it looks like for the Linux server:

Before finishing up, we'd like to emphasize some important points to bear in mind:

In order for the distribution process of BuildGraph to work, your build configuration should contain exactly one active build step.
Currently, the plugin does not manage the retention period of the produced artifacts in the shared storage in any way. It's your responsibility to set it up properly.
Examples in this blog post are by no means production-ready. Your real-world scenarios will likely be more complex. However, they could serve as a good starting point.

Try it out!

You can get the demo code shown in this blog post from the GitHub repository.

If you’d like to try out the Unreal Engine plugin for TeamCity, you can download it from JetBrains Marketplace and install it on a TeamCity On-Premises server.

For TeamCity Cloud users, the Unreal Engine plugin has been pre-installed, allowing you to use it simply by adding an Unreal Engine build step. As a reminder, TeamCity Cloud agents don’t have Unreal Engine pre-installed, so you will need to use a self-hosted agent to run an Unreal Engine build.

An important note for those who have been using a preview version of the plugin (any version that starts with 0.x.x): Please take into account that your existing configurations will be broken, as we have made changes and redesigned several features in the 1.0.0 version. You may need to recreate those configurations using the new version of the plugin.

What’s next?

First off, we’re looking forward to receiving your feedback. Please feel free to get in touch by submitting a ticket via our issue tracker or by commenting on this blog post.

There are also some ideas we have in mind for further work, including:

Providing a more in-depth build log analysis.
Including integration with UnrealGameSync (UGS), with features like build status publication and the potential provision of an out-of-the-box metadata server.
Adding detection of the SDKs available on build machines, possibly using Turnkey.
Looking into Gauntlet Automation Framework and what we can do there.
Looking into how we can leverage recent advancements in the build process introduced by Epic Games, namely, checking out Unreal Build Accelerator (UBA) and coordinating it on the agents from TeamCity.
Open-sourcing the plugin. We believe that this will increase transparency and allow for the creation of a better plugin overall. Furthermore, going open source can be a lot of fun.
Anything else you can think of! If you have an idea of something that would be nice to have, please feel free to reach out via any of the channels we mentioned above or simply by leaving a comment on this blog post.

Make CI/CD Part of Your Development Flow With TeamCity Pipelines

JetBrains TeamCity — Mon, 18 Mar 2024 11:28:35 +0000

Not many developers love configuring their CI/CD pipelines. Describing jobs, defining job dependencies, debugging failed runs… If only there was an easier way to do it all!

As the creators of TeamCity, a powerful and mature CI/CD tool that’s been around for over 17 years, we understand the struggle like no one else. Over the years, we’ve heard our customers praise TeamCity for how potent the tool is at handling pipelines of any complexity. Yet we also heard that the learning curve might be too steep for smaller teams.

That’s why we’re launching TeamCity Pipelines, a tool with a brand-new approach to CI/CD. TeamCity Pipelines reimagines the CI/CD process with its intuitive interface and smart configuration assistance, with JetBrains’ signature intelligence under the hood.

TeamCity Pipelines is engineered to streamline your development flow, helping you accomplish tasks faster and run your CI/CD pipelines more efficiently.

With TeamCity Pipelines, there’s no tradeoff between the simplicity of configuration and the powerful server capable of building complex pipelines.

Let’s see what TeamCity Pipelines is about and how it can make your developers’ lives easier.

Visual Pipeline Editor

In TeamCity Pipelines, we rethink the concept of pipeline configuration. Now, you can use YAML to configure your pipelines as code or take advantage of the visual PipelineEditor. Here, you can easily define commands and dependencies.

Intelligent configuration assistance

TeamCity Pipelines will guide you through the pipeline configuration process, providing you with smart improvement suggestions.

For instance, TeamCity Pipelines can automatically detect specific build tools and suggest smart test parallelization options based on that, bringing you up to 75% faster runtime.

Run your pipelines in any environment

In TeamCity Pipelines, you can choose what type of agent you want to run your job on: Linux, Windows, or macOS.

If your build requires extra tools, you can install them additionally on agents or run your jobs in a Docker container. TeamCity Pipelines features a built-in Docker image search and Dockerfile support.

Need to debug a failed build? Thanks to the Open terminal feature, you can open the terminal and start debugging specific agent issues right from the TeamCity UI.

Smart pipeline optimization

TeamCity Pipelines can also optimize your pipelines on the fly. Choose the option to reuse jobs, parallelize tests, or use build caches. A faster CI/CD experience has landed!

Configure pipelines as code with YAML

Complying with the industry standard, we offer TeamCity Pipelines users the opportunity to configure their pipelines as code by using YAML.

You can easily switch between the YAML and UI configurations.

We hope that TeamCity Pipelines will prove to be a game-changer for your development process. Today, we’ve also launched it on Product Hunt.

We’d really appreciate your support on this big day! If you feel like it, please support TeamCity Pipelines on Product Hunt and give us your valuable feedback.

Yours truly,

TeamCity Pipelines crew

Configuration as Code for TeamCity Using Terraform

JetBrains TeamCity — Thu, 15 Feb 2024 11:06:05 +0000

In DevOps, development and infrastructure tasks are split. Developers and build engineers handle project configuration and scripting, while DevOps engineers on the infrastructure side manage things like TeamCity instances, database maintenance, and permissions.

Although the Kotlin DSL can address CI/CD needs at the project level, it lacks system administration capabilities. It is incapable of setting universal server configurations and managing users, which poses challenges for DevOps engineers, especially those from a systems administration background.

To overcome this, TeamCity has introduced the Terraform Provider for TeamCity on the basis of this widely adopted infrastructure provisioning tool. The Terraform Provider uses a language familiar to DevOps professionals and streamlines global server setup, user management, permissions, and project initiation.

Source: Developer Ecosystem Report 2023

Terraform Provider in action: Setting up OAuth for GitHub accounts

Let’s see how we can use the new Terraform Provider for TeamCity. In the first example, we’ll set up OAuth for GitHub accounts. Here, we need to change the server URL and then add a GitHub App authentication model.

In Terraform, we can describe our web forms as a set of resources. For the global settings, we can change the server URL. All of the other settings are left as default.

Then, we add a module for authentication as shown below.

We also need to add a connection with our custom GitHub App into our root project.

We can also configure SMTP settings for notifications, add license keys, and set up cleaning.

Terraform is a command line tool. Here, we can type in terraform apply, and it will evaluate our configuration files and declare which changes will be made in our infrastructure.

Once the changes are applied, we can log in to TeamCity with a new external account.

Creating accounts and granting permissions in advance

In TeamCity, you can create custom roles and grant only certain permissions to certain roles, following the principle of least privilege. Let’s see how to do so using Terraform Provider.

Let’s create a new custom role, a project, and a new group, and assign project permissions to the group.

When the user connects to TeamCity for the first time, they’ll immediately have access to their projects and custom permissions.

The project that we created is empty. However, when you start developing a new product or microservice, it’s great to provide the team with a working example. Here, we have a repo in GitHub with a basic Kotlin DSL script.

Now, let’s register it in TeamCity. We’ll create a project, and within it, register an SSH key and create a VCS root using SSH authentication.

We can also enable versioned settings there to ensure our Kotlin DSL script is applied.

Managing GitHub resources via Terraform

Terraform doesn’t just refer to GitHub resources but actually manages them. To demonstrate how this feature works, let's create a new repo by cloning it from a template.

In the code, we declare a new GitHub repository and refer to our template as a base for cloning.

This integration can also work the other way around. Namely, Terraform can also reflect changes in your actual infrastructure. Now, let’s see how we can create a repo manually.

Here, we’ve listed all of the repos in their organization and ensured that each of them has a corresponding set of TeamCity resources.

As we can see, Terraform was able to find a new repo that we recently created.

We can also add a set of TeamCity objects, and they will appear automatically in the corresponding TeamCity project.

Importing existing resources

For those who have been working with TeamCity for a while, there is a way to import existing configurations and settings into Terraform without recreating everything from scratch. Here’s an example of how we can customize existing settings.

Here, we’ve specified the ID of a TeamCity object and the ID of a resultant Terraform resource:

Now, we can tell Terraform to go into an actual TeamCity instance, read its state, and generate configuration files.

Terraform will then fetch the list of permissions from TeamCity.

We can then go ahead and add a new permission in Terraform. It will then automatically appear in TeamCity.

Learn more about Terraform Provider for TeamCity

You can find the Terraform Provider for TeamCity in the HashiCorp Terraform registry. Please refer to the documentation for more resources and details.

Also, make sure to check out the project’s GitHub repository.

If you have any questions, feel free to reach out to us – we’ll be happy to help.

Kotlin DSL Examples in TeamCity

JetBrains TeamCity — Fri, 08 Sep 2023 15:24:25 +0000

TeamCity offers the ability to define project settings and configure CI/CD pipelines programmatically with the help of the Kotlin DSL.

To help you take advantage of the Kotlin DSL’s capabilities and simplify the build configuration process, we’ve created extensive Kotlin DSL documentation. It comes with examples that you can simply copy-paste directly into your code base.

How the Kotlin DSL documentation works

Every TeamCity server has its own Kotlin DSL documentation, which is automatically curated to match the TeamCity version and any plugins installed on the server. If you install a new plugin, the documentation compiles again, providing you with relevant examples.

You can also refer to the general Kotlin DSL documentation, which is available in the TeamCity docs.

Accessing the Kotlin DSL documentation from IntelliJ IDEA

The Kotlin DSL documentation is available right from IntelliJ IDEA (both Ultimate and Community editions). You can access it by going to Maven Tool Window | Download Sources and Documentation.

Another way to access the Kotlin DSL documentation directly from your IDE is to run the mvn -U dependency:sources command.

The documentation’s context and examples change when you click on an entity (for example, a build step or a trigger). The information is displayed either in a popup window or in the panel on the left, depending on the settings you’ve selected.

There are a few different ways to open the Kotlin DSL examples from your IDE:

Pressing F1 on Mac or Ctrl + Q on Windows. Refer to this section of the IntelliJ IDEA documentation for more details.
Clicking on the name of an entity (such as a build step or a command). The examples will open in the menu on the right-hand side of the window.
Simply hovering over an entity to access the in-line information window.

How this feature is helpful

Using Kotlin DSL examples can save you time when configuring your pipelines as code. The examples also make it easier to discover all of the things you can do when configuring builds, in addition to helping you identify the scenarios that TeamCity can support.

Working with the Kotlin DSL examples can be a particularly great option when you are just getting started, as they provide a solid foundation on which to build your understanding of the Kotlin DSL.

TeamCity also provides you with an option to view your settings as code with the help of the View as code button, which is available on the build level. This displays your settings as code that you can copy and paste to your codebase.

If your project can’t be configured via the UI and you’d still like to experiment with the View as code feature, consider setting up a sandbox project on your TeamCity server. It will give you a chance to play around with different TeamCity features and see how they look in the Kotlin DSL.

Further resources

If you’d like to learn more about using the Kotlin DSL for TeamCity, here are some additional resources that you might find useful:

Video series “Kotlin DSL: From Zero to Hero”
Blog post “Kotlin DSL for Beginners: Recommended Refactorings”
Blog post “Creating TeamCity Project Templates with Kotlin DSL Context Parameters”
Blog post series “Configuration as Code”

Over to you

Do you have any questions or comments about how we can improve the Kotlin DSL examples and documentation for TeamCity? We’d love to get your feedback! Feel free to share it in the comment section below.

Happy building!

How to Use the Space Git Flow With TeamCity

JetBrains TeamCity — Thu, 27 Apr 2023 15:44:45 +0000

Do you use feature branches or Git flow to ensure code quality?

The powerful combination of JetBrains Space Git flow with TeamCity offers a complete solution to help you achieve better code quality and keep your main branch green with stable builds.

Space provides you with Git hosting, code reviews, and quality gates, while TeamCity offers a build pipeline. The native integration between tools will save you time and effort by ensuring you have a complete and well-integrated flow with a unified UI.

Adopting the Space Git flow with TeamCity is easy and allows you to keep your main branch stable while controlling changes automatically.

In this blog post, we’ll introduce the combo of the Space Git flow and TeamCity, and guide you through the process of using it in your project.

What Is the Space Git Flow, and Why Use it?

The Space Git flow is a branching strategy that is similar to GitHub flow, but with a greater emphasis on safety when introducing changes to the main branch and the ability to scale to large projects and teams.

Getting started with the Space Git flow is easy and doesn’t require going all-in, as you can mirror your existing Git repository without having to migrate, and you can switch back any time.

Using the Space Git flow integrated with TeamCity allows you to:

Achieve higher-quality code and a stable main branch by configuring quality gates based on TeamCity build status for merge requests.
Introduce a code review process your team will love, integrated with IntelliJ IDEA and available on the go with Space iOS and Android apps.
Read this article to learn more about the Space Git flow.

How to Use the Space Git Flow With TeamCity

Mirror or host your Git repository in Space
Start by creating bi-directional mirrors of your repository from GitHub or another Git service and link it to Space in a few clicks.

You can also use Space to host your source code privately. If you host it in Space, you can take advantage of the web-based interface and JetBrains IDE integration for a seamless development experience.

Protect your main branch from accidental pushes

To prevent accidental pushes to the main branch, you can set up branch protection in Space. This helps you protect repositories from unauthenticated commits, restrict who can push or merge changes into the branch, and prevent accidental branch deletion.

Once you’ve set up branch protection, the only way a change can make it to main is through a merge request.

Create a feature branch

To start working on a new feature, you’ll need to either clone the repository to your machine, or use remote development with the IDE hosted on a virtual machine in the Space cloud.

With the Space Git flow, you make changes in a separate feature branch and then request a code review before merging it back to main. From your IDE, you can make changes, add a commit message, and then push this branch back to Space.

Create a merge request

Create a merge request to begin the process of moving your new code into main. You can do this from within your IDE using the native integration, or you can find your branch in Space and create a merge request from there.

You can create a title and description for your merge request and see the commits that will be part of it. The quality gates configured in Space require at least one person to review these changes.

Quality gates are a mandatory and customizable set of conditions that need to be fulfilled before merging. Apart from code reviewer’s approval, you can set up a Space Automation job and a TeamCity check as gates.

Space will require a review based on code ownership**

In your repository, you can create a special file named CODEOWNERS that specifies who is responsible for specific folders and files. Later, you and your colleagues will have the option to select a code owner to review your changes as a part of quality gates.

Wait for your team to review the changes **

Now you wait for your colleagues to review the changes. They can add comments and suggestions, or they can simply approve the request. With turn-based code reviews, Space makes it easy for both the author and the reviewer to understand whose turn it is to take action.

Space allows you to collect your comments as drafts and send them in batches so as not to overload your colleagues with unnecessary notifications.

After you’ve agreed on changes with your reviewer, you can proceed with merging them.

Ensure your build is green
With Space and TeamCity, there are a few options that you can use to make sure your build is green before merging your changes. You can use either or both of them for extra security.

Run CI/CD checks

After you create a merge request, Space Automation and/or TeamCity CI jobs are triggered to validate the changes by building the code and running tests. As part of quality gates, the changes will be merged into main only if a CI/CD server can successfully build the feature branch.

The checks can be set and completed by both Space Automation and TeamCity.

These checks, however, only ensure the new code is valid and integrates well with the main branch at the time of branching. With Safe Merge, validation is done by attempting to integrate the new code with the very latest code from the main branch.

Run Safe Merge

You can use Safe Merge before merging the changes directly into main. Safe Merge acts like a time machine – it runs a preview of what your main branch would look like with the changes.

This can be especially useful for large projects because while you were working on a feature branch, the main branch could have received changes that conflict with your work. Safe Merge allows you to catch these conflicts before actually merging the branches.

Space creates a temporary commit with the latest changes, which can then be used to perform the required quality checks using an Automation job or TeamCity build.

In TeamCity, you’ll see Space trigger a special build, and TeamCity will report the build status back to Space when it’s finished. The changes are merged into the main branch only if the build succeeds. Otherwise, the merge is rejected.

Transition to Native Git in TeamCity Brings 10x Fetch Time Reduction to IntelliJ IDEA

JetBrains TeamCity — Tue, 12 Jul 2022 15:13:03 +0000

Starting from version 2022.04, TeamCity switched to native Git on the server side for Git VCS connections. The switch should positively impact both performance and overall experience of working with Git repositories on the TeamCity server side.

In this blog post, we’ll talk about the reasons for the switch and our own experience with this feature.

Background

In the case of the agent-side checkout, TeamCity agents have always been using a native Git executable to checkout sources. This functionality spawns a separate process for native Git commands like ls-remote, fetch, checkout, and others, so it requires the Git executable to be present on the agent. When it comes to SSH communication and depending on the configuration, native Git may use the OpenSSH, PuTTY, or JCraft JSch Java library as the SSH client.

Meanwhile, the TeamCity server also requires a copy of a Git repository to detect newly pushed commits. These newly found commits later pop up as pending and are included in the following builds. In a regular scenario on the server side, TeamCity first checks if the local repository has the same state (a set of branches pointing to a set of commits) as the remote one. If the local state is out-of-date, missing commits should be obtained from the remote.

In terms of Git, TeamCity server performs an ls-remote operation possibly followed by a fetch. There is also some other server-side functionality which works with remote repositories, like VCS labeling feature that creates Git tags, or versions settings that can push commits to the remote.

Approach with Eclipse JGit and JCraft JSch

Historically, to work with both local and remote repositories on the TeamCity server-side, we chose the Eclipse JGit Java library. For SSH transport and operations with SSH keys, we used the JCraft JSch Java library. Both JGit and JSch libraries did a very good job. However, they have some performance, memory-related, and other limiting issues.

Among the main concerns and complaints were:

Slow fetch for large repositories, which could drag out the overall process of checking for changes on the TeamCity server.
Memory issues when performing fetch inside the TeamCity process, which required more complicated out-of-process solutions.
Notable delay between creating a branch in a Git repository and the moment when TeamCity detects it and starts showing it in the branch list for a build configuration, providing the ability to start a build in the new branch.
Inability to receive a pack file with any object larger than 2GB, which requires switching the whole repository to Git LFS.
Besides, it was often problematic to compare errors reported by the mentioned libraries with messages coming from native Git under the same conditions.

Finally, the official JCraft JSch library is no longer maintained and consequently does not support modern secure algorithms and ciphers. Although an actively developed fork has become available recently, we decided to use the native Git approach.

Native Git approach

These issues brought about the idea of trying native Git with OpenSSH for SSH transport on the TeamCity server side (as we said before, agents have already been using native Git). After a significant amount of work which took a few months, we were finally able to test the native Git approach internally, and the results were better than we had expected.

Our current solution is to use a hybrid approach. For convenience we’re using Eclipse JGit while working with local repositories, e.g. iterating over commits history in the local repository, when collecting modified files, their content, etc. But when it comes to remote communication, such as obtaining a current remote repository state, TeamCity server spawns a process with native Git executable instead of running a JGit inside a Java process.

JetBrains successful experience

As you might know, we in JetBrains dogfood our products as much as possible. Our internal TeamCity server, which builds most of JetBrains products, currently has more than 3,000 VCS roots (mainly Git). Among the biggest Git repositories which are checked out on our TeamCity server is, of course, the IntelliJ Platform hosted by JetBrains Space.

Switching to native Git together with a few other related fixes in 2022.04 solved all the above-mentioned problems on our internal TeamCity server and also showed an impressive fetch time reduction for our Linux node, which is responsible for VCS operations.

The following image shows statistics for the fetch time for IntelliJ Platform repository collected during the time slot covering the switch from JGit to native Git. The fetch time decreased by almost 10 times!

Note: if you’re using Windows, the performance improvements might not be so impressive. But still they can be noticeable.

Try native Git on your server

Before you try native Git on your TeamCity server, we suggest taking a look at the known issues.

To enable native Git, navigate to Administration | Diagnostics and open the Git tab there.

On this page, you can test the connection via native Git for any of the VCS roots on your server. If you choose to test all VCS roots, TeamCity will check whether they successfully connect via JGit and then will test their connection via native Git. This measure helps ensure that none of your pipelines will break after switching to native Git. If the connection test is successful, then you can enable native Git support on your server.

Give it a try and tell us about your experience! We’re looking forward to your feedback.