Forem: Atlassian

Build a Jira comments summarizer app with OpenAI

Anmol Agrawal — Tue, 07 Nov 2023 09:27:55 +0000

In this tutorial, you will build a Forge app that integrates with OpenAI APIs to summarize comments in Jira issues. This app addresses the following challenges:

When a user goes through lots of comments in a Jira issue, it can be overwhelming and time-consuming. It may become difficult to keep track of important details or decisions made during the discussion.
As more comments are added, it becomes harder to find the relevant information needed to take action on the issue at hand.

Before you begin

Make sure you have the following:

A Jira Cloud instance where you can install and test your app.
Basic knowledge of JavaScript, React and the Forge platform.
A completed Build a Jira hello world app, to ensure that you have set up the Forge development environment.
An OpenAI API key which you can obtain from the OpenAI website.

The tutorial uses these components to build the app: Jira Issue Panel module, Permissions, Fragment, Text, fetch API, api.asApp(), .requestJira(), useProductContext, and useState().

About the app

To view a summary of an issue's Jira comments, the user clicks the Summarizer button on an issue page.
The app displays an issue panel containing a summary of all the comments for the issue.

You can find the source code for this demo here - anmolagrwl/forge-ai-jira-comment-summariser.

How does the app work?

Following the numbered points in the diagram:

The user first requests the summary. In this case, it will be as simple as a User clicking a button to get the summary as shown in the above video.
The Summarizer app will get all the comments in the particular Jira issue using the Jira API.
The Summarizer app will then pass on the comments to OpenAI along with prompt to create summary.
Once the app gets the summary, it is returned to the user and rendered in the UI.

Step 1: Create the app

Once your development environment is set up, follow these steps to create an initial version of your app:

Navigate to the directory where you want to create the app.
Create a new project by running the forge create command in the terminal.
Enter a name for your app when asked. For example, summarizer.
After you have done that, Forge prompts you to select a template to help you start building the app. In this case, select the category UI kit and then the template jira-issue-panel.
Once the app is created, deploy it and see how the app looks. In the app directory inside the terminal, enter the command forge deploy to deploy your app.
Enter the command forge install in terminal. You will be asked to provide the site in which you would like to install the app.
After the app is installed, view the app in your Jira instance by opening any Jira issue. There you will see a new button and a new issue panel below Description.

Depending on your Jira site, you may see the name of your app in the menu, as pictured below, or you may see only its icon.

Step 2: Get all the comments of a Jira Issue via REST API

Update the manifest to include the required permissions

manifest.yml

modules:
  jira:issuePanel:
    - key: summarizer-hello-world-panel
      function: main
      title: summarizer
      icon: https://developer.atlassian.com/platform/forge/images/icons/issue-panel-icon.svg
  function:
    - key: main
      handler: index.run
permissions:
  scopes:
    - 'read:jira-work'
  external:
    fetch:
      backend:
        - 'api.openai.com'
app:
  id: <your-app-id>

To call certain Jira APIs and external APIs, you need to give your app permission to do so. This is done by adding scopes and external permissions to the app's manifest.yml file. The app will call two APIs:

Get issue comments API - As per the API documentation, read:jira-work permission is required to call this API.
OpenAI API - external.fetch.backend is used to define external domains your Forge functions can talk to.

The jira:issuePanel module entry was added by the jira-issue-panel template. You can learn more about this module here.

Update index.jsx with the main top-level logic for your app

The top level code calls other functions to interact with the Jira and Chat GPT APIs which youâ€™ll add as you work through the tutorial.

index.jsx

import ForgeUI, { render, Fragment, Text, IssuePanel, useProductContext, useState } from '@forge/ui';
import api, { route, fetch } from '@forge/api';

const App = () => {
  // Getting the issue key in the context.
  const context = useProductContext();
  const issueKey = context.platformContext.issueKey;

  // Getting all the comments of the issue key.
  const [comments] = useState(async () => {
    return await getComments(issueKey);
  });

  console.log("Comments - " + comments)

  // ChatGPT prompt to get the summary
  const prompt = `Here is a sample data where all the comments of a jira issue is joined together: 
  "${comments}". I want to summarize this in a way that anybody can get an idea what's going on in this issue without going through all the comments. Create a summary or TLDR for this.`

  // OpenAI API call to get the summary.
  const [summary] = useState(async () => {
    return await callOpenAI(prompt);
  });

  console.log("Summary - " + summary)

  return (
    <Fragment>
      <Text>{summary}</Text>
    </Fragment>
  );
};

export const run = render(
  <IssuePanel>
    <App />
  </IssuePanel>
);

This is the main part of the app, which contains the top level logic to call APIs and render the UI.

The app first imports the UI components and API methods that will be used in this app.
The run function is the first function that gets executed. This is handled in manifest.yml by defining index.run
App() is then triggered. This is where all the magic happens:
- Try to get the current issue key using useProductContext.
- Then, try to get all the comments in the issue using the getComments() method, which is defined later in this tutorial.
- After the comments are retrieved, create the prompt to be used by ChatGPT.
- Pass that prompt to OpenAI via an API call using callOpenAI() method.
- Return the results using the Text component.

Update index.jsx to call a Jira API to get all comments for this issue

index.jsx

const getComments = async (issueKey) => {

  // API call to get all comments of Jira issue with key 'issueKey'
  const commentsData = await api.asApp().requestJira(route`/rest/api/3/issue/${issueKey}/comment`, {
      headers: {
          'Accept': 'application/json'
      }
  });

  const responseData = await commentsData.json();
  const jsonData = await responseData.comments

  let extractedTexts = [];

  // Extracting all texts in the comments into extractedTexts array
  await jsonData.map(comment => {
    if (comment.body && comment.body.content) {
      comment.body.content.map(contentItem => {
        if (contentItem.type === "paragraph" && contentItem.content) {
          contentItem.content.map(textItem => {
            if (textItem.type === "text" && textItem.text) {
              extractedTexts.push(textItem.text);
            }
          });
        }
      });
    }
  });

  return extractedTexts.join(' ');
}

This function calls the Get issue comments API using the api.asApp() method.

Once the app retrieves all the comments using the Jira API, try to extract only the texts from the comments and exclude other data like created at and author, and join all the comments together into a paragraph.
Join all of the comments together into a paragraph and return it. The app will use this paragraph to construct the prompt variable it sends to ChatGPT.

Step 3: Integrate your app with OpenAI API

Now that the app can retrieve all the comments in a Jira Issue via an API, the next step is to pass it to OpenAI API to get the summary.

Update index.jsx to call the ChatGPT API to summarize the comments

In the previous step, you added the variable prompt, then constructed a prompt using the comments and a command that tells OpenAI what to do with that data (in this case: summarize it). The code passes the prompt variable to a function callOpenAI which would call OpenAI API and return the results.

Here is the code for the callOpenAI function:

index.jsx

const callOpenAI = async (prompt) => {

  const choiceCount = 1;
  // OpenAI API endpoint
  const url = `https://api.openai.com/v1/chat/completions`;

  // Body for API call
  const payload = {
    model: getOpenAPIModel(),
    n: choiceCount,
    messages: [{
      role: 'user',
      content: prompt
    }]
  };

  // API call options
  const options = {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${getOpenAPIKey()}`,
      'Content-Type': 'application/json',
    },
    redirect: 'follow',
    body: JSON.stringify(payload)
  };

  // API call to OpenAI
  const response = await fetch(url, options);
  let result = ''

  if (response.status === 200) {
    const chatCompletion = await response.json();
    const firstChoice = chatCompletion.choices[0]

    if (firstChoice) {
      result = firstChoice.message.content;
    } else {
      console.warn(`Chat completion response did not include any assistance choices.`);
      result = `AI response did not include any choices.`;
    }
  } else {
    const text = await response.text();
    result = text;
  }
  return result;
}

// Get OpenAI API key
export const getOpenAPIKey = () => {
  return process.env.OPEN_API_KEY;
}

// Get OpenAI model
export const getOpenAPIModel = () => {
  return 'gpt-3.5-turbo';
  // return 'gpt-4';
}

Here, the app makes a basic API call to OpenAI. You can learn more about this through their documentation. The steps involved include:

Set the URL of the API to which the call will be made.
Provide the details of the payload which consists of information about the GPT model to which the request will be made and the prompt containing the comments and command.
Set the options that contains extra information like call method, authorisation headers and the payload in JSON format.
Use Forge's fetch method to make the API call. Then, parse through the response to get the text of summary that will be displayed in the Jira UI for that issue.

The getOpenAPIKey() function returns a Forge environment variable called OPEN_API_KEY. Before running the app for the first time, set this environment variable to the OpenAI API key that is needed to interact with their APIs. To create an environment variable in Forge, enter the following command in your terminal:

forge variables set --encrypt OPEN_API_KEY your-key

The --encrypt flag instructs Forge to store the variable in encrypted form.

Step 4: Deploy your app

Once all the above steps are done, you can:

Run forge deploy in the terminal again as the manifest.yml file was updated.
Run forge install --upgrade and select the installation to upgrade. If you have followed along with this tutorial, it should list the development environment for your Jira instance.
Try out the app in your cloud instance. The first time you run it, Atlassian asks you for permission, for that app to access Jira comments and your user information.

Next steps

Great job on finishing the tutorial on developing a Forge app with OpenAI! Take a moment to celebrate this impressive achievement. If you require any additional help, reach out to our developer community. Keep up the excellent work and continue to explore new opportunities for your apps using Forge and OpenAI technology.

Source: Build a Jira comments summarizer app with OpenAI

Assign PRs randomly to a specific list of users in Bitbucket Cloud

Caterina — Wed, 04 Oct 2023 05:06:26 +0000

Every pull request needs a reviewer but selecting the right person is something many teams want to automate. After chatting with a few developers and Atlassian teams, I realized that different teams have different ideas about the logic of assigning a reviewer to a PR.

This blog shows how I implemented one of the most popular options so that you can either use it for your team or take it as a starting point for building your team’s preferred logic.

Forge - Atlassian serverless app development platform

The magic for assigning the reviewers to a PR is implemented using Forge, Atlassian serverless app development platform. Everybody with an Atlassian account can use Forge and you install the app right away on a workspace you administer.

You can always create a new workspace for testing the app and then make it available on your team’s workspace at a later stage.

What does the app do?

With this app installed, every time a pull request is created a user from a predefined list is randomly selected and added as the PR reviewer. A comment is also added mentioning so that it’s clear that the assignment has been performed by the app.

Let's see it in action:

Let’s break down the app into its main components:

the fetchRepository function is used to load the current repository. This is required to get the name of the default branch.
the .reviewers file is loaded by the fetchFile function and the list of possible reviewers is saved in the reviewers variable. The version of the file is the one at the head of the default branch, whose name has been retrieved in the first step.
the getPRAuthor function retrieves the author of the PR
the author of the PR is removed from the list of possible reviewers. This is because the author cannot be set as a reviewer.
a random number is generated and used to select one of the possible reviewers
a reviewer is assigned to the PR via the assignPR function
the commentOnPR function adds a comment to the PR and the reviewer is mentioned

The app is triggered by a PR creation event (avi:bitbucket:created:pullrequest).

Let’s have a look at the source code

The repository for this app with the instructions on how to install it is available here: https://bitbucket.org/atlassian/forge-bitbucket-assign-pr-reviewer

index.js file

The app logic is defined in a single file called index.js which is available in the src folder, here you’ll find all the functions mentioned above.

Because this app is triggered by a PR creation event, the main prtrigger function has access to the event and context parameters which are used throughout the code to access the identifiers for the workspace (event.workspace.uuid), the repository (event.repository.uuid) and the pull request (event.pullrequest.id).

The repository.mainbranch.name retrieved from the fetchRepository function is used as the commit path parameter when calling the /2.0/repositories/${workspaceUuid}/${repoUuid}/src/${commit}/${path}. This is how we retrieve the most recent version of the file on the main branch.

Fetching the file, the PR author, assigning the PR and adding a comment are operations supported by the Bitbucket REST APIs called using the requestBitbucketForge function.

manifest.yml file

Each Forge app comes with a manifest.yml file that describes it.

For this app, it contains the logic to trigger the index.prtrigger (the prtrigger function in the index.js file) every time a new PR is created. This is done using the avi:bitbucket:created:pullrequest event trigger.

  trigger:
    - key: pullrequest-created-event
      function: main
      events:
        - avi:bitbucket:created:pullrequest
  function:
    - key: main
      handler: index.prtrigger

The manifest file also contains all the required scopes for the app to run the REST API endpoints:

permissions:
  scopes:
    - read:pullrequest:bitbucket
    - write:pullrequest:bitbucket
    - read:repository:bitbucket

The read:repository:bitbucket is required for the GET Get a repository and the GET Get file or directory contents endpoint.

For reading the PR author of a PR via the GET Get a pull request endpoint and adding a comment using the POST Create a comment on a pull request endpoint, the read:pullrequest:bitbucket is needed.

While this app doesn’t add any extension points to the UI, Forge for Bitbucket provides a multitude of UI modules that can be used to show your app information.

Wrapping up

If you too, like many others, have your own preferred way to select users to review a pull request, you can build your own Forge app to do that. This page walks you through a few implementation details of an app and feel free to tweak it, change it, build on it - whatever you need to make it work for you.

Check the repository for this app and install it on your workspace by following the instructions in the README.

We can’t wait to see what amazing alternatives you can come up with! Let us know in the comments!

If you need any help, reach out to our community.

Announcing the EAP of Forge in Bitbucket Cloud

Edmund Munday — Thu, 28 Sep 2023 03:19:43 +0000

We're extremely excited to be kicking off the next chapter in the Bitbucket Cloud story. We have a range of exciting changes and enhancements coming to Bitbucket Cloud over the coming months that all start today, with our EAP of Forge in Bitbucket Cloud.

What is Forge?

Forge is Atlassian’s company-wide product extensibility framework that allows customers and partners to add "native like" functionality directly into existing Atlassian products. Imagine if you could write your own native features (back-end & front-end) and inject them directly into Atlassian products – that is what Forge allows you to do.

To learn more about Forge, check out the overview here: Forge Overview

Today, Forge is the platform that underpins the newest and most innovative 3rd party apps in the Atlassian Marketplace – but Forge is not just for building 3rd party apps. We believe this set of capabilities is going to provide a unique opportunity directly to our customers in Bitbucket Cloud, owing to the fact that most of our users are inherently "code first" in terms of how they choose to solve problems. We're extremely excited to bring this capability to both customers and partners in Bitbucket Cloud, and we can't wait to see the amazing solutions you all are able to build.

Why is Forge important in Bitbucket Cloud?

There is a fundamental issue that occurs when software tools grow to support larger and more complex customers and users. As the companies using a product get more complicated and unique, it becomes increasingly (often exponentially) difficult to meet those unique and vastly different requirements with standard "off-the-shelf" functionality.

This often leads to terrifyingly complex enterprise software tools that have 1000 different features, 990 of which only one or two customers ever use (not literally… but you get the idea). This is impossible to maintain, impossible to innovate, and a situation you never want to be in.

Bitbucket Cloud's approach to tackling this challenge as we scale to support large complex organisations is to focus on the core Code & CI/CD experiences we specialise in, whilst also giving customers and users the tools they need to solve entire categories of problems themselves, using Bitbucket Cloud as a foundation.

We will do this by providing clean abstractions, well-defined interfaces, and easy to use extension points that will allow software teams to build exactly the solutions they need to solve their unique sets of problems, and then embed those solutions directly into their core workflow within Bitbucket Cloud.

We have a range of targeted sets of capabilities we are planning to release on top of this Forge powered foundation over the next 3-12 months, including Custom Merge Checks, Dynamic CI/CD Pipelines, and more.

Enough talking… how do I get started?

The first step to getting started building apps in Bitbucket Cloud with Forge is to complete the "Getting started with Forge" guide available here: Forge - Getting Started

This will guide you through the basics of setting up your local environment with the core Forge tooling, required for building apps in any Atlassian product.

Once complete, you can start by working through our example Bitbucket Cloud Forge app tutorial. Instructions on how to do that are available here: Bitbucket Cloud Forge Tutorial

If you're more of a visual learner, there is also a YouTube video where a member of our team walks through building a basic Forge app in Bitbucket Cloud available here: Bitbucket Cloud Forge App Live Code Webinar

The code for the app we build in that video is available in-full, here: Bitbucket Cloud Forge App Repository

For all these examples, you'll need to reference the Bitbucket Cloud Forge Modules, Events, and Endpoints references:

Getting help, providing feedback, sharing ideas?

The Bitbucket Cloud x Forge community space will be the primary place for collaboration and communication during the Bitbucket Cloud x Forge EAP. We will share important updates there like new capability releases, roadmap updates, and any breaking changes.

This will also be where we will provide support to people participating in the EAP. If you have questions or need support, please create a post in that space so that we can help you get going. If you see someone else asking for help, and you have an answer, please get involved, so we can build this community up and make it valuable for everyone.

Please also use this space to provide feedback, request features, and let us know about any bugs or rough edges you find. We're committed to making the Forge experience in Bitbucket cloud the best we can, but we'll need your input to make that happen. Also note that it's just as important for us to get feedback on our documentation, tutorials, and guides – so if you find something that doesn't make sense, please let us know, so we can make it right.

And finally, when you build something exciting (or even just get your first "hello, world" up and running), please, share that progress here, so we can be part of that journey. We're excited to see what people build, and the more ideas we see people implementing, the more ideas we can come up with to unlock even more potential.

Using Apache Solr in Production — Bitbucket

Ash Moosa — Thu, 22 Aug 2019 09:13:54 +0000

Using Apache Solr in Production — Bitbucket

This is a guest post by Pulkit Kedia, a backend engineer at Womaniya.

Solr is a search engine built on top of Apache Lucene. Apache Lucene uses an inverted index to store documents(data) and gives you search and indexing functionality via a Java API. However, to use features like full text you would need to write code in Java.

Solr is a more advanced version of Lucene’s search. It offers more functionality and is designed for scalability. Solr comes loaded with features like Pagination, sorting, faceting, auto-suggest, spell check etc. Also, Solr uses a trie structure for numeric and date data types e.g. there is normal int field and another tint field which signifies the trie int field.

Solr is really fast for text searching/analyzing and credit goes to its inverted index structure. If your application requires extensive text searching, Solr is a good choice. Several companies like Netflix, Verizon, AT&T, and Qualcomm use Solr as their search engine. Even Amazon Cloudsearch which is a search engine service by AWS uses Solr internally.

This article provides a method to deploy Solr in production and deals with creating Solr collections. If you are just starting with Solr, you should start by building a Solr core. Core is a single node Solr server, with no shards and replicas, while collections consist of various shards and its replicas which are the cores.

Implementation

In a distributed search, a collection is a logical index across multiple servers. The part of each server that runs a collection is called a core. So in a non-distributed search, a core and a collection are the same because there is only one server.

In production, you need a collection to be implemented rather than a Solr core, because a core won’t be able to hold production data (unless you do vertical scaling). Apache Zookeeper helps create the connection across multiple servers.

There are two ways you can set this up:

Multiple solr servers and use Zookeeper on one of the servers
Zookeeper on a different server and all the other Solr servers connecting to it

We’ll go through the process of implementing using the second approach. The first approach is similar to the second one but the latter is a more scalable approach.

Installing Solr

Spawn up 3 servers and install Solr on 2 servers (note: you can spawn any number of solr servers — we use 3 in our example). To install Solr, you need to install Java first, then download the desired version and untar it.

Installation: wget http://archive.apache.org/dist/lucene/solr/8.1.0/solr-8.1.0.tgz

Untar: tar -zxvf solr-8.1.0.tgz

You can start Solr by going to the /home/ubuntu/solr-8.0.0 folder with bin/solr start or in the bin folder with ./solr start. This would start solr on port 8983, and you can test it in the browser.

Replicate the exact same steps to install Solr on your 2nd server.

Also remember to setup the list of IP’s and names for each in /etc/hosts

For example :

IPv4 Public IP-solr-node-1 solr-node-1 IPv4 Public IP-solr-node-2 solr-node-2 IPv4 Public IP-zookeeper-node zookeeper-node

Installing Zookeeper

Now the 3rd server would require only zookeeper to which you would push configsets.

Installation: wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

Untar : tar -zxvf zookeeper-3.4.9.tar.gz

If you like, you can add the path to zookeeper to the bashrc file.

Next, in the zookeeper-3.4.9 folder, there is a sample configuration file which comes with zookeeper -> zoo_sample.cfg. Copy this file in the path and rename it to zoo.cfg. The configuration file contains various parameters like “ dataDir ” which specifies the directory to store the snapshots of in-memory database and transaction logs, “ maxClientCnxns ” which limits the max number of client connections etc.

Open the zoo.cfg file and uncomment “ autopurge.snapRetainCount=3 ” and “ autopurge.purgeInterval=1 ” and edit the “ dataDir = data ”

Next start the zookeeper.

bin/zkServer.sh start

Creating A Configset

Configsets are basically the blueprint of the data to be stored. Configsets are stored at server/solr/configsets

You can create your own configset and use it to store your data. Change the managed-schema file content to customise the config.

You can modify the tag to denote the data fields to be stored in one document
you can define the type or create a new type by defining it with the tag.
the id field is compulsory so you cannot delete that

There are many other things you can do in Solr like dynamic fields, copy fields etc. Explaining each of them is beyond the scope of this blog but for more information, here is the official documentation.

Now that you’ve created a config and have chmod -R 777 config folder, push the config to the zookeeper.

bin/solr zk upconfig -n config\_folder\_name -d /solr-8.0.0/server/solr/configsets/config\_folder\_name/ -z zookeeper-node:2181

After pushing the config, start SolrCloud on each Solr servers. To install SolrCloud, refer to this documentation.

Connecting to Zookeeper

To connect to the zookeeper:

bin/solr start -cloud -s example/cloud/node1/solr/ -c -p 8983 -h solr-node-1 -z zookeeper-node:2181

Solr stores the inverted index at this location -> example/cloud/node1/solr/ , so you need to mention that path while connecting. Zookeeper will automatically distribute shards and replicas over the 2 solr servers. When you add some data, a hash would be generated and then it would in a particular shard. This is all handled by zookeeper.

To add data to the server you need to POST to the link http://:8983/solr//update?commit=true .

The IP can be of any server , the data automatically gets distributed among the shards.

To get data from your solr search http://:8983/solr/user/select?q=

Note: If you are using one of the Solr servers as a zookeeper, all the above steps are the same but replace zookeeper ip with that solr nodes ip and port to 9983 instead of 2181

Troubleshooting

Here are a couple common problems that may arise while setting up SolrCloud.

After you have created SolrCloud and are connecting to zookeeper, you may see an error like 8983 or 7574 is already in use.

:fuser -k 8983/tcp -

This would find the process and kill it

Another error you may see is that SolrCloud cannot find the newly created configset.

Solution: Do chmod 777 to the new configset. The more secure approach is to chown the folder to solr user.

Conclusion

Solr has a large community of experienced users and contributors and is more mature when compared to its competitors. Solr faces competition from Elasticsearch, which is open source and is also built on Apache Lucene. Elasticsearch is considered to be better at searching dynamic data such as log data while Solr handles static data better. In terms of scaling, while Elasticsearch has better in-built scalability features, with Zookeeper and SolrCloud, it’s easy to scale with Solr too.

Author bio : Pulit Kidia is a backend engineer with experience in cloud services, system design and creating scalable backend systems. He loves to learn and integrate new backend technologies. Get started, it’s free

Originally published at https://bitbucket.org on August 22, 2019.

Asynchronous JavaScript Under 5 Minutes

Phillip Shim — Sun, 11 Aug 2019 03:54:06 +0000

JavaScript makes use of callbacks, promises, async and await features to support Asynchronous Programming. We won't dive into too many details with each topic, but this article should be a gentle introduction to get you started. Let's commence!

Example setup

Take a look at this simple example. We have an initial array with pre-filled numbers, 'getNumbers' function which loops over the array and outputs each item in the array and the 'addNumber' function to receive a number and add it to the array.

const numbers = [1, 2];

function getNumbers() {
  numbers.forEach(number => console.log(number))
}

function addNumber(number) {
  numbers.push(number);
}

getNumbers(numbers) // 1, 2
addNumber(3);
getNumbers(numbers) // 1, 2, 3

The problem

Now, let's suppose both of our function calls take some amount of time to execute because we are making requests to a backend server. Let's mimic it by using built-in setTimeout methods and wrap our logic inside them.

const numbers = [1, 2];

function getNumbers() {
  setTimeout(() => {
    numbers.forEach(number => console.log(number))
  }, 1000)
}

function addNumber(number) {
  setTimeout(() => {
  numbers.push(number)
  }, 2000)
}

getNumbers(numbers) // 1, 2
addNumber(3)
getNumbers(numbers) // 1, 2 ... Why?

Take a look at the console now. It's behaving differently than before. This is because the 'addNumber' function takes 2 seconds to run and the 'getNumbers' function takes a second to run. Therefore, 'addNumber' function gets executed after two of our 'getNumbers' get called. 'addNumber(3)' function call won't wait for its previous line to finish.

Callbacks

Invoking asynchronous calls line by line won't work in this case. Is there any other way to make sure a function gets executed only after another function finishes executing? Callbacks can help us! In javascript, functions can get passed around as arguments. Therefore, we could have 'getNumbers' function get passed into addNumber function and execute it once a number has been added.

const numbers = [1, 2];

function getNumbers() {
  setTimeout(() => {
    numbers.forEach(number => console.log(number))
  }, 1000)
}

function addNumber(number, callback) {
  setTimeout(() => {
  numbers.push(number)
  callback();
  }, 2000)
}

getNumbers(numbers) // 1, 2
addNumber(3, getNumbers) // 1, 2, 3

Here is the flow of our codebase. 'getNumbers' is invoked after 1 second. 'addNumbers' is invoked after 2 seconds (1 second after 'getNumbers'). After it pushes the number to the array, it calls 'getNumbers' again, which takes an additional 1 second. The program fully terminates after 3 seconds. To learn more about callbacks, I wrote an article in-depth before.

Promises

Here is the rewrite of the same code. We will no longer use a callback and call it directly so let's modify our 'addNumber' function to not take in the 2nd argument anymore. Instead, it will return a promise using new Promise() keyword immediately. A promise is able to use resolve and reject, given from arguments that you can call after certain actions. If everything goes well, you can call resolve().

const numbers = [1, 2];

function getNumbers() {
  setTimeout(() => {
    numbers.forEach(number => console.log(number))
  }, 1000)
}

function addNumber(number) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      numbers.push(number);
      resolve();
    }, 2000)
  });
}

addNumber(3).then(getNumbers) // 1, 2, 3 after 3 seconds

When the promise is actually returned, we can chain it by using then keyword. You can then pass in a function definition to be called after your promise is resolved! Awesome! However, what if there was an error such as a network timeout? We could use the reject keyword and indicate that an action was unsuccessful. Let's manually reject it.

const numbers = [1, 2];

function getNumbers() {
  setTimeout(() => {
    numbers.forEach(number => console.log(number))
  }, 1000)
}

function addNumber(number) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      numbers.push(number);
      const isAdded = false;
      if (isAdded) {
        resolve();
      } else {
        reject("There was an error")
      }
    }, 2000)
  });
}

addNumber(3).then(getNumbers).catch((e) => console.log(e)) // There was an error

Notice that we can pass in a string which is caught by using .catch and is available via its first argument. We could do the same thing with the resolve method as well by passing in some data and receive it inside the then() method.

Async & Await

Let's take the same code and use async and await! Here is a spoiler! We will still need to return a promise, but the way we handle it is different. Take a look.

const numbers = [1, 2];

function getNumbers() {
  setTimeout(() => {
    numbers.forEach(number => console.log(number))
  }, 1000)
}

function addNumber(number) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      numbers.push(number);
      const isAdded = true;
      if (isAdded) {
        resolve();
      } else {
        reject("There was an error")
      }
    }, 2000)
  });
}

async function initialize() {
  await addNumber(3);
  getNumbers();
}

initialize(); // 1, 2, 3

Instead of chaining then and catch to the addNumber invocation, we created a function called initialize. Using the 'await' keyword requires its wrapper function to have 'async' keyword prepended. Also, the 'await' keyword makes our code more intuitive to reason about because our code reads line by line now even though it's async!

Now, How about error handling?

const numbers = [1, 2];

function getNumbers() {
  setTimeout(() => {
    numbers.forEach(number => console.log(number))
  }, 1000)
}

function addNumber(number) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      numbers.push(number);
      const isAdded = false;
      if (isAdded) {
        resolve();
      } else {
        reject("There was an error")
      }
    }, 2000)
  });
}

async function initialize() {
  try {
    await addNumber(3);
    getNumbers();
  } catch (e) {
    console.log(e);
  }
}

initialize(); // There was an error

Let's use try and catch inside our initialize function. If a promise is rejected, our catch block will run.

Summary

We learned a few different ways on how to handle different methods of handling asynchronous JavaScript. As for me, I personally prefer writing async and await at all times of how easy it is to write and think about. But others have their places especially callbacks as some APIs only support them. Thank you for reading and let's write some serious code with our newly acquired knowledge!

This example code was inspired by Brad Traversy's youtube video.

Automate Coverage Reports in Pull Requests with Bitbucket, Jenkins and SonarCloud — Bitbucket

Ash Moosa — Tue, 16 Jul 2019 07:00:37 +0000

Automate Coverage Reports in Pull Requests with Bitbucket, Jenkins and SonarCloud — Bitbucket

At Instaclustr, we’ve experienced significant growth in our team sizes that has been great for increasing the scope and speed of our development. The flip-side to this benefit is that with the increased velocity of projects came increased pressure on approvers to take time from their own tasks, and provide quality feedback at a faster pace.

To address this, we overhauled our existing build systems to:

Automate static code analysis
Expose important metrics (such as test coverage, whether tests have passed); and
Expose it to reviewers within pull requests

Now, our review workflow is:

Developer creates a PR in Bitbucket, targeting the release branch
Jenkins sees the creation of the PR and starts our build-and-test pipeline beginning with unit and system tests. If successful, the pipeline progresses through to our end-to-end tests. At each stage, coverage results are forwarded to SonarCloud for analysis
When an approver views the PR, Bitbucket (via the SonarCloud widget) pulls in the code analysis results and provides relevant context

Approvers can immediately know:

Coverage of new code code in the PR
If there are any common code errors (e.g. not closing resources)
That style guidelines have been followed (e.g. how deep is the inheritance tree)

Best of all, relatively few changes were required to implement this!

1. Jenkins

We use Jenkins as our build system, so we created a multibranch pipeline job that uses the Bitbucket Branch Source Plugin to poll for any new or updated PRs targeting our release branch. The pipeline trigger can then be configured to scan every minute.

Once triggered, the job will run our test pipeline Jenkinsfile.

The relevant parts of our Jenkinsfile are:

mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent test --fail-at-end -DskipTests=false -am

mvn sonar:sonar --batch-mode --errors " + "-pl ${context.env.TEST\_MODULES} -am " + "-Dsonar.projectKey=${Constants.SONARCLOUD\_PROJECT\_KEY} " + "-Dsonar.organization=${Constants.SONARCLOUD\_ORGANISATION} " + "-Dsonar.verbose=true " + "-Dsonar.host.url=${Constants.SONARCLOUD\_URL} " + "-Dsonar.login=${context.env.SONARCLOUD\_TOKEN} " + "-Dsonar.pullrequest.branch=${context.env.BRANCH\_NAME} " + "-Dsonar.pullrequest.base=${Constants.RELEASES\_BRANCH} " + "-Dsonar.pullrequest.key=${context.env.CHANGE\_ID}

2. SonarCloud

Uploaded reports will automatically register a new PR in SonarCloud and can be explored through the SonarCloud Console to show inline views of code issues, test coverage trends and whether the PR meets customisable Quality Gates. It’s important to note that these metrics are calculated against new code introduced by the PR, so developers don’t have to sort through an analysis of the entire codebase.

3. BitBucket

At this point, we have Jenkins automatically testing PRs and SonarCloud providing analysis. To make it as easy as possible for approvers to see that information, we just need to enable the SonarCloud widget in our Bitbucket repository.

Now, whenever a PR is created, the widget will pull in test coverage, bugs and code smell metrics. Using the Bitbucket Jenkins plugin also means that our PRs will show a handy “build passed” status to let us know when a branch has is successfully passing all test cases.

Summary

For Instaclustr, this setup has:

Simplified the basic checks a reviewer would undertake by providing summary metrics for assessing the quality of a PR
Highlighted test cases that don’t work in a parallel setting (e.g. incorrect usage of singleton objects)
Highlighted poor test verification patterns (e.g. count pre- and post-test comparison rather than checking for presence of record)
Code coverage highlights when tests aren’t being run (e.g. names not matching Surefire inclusion patterns)
Ensures highlighting of simple code errors that can have significant impacts, such as not closing resources

Hopefully this has provided a quick guide to getting up and running with static code analysis; if you have any feedback, ideas or questions about this article, I would love to hear it!

Author bio: Alwyn Davis is a Senior Software Developer at Instaclustr where he has worked on multiple infrastructure and development projects, in addition to providing client-facing support and delivering consulting projects. He has focused on the development of Instaclustr’s client-facing management systems implementation of Cassanda, Spark, and Kafka deployment processes. He also has a background in technical consulting experience in search engine, database and CRM implementation, management and application development.

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Scaling your Bitbucket team? Upgrade your plan here

Originally published at https://bitbucket.org on July 16, 2019.

Software development with UML and modern Java

Ash Moosa — Wed, 05 Jun 2019 17:29:52 +0000

This post was written by Bitbucket user Aleksandar Radulović.

With technology evolving fast, there is a need to write and maintain software more efficiently, and better communicate with team members. As developers, we rarely get to think about these things as we rush to meet deadlines.

Current software development practices rarely include software modeling. Even when models are used, they are mostly used as a part of the documentation process and often seem more of a burden.

The purpose of this article is to describe a different approach to software development that puts visual modeling and code generation into the heart of the development process. Visual software models put emphasis on communication and internal software design rather than simply making things work.

I’ll describe how we use code generators to automate software development by using a UML model as a starting point for creating modern Java back-end applications using frameworks such as Spring, Spring Data and Hibernate.

In order to understand the potential of this approach, we need to consider different cornerstones of the development process and the impact this approach has on them:

Unified Modeling language (UML) — used to visualize software systems and flows
Modern Java — essentials of the modern Java ecosystem
Building the Software Product — faster prototyping and maintenance
Team dynamics — better communication and faster onboarding

Let’s briefly go through each of these.

Unified Modeling Language

The Unified Modeling Language (UML) is a standardized, visual language for modeling software. It was developed with an ambitious intention: to provide software teams a standard way to visualize the design of a system and to improve the team’s understanding of the domain of the problem they were solving. Using UML, one can visually model concepts, processes, state machines, interactions or use cases.

The approach we take in our day to day work is to use class diagrams for modeling domain concepts and relations between them and state machines for modeling process flows. We also document different model elements: classes, interfaces, attributes, etc. so that we can derive documentation from the model at any time, using different formats and structures: javadoc or Swagger, just to mention the two.

Here is an example of a UML class diagram.

Modern Java

Modern Java has a vibrant ecosystem. While it takes time to learn a programming language, adopting modern frameworks from that language’s ecosystem is an additional learning curve.

Declarative programming

The emergence of declarative software development practices has silently opened new ways for model-driven development. Unlike the imperative programming flows that are inconvenient to specify using modeling techniques, declarative programming constructs describe structural aspects of the software that can naturally be represented by the class diagrams.

Contemporary Java development heavily relies on declarative constructs: annotations most of all. For example, different frameworks, such as Hibernate and Jackson, use annotations to map object models to relational databases or to different export formats (JSON, XML, Protobuf, BSON, CSV). The Spring Framework, among many other things, brings great support for declarative development of RESTful endpoints and Spring Data introduces many essential constructs for abstracting data store access operations.

Like other types of programming, declarative programming does come at a cost — we introduce complexities of different frameworks and libraries into our applications. While these dependencies bring complexity to the project, developers must learn that they offer a return on developer productivity by letting them focus on high level objectives.

Code generators

Declarative programming allows for code generation. Instead of having to write Java annotations by hand, it is enough to mark a class as persistent in the model and let the code generation tool create Java Persistence API (JPA) annotations for you. Instead of having to write lines and lines of JPA annotations, which can be cumbersome at times, code generation can do the magic without letting you bother with the details. Code generation is either built in to the UML tool you’re using or may be available as a plugin — it’s usually a one click process to go from UML to code.

Here is a sample of the Java code generated from the UML model shown above.

Why use code generators? Code generators translate the language of design (UML) into the language of implementation (Java). It brings automation to our development process, reducing overall development complexity and simplifying maintenance. We can be truly focused on modeling application concepts and services, the core abstractions we are dealing with, while the code generator synchronizes the model with the codebase. Further, it promotes the usage of best practices and significantly impacts the quality as well as the uniformity of the codebase.

Software Product

When the model is completed, the code generator creates a complete starting project that reflects our design — so we can focus on implementing business logic. When it comes to software maintenance, you can change the design and let the generator propagate changes to your codebase. This process of working with a software model and using a code generator allows for rapid prototyping, easier software maintenance and gives you better documentation of your product.

The question that quickly arises when you start working with code generators is: how to synchronize changes that you introduce in the code with the model? Our answer to that question is simple: don’t do that. The model is a set of abstractions and it should be kept separate from implementation.

This one-way transformation is typically referred to as the “model first” approach because it clearly puts emphasis on modeling and not vice versa.

On the other hand, we still want to be able to modify the generated Java code. For that purpose, we rely on preserved sections within Java source files, that keep custom changes intact through multiple code generations.

Development Team Now Has a Visual Map

Team development and communication are often underestimated topics in the everyday hectic run towards achieving results.

Using code generation brings the UML model to the heart of the software development process. The UML model of the product becomes a visual map that evolves as the work progresses. Having this map, different team members can understand the software better and have focused discussions. Onboarding of new team members is now much faster: instead of reading lines and lines of code, they rely upon a live map that communicates backbone ideas without implementation specific details.

This visual software development technique changes the traditional responsibilities of the team members, promoting mutual understanding of the domain and improving team cohesion. When using model-driven development, the role of software developer comes closer to the role of a business analyst. On the other hand, a business analyst clearly understands how the software is being built and the relationships between domain concepts. Finally, QA engineers have a better understanding of the application, and all team members speak the same language.

Conclusion

While it’s possible to use code generation and modeling to automate parts of software development, we do not see this often in practice — either due to lack of awareness or lack of resources to invest in reviewing and researching new ways of working. If the ideas expressed in this article get you interested in model-driven development, there are several ways to go further.

There are multiple providers of low-code development solutions. Mendix is one of them and has a comprehensive guide to low code development.

The best open-source example of this category of products is JHipster, a project that has been embraced by thousands of developers worldwide. The JHipster core team managed to connect experts from different areas of software development to make this amazing application generator.

Our own endeavor is in extending StarUML, our preferred tool for software modeling, with a plugin for code generation — this is the plugin used in the example in this post.

Finally, no matter which tools and methodologies you use, software development is a people business and as such, it has many different sides that are difficult to measure and manage. Model-driven development cannot replace the lack of quality requirements, lack of empathy within the team, or lack of organizational culture in general. It complements agile development methodologies but does not replace them.

Author bio: Aleksandar Radulović is a software developer and architect who developed Rebel, a code generator plugin for StarUML. When he’s not developing software, he enjoys reading classics like Shakespeare or is dancing the tango. Connect with him on Linkedin.

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on June 5, 2019.

How Building An IDE Extension Changed The Way We Ship Code

Ash Moosa — Wed, 15 May 2019 21:53:29 +0000

When our team set out on the adventure of building the Atlassian for VS Code extension, our mission was simple: create an MVP to test if using Bitbucket Cloud and Jira Software Cloud features inside of VS Code would make a better developer experience.

To begin, we did what we all knew: scheduled planning meetings, had daily stand-ups, setup a Slack channel for all of the discussion that happens between meetings, tried to guess at release dates and scheduled retros to discuss what went wrong and what went well.

Over time we discovered that through the use of our own tool, we tackled a notoriously difficult problem: changing developer behavior in ways that make the team more productive while easing the administrative tasks that usually slow them down.

Dogfooding our own extension has helped our team develop a better shared understanding of the code base, and integrate the non-coding tasks required for healthy project management into the dev-loop so tightly that it becomes the preferred way for developers to work.

We’ve realized that if you build tooling that’s fun to use, this will all happen organically.

Iterating in the Dark

The thing about silos is that they’re usually dark inside.

When we first started building our VS Code extension for Bitbucket and Jira users, we were working in a very familiar style where we each went into our “coding caves” for long periods of time and every once and a while came up for air to check the Slack channel we had setup.

It was almost guaranteed that there was a message in the channel begging for someone to review a PR from an hour or two ago, if not more. Then, as any good team member, we’d stop what we were doing, open up Bitbucket and review the code in our browsers. We might add a comment here and there (to be forgotten by it’s author as soon as it was submitted), then certainly take a coffee break, and then come back to crawl into our caves again.

Rinse, Repeat. The problem was, it still felt like separate people working on separate areas of the code base without a lot of cross-functional knowledge of how things worked. We were basically in the dark.

To shed some light (pun intended) on some of the issues:

Nobody likes sitting in a room for hours with a Jira board on the screen while somebody frantically types trying to enter Jira issues while heated discussions conversations are happening.
Once coding commences, developers tend to only take the tasks within their comfortable domain knowledge (silos) and cross-functional learning is lost
The lack of cross-functional code base knowledge means bugs and ideas about other parts of the code base are left up to the developer who works on it lacking collaborative input
Developers hate context switching which in-turn means:
bugs or features may be discovered, but new Jira issues don’t get entered for them
pull requests are submitted, but nagging has to happen to get them reviewed
comments may be entered on a pull request, but replies are either not followed up on or take days to get new responses
updating Jira issues to reflect the state of the project is a chore that happens through nagging, thus leaving Jira in a state of lies
Ensuring Jira units of work are linked to code in every branch and every commit is not consistent at best and doesn’t happen in the worst cases
There’s less collaboration when developers feel like someone else is the expert

Out of the Caves

One day, it happened. We had finally hit the point where we could authenticate against both Jira and Bitbucket and we had the essential features in place to start dogfooding our extension. That first developer exclaimed with triumph “Hey, I just approved that PR with our extension!” and we knew feasibility was no longer our mission…

The new roadmap, almost entirely led by engineers with little oversight and lots of dogfooding, emerged: we would take a selfish approach and focus on making our own lives easier inside of VS Code, knowing that if we got the Jira and Bitbucket experience (mostly) right, we would have a positive impact on other development teams outside of Atlassian. If we have pain points in our development cycle, chances are other teams do as well.

To illustrate some of the successes our team has enjoyed, let’s take a look at our new development practices which integrate the features of our extension to blur the lines between coding, Jira-ing and Bitbucket-ing. And what better place to start than… the middle.

A Shared Cave with Really Nice Lighting

Jira Units of Work: Never Lose Track of Discoveries

OK, let’s say I’m a developer… and I’m working on some code when I notice a bunch of comments littered with Jira issue keys. (we’ll get to why that is in a bit) It’s a bit frustrating to see the keys and a small comment but not have access to the issue details. It’s even more frustrating to hop out of my IDE to paste the key into the Jira web interface just to see the details, and so I add a little TODO comment:

// TODO: add a "quick view" when hovering over Jira issue keys

Even though there are various extensions to list/manage TODO comments, this is going to get lost if I don’t remember to bring it up at the next planning meeting, or stop what I’m doing, jump into the Jira web interface and create an issue for it.

Using the “Create Jira Issue” code link that appears for customizable comment prefixes (TODO, BUG, FIME, ISSUE, etc), I can simply click on the link, create the issue and move along with my coding.

To top it all off, the extension will update the comment and add the newly created issue key as a prefix so other developers can reference it.

// TODO: VSCODE-12324 - add a "quick view" when hovering over Jira issue keys

Turning Ideas into Code: Starting Work

Let’s say there’s another developer on the team that just finished a task and is looking for something to pick up. She sees the Jira issue we made above in her Jira Issue Tree within VS Code and opens it up. After reading through the details she decides she wants to work on it.

Following our best coding practices, getting started on a new task is a little more complicated than just coding away. She needs to:

Create a branch ensuring that the Jira issue key is in the branch name
Link the local branch with a new upstream branch on Bitbucket
Assign the issue to herself
Transition the issue to an “In Progress” state so other developers and “project people” know what’s in flight
Finally checkout the local branch and start coding

This is a small list of things that needs to be done, but after doing it a few (hundred) times, most developers tend to miss a few of these steps.

Using the “Start Work On Issue” button provided on the Jira Details screen within VS Code, this can all happen in a single step.

Pro Tip: Make sure all commits contain the issue key

So what’s this about putting issue keys in branch names?

The idea is simple: if you link your Jira instance in your Bitbucket repository settings, Bitbucket will look for issue keys in your branch names and commit/PR comments and be able to link them in the Bitbucket UI, among other things.

On top of the Bitbucket UI, the Atlassian for VS Code extension checks those same places for issue keys and “magically” gives you lists of issues related to your PRs within various UIs. You can now easily get a better picture of issue/code relationships right within VS Code.

So how to you ensure every commit has a Jira issue key in the comment?

Although outside of the VS Code extension, we use a small Git script that automates the process by finding the issue key in your branch name and pre-pending it to all commit comments on that branch.

Developers never have to type in the issue key in a comment as long as the branch contains the key.

You can grab this nifty script and instructions from our Bitbucket Snippet.

So our developer feels like her feature is complete and now needs to create a pull request. This is going to serve multiple purposes:

Her teammates will do a code review to catch anything she might have missed
Her teammates will have the opportunity to test out the new feature
The entire team can discuss the approach and make any changes as needed
The team has a chance to “sign off” on the pull request by approving it
Finally, the pull request can be merged

Let’s take this step-by-step, first by creating a pull request.

Creating a pull request traditionally meant hopping out of your IDE, navigating the Bitbucket UI, and creating the pull request. With our extension, this can now be done right within VS Code.

A pull request has been created and now is typically when our developer takes (a much deserved) coffee break and when she gets back, checks to see if anyone’s looked at it. Usually the answer is no and so she needs to hop into Slack and gently request that her teammates take a look at the PR. Even this requires some luck that the other developers are in one of their “come up for air” moments.

In our extension we wanted to try to remove as much nagging and waiting as possible, so when she created the pull request, her teammates got a small notification popup right within VS Code and her pull request showed up in all of their “Pull Request Tree” views right within VS Code.

Now her teammates not only know that there’s something to review, but they can see the detailed summary of the pull request, see any and all Jira issues that a related to the pull request, and go through the individual file diffs right withing their IDEs.. you know, where developers like to look at code in whatever crazy theme they have decided to use that week.

As they go through the details and the diffs, they can quickly add comments on a line-by-line basis, and since developers are notified as new comments are added within VS Code, the time spent waiting for replies is greatly reduced, effectively turning pull request into a meaningful communication tool.

Since using these pull request features, our team has enjoyed a huge acceleration in knowledge sharing which in-turn has help break down our silos and turn each team member into a product expert instead of an expert of a smaller domain within the code base.

We all have a deeper understanding of our entire code base and have also become a lot more collaborative simply through greater communication.

Once each teammate finishes reviewing the pull request, they can simply click the approve button from the details screen within VS Code and go back to their coding. Similarly, our developer that submitted the PR can simply merge right from the details screen as well.

Once merged, it’s time for a new task. Our developer can either go through the list of issues in the tree view like before, or now she can simply hover over an issue key she’s found in the code and use the new “Issue Quick View” feature which will allow her to get essential issue details, open up the full detail view and optionally start work on the issue.

Versioning is a Result, Not a Plan

Our team has come a long way since deciding to test out using Bitbucket and Jira features inside of VS Code. We set out to create a useful tool to help ease some of the pain points within common coding cycles. What we ended up with was exactly that. What we didn’t plan on, was that through the use of our own tool, we would organically change the way we worked.

The biggest change we’ve seen so far is that we no longer go into a room armed with a version number for the “next release”, give it a date, and pack it with things we think we can get done. Instead, we code, we discover bugs and features along the way, we are more collaborative in our development, and everyone now feels ownership in the entire code base and are eager to work on any part of it.

Where we used to do a release at some scheduled time period, we now do many more releases as useful features are completed. We don’t wait to ship anything because “1.x is supposed to release at the end of the month”. In fact, we only have a single version we ever work towards and it’s label is vNext. At any point in time, we can say “it’s time.” and then minutes later we release and assign that snapshot in time an appropriate label.

When there’s no deadline, you’re never done

Our team is moving faster than ever and we’ve already see vast improvements to our coding cycles.

By no means is our extension a “silver bullet” and as they say “results may vary”. Our team is passionate (and selfish) about continuously finding ways to use new tooling and new workflows to make every team the best they can be.

We’re excited to learn how teams outside of Atlassian make use of these features and how we can improve.

Install the extension today and let us know what you think!

Written by Jonathan Doklovic: Jonathan has been working at Atlassian for almost a decade and is currently a Principal Developer on the Product Integrations team. He has worked on many Atlassian products as well as the core plugin system and Atlassian Connect.

Originally published at https://bitbucket.org on May 15, 2019.

LAMP vs. MEAN: Which stack is right for you?

Ash Moosa — Thu, 02 May 2019 16:57:51 +0000

A web stack is a collection of software or technologies that are used to build a web application. Choices are plenty, but picking one can be hard.

When chatting with co-workers, developers or customers, suggestions for what technologies and stacks to use couldn’t be more different. When I started off as a web developer, I went the usual way at that time: learning HTML & CSS, exploring some PHP — and of course MySQL. That was, if you were not using Java or ASP.NET, the technology stack of that time. Whether you wanted to host a blog, a bulletin board or become an image hoster — you would more than often need these things: Linux, Apache, MySQL and PHP (LAMP).

Here is a detailed overview of LAMP and the relatively new, MEAN stack, which are currently the most popular open source web stacks and a brief overview of other stacks. Whichever stack you choose, Bitbucket works with them all.

LAMP

LAMP delivers a strong platform for developing and hosting large, performant web applications. With the biggest and oldest community, countless libraries and tools, you get great support and will find developers quite easily.

Its integral components are:

Linux (OS)

Apache (Webserver)

MySQL (Data persistence)

PHP (Programming language)

There are also some derivatives of this stack:

LAMP (with Perl or Python instead of PHP)
LAMP (with MongoDB instead of MySQL)
WAMP (Windows as OS)
MAMP (Mac OS X as OS)
XAMPP (Any OS + Perl or PHP + FTP Server)
LAPP (PostgreSQL as database)

Pros:

LAMP is kind of the dinosaur of web development, used by hundreds of thousands of companies and therefore maintained and supported very well. With endless modules, libraries and add-ons available you can adapt it to your company’s needs.

Being Linux based, you will find help for any topic in the large open source community. MySQL is a very reliable and scalable solution. PHP is in version 7 and is also supported by a mature and big community. PHP is also very fast and integrates well with the rest of the stack.

You can control the server and decide which versions and software you install, so you don’t have to depend on the client’s browser. Best for if you have lots of server-side tasks.

Cons:

Because it’s easy to learn, there are a lot of developers out there who are not following best practices and building garbage apps. Starting with PHP is easy, but mastering it is hard. This is also true for security in these PHP apps. Some would also describe it as a script language instead of a real programming language because it’s not strongly typed and not pre-compiled. I’d recommend diving in deeper into pros and cons of PHP, Python or Perl.

As for MySQL, other options are becoming more mature. NoSQL databases like MongoDB are popular among enterprises today due to it’s scalability. Plus, pure JavaScript Stacks like MEAN gain more traction every year and new developers might not be interested in learning all of the LAMP’s skills.

MEAN

Compared to LAMP, the MEAN stack is fairly new. One of its biggest differences is that MEAN is not dependent on a specific operating system — Node.js takes care of server-side execution. The MEAN Stack is especially recommended for JavaScript enthusiasts — as it uses JavaScript at all levels. This also makes it preferred by new developers.

MongoDB is a popular and flexible document based, NoSQL database, compared to MySQL’s relational database system. Angular helps build progressive and modern web apps.

Its components are:

MongoDB (Data persistence)

Express.js (server-side application framework)

Angular.js (client-side application framework)

Node.js (server-side environment)

This stack has some derivatives too:

MERN (React instead of Angular)
MEEN (Ember.js instead of Angular)

Pros:

Using JavaScript as the primary programming language is a huge advantage. Everything can be set up quickly and done in JS, which makes it much easier to find developers, and LAMP developers typically know JavaScript as well. MongoDB is very popular for its easy schemaless data persistence and is faster than MySQL if you have a lot of read requests. The fact that Angular is maintained by Google is also a big plus. It receives new releases and functions on a constant basis. Another huge advantage is the ability to easily build mobile or desktop apps, for example with Ionic. Code and components can easily be reused or added.

Cons:

Like all new technologies, MEAN’s glamour is creating some hype. Developers fall for this hype and build their apps in JavaScript, just because it’s trendy. Many of these libraries and frameworks are quite new, and new versions get released quickly, so maintaining your app can become quite a hassle. Since many technologies disappear after a few years, sustainability can become an issue. It’s also harder to maintain a clean code base and follow best practices as your app grows. Further, you have to rely on the client and the client’s available technologies e.g. if you are targeting IE users, embedded systems or low end PCs, there may be usability issues.

A few other stacks to consider:

WISA Windows Server / IIS / Microsoft SQL Server /

Not open source, but all components are from Microsoft, so it should work seamlessly.

LAMP (With MongoDB) Linux, Apache, MongoDB, PHP

NoSQL Databases like MongoDB can also be used in a classic LAMP environment.

Ruby Stack Ruby/Ruby on Rails/RVM (Ruby Virtual Machine) / SQLite

This stack is losing popularity. Ruby on Rails was an often used framework once, and thus the whole stack.

Java+Spring

Preferred by large enterprises and shied by indie developers for its complexity, Spring offers an entire full-stack framework written in Java.

Django Stack Python / Django / Apache / MySQL

The Django framework is loved by Python developers, delivers performance and is often referred to as an easy to learn stack.

Which stack is used more frequently?

It’s hard to compare the popularity of stacks, but you can useGoogle Trends to compare programming languages and get a feel for what people are searching for. As the chart below shows, JavaScript is searched for more than PHP right now.

I’d recommend checking development trends using Google’s trend tool from time to time.

I’d also suggest diving deeper on databases (SQL vs. NoSQL) to gain a basic understanding of the two concepts and make a choice.

So, how do you pick a stack?

Picking a stack depends on many factors. If you are a developer or project owner, here are a few questions to ask yourself.

What kind of web-application am I planning to create?
What is its expected lifetime?
What technologies are available at my customer’s/client’s /cat’s/… infrastructure?
How easy is it to find developers to maintain the application?

Let me give you an example. Let’s say you have a website for listing used cars. It was developed using a LAMP stack a while ago. But your website lacks a back-end for used car dealers, where they can manage their listings on your website. Depending on your company size, time and budget, you have to ask yourself all the above questions. If you have a small team, it might make sense to extend your existing application in the LAMP environment. Since your developers know the ecosystem and it’ll be much faster. If you have time and resources to spare, you could take another approach and extend your existing LAMP application with an API. Later, your team could focus on developing a small, standalone (M)EAN application, that can easily be maintained, improved with new features and released using a much faster cycle.

Another example: You want to build a newsletter platform, where people can sign up, upload mailing lists, compose mailings and so on. You could of course use MEAN, but you have large scale and high traffic potential. It may make more sense to use a LAMP stack as your foundation, since Linux, MySQL and Apache provide a stable, scalable environment with lots of community support for any thinkable problem. You will also have lots of server-side tasks and cronjobs and will encounter mailing topics like SMTP and so on. I would recommend a Linux environment customized to your needs in this case.

Here is a summary of things to know/consider.

MEAN

Single code base (JavaScript)
Popular for modern web apps and hybrid apps
Supported by large companies like Google
Better for apps where a lot of the logic happens on the client’s side
Harder to maintain long term due to the rapidly evolving javascript ecosystem
Best for progressive web apps

LAMP

Better for large applications
More mature, huge community
Well-established application frameworks like Symfony, Zend, Laravel
Easier to follow standards and easier to keep code clean

If you are new to programming and web development, ask yourself:

What is the easiest to learn for you and your team?
What technologies are trending and which will win in the long run?
If open source, could you imagine contributing to the project?
Which technologies will serve you personally in the long term?

A great resource for finding answers on JavaScript technologies is the StateOfJS project — https://stateofjs.com/ — it’s a project that conducts a survey every year asking thousands of developers on their opinions on current technologies and salary.

This is a guest post for Bitbucket by Christoph Heike — “I’ve been developing web applications for over 10 years and currently run a web-development agency in Bonn, Germany. I’m also involved in an Amazon-based tech startup. My goal is to always deliver clean, sustainable and high performant software solutions. Connect with me on LinkedIn.”

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on May 2, 2019.

Deploying an Angular app on a Google VM Using Bitbucket Pipelines

Ash Moosa — Tue, 30 Apr 2019 17:27:56 +0000

This is a guest post written for Bitbucket by Suren Konathala.

Angular is one of the most widely used javascript frameworks. But though the builds are easy, developers face issues when configuring deployments and setting up CI/CD pipelines. This post outlines the steps required to deploy an Angular application to a Google VM using Bitbucket Pipelines.

What are Pipelines?

Bitbucket Pipelines allows developers to configure continuous delivery (in the cloud) of source files to test/production servers. These pipelines are configured to connect to the production server using YML scripts.

Why I used Pipelines?

For users to be able to access the application, the source code needs to be deployed to a server. The server from which the web application is rendered/delivered to users is called a Production server. And before the application reaches the production server, it goes through many iterations of development and testing. These iterations are usually deployed to a Development server or a Stage server.

For an application to be deployed to each of the above servers, there are several steps involved that can get cumbersome.

Copy the code files to the server
Run the build and deploy scripts
Repeat the same on each server. Sometimes teams have multiple servers for each stage.

Okay! The above has been automated to some extent by tools like Jenkins but the drawback is that developers/admins have to install another software on top of their servers, and learn to use and administer it.

Pipelines simplifies the above process, and in fact automates the entire build and deploy process — also known as CI/CD (Continuous Integration / Continuous Deployment). The best part with Bitbucket Pipelines is that applications can be built and deployed directly from the Bitbucket repository to any destination server.

The entire copy, build and deploy process can be defined using simple YAML based scripts without the need for any additional software or a solution. All we need to do is to pick the docker image (like NodeJs, Java, etc.) for the pipeline to use to build your project and select the frequency (e.g. manual, or automatically when files are updated in the source repository). This saves a lot of time and resources for teams and organizations.

Tutorial: Configure an Angular Project on Bitbucket

This tutorial covers how to deploy an Angular based web application to a Google Cloud Virtual Machine (VM). Source code for the application is in a Bitbucket repository and this VM will need to be connected to using SSH security keys.

Pre-requisites

External server (or VM) with private and public SSH keys. This will the hosting machine for the website or web application.
Repository onBitbucket with project sources files. These will be used to build & deploy to the server.

Step 1: Setup SSH Keys

Under Bitbucket > Project source repository > Settings > Pipelines > SSH Keys

Add private and public keys. You need to get these from the external server that you need to connect to.

Add Known hosts (this will be the IP address of the external server you want to push the code to. In our example, we used a VM on Google Cloud)

Step 2: Define the YML deployment script

Go to Project source repository > Pipelines > New pipeline and define the script. Here is an example script:

# Sample build file # @author Suren Konathala 
# ----- 
image: node:8 
pipelines:   
  default: 
- step:      
     caches:        
     - node      
     script: # Modify the commands below to build your repository.
       - echo "$(ls -la)"        
       - npm install        
       - npm install -g @angular/cli        
       - ng build --prod        
       - echo "$(ls -la dist/)"        
       - scp -r dist/ user@34.73.227.137:/projects/commerce1

The above script performs the required commands/steps to build an Angular project. Once done, it pushes/deploys the contents of the build files under the dist folder to an external server.

In this example, we used an SCP command to push code to an external server. Since the SSH keys are already set in step 1 above, your local repo can now connect to the server and copy the files over.

Step 3: Run the pipeline

Save and “run the pipeline”. You can see the running log and status of the pipeline.

On successful completion, you can verify if the files actually being copied to the external server by either by testing your live application and checking if the latest updates are reflected or you can manually check if the files on the server were updated (no need to do this for every push).

Troubleshooting/Info

To update NodeJS version in the pipeline.. change the version of the docker node image to node:8 in the YML script.

image: node:8

To know what folder you are in and what files are being generated, you can use echo commands. Some examples..

echo "Starting the deployment..."
echo "$(ls -la)"

This post shows how to add SSH Keys on a Linux server/VM.

The source files for this project are shared here.

Learn more about building Angular projects here.

This is a guest post written for Bitbucket by Suren Konathala. “My mission is to simplify technology adoption for organizations. I’m a developer, architect, consultant and manager. I love to write and talk about technology. Connect with me on LinkedIn or Twitter.”

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on April 30, 2019.

Bitbucket + Bitrise: Configuring Continuous Integration for an iOS app

Ash Moosa — Mon, 29 Apr 2019 17:03:05 +0000

This is a guest post written for Bitbucket by Ivan Parfenchuk. Ivan is an independent iOS and Ruby developer passionate about building delightful experiences. Connect with him on Twitter @gazebushka.

When iOS applications start growing, at some point it becomes essential to have a quick develop-release-test feedback loop. You can create this loop by doing everything manually, but it can be much quicker and more advanced if you use Continuous Integration (CI) tools.

With a CI tool, you can build up a history of releases and quickly see which build contained what. You can run tests for every build automatically and catch some inevitable bugs. You can have consistency in your release notes. And you can streamline your release cycles, which automates your checklists.

Sound interesting? Let’s try to build this feedback loop using Bitbucket Webhooks, Bitrise and fastlane.

Deployment flow

The flow we are going to use for our Continuous Integration is going to look like this:

Create & merge a Pull Request in Bitbucket
Bitbucket performs a “Webhook” HTTP request to Bitrise
Bitrise starts building the process and launches fastlane
fastlane builds the app and sends it to App Store Connect
App Store Connect processes the build, and it becomes available in TestFlight

Bitbucket Webhooks and git branching model

Each deployment starts from us creating a Pull Request.

Let’s say your team is using master git branch for code in the releasable state. It also makes new releases by merging this master branch to the release branch.

The following section describes how to create a Webhook manually. However, if you use Bitrise, it can create a Webhook for you automatically, so, feel free to skip to the Bitrise section.

Manual Webhook configuration

Next, let’s configure Bitbucket Webhooks so that whenever someone pushes to release branch or merges Pull Request to release branch, the Webhook is triggered.

To do that go to your Bitbucket repository and click “Settings” in the side menu. Then click “Webhooks” in the “Workflows” section and then click “Add webhook.”

Fill out Title, URL (see below), set Status to Active, and select “Choose from a full list of triggers” for Triggers. The triggers we are going to use are:

Repository: Push
Pull Request: Created, Updated

To get the URL for our Webhook:

head over to Bitrise, create a new app
open the Dashboard -> Your app -> Code tab
scroll to Incoming Webhooks section and click Setup Manually.
Select “Bitbucket Webhooks” and copy the Webhook URL

Bitrise and automatic Webhook configuration

Bitrise is a platform for Continuous Integration. You can configure different deployment “workflows” in it and have the Bitrise servers build and publish your application. Here are the steps to create a new deployment workflow for our CI setup.

First sign up in Bitrise, go to Dashboard, click “Add New App”
Select “Private” if you want your config and logs to stay private
Select Bitbucket and connect it to your account
Click “Auto-add SSH key” or configure SSH access manually
Enter release as branch name in the “Choose branch” step
In Project build configuration select “fastlane” , check that Fastlane lane is set to ios release.
Select the stack that you normally use to build your app or just the latest available Xcode/macOS and click Confirm.
In the last step “Webhook setup” click “Register Webhook for me”

The last step creates a Webhook in Bitbucket, so you don’t have to do anything manually. You can head over to Bitbucket repository and check the Webhook configuration in Settings -> Webhooks.

In our setup, we are going to use fastlane to build and publish the app to App Store Connect.

Fastlane configuration

Fastlane is a set of tools for automating development and release process.

Follow this guide to install fastlane:Setup — fastlane docs

In short, you need to install Xcode development tools:

xcode-select --install

and then install fastlane via RubyGems

sudo gem install fastlane -NV

or via brew:

brew cask install fastlane

Then open the working directory of your app in the Terminal and initialize fastlane.

cd /path/to/your/app 
fastlane init

Select “3. 🚀 Automate App Store distribution”

Then follow the configuration requests. Fastlane can create and configure the new App Id for you and create a sample deployment “lane.” Lane is just a collection of steps, required to complete some scenario.

Once the configuration is finished, let’s open the Fastfile which has been created and configure our first deployment script. Fastlane has large set of tools for automating various processes like code signing, uploading of screenshots, running tests and so on. However, we are going to start with a simple setup:

default\_platform(:ios)  

platform :ios do   
  desc "Push a new release build to the App Store"   
  lane :release do     
    build\_app(scheme: "CITest")     
    upload\_to\_app\_store(force: true, skip\_metadata: true, skip\_screenshots: true)   
  end 
end

The “release” lane will

perform a build of your project build_app(scheme: “CITest”)
Upload the resulting ipa file to App Store Connect upload_to_app_store. In this guide, we are skipping Fastlane metadata upload.

You can test your setup by opening your project directory in Terminal and running fastlane release :

cd /path/to/your/app/directory 
fastlane release

Code-signing

If you see problems with code-signing start here:Troubleshooting — fastlane docs. You can use fastlane match to manage code signing, but be careful: if you already have generated Certificates and Provisioning Profiles, match can break things. However, if it’s a completely new setup or you don’t care much about the existing profiles, match is going to speed things up considerably.

We’ll use fastlane match in our example:

cd /path/to/your/app/directory 
fastlane match development 
fastlane match adhoc 
fastlane match appstore

Then open the Xcode, turn off Automatic code signing and select provisioning profiles that match has generated.

After that, we can add match to our Fastfile:

default\_platform(:ios)

platform :ios do 
  desc "Push a new release build to the App Store" 
  lane :release do 
      match(type: "appstore", readonly: true) 
    build\_app(scheme: "CITest") 
    upload\_to\_app\_store(force: true, skip\_metadata: true, skip\_screenshots: true) 
  end 
end

Finishing up the Bitrise setup

Bitrise is building the project on its servers which don’t have any of your passwords and credentials required to code-sign and upload your app to App Store Connect. Therefore we’ll have to share some of those, precisely these two:

Login/password for App Store Connect user
Password to decrypt your match repository (if you use match)

The App Store Connect user doesn’t have to be the one you use to control your apps. You can create a new user in App Store Connect, which only has access to the app you automate and has at least a Developer role.

Once you set your new App Store Connect user, head over to Bitrise and open Workflow Editor tab and then Secrets. Add two new secrets:

ITUNES_CONNECT_USER and ITUNES_CONNECT_PASSWORD with App Store credentials for this new user. Plus put the same password into FASTLANE_PASSWORD secret.

If you use match, then add one more secret called MATCH_PASSWORD with the password you used to encrypt match repository.

Conclusion

That should be it. Try to create a new Pull Request and merge it and see if Bitrise triggers the new build. If everything goes well, you will see the new build in TestFlight and will be able to select it for your new iOS app version.

There is much more that you can do with automated deployments, such as:

Testing using a fastlane scan
Automated build number incrementation
dSYM uploads to Crashlytics_Raygun_etc

However, start with simple things first. I hope this guide helps you!

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on April 29, 2019.

Searching DynamoDB: An indexer sidecar for Elasticsearch

Ash Moosa — Thu, 28 Mar 2019 22:43:02 +0000

TLDR;

DynamoDB is great, but partitioning and searching are hard
We built alternator and migration-service to make life easier
We open sourced a sidecar to index DynamoDB tables in Elasticsearch that you should totes use. Here’s the code.

When we embarked on Bitbucket Pipelines more than three years ago we had little experience using NoSQL databases. But as a small team looking to produce quality at speed, we decided on DynamoDB as a managed service with great availability and scalability characteristics. Three years on, we’ve learnt a lot about how to use and how to not use DynamoDB, and we’ve built some things along the way that might be useful to other teams or that could be absorbed by the ever growing platform.

To NoSQL or not to NoSQL that is the question

First off the bat, relational databases aren’t werewolves and NoSQL isn’t a silver bullet. Relational databases have served large scale applications for years and they continue to scale well beyond many people’s expectations. Many teams in Atlassian continue to choose Postgres over DynamoDB for example and there are plenty of perfectly valid reasons to do so. Hopefully this blog will highlight some of the reasons to choose one technology over the other. At a high level they include considerations such as operational overhead, the expected size of your tables, data access patterns, data consistency and querying requirements.

Partitions, partitions, partitions

A good understanding of how partitioning works is probably the single most important thing in being successful with DynamoDB and is necessary to avoid the dreaded hot partition problem. Getting this wrong could mean restructuring data, redesigning APIs, full table migrations or worse at some time in the future when the system has hit a critical threshold. Of course there is zero visibility in to a table’s partitions — you can calculate them given a table’s throughput and size but it’s inaccurate, cumbersome and we’ve found, largely unnecessary if you’ve designed well distributed keys as the best practices developer guide suggests. Fortunately we took the time to understand partitioning from the get go and have managed to avoid any of these issues. As an added bonus, we’re now able to utilize autoscaling without concern for partition boundaries because requests remain evenly distributed even as partitions change and throughput is redistributed between them.

Throughput, bursting and throttling

Reads and writes on DynamoDB tables are limited by the amount of throughput capacity configured for the table. Throughput also determines how the table is partitioned and it affects costs so it’s worth ensuring you’re not over provisioning. DynamoDB allows bursting above the throughput limit for a short period of time before it starts throttling requests and while throttled requests can result in a failed operation in your application, we’ve found that it very rarely does so due to the default retry configuration in the AWS SDK for DynamoDB. This is particularly reassuring because autoscaling in DynamoDB is delayed by design and allows throughput to exceed capacity for long enough that throttling can occur.

DynamoDB auto scaling modifies provisioned throughput settings only when the actual workload stays elevated (or depressed) for a sustained period of several minutes. The Application Auto Scaling target tracking algorithm seeks to keep the target utilization at or near your chosen value over the long term.

Sudden, short-duration spikes of activity are accommodated by the table’s built-in burst capacity.

You’ll still want to set alerts for when throughput is exceeded so you can monitor and act accordingly when necessary (e.g. there’s an upper limit to autoscaling) but we’ve found that burst capacity, default SDK retries, autoscaling and adaptive throughput combine extremely effectively such that intervention is seldom required.

Source

Alternator: an object-item mapping library for DynamoDB

After Pipelines got the green light and refactoring of the horrendous alpha code began, one of the first things we built was alternator: an internal object-item mapping library for DynamoDB (similar to an ORM). Alternator abstracts the AWS SDK from the application and provides annotation based, reactive (RxJava — although currently still using the blocking AWS API under the covers until v2 of the SDK becomes stable) interfaces for interacting with DynamoDB. It also adds circuit breaking via Hystrix removing much of the boilerplate code that was present in early versions of the system.

@Table(
    name = “pipeline”,
    primaryKey = @PrimaryKey(hash = @HashKey(name = “uuid”))
)

@ItemConverter(PipelineItemConverter.class)
public interface PipelineDao {
    @PutOperation(conditionExpression = “attribute\_not\_exists(#uuid)”)
    Single<Pipeline> create(@Item Pipeline pipeline);
}

The migration service

DynamoDB tables are of course schema-less, however that doesn’t mean that you won’t need to perform migrations. Aside from a typical data migration to add or change attributes in a table, there are a number of features that can only be configured when a table is first created such as local secondary indexes which are useful for querying and sorting on additional attributes other than the primary key.

The first few migrations in Pipelines involved writing bespoke code to move large quantities of data to new tables and synchronizing that with often complex changes in the application to support both old and new tables to avoid downtime. We learnt early on that having a migration strategy would remove a lot of that friction and so the migration service was born.

The migration service is an internal service we developed for migrating data in DynamoDB tables. It supports 2 types of migrations:

Same table migrations for adding to or amending data in an existing table.
Table to table migrations for moving data to a new table.

Migrations work by scanning all the data in the source table, passing it through a transformer (that is specific to the migration taking place) and writing it to the destination table. It does this using a parallel scan to distribute load evenly amongst a table’s partitions and maximize throughout to complete the migration in as short a time as possible.

Table to table migrations then attach to the source table’s stream to keep the destination table in sync until you decide to switch over to using it. This allows the application to switch directly to using the new table without having to support both old and new tables during the migration.

Querying in DynamoDB: a tale of heartache and misery

DynamoDB provides limited querying and sorting capability via local and global secondary indexes. Tables are restricted to five local secondary indexes, with each index sharing the same partition key as the table’s primary key, and a query operation can only be performed against one index at a time. This means that on a “user” table partitioned by email address, a query operation can only be performed in the context of the email address and one other value.

Global secondary indexes remove the partition key requirement at the cost of paying for a second lot of throughput, and only support eventually consistent reads. Both types of indexes are useful and sufficient for many use cases and Pipelines continues to use them extensively but they do not satisfy the more complex querying requirements of some applications. In Pipelines this need predominantly came from our REST API which rather typically allows clients to filter and sort on multiple properties at the same time.

Source

Our solution

Our first attempt at solving this problem, MultiQuery, was built in to an alternator. With this approach we queried multiple local secondary indexes (LSI) and aggregated the results in memory allowing us to perform filtering and sorting on up to five values (the maximum number of LSI) in a single API request. While this worked at the time, it started to suffer from pretty terrible performance degradation as our tables grew in size and exponentially so the more values you were filtering on. Before multi query was replaced, the pipelines list page on our end-to-end test repository (one with a large number of pipelines) took up to tens of seconds to respond and consistently timed out when filtering on branches.

A common pattern for searching DynamoDB content is to index it in a search engine. Conveniently AWS provides a logstash plugin for indexing Dynamo tables in Elasticsearch so we set about creating an indexing service using this plugin and the results were encouraging. Query performance vastly improved as expected but the logstash plugin left a lot to be desired taking almost 11 hours to index 700,000 documents.

Some analysis of the logstash plugin and the realization that we had already built what was essentially a high performance indexer in the migration service led us to replace the logstash plugin with a custom indexer implementation. Our indexer, largely based on the same scan/stream semantics of the migration service and utilizing Elasticsearch’s bulk indexing API, managed to blow through almost 7 million documents in 27 minutes.

The indexer sidecar

The custom indexer has since been repackaged as a sidecar, allowing any service application to seamlessly index a DynamoDB table in Elasticsearch. Both the initial scan and ongoing streaming phases are made highly available and resumable by a lease / checkpointing mechanism (custom built for the scan, standard kinesis client for the stream).

We currently utilize the excellent elasticsearch client built by the Bitbucket code search team to query the index and have started work on an internal library which adds RxJava and Hystrix in the same vein as alternator.

Here is the repo with the code and a readme.

Final thoughts

If you haven’t used NoSQL before, it certainly requires a mind shift but overall our experience with DynamoDB has been a positive one. The platform has proven to be extremely reliable over the past three years (I can’t remember a single major incident caused by it) with our biggest challenge coming from our querying requirements. Some might say that’s reason enough to have chosen a relational database in the first place and I wouldn’t strongly disagree with them. But we’ve managed to overcome that issue with a solution that is for the most part abstracted away from day to day operations. On the plus side, we haven’t had to run an explain query in all that time or deal with poorly formed SQL, complex table joins or missing indexes and we’re not in a hurry to go back.

This post was written by Sam Tannous. Sam is a senior software engineer at Atlassian and is part of the team that brought CI/CD Pipelines to Bitbucket cloud. With over 15 years industry experience he has co-authored 3 patents, maintains 1 open source project and has a track record of successfully delivering large scale cloud based applications to end users. Connect with him on linkedin.

Originally published at bitbucket.org on March 28, 2019. To contribute to the Bitbucket blog, apply here.