Forem: Bartosz Gajda

5 Essential skills for becoming a Data Engineer

Bartosz Gajda — Sat, 19 Sep 2020 12:47:05 +0000

The Data Engineer skills is a hot topic, for everyone interested in becoming a one. The rise of data platforms wouldn't be possible without the data engineer skills, that develop the infrastructure and tools. The need for specialists in this area is forecasted to only increase, therefore if your are considering on becoming a Data Engineer, those are my 5 essential skills you have to poses. Enjoy!

Programming - Scala and/or Python

The first and the most important skill - programming. The job of a Data Engineer requires to write a significant amount of custom, tailored code, which cannot be otherwise generated using fancy GUI software. The ability to code and to create readable, clean and scalable code is a must.

I have chosen two programming languages here: Scala and Python. I am not going to lie, that Scala is my personal favorite (if you haven't guessed already by the number of posts on this topic :)) although Python is also a good choice.

Some of the Data Engineering frameworks have been written in Scala (eg. Apache Spark), and it works great when you need more control over the code.

Python is probably much easier to start with and generally more newbie friendly, although it has some performance issues, which in data world might be critical.

A good option is to learn one of those to a proficient stage, and other one to at least basics - it is always better to have another tool under your belt, even tough it might not be as good as others.

If you are looking to learn Scala, be sure to check out my post that explains where to start from.

Data Manipulation - SQL

Despite the fact that in data world there is a flood of semi and unstructured data, SQL is still the de facto data manipulation language. Whether you are working with relation database, data warehouse or a processing engine like Spark - SQL is widely supported.

Some of the technologies use their own variations of SQL, like a CQL in Cassandra distributed database system, and there isn't anything on the horizon that would make SQL go away.

Not all data engineering tasks are focused on building fancy Machine Learning pipelines, or Stream Processing. Lots of simple problems can be solved by writing a SQL query. Modern SQL engines can handle automatic optimization of those queries, which is great if high performance programming is not your strong skill.

Processing Engine - Apache Spark

The third element on the list, is probably my favorite - Apache Spark processing engine. As a Data Engineer you don't want to reinvent the wheel - you want to use the framework that is capable of solving majority of problems in unified and repeatable way.

The days of Hadoop's MapReduce are long gone, and the Apache Spark is something you should focus on. It is the most popular general purpose processing engine, that can to ETL, batch and streaming processing, machine and deep learning and many more.

The nice thing about Spark is that it supports 3 languages: Java, Scala and Python. R is also supported by to lesser extend.

Learning Apache Spark may seem cumbersome at the start, but once you get a good grasp of the basics, it's very pleasant to work with. However, once you will go to the advanced topics, like optimization, various types of testing and deployment, the learning curve becomes quite steep.

If your are looking for a quick intro on how to get started with Spark, check out my post on it.

Cloud - AWS or GCP or Azure

I hope I don't have to convince you that cloud is now everywhere. The data platforms and data engineering pipelines are no different - lots of those utilize cloud services.

Storing data on AWS S3, streaming data with GCP Pub/Sub or storing data in Azure Cosmos DB - those combinations are frequently found in lots of commercial systems. Whether you are just starting with the cloud, or have some previous experience - being a data engineer means using the cloud regularly.

Out of those three, the AWS is probably the safest choice. It's the most popular, has the most services, is widely known and there are lots of tutorials that can guide you through.

My personal favorite however is the Google Cloud Platform - it's extremely developer friendly, has an awesome selection of services and access to the most advanced AI and ML products. Google is pushing strongly into the cloud territory and I for a good reason - companies like the flexibility and agility it gives to developer teams.

For the Azure, I don't have a lot of experience using it, so I leave this choice to you. Although I know that they work closely with Databricks, a company behind Apache Spark, and this might bring some interesting advances in data engineering field.

If you are considering learning GCP and getting ACE certified, be sure to check my post on best resources to prepare for the exam.

Deployment - Docker and Kubernetes

Last but not least - the deployment. And even more precisely - containerization. The Docker and Kubernetes are combination that made the containers a sensible alternative to cumbersome VMs and clusters, like in the old days. Now, companies want to ship their products fast, and be ready to scale when business is getting traction, therefore scalable deployment is a must.

Docker will help you pack your applications into a containers, that can be easily scaled up or down, depending on a need. Kubernetes handles the orchestration of clusters of containers - in the end data platforms are quite complicated beings, with lots of moving parts and you want to ensure they are prepared for extra demand and resilient to failures.

If you are looking for some nice tips and tricks when working with Docker, be sure to check out my post on this.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so :)

RESTful Web scraping in Scala, using Play Framework and Jsoup

Bartosz Gajda — Sat, 19 Sep 2020 12:43:39 +0000

Recently I have encountered a very cool site with cooking recipes, which had extremely poor UI, especially when using a mobile. There was no official API, and so I have decided to build a web service that would web scrape the content out of it, and publish it using RESTful API. In this post I will show you how to use Scala, Play Framework and Jsoup to build such service. Enjoy!

You will need

Scala - I use the version 2.13.1
Sbt - install from official website
IDE of choice - IntelliJ or VS Code, I use latter
Around 15 minutes of your time

Creating empty Play project

We will start off by creating a Play project. You can of course start from the scratch and add the code iteratively, but for the scope of this tutorial, I will suggest using on of the Play's example projects as a starter.

The Scala Rest API project have almost everything we need. This project serves as a nice backbone to our use case, which we will be able to extend with ease.

Let's start by cloning the Play Framework samples repository, and opening the Scala Rest API example like this:

git clone https://github.com/playframework/play-samples.git
cd play-scala-rest-api-example

After this, try running this command - the Play application should shoot off, and you should be able to access it at http://localhost:9000

sbt run

If the application builds and runs correctly, we are done with initializing the project. Now, you can open this folder with your IDE of choice - I will use VS Code

Adding Jsoup dependency

Next, we will add the Jsoup - an open source HTML parser for Java and JVM. To do this, open a build.sbt file and add the following in the libraryDependencies block:

"org.jsoup" % "jsoup" % "1.13.1"

Of course, make sure to use the latest version possible - at the time of authoring this article, 1.13.1 was the latest one.

Now, wait for sbt to refresh the dependencies, and we are ready to write some code!

Defining an endpoint

The first coding step is to define an endpoint to Play framework. This is done, by adding route definitions to conf/routes file. In our case, I want to have a v1/recipes endpoint that is solely handled by RecipeRouter class. We can do this by adding the following:

->         /v1/recipes             v1.recipe.RecipeRouter

Now, any request that hits that URL, will be redirected to our router.

Next of, I am creating a package for my recipe related classes app/v1/recipe. In here, let's create an entry point that will handle the requests: RecipeRouter

package v1.recipe

import javax.inject.Inject

import play.api.routing.Router.Routes
import play.api.routing.SimpleRouter
import play.api.routing.sird._

class RecipeRouter @Inject()(controller: RecipeController) extends SimpleRouter {
  val prefix = "/v1/recipes"

  def link(id: String): String = {
    import io.lemonlabs.uri.dsl._
    val url = prefix / id.toString
    url.toString()
  }

  override def routes: Routes = {
    case GET(p"/") =>
      controller.index

    case GET(p"/all/$id") =>
      controller.showAll(id)
  }

}

Connecting to website

Going further, it's time to finally use Jsoup to get some content out of the website. In the previous section, we have defined showAll function, and that's the one I will be implementing here. We will be making use of the id parameter, which come in handy in many situations.

So, let's move to RecipeController class, and define the first element: sourceUrl. I like to have it defined as a class field, so that it is accessible by all methods:

val sourceUrl: String = "https://kwestiasmaku.com"

Then, inside our method, let's get the content of this website. It can be done like this:

def showAll(pageId: String): Action[AnyContent] = Action { implicit request =>
    val htmlDocument = Jsoup.connect(s"${sourceUrl}/home-przepisy?page=${pageId}").get()

    val r: Result = Ok(htmlDocument)
    r
}

In the snippet above, we are connecting to our website, b using Jsoup.connect method, while passing the correct URL as a string. What we get, is an HTML document, that we just return for now. We will do some finer extraction in the next section.

To test the endpoint, call the following URL in your browser, or any http client, like curl:

http://localhost:9000/v1/recipes/all/1

You should see a response, which is raw HTML document. Now, let's parse it and extract some meaningful data.

Extracting the content

Now, let's extract some specific elements from the raw HTML. For this, we can use the select method of Jsoup. This method requires a CSS like selector syntax - if you have used CSS before, then it is as easy specifying the correct string - in our case, to extract single recipe HTML object, use the following:

import scala.jdk.CollectionConverters._

val recipesDomElements = htmlDocument.select("section#block-system-main .col").asScala

This query returns, collection-like structure, on which we can easily iterate. To extract specific elements from a single recipe, use the following:

final case class Recipe(title: String, href: String, img: String)

val recipeData = for(recipeElement <- recipesDomElements)
      yield Recipe(
        recipeElement.select(".views-field-title a").html(),
        sourceUrl + recipeElement.select(".views-field-title a").attr("href"),
        recipeElement.select("img").attr("src")
      )

In here, we use a simple for loop, which yields the concrete data, by wrapping it into Recipe case class. Now, we have nicely cleaned data, and we can finally return it as JSON.

Returning a JSON

Last step is to return our collection of Recipe case classes as a JSON. We can do this using Play's built in Json class, and its methods:

implicit val recipeWrites: OWrites[Recipe] = Json.writes[Recipe]

val r: Result = Ok(Json.toJson(recipeData))
r

An important thing to have in mind, is that before converting a case class into its JSON representation, we must declare implicit writer, so that the Json class knows how to write this specific data.

And that's it! The complete RecipeController class implementation looks like the following:

package v1.recipe

import javax.inject.Inject
import org.jsoup.Jsoup
import play.api.mvc._
import play.api.libs.json.{Json, OWrites}

import scala.jdk.CollectionConverters._

class RecipeController @Inject()(val controllerComponents: ControllerComponents) extends BaseController {
  implicit val recipeWrites: OWrites[Recipe] = Json.writes[Recipe]
  val sourceUrl: String = "https://kwestiasmaku.com"

  def index: Action[AnyContent] = Action { implicit request =>
    val r: Result = Ok("hello world")
    r
  }

  def showAll(pageId: String): Action[AnyContent] = Action { implicit request =>
    val htmlDocument = Jsoup.connect(s"${sourceUrl}/home-przepisy?page=${pageId}").get()
    val recipesDomElements = htmlDocument.select("section#block-system-main .col").asScala

    val recipeData = for(recipeElement

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Scala collections in a nutshell

Bartosz Gajda — Sat, 19 Sep 2020 12:38:47 +0000

The Scala language, being a mix of object oriented and functional programming paradigms, has a quite unique collection framework, compared to some other languages. Although inheriting a lot from Java and JVM world, the Scala has a much easier and to use API and is overall more polished.

In this post, I will show you a range of Scala collections, how to use them with code samples, and where to use them. Enjoy!

Scala `collection` package(s)

First, let's have a look at the packages that contain all the Scala collections. The top-level package is called as you might expect - collection. What is more interesting, are the sub-packages included in it. Those contain a more specialized version of typical collections, tailored for the specific use case.

Whether you require parallel processing, concurrent access or just want to have a normal key-based lookup - just import the right package, and you are good to go!

The table below presents the name of collections sub-package, as well as brief description of what you can find there.

Package Name	Description
`collection`	Defines the base traits and objects needed to use and extend Scala’s collections library, including all definitions in sub-packages. Most of the abstractions you’ll work with are defined here.
`collection.concurrent`	Defines a Map trait and TrieMap class with atomic, lock-free access operations.
`collection.convert`	Defines types for wrapping Scala collections with Java collection abstractions and wrapping Java collections with Scala collection abstractions
`collection.generic`	Defines reusable components used to build the specific mutable, immutable, etc. collections.
`collection.immutable`	Defines the immutable collections, the ones you’ll use most frequently.
`collection.mutable`	Defines mutable collections. Most of the specific collection types are available in mutable and immutable forms, but not all.
`collection.parallel`	Defines reusable components used to build specific mutable and immutable collections that distribute processing to parallel threads
`collection.parallel.immutable`	Defines parallel, immutable collections.
`collection.parallel.mutable`	Defines parallel, mutable collections.
`collections.script`	A deprecated package of tools for observing collection operations.

Source: Programming Scala, 2nd Edition

As you can see the range of options is very wide. On one hand, it is always good to have an appropriate collection for any use case. On the other hand, it is quite difficult to get started, if you are just looking to implement simple Map.

For the purpose of this post, let's focus on the most widely used collections - immutable

`collection.immutable`

The package of immutable collections is the most popular out of all Scala collections. In fact, this package is included in the scope by default, so we can start using it without any imports.

This package includes lots of both traits and concrete implementations of certain data structures. The image below shows those (blue is a trait, and black is class):

Source: Scala Docs

Now, let's set that aside and actually learn to use some of the basic collections. The following sections focus on some of the widely used implementations, one at a time.

`Set`

The first one on the list is Set. This is a collection, which contains no duplicate elements. If having only unique elements in the collection, that is the one you should choose. A Set is created like that:

// Set of Char type
val set: Set[Char] = Set('a', 'b', 'c')

// Or just empty Set
val setOfInt: Set[Int] = Set()

What's interesting about operations in Set, is the apply method. The apply method behaves like contains method - both check whether the Set contains a given element:

val set: Set[Int] = Set(1, 2, 5, 6, 9)

set.contains(2) // true
set(2) // same as above, returns true

A Set offers a wide range of operations - full reference can be found here.

Where to use Set?

Don't want duplicates
Need to quickly check for the presence of an element
No need to traverse

`Map`

A Map is a collection of key-value pairs. Every key has a corresponding value assigned to it. Keys have to be unique, in order to provide O(1) lookup time. Values, however, do not have to be unique. Let's see how we can create a Map:

// A Map of Int to String mapping
val map: Map[Int, String] = Map(1 -> "one", 2 -> "two", 3 -> "three")

// Or an empty Map
val emptyMap: Map[String, String] = Map()

The elements in the Map, are created using key -> value syntax, although you can also use (key, value) if you prefer (first version is preferred though).

What's interesting about the Map, is how we can get a value, using a key. The default method to get a value is get(key) method. This method, however, returns an Option[T], and not T. That is to accommodate for a situation when a key doesn't exist - instead of an exception, an Option[None] is returned. If you prefer getting straight value, the apply method will give you that (or exception if none found of course). Let's see how it looks:

val map: Map[Char, Int] = Map('I' -> 1, 'V' -> 5, 'X' -> 10)

map.get('I') // returns Option[Int]
map.get('D') // returns Option[None]

map('V') // returns 5
map('C') // throws exception

A Map offers a wide range of operations - full reference can be found here.

You can learn more about Option in my blog post on it.

Where to use Map?

Need key-value storage
Need O(1) lookup and retrieval
Order is not important
No duplicate keys required

`Vector`

A Vector is an indexed, ordered, and traversable sequence. Despite sounding quite complicated, it's quite popular, general-purpose data structure, thanks to its good balance between fast random selection and fast random functional updates. Let's see how to use them in practice:

// A Vector of Integers
val vector: Vector[Int] = Vector(1, 2, 3, 5, 7, 11)

// Or an empty Vector
val emptyVector: Vector[String] = Vector()

The apply method of Vector, simply gets and element, using and index in the vector. In this data structure, an index must be between zero and the length of a vector - otherwise, and IndexOutOfBoundsException is thrown:

val vector: Vector[String] = Vector("one", "two", "three")

vector(0) // returns "one"
vector(3) // exception

A Vector offers a wide range of operations - full reference can be found here.

Where to use Vector?

Need fast random access and updates
Need fast append/prepend/tail operations
Must be traversable
Ordering matters

`List`

The last collection is the easiest and therefore popular among simple use cases - List. This collection is implemented as a linked list, which represents ordered collections. Let's see how to use a List:

// List of Strings
val list: List[String] = List("blue", "red", "yellow")

// Empty List
val emptyList: List[Int] = List()

As mentioned above, a List is implemented as a linked list. This means, that we can create a list, using prepend operator :: and a Nil, which represents empty list. This can be done like this:

// Same as example from above
val list: List[String] = "blue" :: ("red" :: ("yellow" :: Nil))

// And an empty List
val emptyList = Nil

A List offers a wide range of operations - full reference can be found here.

Where to use List?

Have to be traversable
Need O(1) prepending and reading of head element
Can tolerate O(n) appending and reading of internal elements
Don't need random access

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

5 best study resources for GCP ACE exam

Bartosz Gajda — Mon, 10 Aug 2020 13:20:39 +0000

Recently, I have finally had a chance to attempt an exam for Google Cloud Certified Associate Cloud Engineer certification and I have passed in first attempt! This achievement has been possible, thanks to thorough preparation, during which I have used a selection of resources. In this post I will be sharing with you 5 of those that I think have contributed the most to the positive outcome of the GCP ACE exam. Enjoy!

1. Official GCP ACE Study Guide

The first element on the list is probably the most comprehensive source of knowledge you can get to prepare for GCP ACE exam. This book consists of 18 chapters, each covering a large chunk of knowledge you have to possess in order to pass the exam. Chapters include screenshots, so you can easily recreate the steps shown on the paper. Each of those chapters is finished with 20 question mini-exam, that reflects quite good the questions you might encounter on real assignment.

In addition to that, you also get two full practice exams and around 100 flashcards. The flashcards are especially useful in the learning process, but I know it’s quite personal thing.

This books costs typically around £25, which I think is really good bang for the buck.

2. A Cloud Guru Course

The second source I have used was A Cloud Guru course on GCP ACE certification exam. I have used the Udemy to get it, but you can also buy it from an official page (it’s probably better to buy at the source).

The course is quite comprehensive, especially the GKE and Kubernetes related chapters. It takes around 10–12 hours to complete, depending on your studying pace. This course also covers all the chapters like the book, but it comes in much more practical fashion.

Online courses have big advantage over written word, as it is easier to follow what the instructor is doing and getting hands-on experience of the platform.

The price of this course varies — I got it on Udemy for around £15, but that was during sale (which thankfully are quite often on Udemy). If you buy it directly, it is going to cost you a bit more.

3. Linux Academy Course

Another online course comes from Linux Academy. They are known for their high-quality exam prep course and this one is no exception. It takes a bit longer than A Cloud Guru one but in my opinion it’s for the better.

This course covers all of the exam study points (as all of the above sources) but in my personal opinion, does the best work of showing how to actually use the GCP in practice. The lessons are optimally sized and the instructor does a great job of explaining the inner workings of Google Cloud services.

This course is at the time of writing only available on Linux Academy site. The best option there is to go with membership — that will cost you \$49 per month.

4. Google Code Labs

Google Code Labs is my personal favourite on this list. This site is a collection of labs that are completely free and guide you step by step how to implement a real-world solution or application on Google Cloud Platform.

There is a broad selection of labs, covering more Google services than you actually need to pass the ACE exam. Each lab shows you the exact steps you have to complete, in order to finish with complete and working solution to a selected problem (like deploying a Spring Web app on Google App Engine).

It is a great resource not only for ACE exam but for any Google Cloud certification.

5. Google Qwiklabs

The last resource on the list is probably well known to anyone preparing for GCP Associate Cloud Engineer exam. The Qwiklabs provides a so called ‘quests’ which are a form of labs that guide you Google Cloud Platform and show the underlying fundamentals of this environment.

It’s quite comprehensive, although I wouldn’t risk attempting the exam after the Qwiklabs only, but it serves as great supporting material.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Bartosz Gajda

@bartoszgajda55

Want to become @googlecloud Certified Associate Cloud Engineer? This post shows 5 best resources to study and pass the exam with ease!

bartoszgajda.com/2020/07/13/5-b…

#google #googlecloudplatform #gcp #gcpace #asociatecloudengineer #exam #certification #cloud #aws #azure

18:43 PM - 28 Sep 2020

1 0

Exploiting Schema Inference in Apache Spark

Bartosz Gajda — Mon, 10 Aug 2020 13:18:25 +0000

One of the greatest feature of Apache Spark is it’s ability to infer the schema on the fly. Reading the data and generating a schema as you go although being easy to use, makes the data reading itself slower. However, there is a trick to generate the schema once, and then just load it from disk. Let’ dive in!

For the sake of this article, let’s assume that we are using data with complex data structure, where creating a case class or struct type of any kind would be a hundred lines long.

Saving the Schema

Let’s begin with reading some sample data and enforcing a schema inference on it. Using Apache Spark and Scala language, this can be done like following:

val carsDf = spark.read
    .option("inferSchema", "true")
    .json("src/main/resources/data/cars.json")

I read a sample JSON file (cars.json) to a DataFrame called carsDf. A sample record of this dataset looks like the following:

{
   "Name":"chevrolet chevelle malibu",
   "Miles_per_Gallon":18,
   "Cylinders":8,
   "Displacement":307,
   "Horsepower":130,
   "Weight_in_lbs":3504,
   "Acceleration":12,
   "Year":"1970-01-01",
   "Origin":"USA"
}

As we can see above, the schema is fairly straight forward to understand. Now, let’s see what Spark has managed to infer from it:

val schemaJson = carsDf.schema.json
println(schemaJson)

The output of this action can be something like this:

{
   "type":"struct",
   "fields":[
      {
         "name":"Acceleration",
         "type":"double",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Cylinders",
         "type":"long",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Displacement",
         "type":"double",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Horsepower",
         "type":"long",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Miles_per_Gallon",
         "type":"double",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Name",
         "type":"string",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Origin",
         "type":"string",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Weight_in_lbs",
         "type":"long",
         "nullable":true,
         "metadata":{

         }
      },
      {
         "name":"Year",
         "type":"string",
         "nullable":true,
         "metadata":{

         }
      }
   ]
}

That is how Spark defines the schema it managed to infer. Now, this information would have to be recreated every time we launch our Spark job. When doing development, there is a lot of that, and so we are losing a lot of time on recreation of that schema. Let’s try to generate it once and reuse in consecutive reads.

In the Scala I am using, the schema’s JSON file can be saved to local filesystem using the code:

val file = new File("src/resources/schema.json")
val bw = new BufferedWriter(new FileWriter(file))
bw.write(schemaJson)
bw.close()

This way, our inferred schema is saved to local filesystem as a JSON file called schema.json. In the next step, we will see how to read it and speed up the Spark with it.

Loading the Schema

Now, let’s see how to read the JSON file using Spark and Scala environment. This can be done using the following code:

import org.apache.spark.sql.types.{DataType, StructType}

val schemaJson = Source
    .fromFile("src/resources/schema.json")
    .getLines
    .mkString

val schemaStructType = Try(DataType.fromJson(schemaJson))
        .getOrElse(LegacyTypeStringParser.parse(schemaJson))
    match {
      case t: StructType => t
      case _ => throw new RuntimeException(s"Failed parsing JSON Schema")
    }

The section of code above, reads the schema from JSON file and parses it into a StructType instance, thanks to Spark SQL package. I have used pattern matching here, to accommodate for the incorrect path being used.

With the pre-generated schema being available, reading the data in Spark will be way faster. Using our example with cars.json file, we can read this data like this:

val carsDf = spark.read
    .schema(schemaStructType)
    .json("src/main/resources/data/cars.json")

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Scala Type Bounds

Bartosz Gajda — Mon, 10 Aug 2020 13:16:36 +0000

Working with types in Scala can be challenging in many ways. The deeper you dig into the complexity of this subject, the more difficult is to find the right path. In this short post, I will be explaining some of the most fundamental of Scala type system — the type bounds. Enjoy!

What is type bound?

A type bound is nothing more than a mechanism to add restrictions to the type parameters. When passing a type to let’s say a class, we can limit to what types we want to allow. This mechanism allows for greater control over our code and hopefully making it more robust.

There are 3 distinct type bounds available in Scala — upper bound, lower bound and upper and lower combined. The chapters below explain how to use each of them.

Upper type bound

An upper type bound constraints a type A to be a subtype of B. The Scala notation looks like this:

    class SomeClass[A <: B]

The example above defines the type of A as a subtype of B. To understand it better, let's use some better example:

    trait Eatable

    class Vegetable extends Eatable
    class Meat extends Eatable

    class Potato extends Vegetable
    class Beef extends Meat
    class Orange extends Eatable

    class Vegan[A <: Vegetable]
    class Carnivore[A <: Meat]
    class Food[A <: Eatable]

    val vegan = new Vegan[Potato] // works fine
    val mixed = new Carnivore[Potato] // error - type argument doesn't conform to type parameter bound
    val all = new Food[Beef] // all good

As seen on a few examples above, the mechanism is pretty easy to use. The class Vegan, requires its type parameter to a subtype of Vegetable, which is exactly what Potato is. On the other hand, class Carnivore requires its parameter type to be a subtype of Meat, which Potato isn't hence the error. The Food class accepts any class that extends Eatable trait - a Beef is one of them.

Lower type bound

A lower type bounds works as one would expect — it limits the type A to be a supertype of B. The notation looks like this:

    class OtherClass[A >:  B]

The notation is similar to previous bound — only the less than sign is replaced with greater than sign. Let’s use the classes from previous examples to see the lower type bound in practice:

    trait Eatable

    class Vegetable extends Eatable
    class Meat extends Eatable

    class Potato extends Vegetable
    class Beef extends Meat
    class Orange extends Eatable

    class Stew[A >: Potato]
    class BBQ[A >: Beef]
    class Juice[A >: Orange]

    val stew = new Stew[Vegetable] // works fine
    val BBQ = new BBQ[Meat] // fine as well
    val juice = new Juice[Potato] // error

The lower type bound is similarly easy to use. We specify three more concrete classes: Stew, BBQ, and Juice. Stew requires its type parameter to a supertype of Potato, for BBQ the type parameter should be a supertype of Beef and for Juice - supertype of Orange. First to instantiations work fine, but the last one fails - the Potato is by no means a supertype of Orange, therefore an error is thrown.

Lower and upper type bound

Last example is actually a combination of previous two. Scala allows to define both lower and upper bounds in the same time. The notation for this looks like this:

    class LastClass[A <: B >: C]

In this example, the type A, has to be a subtype of B, while also being a supertype of C. As complicated as it may sound, it will be easier to understand using our 'food' examples:

    trait Eatable

    class Vegetable extends Eatable
    class Meat extends Eatable

    class Potato extends Vegetable
    class Beef extends Meat
    class Orange extends Eatable

    class MarisPiper extends Potato
    class Wagyu extends Beef
    class Curacao extends Orange

    class RedMeat[A <: Meat >: Wagyu]
    class RootVegetable[A <: Vegetable >: MarisPiper]

    val stew = new RedMeat[Beef] // all good
    val BBQ = new RootVegetable[Potato] // works fine
    val juice = new RootVegetable[MarisPiper] // error

This example should be fairly self-explanatory — we have two classes that use both upper and lower type bounds for their type parameters: RedMeat and RootVegetable. From the classes defined, the Beef works for the first one, and Potato for the second one. The Meat doesn't meet the upper type bound and Wagyu the lower type bound for RedMeat class type parameters.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Bartosz Gajda

@bartoszgajda55

Scala's type bounds explained - upper bound lower bound and combination of both. Leverage @scala_lang powerful type system mechanism.

bartoszgajda.com/2020/06/14/sca…

#scala #scalalang #fp #oop #programming #tutorial #typesystem #java #jvm

13:10 PM - 15 Sep 2020

0 0

Why I have chosen AWS Lightsail to host my site

Bartosz Gajda — Mon, 10 Aug 2020 13:11:17 +0000

When I have decided to start writing my own programming blog, I wasn’t sure what is the best platform to start on. I was convinced that despite its flaws, WordPress will be my CMS of choice. But that wasn’t the only choice to make — a hosting platform had to be chosen as well. In this post, I will explain why I think AWS Lightsail is the optimal service to do this.

What is AWS Lightsail?

Lightsail is an easy-to-use cloud platform that offers you everything needed to build an application or website, plus a cost-effective, monthly plan.
*AWS Website*

The Amazon Web Services (AWS) ecosystem consists of several hosting options and Lightsail is one of them. As the quote above explains — it’s a managed hosting service that combines computing and storage with affordable pricing. This service itself can support a range of use cases, from test environment for hobby projects to fully-featured web applications.

Now, let’s dive deeper into why I think it’s a great platform for anyone looking for easy and cheap hosting of their site.

1. It’s fully managed

The Lightsail is a managed service, which means that we do not have to think about the underlying infrastructure or platform, updates or anything like it. No need to configure networking, security and what not — it’s all taken care by AWS. What is great as well about Lightsail, is its User Interface.

What you get is a simple and effective dashboard for managing your instances. In reality, you really don’t need to do anything over there, except for making database snapshots from time to time.

If your really need to get connected directly to the instance, there is an option to do it. You can connect using popular clients like PuTTy or use the AWS’s built in Shell.

2. It supports many environments

I use the Lightsail to host the WordPress instance, but that is not the only option. As of writing this post, the AWS allows to choose from 25 different environment options, depending on your needs.

As you can see on the screenshot above, the options are quite diverse and broad. I personally use the WordPress option only, but I assume other environment works equally good if not better.

Hosting the hobby projects, thanks to PHP, Node and Python runtimes is very easy. The instance creation wizard is just a click-through setup that doesn’t require any AWS specific knowledge.

3. It’s cheap

For some this might be the most important point — hosting on AWS Lightsail is just cheap. The cheapest plan you can get (and the one I am actually using) will cost you $3.5 per month. If you really need some computing power, you can up to 8 CPUs and 32GB of RAM for $160 per month.

From my experience, I can say that for hosting a WordPress blog, the cheapest plan will do just fine. I don’t have a massive traffic on my site, and my monthly viewer count ranges around 1000, so 512MB of RAM and 1 CPU is enough.

For hosting load and traffic heavy applications, I would recommend revising other services — $160 per month is quite a lot and other AWS services might provide same computing power with a smaller price tag.

4. It scales automatically

Because AWS Lightsail is a managed service, the scaling is automatic by default. This is a great option if your website gets spikes in traffic or load, or you expect your site to grow rapidly. I have myself noticed that my WordPress instance gets significantly more traffic when I share a new post on the Social Medias. Despite that, I have never had a problem accessing the site, nor any of my visitors.

One thing about scaling that has to be kept in mind is that it costs — a spike in traffic that requires to upscale the website will surely be seen on a monthly bill.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Bartosz Gajda

@bartoszgajda55

Looking for easy and cheap website hosting - @awscloud Lightsail is the way to go. I explain why hosting @WordPress on #AWS Lightsail is the best option.

bartoszgajda.com/2020/06/01/why…

#aws #amazonwebservices #cloud #lightsail #awslightsail #deployment #hosting #wordpress #wp

10:46 AM - 31 Aug 2020

0 0

Partial Functions in Scala

Bartosz Gajda — Mon, 10 Aug 2020 13:09:23 +0000

Writing functions and methods that process the input data and produce some output is the very core of any type of programming. When diving deeper into different types of functions in Scala, there is one type that differs from others — partial function. As the element of Scala’s standard library, it is appropriate to know what it is and where it should be used. Let’s dive in!

What is Partial Function?

A partial function of type PartialFunction[A, B] is a unary function where the domain does not necessarily include all values of type A.
Scala Standard Library

The definition above, taken straight from the Scala lang standard library docs may seem quite obfuscated. Fear not though, as the concept of partial function is very easy to understand.

In essence, a partial function is a function that accepts only a specific set of data as the input. This doesn’t mean that the function takes only integers and not strings — this behaviour is a standard element of any function. The partial function that accepts integer parameter as the input, can specify what exact integers are accepted.

To better understand the concept, let’s see some code examples.

How to write Partial Function

First, let’s write a standard function that accepts any input of type Int:

   val standardFunction = (x: Int) => x % 10

The standardFunction is like any other - takes a parameter and returns the remainder when dividing by 10. Now, what if a developer passes zero as the parameter? Or if we need even more refined restrictions regarding the parameter? One way would be to write a long block of ifs and elses, but that is not what you would want to do. Instead, let's write a partial function, like this:

    val partialFunction: PartialFunction[Int, Int] = {
        case 1 => 100
        case 2 => 120
        case 3 => 130
      }

The partialFunction above is defined using the PartialFunction[Int, Int] trait. The type parameters say that both input and the output will be of type Int. As you can see, this function accepts only three parameters: either 1, 2 or 3. For any other parameter, an exception will be thrown.

The construction of a partial function is quite easy, but the power comes with the different operators you can use with them. Let’s look at some of them:

    partialFunction.isDefinedAt(67)

isDefinedAt() - this method allows to check if the partial function will accepts a certain parameter - in our case, 67 will not be accepted, and the method will return false.

    partialFunction.lift

lift - this method wraps the return type into an Option. For the our partialFunction, that will be Option[Int].

    partialFunction orElse otherPartialFunction

orElse - this one is very neat - it allows to chain multiple partial function, if the first one doesn't accept the input, or in different words - it's isDefinedAt method returns false.

There is plethora of other methods available — be sure to check out the official docs.

`Map` vs `Collect`

One other interesting behaviour of partial functions can be observed when using it for mapping a collection. Let’s try to use it as if it was a standard function:

    List(1, 2, 3, 4, 5).map(partialFunction)

The first three parameters are fine — the partialFunction have a defined output for them. But what happens with parameters 4 and 5. The answer is: MatchError. The map function is not able to determine the output and fails.

The solution to this is to use the collect method. This method uses the isDefinedAt method before applying the mapping function, therefore avoiding the MatchError.

    List(1, 2, 3, 4, 5).collect(partialFunction)

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Bartosz Gajda

@bartoszgajda55

Learn what #PartialFunctions are and how to use them in #Scala. See the difference between #map and #collect when using those on collection.

bartoszgajda.com/2020/05/17/par…

#scala #scalalang #partialfunctions #java #jvm #functionalprogramming #programming #it #development

12:10 PM - 24 Aug 2020

0 0

5 best resources to learn Scala

Bartosz Gajda — Mon, 10 Aug 2020 13:03:11 +0000

Scala is one of the most popular languages used in the JVM ecosystem. The recent post by Li Haoyi explains why Scala is entering the ‘slope of enlightenment’, with ever-growing community and plethora of mature and production-ready tools. It is a great time to start learning Scala and this post will list you, in my opinion, the best places to start your journey with it.

1. Scala Book

There is no better place to start than Scala Book, that can be found on the official page of Scala language. This is a free guide into the world of Scala for beginners who want to get started as quickly as possible. It provides enough details to grasp the basics of Scala programming, without overwhelming you with too much information.

You will learn basic control structures, syntax, collections, classes, traits and a few fundamentals of functional programming. There is a lot of different code examples, so you have a chance to explore it in your way. There is no timing or schedule, so you can learn in anytime you fancy.

My journey with Scala started there, and it helped a lot at the beginning (especially with sometimes weirdly looking syntax :))

2. Scala Exercises

The Scala Exercises is another free to use resource, that can learn you various elements of Scala ecosystem using a series of lectures and exercises. Every chapter in each course, explains in detail what a certain feature is and how to use it in real-life examples. The exercises at the end of the chapter help you to test yourself and improve the learning process.

There is a plethora of courses available, on topics including: Scala Standard Library, Cats, Shapeless, Doobie and more.

You can login to the site using your GitHub account, which helps to track the courses and chapters you have enrolled in and progress within them.

3. Scala for the Impatient

The first of non-free resources is the book by Cay Horstmman that serves as a guide for Java programmers that want to dive into Scala language. Although being aimed at Java devs, this book can be easily understood by developers using any other major programming language.

The huge plus of this book is that it explains the features of Scala language in a very concise and easy to digest way. There is no over lengthy debates over the implementation of let’s say certain classes in collections library — instead, you get a nicely rounded explanation of what it is and how to use it.

The easiest place to get this book is most probably Amazon. You should look for a promotions, as they happen quite often on this book (at least in UK).

4. Scala & Functional Programming for Beginners

The first video course I am recommending is ‘Scala & Functional Programming for Beginners’ that can be found on Udemy platform. This course guides you through the world of Scala in well sized chunks, that covers a great deal of topics that are required from Scala developers.

This course is split into basics, object oriented programming, functional programming and pattern matching. Additionally, there is extra chapter that takes all this information and help you to apply it into a practical project.

Standard price is around £50, but Udemy does regular promotions, so keep an eye on the site and you might this course for as low as £10.

5. Advanced Scala and Functional Programming

The last resource and the second video course is the continuation of the course from point 4. This course however, is aimed at more advanced developers and those interested in getting from good to great in Scala.

The course itself tackles important topics like advanced functional programming, concurrency, implicits and type system. All presented with easy to understand context and lots of code samples.

Same as before, it is best to look for promotions on Udemy website — you should be able to get it at the price around £10.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

Bartosz Gajda

@bartoszgajda55

5 best resources for an aspiring @scala_lang developer. A selection of books, courses, and online sites, which will accelerate your #Scala learning.

bartoszgajda.com/2020/05/03/5-b…

#scala #scalalang #programming #development #jvm #functionalprogramming #learning #jvm #java #udemy

09:51 AM - 17 Aug 2020

0 0

Unit Testing Apache Spark Structured Streaming using MemoryStream

Bartosz Gajda — Mon, 10 Aug 2020 13:00:33 +0000

Unit testing Apache Spark Structured Streaming jobs using MemoryStream in a non-trivial task. Sadly enough, official Spark documentation still lacks a section on testing. In this post, therefore, I will show you how to start writing unit tests of Spark Structured Streaming.

What is MemoryStream?

A Source that produces value stored in memory as they are added by the user. This Source is intended for use in unit tests as it can only replay data when the object is still available.

Spark SQL Docs

MemoryStream is one of the streaming sources available in Apache Spark. This source allows us to add and store data in memory, which is very convenient for unit testing. The official docs emphasize this, along with a warning that data can be replayed only when the object is still available.

The MemoryStream takes a type parameter, which gives our unit tests a very desired type safety. The API of this abstraction is rich but not overwhelming. You should check it out for a full reference.

Before writing any unit test, let’s create some sample job which will be tested.

Writing Spark Structured Streaming job

The job I will be using for the testing, has a simple role — read data from Kafka data source and write it to the Mongo database. In your case, the set of transformations and aggregations will be probably much richer, but the principles stay the same.

First, I read the Kafka data source and extract the value column. I am also applying a prebuilt rsvpStruct schema, but that is specific to my code sample.

import spark.implicits._
val df = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "rsvp")
  .load()
val rsvpJsonDf = df.selectExpr("CAST(value as STRING)")
val rsvpStruct = Schema.getRsvpStructSchema
val rsvpDf = rsvpJsonDf.select(from_json($"value", rsvpStruct).as("rsvp"))

Next, I am outputting the content of parsed data into a Mongo collection. That is done is foreachBatch action, which loops through every batch in the current result set and allows to perform an arbitrary operation. In my case, I am using Mongo Spark connector for persisting the batches.

rsvpDf.writeStream
  .outputMode("append")
  .option("checkpointLocation", "/raw")
  .foreachBatch({(batchDf: DataFrame, batchId: Long) =>
    val outputDf = batchDf.select("rsvp.*")
    outputDf.write
      .format("mongo")
      .mode("append")
      .save()
  })
  .start()
  .awaitTermination()

Now that we have the sample job created, let’s start writing a unit test using MemoryStream class.

Unit testing Spark using MemoryStream

The preliminary step in writing a unit test is to have some sample data that can be used by unit tests. In my case, I am using a stringified JSON, that contains information I am normally receiving from my Kafka data source. You should use any data that fits your use case.

val sampleRsvp = """{"venue":{"venue_name":"Capitello Wines" ...

Next, inside the test case, we create our SparkSession. Nothing special has to be set here, just typical config you use.

class SaveRawDataTest extends AnyFunSuite {
  test("should ingest raw RSVP data and read the parsed JSON correctly") {
    val spark = SparkSession.builder()
      .appName("Save Raw Data Test")
      .master("local[1]")
      .getOrCreate()
...

Going further, we define our MemoryStream using String type parameter. I use string, because that's the type of sample data I am using. To create the MemoryStream, to imports are needed: sqlContext that can be obtained from SparkSession and Spark implicits library, which includes the needed type encoders. Lastly, I convert the stream to DataSet.

implicit val sqlCtx: SQLContext = spark.sqlContext
import spark.implicits._

val events = MemoryStream[String]
val sessions = events.toDS

Converting to DataSet allows us to make first assertion. We can check if the DataSet is indeed a streaming one. You can do this using the following code.

assert(sessions.isStreaming, "sessions must be a streaming Dataset")

Next, I add the transformations that have been included in the sample job. In this case however, I will not be checking if the data has been saved in Mongo — I am only interested if the raw stringified JSON has been parse correctly and I can run SQL queries on it. The transformations I use are:

val transformedSessions = sessions.select(from_json($"value", rsvpStruct).as("rsvp"))
val streamingQuery = transformedSessions
  .writeStream
  .format("memory")
  .queryName("rawRsvp")
  .outputMode("append")
  .start

It is important to use memory format, so that the actual data can be queried in a further step to make the assertions. It is useful to also name those results, using queryName option.

Lastly, I add my sample data into the instance of MemoryStream, process those and commit the offsets, as shown below:

val batch = sampleRsvp
val currentOffset = events.addData(batch)

streamingQuery.processAllAvailable()
events.commit(currentOffset.asInstanceOf[LongOffset])

The very last step is to query the committed data and run some assertions on them. In my case, I run a SQL query, to get the event_name property and I run assertions against that.

val rsvpEventName = spark.sql("select rsvp.event.event_name from rawRsvp")
    .collect()
    .map(_.getAs[String](0))
    .head

assert(rsvpEventName == "Eugene ISSA & Technology Association of Oregon December Cyber Security Meetup")

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂

5 best IntelliJ IDEA shortcuts

Bartosz Gajda — Mon, 10 Aug 2020 12:58:15 +0000

IntelliJ IDEA is awesomely large and complex IDE, mastering which can take years. Knowing the essential IntelliJ IDEA shortcuts can drastically increase your productivity and make using a mouse seem obsolete. In this post I will show you 5 IntelliJ IDEA shortcuts that in my opinion are the best. Enjoy!

Recent Locations — `Ctrl + Shift + E`

This shortcut shows the popup with the recently accessed files or the files that have been recently changed (Figure 1). Pressing the Ctrl + Shift + E shortcut again will show you only the changed files. You can start typing, to filter the files. Useful for quickly jumping through files when working in larger context.

Show JavaDoc Popup — `Ctrl + Q`

Another useful shortcut, for quickly accessing the JavaDoc of a certain class or interface. Just place the caret on a name and press Ctrl + Q which will show the popup, as on Figure 2. The availability of JavaDoc depends of course on the framework or library you are using. It is a useful thing to know when working with large frameworks like Spring.

Replace in Path — `Ctrl + Shift + F`

Replace in path shortcut helped me couple times to avoid going through files when trying to change a variable that occurs many times in the project. The Ctrl + Shift + F shortcut shows you the popup as on Figure 3. In the first input you can provide the term you are want to replace and in the second one - to what replace with.

Highlight Usages — `Ctrl + Shift + F7`

Highlighting the usages is especially useful when working with large files. To do this, place a caret on the word you want to highlight and press Ctrl + Shift + F7. Depending on the theme you are using, the word may be highlighted differently. In my case, it's bright yellow, like on Figure 4.

Show File Structure — `Ctrl + F12`

The last shortcut helped me a lot when working with complex classes. All you need to do is to press combination Ctrl + F12 and a popup like on Figure 5 will be shown to you. Here you can see all method signatures displayed, which is super helpful when you are only interested in skimming through the class. Pressing the Ctrl + F12 again will also show the members inherited from superclass.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally you can follow me on my social media if you fancy so :)

Bartosz Gajda

@bartoszgajda55

Best @intellijidea shortcuts to speed up your work. Learn 5 shortcuts in IntelliJ IDEA that can drastically ease up your code editing tasks.

bartoszgajda.com/2020/03/23/5-b…

#intellijidea #intellij #idea #jetbrains #ide #tips #tricks #shortcuts #webstorm #pycharm #rider #clion

15:48 PM - 03 Aug 2020

0 1

Using Angular and Chart.js to build real-time charts

Bartosz Gajda — Mon, 10 Aug 2020 12:56:05 +0000

Angular and Chart.js is popular combination when creating any data visualization application. The first one can handle a very large throughput of data and the later is capable of rendering the plots in real-time, thanks to Canvas API. In this post I will guide you through the process of creating a real-time chart using Angular and Chart.js

Prerequisites

Before starting to write any code, make sure you have the following:

Node.js — I use version 13.2.0
Angular CLI — I use version 8.3.20
10 minutes of free time

Creating new Angular project

The first step required is to create a new Angular project. As mentioned in prerequisites, I am using Angular CLI to do so and I highly advise you to do the same. Open a terminal window, navigate to desired directory and execute the command:

ng new angular-charts --routing=true --styling=scss

This command creates a new Angular project called angular-charts in directory of the same name. Additionally, I have added two optional flags — routing adds the router module to the app and styling sets the extensions of style sheets used.

With the project created, open it up in your IDE of choice — I will be using Visual Studio Code for this.

Adding a service layer*

The next step of this tutorial is to add a service layer. I have marked this step with asterisk, because it is optional. If you already have one, or you don’t need one, then feel free to skip this section.

Let’s start this section with generating a service that will give access to real-time data source using Observable. To generate a service, use the following command:

ng generate service sse

After executing the command, and SseService gets created and that's where the service layer code will be placed. For this tutorial I am using SSE or Server Sent Events data source, tutorial on which you can find here. If you need more explanation, don't hesitate to read that tutorial. To avoid repetition in this post, I will just paste in the following:

import { Injectable, NgZone } from "@angular/core";
import { Observable } from "rxjs";
@Injectable({
  providedIn: "root"
})
export class SseService {
  constructor(private _zone: NgZone) {}
  getServerSentEvent(url: string): Observable<any> {
    return Observable.create(observer => {
      const eventSource = this.getEventSource(url);
      eventSource.onmessage = event => {
        this._zone.run(() => {
          observer.next(event);
        });
      };
      eventSource.onerror = error => {
        this._zone.run(() => {
          observer.error(error);
        });
      };
    });
  }
  private getEventSource(url: string): EventSource {
    return new EventSource(url);
  }
}

Hooking up the Chart.js

The next step is to hook in Chart.js library into our Angular project. There are couple ways to do so, but I will use a dedicated package, called Ng2-Charts. This package exposes a much nicer API while retaining all the required functionality. In my case, I add the following dependencies to my package.json file:

"chart.js": "^2.9.3",
"ng2-charts": "^2.3.0",

After modifying the package.json file, don't forget to run either npm install or yarn depending on your package manager.

Adding HTML template

Going further, we have to add an HTML template that will render the chart. In the case of this tutorial, you can place it anywhere you fancy — the code is single HTML tag with custom properties which we will explore in next step. I place it in a component HTML template called count-events.component.html. The HTML template should include the following:

<canvas
    width="600"
    height="400"
    [datasets]="countEventsData"
    [chartType]="countEventsChartType"
    [labels]="countEventsLabels"
    [colors]="countEventsColors"
    [options]="countEventsOptions"
></canvas>

I have placed my chart is count-events folder, therefore all variables are prepended with those. In the canvas tag we specify height, width and variable configuration, which will be placed in corresponding .ts file.

Configuring Chart.js

As mentioned in chapter above, we will add some custom configuration to the Chart.js plots. This configuration will be placed in your components’ TypeScript file, in my case it is called count-events.component.ts.

The first thing that has to be set is the datasets property. That is a container that will hold the data displayed on the plot itself. The code for this should look like below:

countEventsData: ChartDataSets[] = [
  { data: [], label: "Number of Events", fill: false }
];

This variable is an array, meaning you can have many data sets displayed on a single plot. Inside of each element there are three core parts:

data - an array that holds the single values to be displayed on the chart
label - label of the data set
fill - configuration option setting the appearance of the data set on chart

Next configuration is the chartType property. That is a single string, flagging the type of chart that should be used. There is a wide variety of options available, including line, bar, graph or pie, but for this tutorial we are going to stick with the simplest one - line:

countEventsChartType = "line";

Going further, labels property has to be set. This element sets what labels the X axis receives. In our case however, we don't want to set them as a constant. We want to be able to update the labels in real-time, in cnjuction with the incoming data. This property therefore is set as empty array:

countEventsLabels: Label[] = [];

Next property is colors. The name itself is probably self explanatory, so I will jump straight to code:

countEventsColors: Color[] = [
    {
      borderColor: "#039BE5",
      pointBackgroundColor: "#039BE5"
    }
];

Last bit of configuration is called options. That is the central configuration point, for all major flags that can be set. The amount of available options is very broad, so please refer to Chart.js docs for complete documentation. In our case, we are only interested in removing the animations - that will optimize the chart and make it run faster. To do this, paste the following into your code:

countEventsOptions: ChartOptions = {
    animation: {
      duration: 0
    }
 };

Connecting Service and Chart.js

Last chapter of this tutorial will show you how to glue the service and the Chart.js together. To make this happen, we will implement couple functions in the count-events.component.ts file.

We start off with subscribing to the data source, which is an SseService in our case. That is done in the ngOnInit hook, so that we connect to data source whenever our component is loaded in the application. In here, we create a Subscription to the endpoint and call pushEventToChartData function.

private countEventsSubscription$: Subscription;
ngOnInit() {
    this.countEventsSubscription$ = this.sseService
      .getServerSentEvent("http://localhost:8082/count-events")
      .subscribe(event => {
        let data = JSON.parse(event.data);
        this.pushEventToChartData(data);
      });
  }

The aforementioned function have a simple purpose — it checks whether the datasets have reached an arbitrary limit (20 in this case) and if so, removes the last element before pushing the new one into this collection. On thing has the kept in mind - if adding or removing elements, it has to be done for both datasets collections and labels collections. Both of them have to by kept synced all the time.

private pushEventToChartData(event: CountEvents): void {
    if (this.isChartDataFull(this.countEventsData, 20)) {
      this.removeLastElementFromChartDataAndLabel();
    }
    this.countEventsData[0].data.push(event.count);
    this.countEventsLabels.push(
      this.getLabel(event)
    );
  }

The last pieces of code include the helper functions calls to which can be found in snippet above. First functions could be used to implement some prettier looking labels. Second one removes the last element from both datasets and labels collections. The third checks whether the a collections has reached its limit, which I have set to be 20 in my case. The snippets for those are as follows:

private getLabel(event: CountEvents): string {
    return `${event.window}`;
  }

  private removeLastElementFromChartDataAndLabel(): void {
    this.countEventsData[0].data = this.countEventsData[0].data.slice(1);
    this.countEventsLabels = this.countEventsLabels.slice(1);
  }

  private isChartDataFull(chartData: ChartDataSets[], limit: number): boolean {
    return chartData[0].data.length >= limit;
  }

Wrapping this all up, the complete code for count-events.component.ts file looks like this:

export class CountEventsComponent implements OnInit, OnDestroy {
  private countEventsSubscription$: Subscription;
  private eventsOnChartLimit = 20;
  countEventsChartType = "line";
  countEventsData: ChartDataSets[] = [
    { data: [], label: "Number of Events", fill: false }
  ];
  countEventsLabels: Label[] = [];
  countEventsColors: Color[] = [
    {
      borderColor: "#039BE5",
      pointBackgroundColor: "#039BE5"
    }
  ];
  countEventsOptions: ChartOptions = {
    animation: {
      duration: 0
    }
  };

  constructor(private sseService: SseService) {}

  ngOnInit() {
    this.countEventsSubscription$ = this.sseService
      .getServerSentEvent("http://localhost:8082/count-events")
      .subscribe(event => {
        let data = JSON.parse(event.data);
        this.pushEventToChartData(data);
      });
  }

  private pushEventToChartData(event: CountEvents): void {
    if (this.isChartDataFull(this.countEventsData, 20)) {
      this.removeLastElementFromChartDataAndLabel();
    }
    this.countEventsData[0].data.push(event.count);
    this.countEventsLabels.push(
      this.getLabel(event)
    );
  }

  private getLabel(event: CountEvents): string {
    return `${event.window}`;
  }

  private removeLastElementFromChartDataAndLabel(): void {
    this.countEventsData[0].data = this.countEventsData[0].data.slice(1);
    this.countEventsLabels = this.countEventsLabels.slice(1);
  }

  private isChartDataFull(chartData: ChartDataSets[], limit: number): boolean {
    return chartData[0].data.length >= limit;
  }

  ngOnDestroy() {
    this.countEventsSubscription$.unsubscribe();
  }
}

And that this tutorial finished. Using Angular and Chart.js is not a rocket science and the benefits of having a real-time charts can be huge.

Summary

I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally you can follow me on my social media if you fancy so :)

Bartosz Gajda

@bartoszgajda55

Using @angular and @chartjs to build a real-time, interactive data charts. Plotting continuous data using @ReactiveX and asynchronous #apis

bartoszgajda.com/2020/03/17/usi…

#angular #chartjs #frontend #web #webdevelopment #javascript #typescript #realtime

12:23 PM - 24 Jun 2020

3 1

Forem: Bartosz Gajda

5 Essential skills for becoming a Data Engineer

Programming - Scala and/or Python

Data Manipulation - SQL

Processing Engine - Apache Spark

Cloud - AWS or GCP or Azure

Deployment - Docker and Kubernetes

Summary

RESTful Web scraping in Scala, using Play Framework and Jsoup

You will need

Creating empty Play project

Adding Jsoup dependency

Defining an endpoint

Connecting to website

Extracting the content

Returning a JSON

Summary

Scala collections in a nutshell

Scala collection package(s)

collection.immutable

Set

Map

Vector

List

Summary

5 best study resources for GCP ACE exam

1. Official GCP ACE Study Guide

2. A Cloud Guru Course

3. Linux Academy Course

4. Google Code Labs

5. Google Qwiklabs

Summary

Exploiting Schema Inference in Apache Spark

Saving the Schema

Loading the Schema

Summary

Scala Type Bounds

What is type bound?

Upper type bound

Lower type bound

Lower and upper type bound

Summary

Why I have chosen AWS Lightsail to host my site

What is AWS Lightsail?

1. It’s fully managed

2. It supports many environments

3. It’s cheap

4. It scales automatically

Summary

Partial Functions in Scala

What is Partial Function?

How to write Partial Function

Map vs Collect

Summary

5 best resources to learn Scala

1. Scala Book

2. Scala Exercises

3. Scala for the Impatient

4. Scala & Functional Programming for Beginners

5. Advanced Scala and Functional Programming

Summary

Unit Testing Apache Spark Structured Streaming using MemoryStream

What is MemoryStream?

Writing Spark Structured Streaming job

Unit testing Spark using MemoryStream

Summary

5 best IntelliJ IDEA shortcuts

Recent Locations — Ctrl + Shift + E

Show JavaDoc Popup — Ctrl + Q

Replace in Path — Ctrl + Shift + F

Highlight Usages — Ctrl + Shift + F7

Show File Structure — Ctrl + F12

Summary

Using Angular and Chart.js to build real-time charts

Prerequisites

Creating new Angular project

Adding a service layer*

Hooking up the Chart.js

Adding HTML template

Configuring Chart.js

Scala `collection` package(s)

`collection.immutable`

`Set`

`Map`

`Vector`

`List`

`Map` vs `Collect`

Recent Locations — `Ctrl + Shift + E`

Show JavaDoc Popup — `Ctrl + Q`

Replace in Path — `Ctrl + Shift + F`

Highlight Usages — `Ctrl + Shift + F7`

Show File Structure — `Ctrl + F12`