Forem: Anniina Sallinen

Understanding Time Complexity with the First Puzzle of Advent of Code

Anniina Sallinen — Wed, 02 Dec 2020 18:22:32 +0000

First before anything else: ⚠️ SPOILER ALERT! ⚠️ This blog post will discuss possible solutions to the first puzzle of Advent of Code. If you haven't completed it yet, and want to do it before looking into any solutions, don't read this blog post. Also if you're just a beginner in coding and you need to spend a lot of time figuring out how to solve the puzzle, this might get over your head and that's fine. I want to say that it's totally fine not to optimize your solution, since Advent of Code doesn't have any efficiency criteria for solutions.

This is my first year to participate Advent of Code, and I find it very enjoyable. I have a degree in CS, but since I graduated in 2018, I haven't done any coding puzzles. I solve the puzzles with some brute-force method first, but then I find myself thinking about if there's a more efficient solution. We have a group of people in our community that solve the puzzles, and when I mentioned the time complexity of my solution, someone said they had never thought analyzing the time complexity of their code. That's why I thought I'd write a blog post about big O notation and how to analyze your solution to understand it better. I use the first puzzle of Advent of Code to demonstrate how to analyze the code.

The assignment

In the first part of the puzzle the assignment is the following:

Before you leave, the Elves in accounting just need you to fix your expense report (your puzzle input); apparently, something isn't quite adding up.

Specifically, they need you to find the two entries that sum to 2020 and then multiply those two numbers together.

The second part is the same, but with three numbers, that need to sum to 2020. I focus on the first part only, because I didn't optimize the solution of the second part, and also because it would make this blog post even longer. I used Python to solve the puzzle and will use that in the examples, too.

Big O notation

So big O notation is used for indicating the time complexity of an algorithm, and more specifically the worst case. It means that for example if you have a list of numbers, and you're searching for number 2 in the list, the worst case is that it's not in the list or it's the last element, and you have to go through the whole list. The time complexity is then O(n), where n is the number of elements in the list.

Time complexity shouldn't be thought of as the actual duration of the algorithm execution, but rather as how fast the duration grows in relation to input size. There are multiple classes that are typically used. I've listed some of them in the table and will show examples of them later.

Time complexity	Meaning
O(1)	Constant time complexity
O(log n)	Logarithmic time complexity
O(n)	Linear time complexity
O(n log n)	Log-linear time complexity
O(n^2)	Quadratic time complexity
O(n^c)	Polynomial time complexity

Brute force solution

So the easiest way to solve the puzzle is a brute force solution. As I said earlier, Advent of Code doesn't have any requirements for how efficient your solution is, it just wants an answer, so brute force is a completely valid approach to the problems. I first used the brute force solution to get the answer, and then later optimized my solution, so my first version of the code looked like this:

def find_two_numbers_that_sum(desired_sum, list_of_expenses):
    for expense in list_of_expenses:                   # Iterating over a list: O(n)
        for expense_to_compare in list_of_expenses:    # Iterating over a list: O(n)
            if expense + expense_to_compare == 2020:   # Comparison: O(1)
                return expense * expense_to_compare    # Multiplication: O(1)

This is a very simple solution, and it works. The size of the list is 200, so it's not too slow to be solved like this. Let's get to the analysis of that solution then.
As I wrote as a comment to the code, iterating over a list is an O(n) solution. Because there are two loops, and the second one is inside the first one, they are multiplied. We multiply also the comparison complexity and the multiplication complexity, but since they're both constant, they don't affect the complexity of the solution. Thus the complexity of the solution is n*n = O(n^2). You can test this with a small list (for example of size 5) and calculate how many iterations it takes with that function. Remember to calculate the worst case, which is in this example that there are no two numbers that sum to 2020, and you go over the whole list.

In the picture below you can see how the number of iterations grows as the size of the list grows. The number of elements in the list is on the x-axis and the number of iterations is on the y-axis.

But since there's an early exit if the two numbers are found, wouldn't it mean that it's better than O(n^2)? Unfortunately not. Because big O notation is about the worst case, and the worst case is that either there are no such numbers, or they are the last pair to compare, the time complexity is still the same.

How about if we never look into the same combination once? Wouldn't it make the solution faster? Well in practice the solution is faster, but the growth of the duration still remains on the same scale. If you get a pen and paper (which I suggest you have always when you try to solve a puzzle) and simulate the algorithm you notice that if you have a list of five numbers, the number of elements you check is 5 + 4 + 3 + 2 + 1 = 15. When there's 10 elements, the number of elements you check is 10 + 9 + 8 + 7 + 6 + 5 + 4 + 3 + 2 + 1 = 55. The time complexity is roughly O((n^2 + n)/2), but because we ignore the constants, we have only O(n^2 + n) which is essentially the same as O(n^2). In the picture you see a similar growth in the number of iterations when the size of the list grows:

So you see that although the number of iterations is lower, the growth is as fast as with the simple version.

Enhanced solution

So after I had given the correct answer to Advent of Code and got a confirmation that I'd solved it, I started to think about how to optimize it. I came up with an idea that maybe I could sort the list of numbers and then use a binary search to find the number we were looking for. Python has a sorting function as built-in, so I didn't need to implement that myself, but I implemented binary search myself. I first implemented the iterative version of it but later refactored it to be recursive. Here's the code:

import math 

def recursive_binary_search(sublist, first_elem, desired_sum):
    if len(sublist) == 0:
        return
    beginning = 0
    end = len(sublist) - 1
    selected_elem = math.floor((end - beginning)/2)
    elem_sum = first_elem + sublist[selected_elem]
    if elem_sum == desired_sum:
        return first_elem * sublist[selected_elem]
    if elem_sum > desired_sum:
        end = selected_elem - 1
    if elem_sum < desired_sum:
        beginning = selected_elem + 1
    recursive_binary_search(sublist[beginning:end], first_elem, desired_sum)


def find_subset_sum(expenses_list, desired_sum):
    sortedd = sorted(expenses_list)  # Python built in sort algorithm, O(n log n)         

    for elem in sortedd: # Iterating over the elements, O(n)
        binary_search_list = sortedd.copy()
        beginning=0
        end = len(binary_search_list)-1
        # Binary search, O(log n)
        return_val=recursive_binary_search(binary_search_list, elem, desired_sum)
        if return_val:
            print(return_val)
            return

Now recursion makes the analysis of the algorithm a bit more complex, but as binary search is a commonly known algorithm, we know its time complexity is O(log n). If you are familiar with logarithm, it's easier to understand, but basically with each iteration the size of the list that we need to go through halves, so in total, the number of iterations is log n.

What is the time complexity of the solution? First, we sort the elements, and it has O(n log n) complexity. Then we iterate over the elements, with time complexity O(n), and inside the loop, we perform the binary search. Let's start with calculating the complete time complexity of the loop. Because binary search is inside the loop, we multiply O(n) with O(log n). It results in O(n log n). Because sorting is not done inside the loop, we can just sum O(n log n) + O(n log n). The total complexity is thus 2 O(n log n), which is essentially the same as O(n log n). Thus the time complexity for the solution is O(n log n). It's good to note that this solution only works when there are two numbers that we need to find, so it doesn't help with part 2 of the puzzle.

Optimal solution

After making the improvements described above, I went to see how other people had solved it. On Twitter, I saw some vague tweet about using a set data structure, but couldn't wrap my head around it. Then I came here to see if someone had posted their solution and found Christopher's post. The solution in the blog post is simple and elegant, and it was more efficient than the sort + binary search solution. You can check the blog post but I also explain it shortly here.

Iterate over the elements
Subtract the value of the element from 2020 to get the number you want to find
Insert all other elements into a set
Check if the set has the number your looking for, return the result of the multiplication

So this solution is very smart because it uses a set as a data structure. Inserting a new element, removing an existing one, and finding a value in a set has constant O(1) time complexity. This means that only step 1 has larger time complexity, O(n), and all the other steps have time complexity O(1). When we multiply all of these together, we get O(n).

In the picture below you can see different solutions in a one plot, it might make it easier to understand how large difference there is:

Summary

So when you analyze your solutions for Advent of Code or some other puzzle (or even your work!), the most important thing is to remember that big O notation doesn't describe duration, it describes the growth of the number of iterations in relation to the size of an input. In practice, some optimizations might make your code faster, but the growth in relation to the input size doesn't change. When you calculate the complexity, you can define complexity for each row and then either sum them or multiply them, depending on whether they happen in a row or a nested manner.

If you want to compare your solutions with others, comparing time complexity instead of duration might be a good idea if it doesn't feel too heavy. It's because when you measure duration, the context is important. The duration depends on for example your computer, if you have to same data, if the data is in the same order, the programming language you're using, etc. If you haven't analyzed your solutions and their time complexity before, I think Advent of Code provides nice puzzles to start with!

Photo by Markus Spiske on Unsplash

Datetimes Are Hard: Part 2 - Writing and running code

Anniina Sallinen — Sun, 29 Nov 2020 13:21:05 +0000

In this part of the series, we go through how to parse and manipulate datetimes. This can be done in data pipeline with code or you can specify the format when you load data into database. In this blog post we focus on the first case.

Usually, data pipelines code are written with Python. In this blog post I'm using Clojure, though, mainly because I use it at work and it has nice library for handling datetimes. I'm using clojure.java-time library, version 0.3.2. As one can guess from the name of the library, it's a wrapper library for Java 8 time API. One of the good things about Clojure is that you can use java libraries and classes. Java happens to have a nice time API, so why not use it?

At this point, I want to point out that this is not a coding tutorial, and the purpose of the blog post is not to teach anyone how to program with Clojure. If you're not familiar with Clojure, the syntax might look quite overwhelming at first. I suggest you take a look at the Clojure syntax documentation if the syntax prevents you from understanding the examples given in the blog post.

Datetime data

I'm going to introduce the datetime classes I've used at work to represent datetimes. What they have in common is that their string presentation includes both date and time. Some classes in time API include only date or only time, but I'm not going to discuss them here. The code in examples are run in a REPL.

First, we are going to require the library, and alias it as t.

(require '[java-time :as t])

Local datetime

Local datetime is a datetime without timezone information. When there is no information about the timezone, datetime is assumed to be in local time.
For example:

(def now-time (t/local-date-time))
(str now-time)

would output for example

"2020-11-22T18:49:20.446"

You have to be mindful about when you can use local datetime and when to use some other class, which has a timezone. For example, birthday is something you could use local datetime for, but you wouldn't want to use it for example for scheduling a meeting, would you?

Local datetime is always relative to what local means. I might run some code locally, and check that datetime looks correct, but when I deploy the solution to for example to AWS, local might mean a different thing. Method now returns a different time depending on the system clock, unless you provide a timezone as an argument for it.

Instant

Instant is a single point on the timeline. The point is represented as nanoseconds since the epoch, but in string format, it looks like this:

(def instant-now (t/instant))
(str instant-now)

"2020-11-22T17:09:43.456Z"

As you may remember from the previous part of the series, Z at the end means datetime is in UTC. Instant can be used for example to record an event, for example when the event was received from an external system. Because instant is point on the timeline, it doesn't store date and time fields, only a number representing the point in time relative to epoch.

Offset datetime

Offset datetime is a datetime with an offset from UTC. As you may remember from the previous part of the series, it may have a negative or positive value, depending on whether it's less or more than UTC.

(def offset-now (t/offset-date-time))
(str offset-now)

"2020-11-22T18:58:54.961+02:00"

The offset datetime doesn't include the timezone information. Due to daylight savings, the same timezone might have a different offset depending on the time of the year. The timezone can be deducted from the offset, but the class itself doesn't store information about it.

Zoned datetime

Zoned datetime is a datetime with an offset and a timezone. It contains the most information about date and time.

(def zoned-now (t/zoned-date-time (t/zone-id "Europe/Helsinki")))
(str zoned-now)

"2020-11-22T19:13:54.682+02:00[Europe/Helsinki]"

Zoned datetime is needed for example when you need to convert local datetime to instant, since you would need the offset to calculate the time since epoch.

Parsing and formatting datetimes

If the datetime in data happens to be in a wanted format already, you can just give the datetime as an argument to the constructor

(def parsed-local-date-time (t/local-date-time "2011-12-03T10:15:30.234"))

Instead, if the datetime has space as a delimiter instead of character T, you need to pass the datetime format as an argument, too

(def parsed-local-date-time (t/local-date-time "yyyy-MM-dd HH:mm:ss.SSS" "2011-12-03 10:15:30.234"))

Otherwise local-date-time is not able to parse it.

Sometimes the datetime doesn't contain timezone information, but you know which timezone it should have. It might be that the documentation of the API tells you that, or you have confirmed it from the API developer, so you can set it as you parse the datetime:

(def parsed-zoned-date-time (t/zoned-date-time (t/local-date-time) (t/zone-id "Europe/Helsinki")))

Now let's have a look at how it works the other way around. The query might return for example an instant, and you need it to be in a certain format for sending it to an API.

You can do it as an one-liner

(->> (t/instant) (t/format (t/with-zone (t/formatter :iso-local-date-time) "Europe/Helsinki")))

but I personally think it looks a bit nasty, so I created a function for it:

(defn instant->iso-local-date-time-str [instant-to-format]
    (let [iso-formatter (t/formatter :iso-local-date-time)
          formatter-with-hki-zone (t/with-zone iso-formatter "Europe/Helsinki")]
        (t/format formatter-with-hki-zone instant-to-format)))


; More generic function, for formatting instant with any predefined formatter and timezone:
(defn instant->formatted-str [instant-to-format predefined-formatter time-zone]
    (let [iso-formatter (t/formatter predefined-formatter)
          formatter-with-hki-zone (t/with-zone iso-formatter time-zone)]
        (t/format formatter-with-hki-zone instant-to-format)))

Because in the example we used an instant, we need to provide timezone information before formatting it. That's because an instant has a different string presentation depending on the timezone. If you want to have the string presentation in UTC, you don't need the timezone and you can just go with

(t/format (t/formatter :iso-instant) (t/instant))

which gives you the same string as you would just

(str (t/instant))

You can find all predefined formatters in Java DateTimeFormatter documentation.

Manipulating datetimes

Once the datetime data is parsed into an object, manipulating it is quite straightforward. For example, if you want to add two weeks to the date:

(-> parsed-local-date-time
    (t/plus (t/weeks 2)))

Getting the start of the last month would be like this:

(-> parsed-local-date-time
    (t/adjust :first-day-of-month) 
    (t/adjust (t/local-time 0)))

Possible adjusters (such as :first-day-of-month) are:

:day-of-week-in-month
:first-day-of-month
:first-day-of-next-month
:first-day-of-next-year
:first-day-of-year
:first-in-month
:last-day-of-month
:last-day-of-year
:last-in-month
:next-day-of-week
:next-or-same-day-of-week
:previous-day-of-week
:previous-or-same-day-of-week

It is good to note, that you cannot use the adjusters for instant, because instant doesn't have information such as year or day of month, so you would need to convert instant to another datetime before using adjuster.

For converting datetime from a timezone to another you would need a zoned datetime. Converting the datetime from for example Helsinki timezone to UTC can be done like this:

(-> (t/zoned-date-time (t/zone-id "Europe/Helsinki"))
    (t/with-zone-same-instant "UTC"))

If the provided datetime is not a zoned-date-time but for example offset-date-time, you would first need to convert it into zoned-date-time before changing the timezone.

(-> (t/offset-date-time "2020-06-30T20:30:21.145+05:00")
    (t/zoned-date-time)
    (t/with-zone-same-instant "Europe/Helsinki"))

Pitfalls with handling datetimes

It's easy to mess up the datetimes. It's easy to forget about the timezones and offsets, and just use local datetime only to realize later datetimes are different from what you'd expected. In some cases that's fine, but in most data and analytics platforms, timezone and offset matters. They're also important if there is any kind of scheduling involved.

If you've set up for example Black Friday offers to open at midnight, and you're in Finland, you can find yourself in a situation when the offers are opening at 2 AM instead of at midnight because the system timezone is UTC. Probably not the most horrible situation, but you might get some angry feedback from the customers that have been waiting for the offers.

Another thing that might cause trouble is daylight savings. Once a year, you might have a situation where you're missing data from one hour. And once a year, you might find yourself in the situation that you have one hour at night that has more data than usual. This might not cause any issues, but it's a good thing to acknowledge.

And how about traveling? When we talk about some wearables that produce data, the local might change to another in the middle of a day. I do not envy the developers that need to face that type of issue when they build the system.

I hope this blog post has provoked some toughts and gave you a good overview of how datetimes can be handled with Clojure. In the next part I'm going to discuss about datetimes from the perspective of the database.

Photo by Lucian Alexe on Unplash

Datetimes Are Hard: Part 1 - Incoming data and formats

Anniina Sallinen — Sun, 22 Nov 2020 10:09:06 +0000

I have worked in the industry since 2014, first as a software developer and from 2018 on as a data engineer. Especially nowadays I have to work a lot with datetimes, and have come to realize that they are one of the hardest things in my job. I thought I'd write a series about datetimes from a data engineer's point of view. This is the first part of the series.

As a data engineer, one of my responsibilities is to build data pipelines. Multiple internal and external systems are producing data that we want to import and use. Since we cannot control the source systems and how they have designed their systems, we need to adjust to the decisions they have made. This means we need to, almost always, do some transformations to the data and the datetimes in it, before inserting the data into a database.

Datetime formats

There is a standard, called ISO 8601, which standardizes presenting date and time. It uses the Gregorian calendar and 24-hour clock and the purpose of the standard is that the dates and times are unambiguous. For example, 01/11/19 can mean either November 1st, 2020, or January 11th, 2020, so it is good to have a standard. I'm happy that I haven't met datetimes that are not following the standard, since it would make all of it even harder. This post doesn't aim to discuss the complete standard, but rather the formats I've seen in my everyday work.

Although there is a standard, it doesn't mean that handling datetimes would be simple. Although ISO 8601 defines a standard, there are still multiple formats available to use. Basically in the standard Y stands for a year, M stands for month and D stands for day. This means that YYYY-MM-DD is a datetime with a four-digit year, two-digit month, and two-digit day of the month. An example of such date could be for example 2021-01-20.

In the standard h stands for hour, m stands for minute and s stands for second. Using those, time can be represented for example as hh:mm:ss. This is a simple case. There might be fractions of a second, too. When there are fractions of a second, they are marked with s. They are separated from seconds with comma or dot, so the format becomes hh:mm:ss.sss. This is not how the time is always presented, though. The separator : is optional, so it could be hhmmss as well. Here's a table to summarize what I just explained:

CHARACTER	MEANING
Y	Year
M	Month
D	Day of the month
h	Hour
m	Minute
s	Second
s	Fractions of a second

Timezones

Usually, datetimes also include timezone information. ISO 8601 represents timezone as local time, UTC, or offset from UTC. The simplest case is UTC (Coordinated Universal Time). In this case it is represented as Z at the end of timeformat. An example of this is 23:59:59Z.

Then we have an offset from UTC. UTC is the same time as GMT (Greenwich Mean Time) which means it has a value of 0. The offset can be either positive or negative, meaning hours and minutes less or more than in zero timezone. The most common ways I've seen it is hh:mm or just hh. In this case, an example of time with offset is 23:59:59+02 or 23:59:59+02:30. Without : separator it could be 235959+0230.

The most tricky case is the one that doesn't have any timezone information. The time can then be assumed to be a local time, but it's a big assumption, and sometimes it is not clear what the local time is in the source system.

Finally, we can put the date and time information together. Usually separator T is used to separate date and time. In the older version of ISO 8601, it was permitted to omit T, but the decision was retracted later. I have seen both used, though. So to put everything together, these are few examples of how the source system can represent datetime:

WITH SEPARATOR	WITHOUT SEPARATOR
2021-01-20T19:00:00	20210120T190000
2021-01-20T17:00:00Z	20210120T170000Z
2021-01-20T12:00:00-05	20210120T120000-05
2021-01-20T19:00:00.125	20210120T190000.125
2021-01-20T19:00:00.125Z	20210120T190000.125Z
2021-01-20T19:00:00.125-05	20210120T190000.125-05
2021-01-20T19:00:00.125-05:30	20210120T190000.125-0530

So this was the first part of the series. After you've identified the date format that incoming data uses, you can parse it using a programming language of your choice. In the next part I'm going to show how to parse and manipulate datetimes with Clojure.

Photo by Kevin Ku on Unsplash

Story of Koodikahvit -podcast

Anniina Sallinen — Fri, 13 Nov 2020 17:34:22 +0000

Koodikahvit is a Finnish podcast, in which we gather around a cup of coffee to discuss IT, programming and everything related. I host the podcast with a friend of mine, Pauliina Solanne.

How the idea was born

I met with Pauliina for the first time when Pauliina became a leader of the community for women and people in gender minorities (which I mentioned in my earlier blogpost). In autumn 2019, we started talking about tech podcasts, and how there are not that many in Finnish. That's when we got the idea of our own podcast.

The idea for the podcast was to provide information also for beginners. One of our favorite podcasts in English is Ladybug podcast and we felt like there was room for a similar podcast in Finnish: technical subjects, but also episodes about recruitment and career, studying and working in IT.

Colleagues to the rescue

I knew that we had a meeting room at my workplace which doubles as a recording studio for other podcasts, so we had all equipment we needed there. I have taken a studio recording course in high school, but basically, we had zero experience in recording and especially editing audio. I'm sure you'd like to know about the equipment, but unfortunately I have no idea what they are and I cannot check because we're not supposed to visit the office.

Lucky me, I have amazing colleagues and one of them, Aki Kiminki (producer of multiple podcasts), promised to help to get started with both recording and editing. I also found another colleague of mine, Henry Jalonen who was willing to help. They showed me how to connect everything, how to do a soundcheck, and how to do basic editing after recording. Aki made our theme jingle, too!

We also needed help with the visuals. I could draw a box plot with python, but when it comes to designing a logo, I'm no use there. Pauliina also felt that it's not her area of expertise, but luckily we got help again. This time Pauliina's colleague Niko Salkola, lead designer was more than happy to help us.

In the podcast episodes we have emphasized the importance of a community, and I think the help we got proves it.

Further planning

I and Pauliina have very different backgrounds when it comes to the career. I studied computer science at a university in Finland, Pauliina participated in a bootcamp in Australia. Pauliina has also worked in a different industry before, and I haven't. Despite that, we knew the podcast would provide the most value and insights to the listeners if we had guests. We wanted to have guests with diverse backgrounds and different areas of expertise in IT. We started gathering a list of potential guests.

Another thing we had to decide was the name of the podcast. For me personally, titles are very hard to come up with. It's like naming variables! We finally came up with a name we were very happy with: Koodikahvit. The origin of name Koodikahvit is something I always want to explain since it was not originated by me. Koodikahvit is originally from my workplace, Solita, and it means a get-together, where some technical presentation is given and then the subject is discussed. It's a knowledge sharing session amongst the employees at Solita. That sounds like a perfect name for a tech podcast, amirite?

Current situation

The first episode of Koodikahvit was published in March 2020. We have now published four episodes and have about 3850 listens. Unfortunately, we underestimated how long we're going to have COVID-19 around, and haven't published an episode since July. In the latest episode, we had our first guest, when we were talking about studying computer science at university. We hope to get back to recording soon, but meanwhile, feel free to share tips for remote recording. As I told before, we are not very experienced with recording equipment, so nothing too fancy would be great! We feel that we only got started, and super eager to get new episodes recorded!

And finally, if you speak Finnish, here are links to our current episodes:

In addition to audioboom, you can listen to the podcast for example on Spotify and Apple Podcasts, but also on many other platforms. Check the one you use when you listen to podcasts!

Also: would you be interested in seeing how it looks like when I edit an episode?

I have written this same blog post published in Finnish, you can find it here.

Koodikahvit -podcastin tarina

Anniina Sallinen — Fri, 13 Nov 2020 17:33:02 +0000

Koodikahvit on suomalainen ja suomenkielinen podcast, jossa istutaan kahvikupin ääreen keskustelemaan IT-alasta, ohjelmoinnista ja kaikesta siihen liittyvästä. Hostaan podcastia ystäväni Pauliina Solanne kanssa.

Mistä idea syntyi?

Tapasin Pauliinan ensimmäisen kerran kun Pauliina liittyi mukaan adminiksi yhteisöön, joka on tarkoitettu naisille ja sukupuolivähemmistöön kuuluville (mainitsin yhteisöstä jo aiemmassa blogipostauksessani). Syksyllä 2019 aloimme jutella teknologiapodcasteista, ja kuinka niitä ei ole kovin montaa suomenkielistä. Saimme silloin ajatuksen omasta podcastista.

Podcastin ajatuksena oli tarjota tietoa myös alottelijoille. Yksi lempipodcasteistamme (englanniksi) on Ladybug podcast ja uskoimme, että samantyyppiselle podcastille oli kysyntää: teknisiä aiheita, mutta myös jaksoja rekrytoinnista ja urasta, opiskelusta ja työskentelemisestä IT-alalla.

Työkaverit apuna

Tiesin, että meillä on töissä neuvotteluhuone, jota käytetään myös äänitysstudiona muille podcasteille, joten meillä oli siellä kaikki tarvittava laitteisto. Olen käynyt yhden studioäänittämisen kurssin lukiossa, mutta periaatteessa meillä ei ollut lainkaan kokemusta äänittämisestä tai varsinkaan audion editoimisesta.

Olen onnekas, koska meillä on töissä ihania työkavereita ja yksi heistä, Aki Kiminki (tuottaa useaa podcastia) lupasi auttaa alkuun pääsemisessä äänittämisen ja editoimisen suhteen. Myös toinen työkaverini, Henry Jalonen, lupautui auttamaan. He näyttivät kuinka laitteet kytketään, kuinka soundcheck kannattaa tehdä ja kuinka peruseditointi tehdään. Aki teki jopa meidän tunnusluritukset!

Tarvitsimme apua myös visuaalisen puolen suhteen. Voin piirtää box plotin Pythonilla, mutta kun on kyse logon suunnittelusta, minusta ei ole apua. Pauliinakin oli sitä mieltä, ettei se ollut hänen alaansa, joten onneksi saimme taas apua. Tällä kertaa Pauliinan työkaveri, Niko Salkola, yrityksen lead designer auttoi meitä.

Podcast-jaksoissa olemme korostaneet yhteisön merkitystä, ja mielestäni kaikki saamamme apu todistaa tämän.

Vieraat ja podin nimi

Minulla ja Pauliinalla on hyvin erilaiset urataustat. Minä opiskelin tietojenkäsittelytiedettä yliopistossa Suomessa, Pauliina osallistui bootcampille Australiassa. Pauliina on myös työskennellyt aiemmin muulla alalla, minä en. Siitä huolimatta tiesimme, että podcast tuottaisi kuuntelijoilleen eniten arvoa jos meillä olisi vieraita. Halusimme, että mukana olisi vieraita monipuolisesti erilaisilla taustoilla ja osaamisalueilla. Aloimme keräämään listaa potentiaalisista vierailijoista.

Toinen asia, joka meidän piti päättää oli podcastin nimi. Otsikot ovat minulle hankalia keksiä. Se on vähän sama kuin muuttujien nimeäminen! Lopulta keksimme nimen, johon olimme tyytyväisiä: Koodikahvit. Haluan aina avata nimen alkuperää, koska se ei ole alunperin meidän keksimä. Koodikahvit on peräisin työpaikaltani Solitalta ja se tarkoittaa yhteistä tilaisuutta, jossa joku pitää teknisen esityksen ja sen jälkeen aiheesta keskustellaan yhdessä. Se on tiedonjakoa Solitan työntekijöiden kesken. Täydellinen nimi teknologia aiheiselle podcastille siis, eikö?

Nykyhetki

Ensimmäinen Koodikahvit-jakso julkaistiin maaliskuussa 2020. Olemme nyt julkaisseet yhteensä neljä jaksoa, ja niitä on kuunneltu yhteensä n. 3850 kertaa. Valitettavasti aliarvioimme kuinka kauan COVID-19 on täällä kiusaamassa meitä, joten emme ole julkaisseet uutta jaksoa heinäkuun jälkeen. Viimeisimmässä jaksossa meillä oli ensimmäinen vieraamme, ja juttelimme alan koulutuksesta korkeakouluissa. Toivomme, että pääsemme pian taas äänittämään, mutta sillä välin otamme mielellämme vastaan neuvoja etänä äänitykseen. Kuten aiemmin kerroin, emme ole kovin kokeneita, joten ei mitään liian hienoa! Meistä tuntuu, että pääsimme vasta vauhtiin, joten olemme superinnokkaita äänittämään lisää jaksoja!

Ja lopuksi, tähän mennessä julkaistut jaksot Audioboom-palvelussa:

Audioboomin lisäksi voit kuunnella podcastia esimerkiksi Spotifyssa ja Apple Podcasts-palvelussa. Löydät podcastin myös monilta muilta alustoilta, tarkista se, jota itse käytät podcastien kuunteluun!

Kiinnostaisiko teitä nähdä miltä näyttää jakson editointi (englanniksi)?

Ihan loppuun vielä linkkejä muihin suomenkielisiin teknologiapodcasteihin:

Deploying containerized NginX to Heroku - how hard can it be?

Anniina Sallinen — Mon, 02 Nov 2020 07:40:58 +0000

I'm currently taking a Docker MOOC course. In part 3 of the course, there is an exercise about deploying a dockerized software to Heroku with CI/CD pipeline. The software can be anything, so I decided to deploy the course materials, and use Github Actions to make the deployment. Github Actions was very straightforward to use in my opinion, and I had no problems with that. Instead, I ran into some issues to get the materials available in Heroku. In this blog post, I thought I'd share the challenges I faced and tell how I solved them.

Course materials and how they're built

So the course material is a simple website built with Jekyll. The website was containerized, so it already had a Dockerfile in the course material repository. I forked the repository to be able to set up a CI/CD pipeline for it, and to deploy it to Heroku.
The Dockerfile that already existed in the course repository looks like this:

FROM jekyll/jekyll:3.8.3 as build-stage

WORKDIR /tmp

COPY Gemfile* ./

RUN bundle install

WORKDIR /usr/src/app

COPY . .

RUN chown -R jekyll .

RUN jekyll build

FROM nginx:alpine

COPY --from=build-stage /usr/src/app/_site/ /usr/share/nginx/html

So from the Dockerfile you can see that it uses a multi-stage build: in the first phase it uses Jekyll image, and in the second phase NginX. The repository doesn't contain any configuration for NginX, so it just uses the default configuration file.

The problem with NginX and Heroku

NginX is a widely used web server that can also act for example as a load balancer or reverse proxy.
As a reverse proxy, it takes a request coming from a client and forwards it to a server. This way the client doesn't communicate with the server itself, because the proxy is internet-facing, not the server. It can also act as a load balancer and send requests from clients evenly to different servers if there's multiple.
In this case, NginX is used as a basic web server, serving the static pages produced with jekyll build command.

Heroku is a PaaS (platform as a service) where you can deploy your software and host it in the cloud. It supports multiple programming languages, and also docker containers.

By default, NginX listens to the port 80. Now when I build the docker image locally on my laptop, and then run it with the command docker run -p 8080:80 docker-course-material I can open localhost:8080 in browser, and it will display the course materials. Now when I first deployed the materials to Heroku, the deployment to Heroku was successful, but opening the website showed a message that something went wrong. I noticed an error in the logs:

State changed from starting to crashed
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

After googling a while, I found the information that Heroku assigns a random port for the web apps when the application is deployed. It says in the Heroku documentation that

Each web process simply binds to a port, and listens for requests coming in on that port. The port to bind to is assigned by Heroku as the PORT environment variable.

This means that I would need to make NginX listen to the random port that Heroku assigns for the application. How can I accomplish that?

First version of the solution

Let me be clear, although I have worked as a software developer before I went to data engineering, I had no experience in NginX whatsoever. I however realised I had to create a configuration file for it to make it listen to another port. My first goal was to make it listen to some other port, such as 8080 locally so that I could test the configuration.

I read that NginX uses configuration file in path /etc/nginx/conf.d/default.conf. My first solution was to create a configuration file that would replace the default file parent image had added, and that in the file a different port was defined. I struggled with the configuration a bit, but the first working version of the configuration file with hardcoded port looks like this:

server {
  listen 0.0.0.0:8080;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }   
}

The line with listen 0.0.0.0:8080; tells the NginX to listen localhost and port 8080. After copying the configuration file in Dockerfile, I was able to bind to the port 8080 and access the materials. This didn't solve the problem with Heroku yet, though.

Binding PORT environment variable

So now that I had succeeded in changing to port to 8080, I would still need to figure out how to define the port dynamically when the Heroku assigns it. I couldn't then just hardcode the port number, but instead I needed to use some placeholder and replace it when the container is run.

So I replaced the hardcoded 8080 in the configuration with $PORT and added a line to Dockerfile that would replace it with the environment variable. Because it needed to be done at runtime, the only option was to use CMD. I couldn't use RUN command in the file, because the environment variable PORT set by Heroku is available only when the container is started. RUN command is executed once at build time.

For replacing the placeholder with the value of environment variable I used sed s command. Sed is a simple stream editor and can be used to transform text. sed s is probably to most commonly known sed command, and it can be used to replace text with some other using regular expression.
The example of sed s command
sed -i 's/cat/dog/g input-file.txt'
replaces all occurrences of a string cat with string dog in the file input-file.txt. Option -i means that the substitution is done in-place.

Final version

So in the end, this is how the configuration file looks like:

server {
  listen 0.0.0.0:$PORT;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }   
}

and this is how the Dockerfile looks like:

FROM jekyll/jekyll:3.8.3 as build-stage

ARG PORT

WORKDIR /tmp

COPY Gemfile* ./

RUN bundle install

RUN echo $PORT

WORKDIR /usr/src/app

COPY . .

RUN chown -R jekyll .

RUN jekyll build

FROM nginx:alpine

COPY --from=build-stage /usr/src/app/_site/ /usr/share/nginx/html

COPY nginx.conf /etc/nginx/conf.d/default.conf

CMD sed -i -e 's/$PORT/'"$PORT"'/g' /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'

Dockerfile has only two new lines: copying nginx configuration file, and replacing the placeholder with value of $PORT. The final command nginx -g 'daemon off' is needed when the Docker is used, so that the NginX stays on the foreground and the container is not stopped immediately.

The final thing that I need to mention is that the problem I encountered is not, in fact, NginX-specific, but rather Heroku-specific. This problem can occur with any web server and Heroku, and needs to be solved by binding on port that Heroku assigns. I just happened to encounter this problem for the first time with NingX.

The solution is pretty simple, but I managed to use a couple of hours with this problem. If you do encounter the issue, I hope this blog post helps!

Hello dev.to, I am Anniina!

Anniina Sallinen — Fri, 23 Oct 2020 17:30:40 +0000

Hi all, my name is Anniina and I live in Finland. My pronouns are she/her.

I'm so excited to join the community! I've been meaning to join earlier already but got the final push once @sjarva said they had just joined. This first post is just to introduce myself, from a tech perspective and outside of it. Nice to meet you all! 👋

My journey to tech

Before I started studying at university I had never been actually interested in IT or coding. My only experience was that my cousin had made me a website, and I edited it a bit. I had always been sure I'd be studying some field seen more "human-centered" such as psychology or media. Well, things didn't work out as expected, and I found myself studying computer science in 2011.

When I applied to uni to study computer science, I didn't even know what it meant. I had some friends who studied it and one of them pushed me to apply. So I did and started without even knowing what it is. At this point, I'd like to point out the fact the in Finland studying at university is free for all EU citizens. I understand how ridiculously privileged I am to be able to study for free, and how that factor allowed me to get into tech by accident.

So I started my studies in 2011 and landed my first job as a software developer in 2014. I had used mostly Java during my studies, and that was the language used in the company, too. In 2016, I quit there to start working in a consultancy company I work nowadays. I started as a software developer there, again working with Java, but as I was completing my studies, I realized I was more interested in data. I graduated from university in 2018, algorithms and machine learning as my master's program, and moved to the company's data unit. I am passionate about machine learning and especially MLOps, but find myself getting excited about a lot of different stuff.

So, to summarize my journey, I came to the field by accident but stayed because I love it!

My other life

Nowadays I'm one of the leaders in a community for women and gender minorities who are interested in tech. The group helps to find valuable resources, helps with the struggling, and provides contacts with like-minded people. We have organized workshops and physical get-togethers and encourage group members to share their knowledge. At the time of writing this, there are more than 4000 members in the Facebook group. Some members are more active than others, and it's ok.

I started a podcast with another community leader of the group last winter. It is a tech podcast in Finnish, called Koodikahvit (Code coffee in English), and it aims to provide information about the industry. We also speak about our experiences and have guests sharing their experiences and expertise. We have published four episodes so far, but unfortunately, because of the pandemic, we haven't been able to record and publish more. If you have any tips for remote recording, feel free to share them!

Other than an IT professional, I am a dancer. I started dancing at the age of seven I think and continued until I was 18. During the years I danced hip hop in several different groups that performed and competed and it was my passion. At the age of 18, I moved from my childhood home and had to start paying for my hobbies, which basically meant I had to stop dancing. I continued it later as an adult and in addition to the hip hop, I started voguing. As a cis, white woman I do my best in educating myself and respecting the roots of voguing. I know I have still a lot left to learn.

Some random facts about me:

I'm not good company in the mornings. I get up right away but prefer not to talk anyone and may look a bit angry
I know four languages: Finnish (native), English (pretty well, I'd say), Swedish (not that well but some basics) and Spanish (basics at well)
I'm a minimalist. It doesn't mean I throw a lot of stuff away, it means I don't buy a lot of stuff.
Sustainability (environmental and ethical perspective) is very close to my heart. I could talk about sustainable clothing for hours. For this reason, I'm also vegetarian (not vegan yet).

I think that's it about me, hope you got an image of what kind of person is writing here! You can find me on Twitter with the handle @annisalli. One last thing about me: I and my partner hopefully get a dog next year. It's going to be a Samoyed puppy, and I will definitely post (way too many) pics of him to Twitter when we get him home!