Forem: Codegram

Crawling websites with elixir and crawly

Codegram — Thu, 13 Aug 2020 12:00:00 +0000

In this post, I'd like to introduce crawly which is an elixir library for crawling websites.

My idea for this post is to be a quick introduction to how we can use crawly. For this little example we're going to extract the latest posts titles on our website and write it to a file, so let's do it.

First of all, we need elixir installed, in case you don't have it installed you can check this guide.

Once we have elixir installed let's create our elixir application with built-in supervisor

mix new crawler --sup

In order to add the crawly dependencies to our project, we are going to change the deps function in the file mix.exs and it should look like this

defp deps do
    [
        {:crawly, "~> 0.10.0"},
        {:floki, "~> 0.26.0"} # used to parse html
    ]
end

We need to install the dependencies that we just added running the command below

mix deps.get

Let's create a spider file lib/crawler/blog_spider.ex that is going to make a request to our blog, query the HTML response to get the post titles, and then returns a ParsedItem which contains items and requests. We are not going to leave requests as an empty list to keep it simple.

defmodule BlogSpider do
  use Crawly.Spider

  @impl Crawly.Spider
  def base_url(), do: "https://www.codegram.com"

  @impl Crawly.Spider
  def init(), do: [start_urls: ["https://www.codegram.com/blog"]] # urls that are going to be parsed

  @impl Crawly.Spider
  def parse_item(response) do
    {:ok, document} = Floki.parse_document(response.body)

    items =
      document
      |> Floki.find("h5.card-content__title") # query h5 elements with class card-content__title
      |> Enum.map(&Floki.text/1)
      |> Enum.map(fn title -> %{title: title} end)

    %Crawly.ParsedItem{items: items, requests: []}
  end
end

Now that we have our spider created it would be nice to save what we're extracting into some file. To do this we can use a pipeline provided by crawly called Crawly.Pipelines.WriteToFile. For that, we need a config folder and a config.exs file:

mkdir config # creates the config directory
touch config/config.exs # creates an empty file called config.exs inside the config folder
mkdir -p priv/output # creates output folder inside priv where we are going to store our files

Now let's create the configuration to save the response from our spider into a file.

use Mix.Config

config :crawly,
  pipelines: [
    Crawly.Pipelines.JSONEncoder, # encode each item into json
    {Crawly.Pipelines.WriteToFile, folder: "priv/output/", extension: "jl"} # stores the items into a file inside the folder specified
  ]

Now that we are good to go, we can open the elixir repl

iex -S mix

And then we can execute our spider

Crawly.Engine.start_spider(BlogSpider)

The spider is going to be executed by a supervisor and then we should see a new file inside priv/output folder. In my case the lasts posts showing in the first page are

{"title":"\"High tech, high touch\": A communication toolkit for virtual team"}
{"title":"My learning experience in a fully remote company as a Junior Developer"}
{"title":"Finding similar documents with transformers"}
{"title":"UX… What?"}
{"title":"Slice Machine from Prismic"}
{"title":"Stop (ab)using z-index"}
{"title":"Angular for Junior Backend Devs"}
{"title":"Jumping into the world of UX 🦄"}
{"title":"Gettin' jiggy wit' Git - Part 1"}

This is just a simple example of what is possible to do using crawly. I hope you enjoyed this introduction and remember to be responsible when extracting data from websites.

My learning experience in a fully remote company as a Junior Developer

Codegram — Tue, 14 Jul 2020 08:00:00 +0000

Three years ago I decided that I wanted to change my profession and look for better opportunities. I’m an Industrial Engineer and IT was, among other sectors, the one that since the beginning drew my attention the most, in particular the world of application development. I could still remember some of my development skills from my studies at university, so I decided to give it a try and start studying a language called Ruby.

The first steps were not easy, I couldn’t understand many concepts, and my English skills back then slowed down my progress. After improving my English skills and completing some courses and books and after a lot of effort and frustration, I managed to land my first job 2 years ago.

Due to my personal situation I needed to find a remote job, but I would usually get the same answer: "with your profile you must work in the office", another unattainable job for me.

I started wondering what was the difference between talking to a person next to you, or someone on the other side of the screen when they can both hear you and see your code?. My conclusion was, after speaking with experienced remote developers, that there was none… if they really want to work with you.

My very first job wasn't remote, the office was located a 5-minute walk from my place and that's why I decided to give it a try, but in the end, it only lasted one month, a really long month. They didn't know how to work as a team, which led them to use strange work methodologies and management also had a weird relationship with the employees that thankfully, I never saw again. It wasn't a good place to start my professional career and I didn't have the feeling that anything that I did couldn't be done remotely, so I decided that my next job has to be fully remote no matter what.

During the next months, I rejected a good number of offers, where working from home with my profile was not an option. Finally, I managed to find a company where they wanted to try remote work with juniors!

It was much better than my previous experience, but it wasn't my place either. I worked for this company for more than 6 months, but we couldn’t figure out how to make it work. The company managed to sign a bunch of deals with new clients and they didn't have resources to help me to grow. I felt stuck and not performing as I would like to so we decided that it was time to move.

Those first experiences didn’t help me to clearly see my future in development, but everything changed when I joined Codegram…

The Interviews

I met the three founders, Oriol, Josep, and Txus, through 3 very pleasant and entertaining interviews that managed to get my attention for the company.

I admit that it was not the first time I heard some of the things they told to me; everyone wants to teach you and help you to improve, but there is a long way between wanting to do something and doing it. Nevertheless, the way they behaved and how openly they spoke gave me more confidence than my previous interviews, so I decided to follow their hiring process. Finally, I managed to get the job!

My first days at Codegram

The evening before my first day of work, I was eager to get started as well as worried about what I might find in this new adventure. If it went wrong, I had lost nothing, but I wanted it to go well.

The big day came and when it was over I felt I had nothing to worry about.

Day 1

For me, the hardest thing about working remotely is the feeling of being alone in front of something unknown. When I start a new job, a feeling of being lost usually invades me and the days turn into a research job. This time I had my new computer ready to start working, and in my email a document with the first tasks to be performed within the company, in addition to the name of the person who was going to help me whenever I needed it, that seemed like a dream!

Having a reference person, Arnau in my case, is very useful. Of course, there are other people I can ask for help, but, especially at the beginning, I didn't know who was the right person for different topics, or if they have a high workload and you are bothering them. In addition to this, the existence of a person who knows your defects and your strengths at work can help you to learn faster. Something I did not expect that day was a meeting with Josep and Oriol, to find out how my day had gone, it was the first time that a C-Level cared about how I felt at work. In this sector, it is very important to know if the people in your company are happy. I have know companies that have suffered massive developer losses, with the risk that this entails for their own existence.

The first day of work gives you a fairly accurate idea of what your day to day will be like and my day ended with a desire to learn more.

My first months

In my day to day, there are many concepts that I do not know, and my big problem when learning as a Junior is having too many open fronts and not knowing what I should study.

As I discover this world I am more aware of how vast it is and of everything that I do not know, which is really overwhelming, that is why I need an easy goal to achieve so as not to block myself and continue learning. I have different meetings in which I explain what I would like to improve or what is a problem for me, and in this way, I can direct my work and my learning in the same direction.

Currently, my job is to fix small bugs or add small features, without the obligation to be productive from day one, it helps me to discover new concepts and acquiring better practices in my day to day, such as learning to research, debug or manage repositories with agility.

One of the most efficient ways of learning is to alternate studying something new by working on what you have learned, for which I have 5 hours each week to study whatever interests me. Also, every 2 weeks we enjoy listening to our colleagues talk about an interesting topic, what better way to discover something new than when someone explains it to you from their own experience!

Nowadays

After 4 months I am still curious to know how much I will be able to grow here, I am learning new things and improving my coding speed, although not as fast as I would like.

After this experience, I no longer doubt my future as a developer.

Lessons learned

This is what I have learned from my experience:

It is possible to work as a junior remotely if the company is willing to help you.
Having a dedicated mentor you can turn to at any time increases your learning speed and reduces downtime.
Carrying out small tasks makes you aware of your strengths and weaknesses, letting you establish a more targeted training plan.
To learn at a higher speed, it is important not to have the obligation to be productive at the beginning, since that only frustrates you. It is more important to acquire a stable knowledge base to gain more speed over time.
A disgruntled worker does not learn, or does so at a reduced speed.

In short, remote work can be facilitated and also help improve your career plan, provided you are lucky enough to find a company willing to make it possible.

📌 Codegram is always looking for great talents. Visit our career page.

Cover photo by Veronika Hradilová

Let's read! Book recommendations for World Book Day

Codegram — Thu, 23 Apr 2020 12:49:00 +0000

Sant Jordi, Catalan St George's day on April 23rd, is a very special tradition in Catalonia. Streets fill with stands, where people buy books and roses as a present to those they love. This year we are a bit sad, since due to COVID-19 all the celebrations have been cancelled.

We want to make this day special despite the current situation, so a few codegrammers wanted to share some of the books that have helped us during our careers as programmers, project managers or designers. The best part of it is that most (if not all) are available online, so you'll be able to read them safely from your home. Let's celebrate reading! 📚🌹

(Please note that none of the links are affiliate links)

Being Geek: The Software Developer's Career Handbook, by Michael Lopp

Recommended by Núria

I read this book back when I did not consider myself a programmer yet, but I was working in the industry (I was more focused on design, and HTML and CSS development). It helped me realise that I was not happy in the company I was working at the time, I learned to recognize red flags in an organization, and it empowered me to start my journey into programming.

Test Driven Development: By Example, by Kent Beck

Recommended by David

A few years ago I started using Test Driven Development (or TDD for short) and I hated it! It's not like I didn't want to test things, because I think testing is extremely important for big systems, but the methodology itself was a pain for me. I decided to read a few books about it but most of them were technology-oriented. I finally found this book and I liked it! I still don't like TDD but this book changed my vision and I use it most of the time while developing new features. I don't follow a strict TDD approach but close! The examples of the book are great and easy to follow.

Functional Programming in Scala, by Paul Chiusano and Runar Bjarnason

Recommended by Oriol

Even though I don't use Scala usually, this book really helped me get into functional programming. It's quite accessible and it combines theory with quick exercises to keep learning and understanding the principles of FP. If concepts like generics, monads, monoids, or functors sound like total gibberish and you'd like to learn more about them, this is your book!

Eloquent JavaScript 🦜, by Marijn Haverbeke

Recommended by Anna

This was the first programming book I read and was recommended to me when I first started learning. I found it very logical and liked how the author used examples to explain the code. There are also exercises at the end of each chapter to test your understanding. It starts off basic but get’s pretty involved once you get into it, so I think it’s a book you could take your time reading as you’re learning. The best part is it’s available free online so if you’re just dipping your toe into programming to see if you like it, you don’t need to fork out for an expensive book!

The Infinite Game, by Simon Sinek

Recommended by Eva

I read this book after watching this talk. The book is about strategy and balancing short-gains with long-term ideas. Building products with shared values and propose as a team make an impact and are the basics of an infinite mindset.

In these times of pandemic, this has also come to mind:

"a company which builds with an infinite mindset is resilient, not stable. Resilient to an unpredictable change, markets shift, or new competitor — a resilient company can withstand and be transformed by the change"

Practical Object Oriented Design, by Sandy Metz

Recommended by Elisa

It’s the book that I’m reading currently and it’s one of those books that grips you from the first moment.

It teaches you to think before programming, it shows us how we should organise our code to make it readable and easy to maintain over time, all backed by a large number of examples.

It’s one of those books that when you re-read it continues to teach you new things.

Learn You a Haskell for Great Good! by Miran Lipovača

Recommended by Armand

Haskell is what I expected programming would be before my first acquaintance with it. I had no idea yet that it can be an imperative affair most of the time (of course, I would not have used these terms by then). When I heard about it later on, it was love at first sight.

Regardless of its real-world applications, Haskell is a delight to play with. Once you understand what is going, it gives you both that rare feeling of elightenment and some insights that will help you elsewhere in your day-to-day programming.

However, both the Haskell ecosystem and community are not known for being the most welcoming ones. That is why I am recommending this book, as it is the one that finally helped me take the first steps into it.

And it's witty, it has cute drawings and it's free on the internets! 🎉

Crossing the chasm, by Geoffrey A. Moore

Recommended by Divins

The point of greatest peril in the development of a high-tech market lies in making the transition from an early market dominated by a few visionary customers to a mainstream market dominated by a large block of customers who are predominantly pragmatists.

The gap between these two markets is what we call the chasm and must be a primary focus of any long-term high-tech marketing plan.

This book gives imho a nice marketing vision that has helped me understand the need of changing strategies while your product matures and the target audience changes.

Clean Code, by Robert C. Martin

Recommended by Edgar

When I read this book for the first time it completely changed the way I was writing code.For me it's such a great book because it explains how you can write more readable and easier to test code and it also shows the importance of doing so.

Hello Ruby, by Linda Liukas

Recommended by Svenja

First of all, I have to admit that I actually didn't read these books (there's more than one and they are available in more than 20 languages) but I love everything about the idea behind: showing kids the world of computers, programming and technology. And teaching this knowledge in such a marvellous way with (in my opinion) gorgeous illustrations and really nice examples (you can find some on her website).

why's (poignant) Guide to Ruby, by why the lucky stiff

Recommended by Agustí

An entertaining introduction to Ruby, an unusual programming book with cartoons and humor.I recommend to read it just for the sake of reading it.

The book is distributed under the Creative Commons Attribution-ShareAlike license.

Inventing Bitcoin, by Yan Pritzker

Recommended by Faith

This is a great short book explaining what Bitcoin is, Why it is important and who controls it. It gives high-level explanations of the concepts so it's a great resource for beginners to understand the history of bitcoin and where it's headed.

Cover image by Ajuntament de les Franqueses del Vallès