Forem: lukaszkuczynski

Spark is Pandas on steroids

lukaszkuczynski — Tue, 17 Sep 2019 00:00:00 +0000

Read it if you fell in love in Pandas, and started to think about whether parallel processing can be easy

Who are you

Let me guess. You are a data scientist who runs preprocessing and modeling on your machine using pandas and other cool libraries. Or you are just a mature software engineer who once heard about parallel processing but have never seen the real use-case for that. Even if you don’t belong to either group think of it:

why do I need parallel processing?

Why it may be of your concern, even if you did not realize that before? Let me share with you my experience.Lately, we had a pipeline when fbprophet was used to predict several thousand of series. It took ages to complete the whole process. A thought: maybe running it on some Azure fast computing set will speed it up? Yes, but still it is hours of waiting. So what is the bottleneck here? fbprophet and all of that processing is one-machine solution. You cannot just spread it through nodes to be processed in parallel. But it would be nice, huh? You say: these 10k rows go to compute #1, the other 10k rows go to #2 and so on. Want to make it happen?

Spark is the answer

Apache Spark was created to make parallel processing possible for you and me. But what was before?The era of Hadoop was there. The big ecosystem of tools to process big data. Maybe now some of the experts claim: it is out of date, but it was a big leap in the future - you can split your big job into smaller chunks and send them to be processed on several machines. Then the result is collected and here you go!

You can utilize the commodity machines you have without a need to invest in a top-class machine, which can be very expensive. You can scale out (as opposed to scaling up)! But to use Hadoop there were some limitations involved. You had to stick to MapReduce and Java language which was the only building and API language. Moreover, all your computation steps were using disk heavily, which was one of the biggest bottlenecks of the architecture.

So this is why Spark was introduced. The main advantage of this engine is - as compared to Hadoop - processing all of its data in memory. And you do not think in terms of the machine which drives the process (driver). You can instantiate as many workers as you need. All of these - making up a cluster - are quite a powerful tool in your hands. What is more, you are not glued to Java-based MapReduce, but there is nice DataFrame API at hand, having support for multiple languages, like Python, Scala, Java, R, and even SQL.Sounds expensive and hardly available? On the contrary!

How can I get it

Spark or hosted Spark?

You could install Spark on your machine. But why if you may try it online? For free! No more installing binaries on your machine to try something out. Even there is no need to have docker daemon running somewhere in the background. Why? There is a blessing (or a curse) of PaaS and SaaS abundance all around you. So how about you. Would you like to try Spark yourself?

Databricks

Let me introduce Databricks. This is a product available online, created for widely-understood analytics. It is also hosted by Azure (Microsoft cloud offering), where you can just try it and run. I mentioned Azure here because it was the first place when I encountered Databricks.

Especially for the sake of this article I created a new account on community version of Databricks to check how easy it is. And yes, it took me roughly 2 minutes to create and start my first Databricks account without any credit card information and so on, just fill in the form on this page.When you have your cluster ready, just may play with it using Spark API.

use-case

I do not like learning for learning. I like to notice how some tools empower me to solve the problem faster. Maybe as an IT-person, you were traversing long, boring log files to find out what happened to the application. You have Gigabytes of logs and you wished to see patterns in it. You may continue grepping, but no… how are you going to make it happen with Spark?

You don’t have to (however you may, if you wish) to upload your own log files to start playing with it in Community version of Databricks. There are many datasets already available on Databricks to use them, including sample log files.First, we will read them all to the structure called RDD:

rdd_original = sc.textFile('/databricks-datasets/sample_logs/')
rdd_original.take(10)

The previous command just reads all files from the location provided to an abstraction called RDD. This one can be used for further transformations (map/reduce/filter) or transformed into a data frame. There were some operations done to parse data out of file rows. For the sake of simplicity, I skip it in this article - the full notebook however is also available there. To create a data frame, we just need to provide the schema.

from pyspark.sql.types import StructField, StructType, IntegerType, StringType, TimestampType

schema = StructType([
  # your data types definitions go here
])

df = rdd_mapped.toDF(schema)

With a data frame ready we may start doing some aggregations and draw conclusions. For example, you may be interested in how often different errors occurred for specific users. We may learn about it using Python API or SQL.

df_identified = df[df['user'] != '-']
df_grouped = df_identified.groupby(['user','status']).count()

For you, my dear Pandas developer, these operations should look pretty familiar, am I right?Of course, you may express your query with SQL syntax.

df.createOrReplaceTempView('log')

select user, status, count(*)
from log
where user <> "-"
group by user, status

Databricks gives you not only Spark on Cloud experience, but also contains basic visualization capabilities. I encourage you to use them for your scientific purposes, not as a tool to present data to your customers.

Summary

The data analysis done here is available publicly for some period of time under this public link. I also uploaded that as a downloadable asset here.With Spark features and DataFrame API we can do really a lot. What is important the full advantage you can see when dealing with big amount of data that is hard to ingest by a single machine.

Predicting dev.to posts

lukaszkuczynski — Mon, 18 Feb 2019 00:00:00 +0000

The following is the short description of the full small end to end data science project I chose for myself. I always wanted to play with text data, so I chose to create my own corpus. I will use it later to apply some machine learning on top of it. Take a look at how do I predict a category of some blog posts.

Goal

Recently I learn about Machine Learning (ML). There are lots of possible areas of use: analyzing video, images, speech and text. The last one seems to be the most appealing to me. Online content grows every day. We have too much to read. Natural Language Processing (NLP) is a data science response to this problem. The content of my small project comes from a great developer community blog: dev.to. To show basic ML I will guess the tag based on the dev.to post content. Thus, I will run supervised learning. In simple words: I am pasting the content of a post and it should tell me is this about “java” or maybe “python”. The effect of my work is a web app, so feel free to play with it here. For the sake of simplicity, I used 4 major categories only. Take a look on this life demo gif:

Corpus

What is a corpus? In the NLP world, it is a group of documents. We will analyze these documents later on. Doing any ML project you have to start with something to analyze. With some data. In NLP you have to get some texts. It is not obligatory to scrape them from some internet portal. You can start with built-in corpora as these being part of nltk library described here. In my case, I want use dev.to data, so I will use API they expose. I will create my own corpus. The process of data acquisition is visible as part of my GitHub project here. Doing it with Python is so easy, I can use great libraries, such as BeautifulSoap and requests.

BoW

The BoW acronym stands for Bag of Words. This is a simple way of vectorization of text. Its name comes from the idea of having a bag where we put all the words. We don’t care how are they ordered. We just need to vectorize our text. This is a prerequisite before applying any ML algorithm on text. So how does this vector look like for a simple sentence? Taking a CountVectorizer available as part of Scikit Learn library let us see how does it transform text to vector.

sentences = [
  "I like playing the violin",
  "Playing football is nice",
  "How do you like playing football?"
]

from sklearn.feature_extraction.text import CountVectorizer
import operator
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(sentences)
vocabulary = vectorizer.vocabulary_.items()
vocabulary_sorted = sorted(vocabulary, key=operator.itemgetter(1))
print("vocabulary is:")
print(vocabulary_sorted)
print("matrix resulted is:")
print(X.toarray())

The result would be:

vocabulary is:
[('do', 0), ('football', 1), ('how', 2), ('is', 3), ('like', 4), ('nice', 5), ('playing', 6), ('the', 7), ('violin', 8), ('you', 9)]
matrix resulted is:
[[0 0 0 0 1 0 1 1 1 0]
 [0 1 0 1 0 1 1 0 0 0]
 [1 1 1 0 1 0 1 0 0 1]]

And let us see how does it look like when we visualize it as a heat map:

As we will produce a model soon we want to make it of good quality. We will perform a Tf-Idf (Term Frequency - Inverse Document Frequency) transformation. It means we take into consideration how frequent the word is in a document (TF) and how rare it is among other documents (IDF). Applying this transformation in sklearn results in a matrix little different from what we saw before:

vocabulary is:
[('do', 0), ('football', 1), ('how', 2), ('is', 3), ('like', 4), ('nice', 5), ('playing', 6), ('the', 7), ('violin', 8), ('you', 9)]
TfIdf matrix resulted is:
[[0. 0. 0. 0. 0.44451431 0.
  0.34520502 0.5844829 0.5844829 0. ]
 [0. 0.44451431 0. 0.5844829 0. 0.5844829
  0.34520502 0. 0. 0. ]
 [0.4711101 0.35829137 0.4711101 0. 0.35829137 0.
  0.27824521 0. 0. 0.4711101 ]]

And the visualization

The output matrix is little different now. It is because we care how often terms occur in the whole data set. In our case TfIdf promotes rare ones like violin and deprecates the popular, like playing.

Machine Learning

Having vector ready we can apply math on top of that. In supervised learning we have to find a function that matches input vector X with label y. Having structure and kind of data we process there is a need to apply the correct algorithm, not just a random one. If you’re facing this problem find this sklearn cheatsheet. In my case, I am doing the classification of textual data. I found many experienced data scientists tend to use Naive Bayes for that. After several trials I found it useful, too. You can check my notebook here. Remember, with textual data we use the Multinomial and not the Gaussian algorithm. So there is a snippet from the model training process.

import pandas as pd
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split

def train_naivebayes(X, y):
  X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
  bayes = MultinomialNB()
  bayes_model = bayes.fit(X_train, y_train)
  y_pred = bayes_model.predict(X_test)

Please find the full codebase in my repo.

Deploy

Once your model is ready there is a time to share it with others! You can do it with Azure or AWS. They usually have ready-to-use Docker containers where you just have to put your model inside, and they magically expose it as a REST service. However, the 1st time I expose some model I wanted to have everything under control. This is why I decided to build my model inside of a web application myself. It is as easy as serializing building blocks to files and then uploading these files to a server. You can go there and check my app deployed.

Lessons learned

I do not think the project is perfect. I measured the accuracy of the model and it is around 80%. To have better results we could:

use another algorithm, f.e. ensemble algorithm, or tune the existing one
have more data than hundreds of entries (more data always means: better)
clean data better

I am happy to see how a nice experience working with data with sklearn was. Python provides a must-have ML tool belt. I also tasted the full stack of the ML problem. I collected, analyzed, fitted model and finally deployed it.

Clustering snacks and vegetables

lukaszkuczynski — Thu, 10 Jan 2019 00:00:00 +0000

Supervised vs Unsupervised:

Lesson no 9 of Udacity’s Introduction to Machine Learning showed another aspect of machine learning to me. I could strengthen my knowledge in an area of clustering. This is topic particularly interesting as is it reveals insights, which may inspire you to further analysis. As I am lately into NLP, one possible use case is that it can group similar documents together and then you can discover the connections between them. Supervised learning is about having labels and then checking if these labels fit features of newly acquired data. Unsupervised learning has no comparison phase, as no labels are known at the beginning.

K-Means and sklearn

Clustering using K-means algorithm is one of the most widely used unsupervised techniques. It is about finding such cluster centres that will allow the whole system to be in a “harmony”. Distances are calculated using Euclidean distance.

There is a nice visualization that was advertised during the training, you can take a look at fantastic work was done at naftaliharris blogAs with every algorithm we have to be aware of its limitations. One of these is the fact that K-means is a hill climbing algorithm. This very fact has its own Wikipedia page, that you can check. So the algorithm is very sensitive to local minima. Thus, some specific way of choosing initial points (centroids) can lead to clusters we would like not to have. This is why in sklearn implementation you are encouraged to do the clustering several times and then the best clustering is chosen.

Use case : veggies vs snacks.

The Kaggle for the following you can find here. I thought: maybe I could use clustering somewhere? Maybe a computer can be smart enough to distinct junk food from a nice one? So I found this dataset, that is the Australian Food Nutrient Database. I played around a bit, and you know what? It works. Of course, if I spent more time on the careful assignment of snack and vegetable categories it would be more meaningful, without that I have some outliers, like tomatoes with a lot of fat, as an example. Anyhow KMeans was able to mark a clear distinction between high calories, low Vitamin C snacks and low calories, healthy vegetables.

2018 Accomplishments

lukaszkuczynski — Fri, 04 Jan 2019 00:00:00 +0000

what is in the past

It is good to ‘find enjoyment for all your hard work’. I can see many people doing their year summaries. Obviously, there are areas more important for than job matters, but here I will focus on business only. What has changed?

machine learning

There was a lot of learning this year. I always thought this is too much for me. I always liked Math, but when I started to dig into the math behind the machine learning the task seem to be daunting. But the Udacity and Pluralsight brought much to the table. Especially the Udacity one. With simple examples and lots of pictures, they were able to explain difficult matter easily. I immediately thought about the business use cases. So there was a need and motivation. I took some data and I was successful to apply this knowledge in my 2 projects. Both were about Text Analysis. The first was supervised machine learning with automatic team assignment for ticketing tool we have. I was able to achieve pretty high accuracies for teams that had distinct areas of responsibility. The second was applying clustering as an example of unsupervised learning with the famous K-Means algorithm. Of course, I cannot share this data. But when playing with my machine learning basics I am happy to share some of insights in my Kaggle account based on public datasets. You can see some of my kernels here. By the way, learning about Kaggle is another big event for me this year as this is a great place to share my knowledge and progress. But not only this, I can gain inspiration from others, too. Some of the posts are proof of this, check this about beer in Brazil.

cloud

This year I understood some of the good aspects of “everything into the cloud” hype. When thinking of one’s local or on-premises environment I can see problems with scalability and lack of fluency. Everything seems to be so hard to get when for every change you have to raise the incident to a dedicated team. How different it is when using cloud infrastructure! This year I had a chance to work with Openshift as PaaS. All you need is to close your dependencies inside of a Docker image and reuse it whenever you need.

I also played with big guys on the market: AWS and Azure. I even made a little comparison of their services deploying similar infrastructure on both of them, because I wanted to compare their Serverless support. I built a simple notification engine leveraging both Azure Functions and AWS Lambdas.

The main part of my current billable assignment was to conduct several data analysis tasks. I will take a closer look at it the next section but I can tell that without Azure Databricks this journey would not be that easy.

In the cloud these my 3 favorites:

it is unlimited
pay as you go
don’t care about the infrastructure

data analysis

As I mentioned before I started to use Spark some years ago, but that time it was – to put it mildly – not so justified. So what has changed lately? I was given several tasks to make use of big data we have in our log files when it comes to user activity and I noticed when does the parallel computing unleashes its power: when the volume is greater than Mbytes. Of course, it is not yet the TB of processed data but we are getting there. Spark is cool when you run it somewhere where the platform is available. In my current assignment, I can leverage the power of Databricks, as we have some Azure account to utilize. Personally, Databricks make things really easy and fast. Why?

First, I love playing with Python notebooks. It makes your job so descriptive, I think the idea of mixing code with some textual explanations is great.

Second, it is so nice to use the flexibility of fast provisioning available there. You just choose the cluster you need and it is ready within minutes. If you are dissatisfied – or you are running out of money! – just dismiss the one. Simple as that.

I was able to gain really nice insights that were hidden before the stakeholders, so then we can do the decision process better. We could answer the following questions: what are the areas are a user’s favourite? how a user is responding to a new feature? what are locations of users, and does it affect their choices?

Gaining insights from data is important and I think data analysis with a help of Azure Databricks play is crucial here. This work can open the eyes of decision-makers to some challenges or successes. It can make complicated data an easy one.

more blogging

This year I could do some blogging. Only at dev.to I published 11 entries starting May 2018. This year I left my Wordpress account for the sake of Github pages portal which I Ioved because I can write markdown only, and I like git-pushing of my blog entries. Not only I was able to write about my progress but also I could report 2 nice events I was part of, Python Conference in August PyConPL and the internal AI days at Volvo. Of course, dev.to plays a very important role here because the community there is great and I like the User Experience when dealing both with content and its design.

what comes next?

What about my business-related plans for this year? In the 2019 year I do not plan to learn any new language. I am not to going to change my direction. Rather I would like to continue the progress I started 2018 year. The year that I understood even more my change from Java to Python was a good choice. With Python I feel I am young again, I like programming and writing the code I feel like a poet now, not like a journalist forced to write a few articles before the end of the month.

I am going to read new books, I don’t know which yet… But for sure there is a goal to finish the great one I started to read lately about storytelling with data. Wow, it really opens my mind to the fields that are so green. I want to finish the Udacity in February and start playing with text data, seriously. By the way, you can check my progress of it on a dedicated git repo.

To keep things short:

I want to be a help and use the help of others.

I want to communicate to bring results.

I want to make things simpler with visualization.

I want to automatize to make things faster.

Regression and Outliers and Beer

lukaszkuczynski — Wed, 19 Dec 2018 21:22:12 +0000

Distribution

It is all about data distribution. Before applying any machine learning algorithm you have to answer the question. Do I know what I want to know? Having this question clear in mind is required when one wants to stay focused. So what is your data distribution? In lesson no 7 of Introduction to Machine Learning a student is given several introductory tasks to comprehend is data distribution continuous or discrete one. The discrete distribution can be likened to a categorical dataset with elements like human names or car brands. The continuous one is connected to variables that can have any value within a range, like age or volume of something.

Regression Parameters

Regression is a way to find the pattern in a continuous distribution. Given X values (one or more features) you are finding what outcome it is producing. People sometimes talk about a regression when saying: the more something happens the more some value grows. The linear regression is about having the relationship between the two figures. Say you want to know how much the air temperature affects the beer consumption. For this study please check the next sections of this entry. There are extensive and nice explanations what does a regression mean mathematically (f.e. this blog or even simpler here ), so I will not go into details in this post. But to be concise I can tell that I was taught about two main parameters of regression. These are slope and intercept. Because the regression is a mathematic formula it can be represented as the following figure shows

Y = a + bX

In scikit-learn this two parameters can be easily fetched, what I will show in the following section.

Sklearn is fitting

Simple Example

Having data imported to a data science toolbox you can leverage the power of ‘scikit-learn’. It has major machine learning algorithms already built-in, so you just have to follow a simple flow of:

data preparation
choosing X and y
fitting your data
the algorithm evaluation
visualization (optional)

With sklearn doing so is as simple as the following (imports excluded):

x = np.array([1,2,3,6])
y = np.array([2,4,6,12])
plt.scatter(x,y)
reg = LinearRegression()
X = x.reshape(-1,1)
reg.fit(X, y)
print("Score is %f" % reg.score(X, y))
a = reg.coef_
b = reg.intercept_
print("Regression calculated for equation y=ax+b. Params are a (slope)=%.1f, b (intercept)=%.1f" % (a,b))

For the complete notebook please refer to my Kaggle kernel. Normally we will not use the training values (X and y) for scoring, but this snippet works just for a presentation purpose.

The real story: Kaggle and Beer consumption

I was thinking which dataset will be a nice visualization of linear regression. And this is how I reached Beer consumption in Sao Paolo dataset. I did some calculations harnessing scikit-learn and visualization given by matplotlib integration built-in into pandas. If you are interested in how was a data prepared and my steps please refer to my Kaggle notebook.

Outliers

Lesson no 8 from the course teaches about outliers. It is an important factor, especially when calculating a linear regression. Most times you can just take a look at data doing the scatter plot and you’re done. It was the way in the beer consumption dataset. I didn’t see any major anomalies so no outliers removal technique was used. But it is good to remember that we do have them, to make use of this when needed. During this lesson, I saw how practical outlier removal is when dealing with Enron data.

Summary

Lessons 7 and 8 of Udacity’s Intro to Machine learning focused my attention to continuous distributions. I had a chance to work with real-world data and the outcome was refreshing my Kaggle account. It was another proof of viewing Python as a primary language for data analysis. Of course, I continue update of GitHub repo that serves as an insight into my coding while studying this course.

AI days

lukaszkuczynski — Fri, 30 Nov 2018 00:00:00 +0000

Intro

The following is the report about the event that took place Nov 29th, this year, inside the Volvo. We could listen to many talks concerning the AI revolution in both world and local scope. We are already using AI. Where and what tools do we use? Let me explain.

AI application, trends

There are some misconceptions about what AI is and how different it is from Machine Learning. Thanks to Jair Ribeiro we could learn that AI is currently a popular trend, and we have to face it. In Volvo, there are some use-cases with the famous Vera vehicle and other projects using data to draw conclusions and to be informed.

Chatbots

Virtual assistant: this is a topic which becomes more popular recently and it is surely not a dream! Whenever you visit some website and there is a nice chat window popping out at the bottom right of your browser – it is probably a machine talking to you. In Volvo, there is a use-case of a chatbot in the ticketing system. We were shown by Singh Kumar the chatbot that helps you with internal knowledge database, helping you to choose the correct person or team to help you with your problem. It can also answer you directly if he knows the answer. There are many possible challenges we can face when installing a system like this one. F.e. you have to teach your chatbot when it should give up and say “I don’t know”.

AI Mindful?

Thanks to Patrick Kozakiewicz we could have a moment of reflection on what kind of person we are and how does it affect the software we produce. Of course, AI is another type of software and it was interesting to learn that it could be biased by the way it was produced. There were some tricky questions about the “bad AI” and of course it can happen if we fed AI with a bad content. Should we teach AI our normal, human behaviors? I believe we should. And we all should be aware of the fact that when we are under stress, our productivity is worse, so take care of your surroundings.

Data collection

The very important and the most time-consuming part of a machine learning process is the preparation phase. As Bartek Starościak explained, we do have the project of an improvement of a production process. We want to predict which features have an impact on the future vehicles failures. To collect all the data you need to face many kinds of inputs. Apart from nicely formatted files, we do have some hand-written notebooks and the variety of different systems and databases.

Qlik

Having lots of data you will sooner or later face the issue of choosing what visualization is the best for explaining your data to a business person. As told by Tiago Hubner and his friends we can leverage QlikSense for that purpose. There are lots of features of that platform but one really caught my attention: Insights. It is a nice feature suggesting you what you can find in your data and how can you visualize it.

Black box

There are some machine learning algorithms that are pretty hard to explain. When the model explanation is important? As Bartosz Kurlej explained, there are some business areas when there is a need for a detailed explanation. F.e. if you want to prove the bank management that someone should not be given a loan, it is better to explain what the reasons are for that. To make it happen we can use Lime algorithm.

Tools

Azure and Databricks

I cannot imagine AI without tools. At some point of the time, you will face the following challenge: your local machine is not powerful enough. Training models require a lot of the computational power which – the more likely – you do not have. What should I do then? Use clusters and make your job distributed. It is far easier to do find lots of regular computers than just one big machine with a big processor.

Damian Kowalczyk was explaining what is the Microsoft’s response toward AI and how can we use Azure to do our AI stuff. Azure really has a wide variety of tools that can be used, you can find:

trained models ready to be used like computer vision models or speech models
frameworks, f.e. Tensorflow, Keras
services: ready to use, f.e. Databricks (my favorite)
infrastructure: you don’t care about how it is running
deployment: your model can be easily packaged in a container and exposed as a web service

I am particularly interested in the last element. When developing the model sooner or later you would like to share with others the great results of your work. We could also learn about Spark on Azure so Databrick platform. Running your data processing in a distributed environment is not so old because everything started to happen around 15 yrs ago when Google published its papers about MapReduce (2004). Then things changed dramatically when Spark was introduced as it was different from its precursor: Hadoop. Spark can do its job fully in-memory so it makes its computation more powerful. And this is why Databricks offers easy access to this technology using data-science standard: Jupyter notebooks.

NLP

Maciej Szymczak provided an introduction to NLP for developers. I am particularly interested in NLP lately, so I was listening intensively and I can assure he did a great job. I like the way he explained NLP for developers so it was led in such a level that anyone who starts playing with text could be encouraged. We were able to see the live demo of a sentiment analysis using the standard Bag-of-Words technique. But it is not everything, we could learn the current trends of NLP with the following names as spaCy or gensim to be used in the modern NLP projects. Obviously, Bag of Words is not perfect, and we have alternatives to that, so there was mentioned Word embeddings with a continuous version of Bag of Words. Thus, we can avoid skipping the importance of the context of words we analyze.

Advanced analytics

Is advanced analytics something that your business needs? This is the question Fabio Bezerra answered. We live in a world of big data. And sooner or later we have to face it. There are other companies making good use of that, so we have to follow up. The analytics of a big data has several forms. We have descriptive analytics when we do want to make our data more explainable. There is also prescriptive analytics when I am analyzing my data to make it useful in making a decision later on. What are the Business Intelligence tools making that analysis possible? We could learn about some of them, including Excel, Qliksense and Knime.

Machine Learning with Udacity

lukaszkuczynski — Thu, 22 Nov 2018 00:00:00 +0000

I am studying Machine Learning with Udacity. Let me explain why, and why do I think Python nicely fits into the picture.

My motivation

I am studying Machine Learning. I want to explore the topic of Natural Language Processing. Before going deeper in NLP insights I felt I need to learn the basics of ML. I have to understand supervised techniques better. And this is course seems to be helpful. So far I learned about SVM and Naïve Bayes and what makes the difference between both. Before, it was not easy for me – I admit – but the explanation given by speakers was more than satisfying. Really, it is lots of graphics and examples with enough level of information. Very good job Udacity guys! I will try to regularly update my Git repo forked from Udacity, so stay tuned!

Python to play with data

Every time I create something in Python I have this feeling I am using the right tool for the right task! That is interesting: I do not have that feeling when trying to fight with Spark ML. Every line of code I write here is just a pleasure. Didn’t you experience that? If you did not, try to use Python for data exploration. After several years of experience in Java, I cannot imagine myself doing the same in Java.

Choose a model

The mindful choice of an ML algorithm is important crucial. You cannot just take the first and try to manipulate it to the point it will be OK. You have to mind the use case from the very beginning. And when you are ready, it is time to apply the algorithm. Every choice has to be justified.

Tune params

When learning about SVM there was a nice hint to separate a small amount of data to check your model params, and then, when they seem to be good, try to apply them on a full set. It is because the training model requires a lot of try-and-fail attempts. So decrease your waiting time to the minimum! Parameters also can be nicely visualized by diagrams, so when you applied some change, try to see it on a plot. Why not, if you have so many visualization libraries like matplotlib or seaborn.

Udacity is cool

I don’t know if Udacity is the best but compared to other platforms I can see its benefits. Let me a show you a few of them. They made a difference for me, being a programmer: Git repo with prepared data from the course! You just fork it and start doing things! Very interactive training. Many questions and lots of graphics The ideal tradeoff between the professional look and level of knowledge

Short Summary

Why do I study ML? The reason is: I want to understand NLP. And everything is Git-visible. Read the story of me studying the first 3 lessons from Udacity’s “Introduction to Machine Learning” training.

NLTK revisited

lukaszkuczynski — Wed, 31 Oct 2018 00:00:00 +0000

NLTK revisited: why

When you start working with some text-analysis project, sooner or later you will encounter the following problem: Where to find sample text, how to get resources, where should I start. When I first had a contact (Polish language post) with NLP I didn’t appreciate the power that lies behind the NLTK - the Python first-choice library for NLP. However, after several years, I see that I could use it earlier. Why? NLTK comes with an easy access to various sources of text. And I am going to show you what particularly I like and what caught my attention when studying the first 3 chapters of an official book.

But the main business is what? I would like to (finally) build some Suggestion tool that will allow providing some Virtual Assistant to help in decision making progress.

Bundled resources available

brown and its categories

NLTK comes with various corpora, so big packs of text. You can utilize them as shown in the following example. All you need to do is to download appropriate corpus and start exploring that. Let us see now.

import nltk
nltk.download('brown')
files = nltk.corpus.brown.fileids()
print(f"'Brown' corpus contain {len(files)} files")


[nltk_data] Downloading package brown to /root/nltk_data...
[nltk_data] Unzipping corpora/brown.zip.
'Brown' corpus contain 500 files

In this corpus you will find different text categorized into categories. So it nicely fits into classification area of machine learning. Following there are categories of these texts together with some samples.

print("'brown' contains following categories %s" % nltk.corpus.brown.categories())
brown_adventure = nltk.corpus.brown.sents(categories='adventure')[0:5]
brown_government = nltk.corpus.brown.sents(categories='government')[0:5]
print("Following we have some sentences from 'adventure' category:")
for sent in brown_adventure:
  print(" > "+ " ".join(sent))
print("And here we have some sentences from 'government' category:")
for sent in brown_government:
  print(" > "+ " ".join(sent)) 


'brown' contains following categories ['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']
Following we have some sentences from 'adventure' category:
 > Dan Morgan told himself he would forget Ann Turner .
 > He was well rid of her .
 > He certainly didn't want a wife who was fickle as Ann .
 > If he had married her , he'd have been asking for trouble .
 > But all of this was rationalization .
And here we have some sentences from 'government' category:
 > The Office of Business Economics ( OBE ) of the U.S. Department of Commerce provides basic measures of the national economy and current analysis of short-run changes in the economic situation and business outlook .
 > It develops and analyzes the national income , balance of international payments , and many other business indicators .
 > Such measures are essential to its job of presenting business and Government with the facts required to meet the objective of expanding business and improving the operation of the economy .
 > Contact
 > For further information contact Director , Office of Business Economics , U.S. Department of Commerce , Washington 25 , D.C. .

There is a very important component of NLP in the brown corpus, namely Part Of Speech tagging (POS). How is it organized? Let us see the example from one of the sentences printed just minutes ago.

adv_words = nltk.corpus.brown.words(categories='adventure')
print(adv_words[:10])
adv_words = nltk.corpus.brown.tagged_words(categories='adventure')
print(adv_words[:10])


['Dan', 'Morgan', 'told', 'himself', 'he', 'would', 'forget', 'Ann', 'Turner', '.']
[('Dan', 'NP'), ('Morgan', 'NP'), ('told', 'VBD'), ('himself', 'PPL'), ('he', 'PPS'), ('would', 'MD'), ('forget', 'VB'), ('Ann', 'NP'), ('Turner', 'NP'), ('.', '.')]

Yes! It is tagged and ready to be analyzed. What does these symbols mean? They are symbols of parts of speech, nicely described here. F.e. NP stands fo Proper Noun and VBD is Verb, past tense.

sentiments

One of the areas where NLP is used is a Sentiment Analysis, playing an important role in digital marketing. Imagine how nice it is to process vast amount of opinions and instantly recognize whether the product is approved or rejected by a community. Seeing trends is also possible with that. So what is the one of the NLKT bundled tools to deal with sentiments? This is opinion_lexicon

from nltk.corpus import opinion_lexicon
nltk.download('opinion_lexicon')
negatives = opinion_lexicon.negative()[:10]
positives = opinion_lexicon.positive()[:10]
print("If you find some negative words, here you are: %s" % negatives)
print("But let us try to see the positive side of life! Described with these words: %s" % positives)


[nltk_data] Downloading package opinion_lexicon to /root/nltk_data...
[nltk_data] Unzipping corpora/opinion_lexicon.zip.
If you find some negative words, here you are: ['2-faced', '2-faces', 'abnormal', 'abolish', 'abominable', 'abominably', 'abominate', 'abomination', 'abort', 'aborted']
But let us try to see the positive side of life! Described with these words: ['a+', 'abound', 'abounds', 'abundance', 'abundant', 'accessable', 'accessible', 'acclaim', 'acclaimed', 'acclamation']

and much more…

There is a lot of other corpora available there. This is not the task of this post/notebook to repeat sth what one can read him/herself in the official papers as here. With NLTK after downloading some material, you have access to such materials as:

multilingual corpora (like _Universal Declaration of Human Rights_with 300+ languages)
lexical resources (WordNet)
pronouncing dictionaries (CMU Pronouncing Dictionary)

Lots of things to browse. But it is worth just to take a look on some of them, to have this feeling “I saw it somewhere..” when facing some NLP task.

Fetch anything

If attached resources will not be enough for you, just start using different resources.

requests

Python has libraries for anything so this is possible to use available NET resources in your app just having their URL available.

import requests
url = 'https://databricks.com/blog/2018/09/26/whats-new-for-apache-spark-on-kubernetes-in-the-upcoming-apache-spark-2-4-release.html'
resp = requests.get(url)
blog_text = resp.text
blog_text[:200]


'\r\n<!DOCTYPE html>\r\n<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# video: http://ogp.me/ns/video# ya: http://webmaster.yandex.ru/vocabularies/">\r\n<head>\r\n <meta charset="UTF'

RSS

Ready to consume RSS? With Python, nothing is easier. You can create an instance of nltk.Text having RSS feed as the input. This is a snippet how it could be done

!pip install feedparser
import feedparser
url = 'http://jvm-bloggers.com/pl/rss.xml'
d = feedparser.parse(url)
title = d['feed']['title']
entries = d['entries']
print("Look ma! I've just parsed RSS from a very nice Polish blogging platform. It has a title %s" % title)
print("And there we go with 5 exemplary entries:")
for entry in entries[:5]:
  print(' > ' + entry.title)


Collecting feedparser
Successfully built feedparser
Installing collected packages: feedparser
Successfully installed feedparser-5.2.1
Look ma! I've just parsed RSS from a very nice Polish blogging platform. It has a title JVMBloggers
And there we go with 5 exemplary entries:
 > Odpowiedź: 42
 > Thanks for explaining the behaviour of dynamic (partition overwrite) mode.
 > Non-blocking and async Micronaut - quick start (part 3)
 > Strefa VIP
 > Mikroserwisy – czy to dla mnie?

cleaning

When your HTML doc is fetched you probably have a doc that is full of HTML mess and there is no added value of having <body> in your text. So there is some clean-up work that can be done and there are tools that can make it happen, but they are not part of nltk package. So let us have BeautifulSoap as an example.

from bs4 import BeautifulSoup
soup = BeautifulSoup(blog_text)
content = soup.find("div", {'class':"blog-content"})
text_without_markup = content.get_text()[:100]
text_without_markup


'\n\n\nWhat’s New for Apache Spark on Kubernetes in the Upcoming Apache Spark 2.4 Release\n\n\nSeptember 26'

Normalization - languages are not easy

Your language is not easy. If you are Polish like me, it is soooo true. But even English and other European languages add complexity to NLP. Why? Words have different forms and we have to conform grammar rules to be respected when analyzing text by the machine. Taking English going word as an example, you mean go verb, but there is also its lemma -ing that has to be recognized and skipped for the moment of analysis. What are 3 processes that have in-built support in NLTK? Read the following.

Tokenization

Text consists of sentences and these contain words. Often times, we would like to have words presented as vectors as we will apply some algebra to that. The simplest approach of tokenization can be implemented as follows, however, there are some limitations. You can use a variety of tokenizers available in nltk.tokenize package.

# write tokenizer yourself?
import re
text = "Two smart coders are coding very quickly. Why? The end of the sprint is coming! The code has to be finished!"
tokens_manual = re.split(r"[\s+]", text)
print("Tokens taken manually %s " % tokens_manual)

# or maybe choose the one from the abundance in `nltk`
import nltk
from nltk.data import load
from nltk.tokenize.casual import (TweetTokenizer, casual_tokenize)
from nltk.tokenize.mwe import MWETokenizer
from nltk.tokenize.punkt import PunktSentenceTokenizer
from nltk.tokenize.regexp import (RegexpTokenizer, WhitespaceTokenizer,
                                    BlanklineTokenizer, WordPunctTokenizer,
                                    wordpunct_tokenize, regexp_tokenize,
                                    blankline_tokenize)
from nltk.tokenize.repp import ReppTokenizer
from nltk.tokenize.sexpr import SExprTokenizer, sexpr_tokenize
from nltk.tokenize.simple import (SpaceTokenizer, TabTokenizer, LineTokenizer,
                                    line_tokenize)
from nltk.tokenize.texttiling import TextTilingTokenizer
from nltk.tokenize.toktok import ToktokTokenizer
from nltk.tokenize.treebank import TreebankWordTokenizer
from nltk.tokenize.util import string_span_tokenize, regexp_span_tokenize
from nltk.tokenize.stanford_segmenter import StanfordSegmenter
from nltk.tokenize import word_tokenize

nltk.download('punkt')
tokens = word_tokenize(text)
print(tokens)


Tokens taken manually ['Two', 'smart', 'coders', 'are', 'coding', 'very', 'quickly.', 'Why?', 'The', 'end', 'of', 'the', 'sprint', 'is', 'coming!', 'The', 'code', 'has', 'to', 'be', 'finished!'] 
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data] Unzipping tokenizers/punkt.zip.
['Two', 'smart', 'coders', 'are', 'coding', 'very', 'quickly', '.', 'Why', '?', 'The', 'end', 'of', 'the', 'sprint', 'is', 'coming', '!', 'The', 'code', 'has', 'to', 'be', 'finished', '!']

Stemming

So one of the tasks that can be done is stemming, so getting rid of the words ending. Let us see what does the popular stemmer does to the text we already tokenized before.

porter = nltk.PorterStemmer()
tokens_stemmed = [porter.stem(token) for token in tokens]
print(tokens_stemmed)


['two', 'smart', 'coder', 'are', 'code', 'veri', 'quickli', '.', 'whi', '?', 'the', 'end', 'of', 'the', 'sprint', 'is', 'come', '!', 'the', 'code', 'ha', 'to', 'be', 'finish', '!']

Lemmatization

If stemming is not enough, there has to be lemmatization done, so your words can be classified against the real dictionary. Following there is an example of running this for our text sample.

nltk.download('wordnet')
wnl = nltk.WordNetLemmatizer()
lemmas = [wnl.lemmatize(token) for token in tokens]
print(lemmas)


[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data] Unzipping corpora/wordnet.zip.
['Two', 'smart', 'coder', 'are', 'coding', 'very', 'quickly', '.', 'Why', '?', 'The', 'end', 'of', 'the', 'sprint', 'is', 'coming', '!', 'The', 'code', 'ha', 'to', 'be', 'finished', '!']

Summary

So where did we go? I have just analyzed NLKT book available online, chapters no 2 and 3. I gave a try for few tools from the big number available in this Natural Language Toolkit. Now there is a time to explore other chapters over there. Stay tuned. I have to build my Suggestion tool.

All of these was created as a Jupyter notebook.

Innovation day at Volvo

lukaszkuczynski — Wed, 17 Oct 2018 00:00:00 +0000

As a Volvo employee, I had a chance to attend several sessions during the Innovation Days summit held a few days ago in Wroclaw, Volvo. There were several stories told, and it is nice that these stories are not just a mere theory; they are practised in some rooms of Volvo buildings.

Automation

Automation is visible in many business areas. One of these is a process automation. There is always some mundane work that can be automated, if you can use the browser to perform an action for sure you can use an automat to click on a webpage. One of the automation tools is UIPath and what is nice there is also BigData tool linked to it that allows for better control over the metrics.

convince them

Another presentation was about how to encourage your colleagues that they need an innovation. Of course, always there is a question about money and time. When may the busy workers have time to work on innovations? Maybe in their free time, they can find a slot. There is always some period-of-doing-nearly-nothing, so why not to utilize it?

UX

User experience is another hot topic. Why do you like software tools that you often use? Mostly because they are easy to use, you don’t need to read a manual to start using it. So when creating a tool remember, about not falling into aforementioned traps: don’t be too trendy, don’t overdesign, and don’t allow your user to be designer. The simple example was shown when a user had to use some complicated HTML form, and when it was simplified (fewer HTML elements, more open search capabilities present), the usage of a tool increased, and more user became happy ones.

Artificial and Business Intelligence

AI is really here. As you can see the global results here in Vera Car there are other teams utilizing Artificial Intelligence. The intro was very nice. At the beginning the question was posed: how do you know that an apple is an apple? Because you are able to properly identify and measure its properties, features. This is what AI and machine learning is about, but it is done by machine and not you. Do you fear AI progress in our lives? You should, cause soon you can learn, your job can be done by a machine better, faster, and cheaper! And at the end, I learned about QlikSense. There is no value from data that is not easily viewable and discoverable. The data you have brings value to your business if it can help you to conclude what your action is and make the right decisions. QlikSense brings value to you as it makes it possible to create dashboards and analyze your datasets, having easy to use dashboards and visualizations.

Monitoring Wrocław weather with AWS

lukaszkuczynski — Tue, 18 Sep 2018 00:00:00 +0000

What if I wanted to compare AWS Lambdas with Azure Function apps? The previous weeks I did some Azure sample app. There is some article on that. But I always felt AWS compared to Azure is smarter, GUI is lighter and I love Python. So why not to copy my project to AWS?

Plan

Create a notification mechanism telling me about changes of weather in Wrocław. This will be done using Lambda architecture in AWS mechanism.

Implementation

Check the weather

Lambda function can be fired by CloudWatch Events, using which I can set up some cron expression. Thus I will check weather exposed by openweather API and put it on a queue. Queues in Amazon are part of so-called SQS. Handling SQS in Amazon is pretty easy using Python boto clients. Following there is my function to fetch data. In my case every 10 minutes I will call openweather API to query the current weather and I will put its description in a queue.

import os
from datetime import datetime
from urllib.request import Request, urlopen
import json
import boto3

SITE = os.environ['site'] # URL of the site to check, stored in the site environment variable

def validate(res):
    return EXPECTED in res

def lambda_handler(event, context):
    print('Checking {} at {}...'.format(SITE, event['time']))
    try:
        req = Request(SITE, headers={'User-Agent': 'AWS Lambda'})
        response = urlopen(req).read()
        print(response)
        data = json.loads(response.decode("utf-8"))
        weather = data['weather'][0]['main']
        dt = data['dt']
        dt_iso = datetime.isoformat(datetime.fromtimestamp(dt))
        print(f"weather is {weather} with UTC time {dt_iso}")

        queueUrl = os.environ['SQS_NEW_WEATHER']
        client = boto3.client('sqs')
        response = client.send_message(
            QueueUrl=queueUrl,
            MessageBody=weather
        )

    except:
        print('Check failed!')
        raise
    else:
        print('Check passed!')
        return weather
    finally:
        print('Check complete at {}'.format(str(datetime.now())))

Write to persistence

When received the value I would like to compare it against the previous one to decide whether there was a change. I have to put state somewhere. I will use default database for AWS: DynamoDB.

The function is triggered for a new message on SQS, and it is easily configurable with AWS lambda GUI. The message received on SQS is available as event parameter of lambda.

import boto3
from boto3.dynamodb.conditions import Key
from datetime import datetime

def lambda_handler(event, context):
    print("received!")
    print(event)
    new_value = event['Records'][0]['body']
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('weather')
    table.put_item(
       Item={
            'created_at': datetime.utcnow().strftime('%Y%m%d_%H%M%S'),
            'type': 'wroclawWeather',
            'text': new_value
        }
    )
    return "time created with value "+new_value

So the above function will create a new document in DynamoDB instance. As you can see boto3 library helps to handle DynamoDB connections. All you need is to call resource and table and you are ready to put_item into the table. This will enable the comparison in the near future. Behold!

Compare time!

Having the new document inserted into the table, I can compare it against the previous one of the same type. I will use again the appropriate trigger, this time it will be “New row” trigger for a DynamoDB. The last 2 items will be fetched using this complicated query with KeyConditionExpression because I want to get only two previous values of the exact type (like a WHERE clause in SQL).

import os
import json
import boto3
from boto3.dynamodb.conditions import Key

def lambda_handler(event, context):
    # print("Received event: " + json.dumps(event, indent=2))
    records = event['Records']
    if len(records) != 1:
        print(f"len records != 1, len = {len(records)}")
        print(records)
        return "len(REC) <> 0"
    record = records[0]
    current_type = record['dynamodb']['NewImage']['type']['S']
    print(f"current_type = {current_type}")
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('weather')
    response = table.query(
        KeyConditionExpression=Key('type').eq(current_type),
        ScanIndexForward=False, 
        Limit=2
    )
    new_value = record['dynamodb']['NewImage']['text']['S']
    old_value = response['Items'][1]['text']

    if old_value != new_value:
        change_text = "changed IN DYNAMO trigger! "+old_value+"->"+new_value
        print(change_text)
        queueUrl = os.environ['SQS_CHANGED_WEATHER']
        client = boto3.client('sqs')
        response = client.send_message(
            QueueUrl=queueUrl,
            MessageBody=change_text
        )
    print(f"Current FROM DYNAMO {new_value}, previous {old_value}")

If the value was changed the message will wander to another SQS topic, and this one I will use to notify the user about the changes.

Notification with SNS

When the weather is changed, it will go to Simple Notification Service , that can be used to… apply notification rules. In my case, I will simply use an email to be sent with the text of a change.

And once again, thanks to boto3 library enables I create a SNS client so I can send SNS message super-easily from within the lambda.

import os
import json
import boto3

def lambda_handler(event, context):
    print("called weatherChanged")
    print(event)
    change_text_value = event['Records'][0]['body']
    print("I received a change: "+change_text_value)
    arn = os.environ['SNS_TOPIC']
    message = change_text_value
    client = boto3.client('sns')
    response = client.publish(
        TargetArn=arn,
        Message=message
    )
    return change_text_value

After that one I just needed to configure my SNS connection.

Configuring SNS

To receive emails I had to create a topic. This topic has SNS address that I used in lambda function code. Then there is a need to create a subscription. You can use subscription of “Email” type, so then AWS will push the messages to your email inbox.

AWS or Azure?

I configure the same business logic in both AWS and Azure. Time for a comparison.

Azure pros:

security made easier

I wasn’t forced to care about securty too much, all the functions were just connected to each other out-of-the-box.

functions bundled together

For a reason Function App in Azure is a root for all the functions inside, I could nicely put all the related functions in one resource.

AWS pros:

python

The language is waaay better for me. Have no experience in C# and Python is just beautiful for playing with functions.

no hassle with param bindings

Binding in Azure is not so straighforward for me as AWS triggers and handling params for AWS Lambdas. Just better.

boto3

I had to make some strange returning and AsyncCollectors in Azure while AWS gives boto3 that makes everything just simple.

My choice? AWS.

My girl in IT

lukaszkuczynski — Fri, 14 Sep 2018 06:05:24 +0000

Recently I was asked by my wife, is something she could learn/do to help me in my IT job. What options do you see?
Say: You want to work 4 days and your girl 1 day. And you want to co-operate.
What IT stuff can she do? What should I teach her?
I am cloud/python/java/devops person.

PyConPL 2018 report

lukaszkuczynski — Sun, 26 Aug 2018 00:00:00 +0000

It was time for a Python conference. Attending many JVM ones I never had chance to give Python its place. Now, whenever the problem stands before me – Python is default language of choice, I love the freedom and possibilities it gives, especially in data science world. Let’s be honest, it has become my “love” since 2 years, so attending PyCon was a must. What kept my focus? I will share it with you:

Speed is what matters

One of talks I attended was about pypy. Mr Cuni showed it as the default choice when you want to write fast Python code with that library. If you want to do so, execute your code with pypy and not python. The use case shown during the presentation was video processing, that is true, the version of program run by pypy was much faster. But the next day, when I discussed the topic with video processing people. They said comparison chosen for the presentation was not quite fair. Python is just a scripting language not a real-time video processing tool.

Cloud is everywhere

We were shown how cloud services are important. Two main players were mentioned. Who won? In my opinion stage is of AWS. It is hard to imagine how now we could handle processing great amounts of data. First of all, Suryansh Tibarewala presented Alexa, which is a service on AWS. Unfortunately we didn’t see any live demo, but at least the theory was introduced. Beware, speech recognition brings lots of difficulties, one of these is handling consecutive questions within the context. The next day, Michał Smereczyński shed some light on Microsoft as a cloud provider. He was convincing us, Azure is not only about Windows. Azure Batch is a nice Azure service, thanks to what you can run your scripts in parallel. If you have some batch job to be executed occasionally, there is no need to buy and maintain big servers. Furthermore you are not forced to set up your environment manually each time, as ACR allows you to store and use your Docker images. The Friday’s Ansible task given by Paweł Kopka also made it clear, having professional configuration management tool, I can play with any cloud provider using the same (or similar) scripts.

Are you healthy?

Two of the presentations were especially cool in my humble opinion. And important! They proved how Python may be a help in disease prevention. On Friday I learned about Python use in bioinformatics in talk given by Jacek Śmietański. We learned what make DNA, RNA and proteins different and how can I fit very loooong genome sequences, and analyze them with Biopython that you can fetch from here. Unfortunately the talk was not given in English, so it blocked some foreign audience from listening to that talk, although they were interested. On Saturday in MRI talk we were shown how does Python help scientist to analyze MRI pictures. The libraries used for that purpose are nipy, but you can make good use of other non-Python apps as FreeSurfer. The main use case of Mikołaj Buchwald was to learn, what parts of our brain are stimulated when we are shown particular images. What motivates me to have a positive thinking is the following sentence I extracted from that talk:

An action and visualizing it is almost the same, so keep thinking of positive attitude

Be safe!

Ms Christine Bejerasco showed the way of FSecure way toward security standards. She started as Perl developer and then switched to Python as she could continue working with regex there, and well… we all know it: Python is far better. In my opinion her Cybersecurity talk was one of the best, as it was of great quality, no extensive details were enforced and a lot of practical examples were shown. When the Internet was not so popular, evil code was to be linked to existing exe files and it could infect machines via physical drives. Now, living in times of so popular “online life” we have to be careful about what we browse and what we do open. I learned that even visiting dangerous sites, without actually downloading something intentionally, can lead to disaster. My browser can be probed for vulnerabilities and then injected with some malicious code. Viruses can even live for some time deactivated and then run (wake up) in the most critical moment (Stuxnet). How to play with viruses, and not to harm your machine? They are using sandboxes to do so.

How to teach a machine

Machine learning is not an obvious task. And this is not only a buzzword nowadays. The words like numpy scipy sklearn were mentioned heavily during days of the conference. I was happy to attend Saturday’s workshop that showed me how neural networks help to recognize picture. The challenges and solutions for computer vision were presented by Prakhar Srivastava. It was pretty new for me to learn, that in 2015 year Convolution Neural Networks already surpassed human ability to recognize simple pictures. Sounds a little creepy. I learned that before using some advanced libraries as keras you would better understand how it works under the hood. Then you would be better prepared to 1) cherish what import sklearn enables you to do and 2) use well these tools. You can also play with nice Neural Network demo online, here.