Forem: Odhiambo Paul

Bitcoin Vs. Ethereum: Technical Perspective

Odhiambo Paul — Wed, 04 Oct 2023 17:12:59 +0000

This article will examine the critical differences between the two largest cryptocurrencies in circulation and market value. We will cover some of the key concepts that enable these cryptocurrencies. Advantages such as decentralization, peer-to-peer architecture, secure and anonymous transactions have boosted cryptocurrencies' adoption.

Both Bitcoin and Ethereum use cryptography to enable transactions. Let us quickly go through the different types of cryptographic techniques used.

Cryptography

Cryptography plays a crucial role in ensuring transactions are secure and anonymous. The various methods used in cryptocurrencies are as follows:

Symmetric Encryption Cryptography: A secret key is used to encrypt and decrypt the message at the source and destination addresses respectively. Identifying the secret key is a challenging task and probabilistically infeasible to determine. However, this approach has many issues with security and scalability.
Asymmetric Encryption Cryptography: In this method, instead of using a single secret key, two keys are used: a public key and a private key.

Image Source

Through the use of key pairs, we can ensure enhanced security. The public key is shared with the public. Bitcoin addresses are an example of public keys. The private key, on the contrary, is known only at the source. The message can be decrypted only using the private key. This is the reason for the enhanced sense of security.

Hashing is used in blockchain significantly.
Digital signatures are complementary methods that allow anonymous users to display genuineness.

They possess the following features:

Verifiable
Anti-forgeable:
Bound by contracts this eliminates the possibility of denying the commitment once signed.

Bitcoin

Termed as digital gold, Bitcoin recently passed a market valuation of 1 trillion dollars. The valuation exceeds that of many tech giants and multi-national corporations. Let us understand the technical aspects of Bitcoin running transactions at such a large scale with ease.

Timestamped logs

Blockchain derives its name from the concept behind it: a chain of blocks. The blocks contain logs of transactions that are timestamped. Once a transaction is made, no stakeholder in the network can reverse it. In the current banking system, banks can mediate between two parties in case of any disputes.

Banks charge for this assurance of mediation. It is called the cost of trust borne by the customers. In Bitcoin, the cost of trust is minimized by making the blocks immutable and adding a heavy penalty to act as an adversary in the system. This property is referred to as append-only timestamped logs.

Decentralized network consensus

As discussed in an earlier article on various consensus protocols, Bitcoin uses the Proof of Work protocol. Proof of work requires that a cryptographic puzzle be solved. A distributed ledger is maintained across all the nodes verifying the transactions.

Proof of work protocol

Each block contains a highly selective hash value computed based on the previous blocks in the blockchain. The term highly selective is of importance here. The hash values are computed such that a specified number of trailing zeroes are present in it.

The data is coupled with a value called the nonce. Various values of the nonce are tested to generate the hash value thus, satisfying the specified condition. In the example below, we implement a Python code to understand the importance of hashing.

from hashlib import sha256
import random

nounce = 0
numer_of_trailing_zeros = 4
for nounce in range(100000):
    text =  'transaction' + str(random.randint(0,1000)) + str(nounce) 
    hash = sha256(b'' + text.encode()).hexdigest()
    if hash.startswith(numer_of_trailing_zeros*'0'):
        print('nounce:',nounce)
        print('hash:',hash)
        break

Running the following code generates the following output:

nounce: 66053
hash: 0000a17d817a5b5414cce89e889278fcb60c6232208b1cc8933014009fd8fa2e

For more details on the Proof of Work protocol, refer to my previous article on various consensus protocols.

Ledgers and transaction code

The ledger is the critical component upon which all transactions are allowed and verified. Bitcoins work on distributed ledger systems. However, we should define ledgers to understand them critically.

Ledgers are analogous to state transition systems. It contains the ownership status of all the bitcoins mined. For example, if one wants to transact with Bitcoins, the state transition function checks for available balance. If there is a sufficient amount, the Bitcoin is deducted from the sender and added to the receiver.

Unspent Transaction Outputs, abbreviated as UTXO, is a collection of all the coins that have been mined and yet to be spent. Each UTXO contains the value and the owner's public key. A new transaction shall contain a reference to an existing UTXO containing references to the previous transactions made. This system is put in place to solve the double-spending problem.

Ethereum

While Bitcoin was introduced in 2009, Ethereum was introduced in the year 2013. Quoting from the whitepaper, Ethereum's goal is given as follows: Ethereum aimed to provide a blockchain with built-in fully fledged Turing-complete programming language that developers can use to create contracts.

These contracts are used to encode arbitrary rules for transactions in the form of state transition functions thus, allowing users to create several decentralized applications. In other words, it is regarded as a "programmable blockchain."

Scripting

Ethereum uses a Turing-complete scripting language to encode terms of usage. Smart contracts are contracts that bind the buyers and sellers. Ethereum aims at creating custom contracts that bind these two parties. Solidity is a Turing complete programming language used to define and code smart contracts.

Migration to Proof of Stake

Proof of Work protocol has challenges related to energy consumption and time taken to process each block. Ethereum has ambitious plans to migrate to Proof of Stake protocol, reducing energy usage dramatically, and offering better security features.

Technical differences

Properties	Bitcoin	Ethereum
Founder	Satoshi Nakamoto	Vitalik Buterin
Genesis	2009	2013
Coding Contracts	Non-Turing (Script)	Turing Complete (Solidity, Serpent, LLL, or Mutant )
Ledger	UTXO Transactions	State - Account-Based
Transactions	Merkle Trees based Transactions	Transactions, state, storage, receipts(w/nonces)
Block time	10 minutes	14 seconds
Consensus	Proof of Work	Proof of Stake
Hash function	SHA 256	Ethash

Conclusion

In this article, we considered the differences between the two largest cryptocurrencies. In the future, a cryptocurrency with better technical features will dominate and facilitate transactions.

The dominance of Bitcoin and Ethereum is not guaranteed forever. However, until the next disrupting cryptocurrency, these cryptocurrencies will maintain their dominance.

Happy learning.

How to create Django REST APIs

Odhiambo Paul — Thu, 20 May 2021 12:12:12 +0000

In this tutorial, we will create a fully working to-do CRUD Django API using Django and Django rest framework. Restful API endpoints make it possible to perform CRUD functionality in the backend from within the mobile app or website.

Prerequisites

Knowledge of Django web framework.
Basics of Python programming language.
Postman installed.
Code editor of your choice installed.
Python and Virtualenv installed.

To verify if Python is installed and configured correctly on your system, Open the terminal and type in the command python3 --version on Linux/Mac or python --version if you are on Windows.

$ python3 --version
Python 3.8.5

To verify virtualenv installation execute the command virtualenv --version on the terminal.

$ virtualenv --version
virtualenv 20.0.35 from /home/user/.local/lib/python3.8/site-packages/virtualenv/__init__.py

Creating the Todo project

We will start by creating our project's work directory and a virtual environment for our project. The virtual environment makes it possible to run our project and its dependencies in an isolated environment.

Run mkdir ~/todo to create our working directory.

$ mkdir ~/todo
$ cd todo

To create a virtual environment for our project run:
```
  $ virtualenv venv
```

venv in in the command virtualenv venv is the name of our virtual environment

To activate the virtual environment for our projected run:

   $ source venv/bin/activate

To install Django and Django rest framework in our virtual environment run:

   $ pip3 install django

Running the command will result in something similar to the code block below

   $ pip3 install django
   Collecting django
   Using cached Django-3.1.3-py3-none-any.whl (7.8 MB)
   Collecting asgiref<4,>=3.2.10
    Downloading asgiref-3.3.1-py3-none-any.whl (19 kB)
   Collecting pytz
     Using cached pytz-2020.4-py2.py3-none-any.whl (509 kB)
   Collecting sqlparse>=0.2.2
    Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
   Installing collected packages: asgiref, pytz, sqlparse, django
   Successfully installed asgiref-3.3.1 django-3.1.3 pytz-2020.4 sqlparse-0.4.1

To Django rest framework in our virtual environment run:
```
$ pip3 install djangorestframework
```

Running the command will result in something similar to the code block below:

   $ pip3 install djangorestframework
   Collecting djangorestframework
     Downloading djangorestframework-3.12.2-py3-none-any.whl (957 kB)
        |████████████████████████████████| 957 kB 595 kB/s
   Requirement already satisfied: django>=2.2 in ./venv/lib/python3.8/site-packages (from djangorestframework) (3.1.3)
   Requirement already satisfied: asgiref<4,>=3.2.10 in ./venv/lib/python3.8/site-packages (from django>=2.2->djangorestframework) (3.3.1)
   Requirement already satisfied: sqlparse>=0.2.2 in ./venv/lib/python3.8/site-packages (from django>=2.2->djangorestframework) (0.4.1)
   Requirement already satisfied: pytz in ./venv/lib/python3.8/site-packages (from django>=2.2->djangorestframework) (2020.4)
   Installing collected packages: djangorestframework
   Successfully installed djangorestframework-3.12.2

Let's create our django_todo project now by running the command django-admin startproject django_todo.

$ django-admin startproject django_todo

After successfully creating the project we change our working directory to our project folder and run the Django development server.

$ cd django_todo
$ ./manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run `python manage.py migrate` to apply them.
November 17, 2020 - 10:14:01
Django version 3.1.3, using settings 'django_todo.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

When we visit http://127.0.0.1:8000/ we should see the Django welcome page. Django makes it easy for us to organize our code into reusable apps. We are going to create an app for the todo endpoints.
To create an app, run:

$ python3 manage.py startapp todo

The command above creates a directory named todo and generates boilerplate code for a todo app. Now we can plug our todo app into our django_todo project.
Open settings.py in the project directory django_todo and add our todo app to the INSTALLED_APPS list. We will also add the rest_framework app to make it available for use in our project.

# ./django_todo/settings.py
...
INSTALLED_APPS = [
    ...
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'todo',
    'rest_framework',
]

Creating the Todo model

In Django, models are python classes that represent a table in the database. The Todo Model represents a Todo table in the database that gets created by Django whenever we run the command python manage.py migrate.
In the models.py in the todo app, we will create our Todo model will the below code:

from django.db import models
# Create your models here.
class Todo(models.Model):
    title = models.CharField(max_length = 100)
    body = models.CharField(max_length = 100)
    is_completed = models.BooleanField(default=False)
    date_created = models.DateField(auto_created=True)
    last_modified = models.DateField(auto_now=True)
    def __str___(self):
        return self.title

To create an SQLite database and Todo table in the database, we will first run the command ./manage.py makemigrations to create SQL queries that will be used to create a Todo table in the database. To create the Todo table in the database will run the command ./manage.py migrate.
To make migrations run:

./manage.py makemigrations

To apply the migrations run:

./manage.py migrate

To get a better understanding of the Django migrations, read the documentation here.

Serializing the Todo model

To convert the Python objects obtained from the database to the JSON format needed by our endpoints and back to the Python objects that can be mapped to our database tables, we will subclass the Django rest framework Serializer.ModelSerializer class for easier conversion.

In the todo app directory, create a file serializers.py, where we will write our TodoSerializer code.

from rest_framework import serializers
from todo.models import Todo
class TodoSerializer(serializers.ModelSerializer):
    class Meta:
        model = Todo
        fields = "__all__"

We only need to pass in the model that we need to serialize and the fields to serialize as meta fields. In our case, we are serializing all of the fields from the Todo model that's why we are passing __all__ to our fields variable.

Creating the TodoAPIView

In the views.py file in our todo app directory, we are going to write the logic for the CRUD functionality for our app. Django Rest framework comes with inbuilt classes that make building the CRUD functionality very easy.

We start by importing the Django rest framework classes that we'll use to create our CRUD API. We just need to create a class for each of our crud endpoints and add in the queryset and the serializer_class for our Todo model.

from django.shortcuts import render
from rest_framework.generics import ListAPIView
from rest_framework.generics import CreateAPIView
from rest_framework.generics import DestroyAPIView
from rest_framework.generics import UpdateAPIView
from todo.serializers import TodoSerializer
from todo.models import Todo
# Create your views here.
class ListTodoAPIView(ListAPIView):
    """This endpoint list all of the available todos from the database"""
    queryset = Todo.objects.all()
    serializer_class = TodoSerializer
class CreateTodoAPIView(CreateAPIView):
    """This endpoint allows for creation of a todo"""
    queryset = Todo.objects.all()
    serializer_class = TodoSerializer
class UpdateTodoAPIView(UpdateAPIView):
    """This endpoint allows for updating a specific todo by passing in the id of the todo to update"""
    queryset = Todo.objects.all()
    serializer_class = TodoSerializer
class DeleteTodoAPIView(DestroyAPIView):
    """This endpoint allows for deletion of a specific Todo from the database"""
    queryset = Todo.objects.all()
    serializer_class = TodoSerializer

Creating URL paths for our Todo endpoints

The URLs allows us to interact with the various Todo crud views. In the todo app directory create a file urls.py where we will write URLs for various endpoints.

from django.urls import path
from todo import views
urlpatterns = [
    path("",views.ListTodoAPIView.as_view(),name="todo_list"),
    path("create/", views.CreateTodoAPIView.as_view(),name="todo_create"),
    path("update/<int:pk>/",views.UpdateTodoAPIView.as_view(),name="update_todo"),
    path("delete/<int:pk>/",views.DeleteTodoAPIView.as_view(),name="delete_todo")
]

In the urls.py in the django_todo project directory, let's added the base URL for our app so that the django_todo project can be aware of the todo app URLs.

from django.contrib import admin
from django.urls import path
from django.urls import include
urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/v1/todo/',include("todo.urls"))
]

Testing the endpoints

Before testing our API endpoints let's make sure our development server is up and running. To start the development server run the command ./manage.py runserver in the root folder of our project where manage.py file exists.

$ ./manage.py runserver

To create a new Todo we make a POST request to
http://localhost:8000/api/v1/todo/create/ with the new Todo object.

{
    "date_created": "2020-11-19",
    "title": "Go to School",
    "body": "Go Buy Goods from the market",
    "is_completed": true
}

Sample postman POST request:

Making a GET request to http://localhost:8000/api/v1/todo in postman returns a list of todos.

[
    {
        "id": 1,
        "date_created": "2020-11-19",
        "title": "Go to market",
        "body": "Go Buy Goods from the market",
        "is_completed": true,
        "last_modified": "2020-11-17"
    }
]

Sample postman GET request:

To Update a Todo we make a PUT request to http://localhost:8000/api/v1/todo/update/1/ with the Todo object fields to update and pass in Todo ID as a URL parameter.

{
    "date_created": "2020-11-19",
    "title": "Go to Town",
    "body": "Go Buy Goods from the market",
    "is_completed": false
}

Sample postman PUT request:

To Delete a Todo we make a DELETE request to http://localhost:8000/api/v1/todo/delete/1/ passing the ID of the Todo to delete as URL parameter.

Sample postman DELETE request:

Documenting Todo endpoints

It's a good practice to provide documentation for the various endpoints that we create, this makes it easier for other people to use our API endpoints.

We will use coreapi to document our endpoints. To install coreapi and plug it into our app we run the command pip3 install coreapi in the terminal.

On the setting.py file in django_todo project directory add coreapi to the installed apps list and the add the below rest framework configuration to enable documentation autogeneration.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'todo',
    'rest_framework',
    'coreapi',
]

REST_FRAMEWORK = {
    'DEFAULT_SCHEMA_CLASS': 'rest_framework.schemas.coreapi.AutoSchema'
}

Finally we create a URL to our documentation page by adding a URL configuration in the urls.py file in the django_todo project directory.

from django.contrib import admin
from django.urls import path
from django.urls import include
from rest_framework.documentation import include_docs_urls
urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/v1/todo/',include("todo.urls")),
    path('docs/', include_docs_urls(title='Todo Api')),
]

By visiting http://127.0.0.1:8000/docs/ in the browser we'll be able to see the full documentation of our Todo CRUD API endpoints.

Conclusion

We now understand how to create and document our restful endpoint APIS in Django. Go ahead and clone the repos django_todo to view the full source code of the project and add new fields to our Todo model.

In our next article, we will secure our endpoints and add social authentication to our app.

Getting started with Spring Boot and Docker

Odhiambo Paul — Thu, 20 May 2021 11:48:32 +0000

Docker is a container engine that makes it possible to bundle, ship, and distribute applications across distributed servers.

Prerequisites

Have Java Development Kit 11 or newer installed on your computer.
Make sure Docker is installed on your computer.

To get started with Docker, visit Getting started with Docker.

Project setup

We will bundle and containerize the application that we created here. The application allowed us to perform CRUD functionalities on the Todo items via a REST API.

To check if JDK is installed and configured correctly on your computer, execute the command below.

$ java --version
java 15.0.2 2021-01-19

Execute the command below to create a working directory for the project.

$ mkdir todo
$ cd todo

In the directory created above, execute the command below to clone the project to your computer.

$ git clone https://github.com/paulodhiambo/springbootcrud.git

Execute the command below to move into the project folder.

$ cd springbootcrud

Run the application to verify if the project is correctly set up and the application runs without errors.

$ gradle build

The project structure is shown below.

└── springbootcrud         # < project root package
    ├── src
    │   ├── main
    │   ├── test
    ├── build.gradle
    ├── gradlew
    ├── Dockerfile # Dockerfile where we write all the container build instructions
    └── gradlew.bat    
    └── settings.gradle

Creating a Dockerfile

A Dockerfile is a plain text file that contains instructions from which a Docker image is built. A Dockerfile contains the commands below.

FROM: directive sets the image on which the container will be based on.
RUN: executes commands in the container.
COPY: creates a copy of files from the file system in the container.
CMD: sets the executable commands within the container.

Create a Dockerfile in the project's root folder.

Add the code snippets below to the Dockerfile created above.

FROM openjdk:8-jdk-alpine
RUN addgroup -S springdocker && adduser -S springdocker -G springdocker
USER springdocker:springdocker
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

FROM openjdk:8-jdk-alpine sets the base image from which the Docker container will be created.
RUN addgroup -S springdocker && adduser -S springdocker -G springdocker add a user profile to the container.
USER springdocker:springdocker sets the user in the container making it possible to execute commands in the container as an administrator.
ARG JAR_FILE=target/*.jar sets the path to jar generated by gradle build command to ARG_FILE variable.
COPY ${JAR_FILE} app.jar copies the jar file created after running gradle build command to the app.jar file inside the docker container.
ENTRYPOINT ["java","-jar","/app.jar"] indicates that the container has an executable file.

Building the Docker image

To build a Docker image from the Dockerfile we have created above, execute the command below.

$ docker build --build-arg JAR_FILE=build/libs/\*.jar -t spring-boot-docker:latest .

--build-arg JAR_FILE=build/libs/\*.jar sets the path to the jar file generated by running gradle build command.
-t sets the tag for the image.
. indicates that the Dockerfile is in the current directory.

Execute the command below to list all the Docker images available on your computer.

$ docker image ls
REPOSITORY    TAG            IMAGE ID       CREATED         SIZE
spring-boot-docker               latest              b7a1056c3eb2   31 minutes ago   143MB
springio/gs-spring-boot-docker   latest              b7a1056c3eb2   31 minutes ago   143MB
<none>                           <none>              4f8c8633df3e   2 days ago       105MB
latest                           latest              6c802830c996   4 weeks ago      414MB

From the list above, we see the spring-boot-docker image that we have created.

Creating and running the Docker Container

Execute the command below to create a Docker container from the Docker image we have created above.

$ docker run --name spring-boot-docker -d -p 8080:8080 spring-boot-docker:latest

--name flag sets the Docker container name.
-d flag makes the image run in the background.
-p 8080:8080 maps port 8080 in the Docker container to port 8080 in localhost.
spring-boot-docker:latest specifies the image from which the Docker container is built.

Execute the command below to list all the running Docker containers.

$ docker container ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED       STATUS       PORTS                        NAMES
792fce534e3a   spring-boot-docker   "java -jar /app.jar"   31 minutes ago   Up 31 minutes   0.0.0.0:8080->8080/tcp   spring-boot-docker

Visit localhost on port 8080 to confirm if the spring-boot-crud application is running in the container.

Pushing the Docker image to Docker Hub

Docker Hub is an online repository of Docker container images. Docker hub stores Docker containers and also makes it easier to share Docker images with other developers.

Create a Docker Hub account here if you don't have one.

After creating an account, log in to Docker Hub, create a repository with the name spring-boot-crud and the description Spring Boot Docker Image.

Execute the command below to push the Docker image to the repository that we have created in the Docker Hub.

$ docker login
$ docker tag spring-boot-docker:latest <username>/spring-boot-docker:latest
$ docker push <username>/spring-boot-docker:latest

The docker login command authenticates you into your Docker Hub account on the command line, making it possible to push Docker images to Docker Hub.
docker tag spring-boot-docker: latest <username>/spring-boot-docker: latest this command modifies the tag of our local Docker image to include the Docker Hub username.
docker push <username>/spring-boot-docker: latest pushes the image to Docker Hub repository we created earlier.

Note: Replace <username> with your actual Docker Hub username.

The source code to the application is in this GitHub repository.

Conclusion

Docker is a great tool for building and packaging applications with the required dependencies to run them both locally and in production. Docker playground is a browser-based environment that you can use to test various Docker commands without installing Docker locally on your computer.

Happy coding!

How to upload files to Amazon S3 in Spring Boot

Odhiambo Paul — Thu, 20 May 2021 11:44:56 +0000

Amazon simple storage (Amazon S3) is a service offered by Amazon web services that offers scalable, secure, and well performing object storage. This article will go over how to upload files into Amazon S3 using Spring Boot.

Prerequisites

Some knowledge in Java and Spring Boot.
Java development kit installed on your computer.
Your favourite IDE. I use Intellij community edition which is free.

Amazon web services account

Before we start creating our application, head over to Amazon console, and create an account. You will be given 12 months of free access to various Amazon web services that you can use to test various Amazon services.

After signing up, head over to Amazon console and search for Amazon S3 in the search box provided in the console.

Amazon S3 bucket

After selecting Amazon S3 from the step above, create a new S3 bucket that we will use to store the files we will be uploading from our application.

Name the bucket as spring-amazon-storage and leave all other settings as default then create the bucket.

Access and secret keys

Create a new access key from My Security Credentials navigation menu as shown in the image below. Copy the access and the secret key generated as we will be using them to access the bucket from the application we will be creating.

Creating the application

We will be using spring initializr to create our application. Head over to spring initializr and create a new Spring Boot application adding h2, spring boot dev tools, spring data jpa and spring web as dependencies then generate the project.

Unzip the downloaded project and open it in your favorite IDE.

Adding Amazon SDK dependency

Amazon SDK makes it possible to interact with various Amazon services from our applications.
In the pom.xml file add the Amazon SDK dependency as shown below.

    <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.931</version>
    </dependency>

Project structure

config/
|--- AmazonConfig.java
|--- BucketName.java
controllers/
|--- TodoController.java
domain/
|--- Todo.java
repositories/
|--- TodoRepository.java
service/
|--- FileStore.java
|--- TodoService.java
|--- TodoServiceImpl.java
SpringAmazonApplication.java

Configuration package

In the configuration package, we have two Java files, one that authenticates with Amazon S3 and the other which contains the bucket name.

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class AmazonConfig {
    @Bean
    public AmazonS3 s3() {
        AWSCredentials awsCredentials =
                new BasicAWSCredentials("accessKey", "secretKey");
        return AmazonS3ClientBuilder
                .standard()
                .withRegion("ap-south-1")
                .withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
                .build();
    }
}

The AmazonConfig class above is annotated with @Configuration annotation to make it available to the Spring context as a configuration class. With the Amazon credentials we got from the Amazon console earlier we will authenticate to S3 using the AmazonS3ClientBuilder available in the Amazon-SDK that we added to our pom.xml ealier.

import lombok.AllArgsConstructor;
import lombok.Getter;
@AllArgsConstructor
@Getter
public enum BucketName {
    TODO_IMAGE("spring-amazon-storage");
    private final String bucketName;
}

In the BucketName enum above we pass in the bucket name that we created in the Amazon console earlier. The bucket will be used to store all of our file uploads.

@AllArgsConstructor annotation generates a constructor with the bucketName variable in the enum.
@Getter annotation generates a getter for the bucketName variable in the enum.

Domain package

In this package we have the Todo model that represents our Todo in the database.

@Data
@AllArgsConstructor
@NoArgsConstructor
@Builder
@Entity
public class Todo {
    @Id
    @GeneratedValue
    private Long id;
    private String title;
    private String description;
    private String imagePath;
    private String imageFileName;
}

@Data annotation generates getters, setters, toString and equals methods for the Todo class.
@AllArgsConstructor annotation generates a constructor with all the arguments for the Todo class.
@NoArgsConstructor annotation generates a constructor with no arguments for the Todo class.
@Builder annotation creates a builder pattern for the Todo class.
@Entity annotation makes the Todo class a database entity.
@Id annotation marks the id field as a primary key in the database.
@GeneratedValue annotation makes the id field auto-increment whenever a new todo is saved into the database.

Repository package

In this package, we have the repository class that extends the JPA CrudRepository interface that makes it possible to perform various database queries.

public interface TodoRepository extends CrudRepository<Todo, Long> {
    Todo findByTitle(String title);
}

Service package

@AllArgsConstructor
@Service
public class FileStore {
    private final AmazonS3 amazonS3;
    public void upload(String path,
                       String fileName,
                       Optional<Map<String, String>> optionalMetaData,
                       InputStream inputStream) {
        ObjectMetadata objectMetadata = new ObjectMetadata();
        optionalMetaData.ifPresent(map -> {
            if (!map.isEmpty()) {
                map.forEach(objectMetadata::addUserMetadata);
            }
        });
        try {
            amazonS3.putObject(path, fileName, inputStream, objectMetadata);
        } catch (AmazonServiceException e) {
            throw new IllegalStateException("Failed to upload the file", e);
        }
    }
    public byte[] download(String path, String key) {
        try {
            S3Object object = amazonS3.getObject(path, key);
            S3ObjectInputStream objectContent = object.getObjectContent();
            return IOUtils.toByteArray(objectContent);
        } catch (AmazonServiceException | IOException e) {
            throw new IllegalStateException("Failed to download the file", e);
        }
    }
}

In the FileStore class above, we have the logic used to upload and download files from Amazon S3.

In the upload method we pass in:

path is the path on the Amazon S3 bucket where the file will be stored.
fileName is the actual name of the file being uploaded. It will be used as the key when downloading the file from S3.
optionalMetaData map contains the details of the file i.e file type and file size.
inputStream contains the actual file that should be saved to Amazon S3.

      ObjectMetadata objectMetadata = new ObjectMetadata();
      optionalMetaData.ifPresent(map -> {
            if (!map.isEmpty()) {
                map.forEach(objectMetadata::addUserMetadata);
            }
        });

The above code block loops through the optionalMetaData map adding all of the file information to the S3 objectMetaData.

amazonS3.putObject(path, fileName, inputStream, objectMetadata); saves the file to Amazon S3 bucket.

In the download method:

S3Object object = amazonS3.getObject(path, key); downloads the file from the path passed in and with the file name similar to the key passed in the getObject method.
S3ObjectInputStream objectContent = object.getObjectContent(); gets an inputStream from the object returned from Amazon S3.
IOUtils.toByteArray(objectContent) converts the input stream to byteArray that can be sent over Restful APIs.

public interface TodoService {
    Todo saveTodo(String title, String description, MultipartFile file);
    byte[] downloadTodoImage(Long id);
    List<Todo> getAllTodos();
}

The TodoService interface above contains various methods that we will implement to be able to save and get todos.

@Service
@AllArgsConstructor
public class TodoServiceImpl implements TodoService {
    private final FileStore fileStore;
    private final TodoRepository repository;
    @Override
    public Todo saveTodo(String title, String description, MultipartFile file) {
        //check if the file is empty
        if (file.isEmpty()) {
            throw new IllegalStateException("Cannot upload empty file");
        }
        //Check if the file is an image
        if (!Arrays.asList(IMAGE_PNG.getMimeType(),
                IMAGE_BMP.getMimeType(),
                IMAGE_GIF.getMimeType(),
                IMAGE_JPEG.getMimeType()).contains(file.getContentType())) {
            throw new IllegalStateException("FIle uploaded is not an image");
        }
        //get file metadata
        Map<String, String> metadata = new HashMap<>();
        metadata.put("Content-Type", file.getContentType());
        metadata.put("Content-Length", String.valueOf(file.getSize()));
        //Save Image in S3 and then save Todo in the database
        String path = String.format("%s/%s", BucketName.TODO_IMAGE.getBucketName(), UUID.randomUUID());
        String fileName = String.format("%s", file.getOriginalFilename());
        try {
            fileStore.upload(path, fileName, Optional.of(metadata), file.getInputStream());
        } catch (IOException e) {
            throw new IllegalStateException("Failed to upload file", e);
        }
        Todo todo = Todo.builder()
                .description(description)
                .title(title)
                .imagePath(path)
                .imageFileName(fileName)
                .build();
        repository.save(todo);
        return repository.findByTitle(todo.getTitle());
    }
    @Override
    public byte[] downloadTodoImage(Long id) {
        Todo todo = repository.findById(id).get();
        return fileStore.download(todo.getImagePath(), todo.getImageFileName());
    }
    @Override
    public List<Todo> getAllTodos() {
        List<Todo> todos = new ArrayList<>();
        repository.findAll().forEach(todos::add);
        return todos;
    }
}

In the TodoServiceImpl above, we provide the implementation for the methods for saving and getting all todos.

Controllers package

In this package, we have TodoController class which handles the incoming HTTP requests.

@RestController
@RequestMapping("api/v1/todo")
@AllArgsConstructor
@CrossOrigin("*")
public class TodoController {
    TodoService service;
    @GetMapping
    public ResponseEntity<List<Todo>> getTodos() {
        return new ResponseEntity<>(service.getAllTodos(), HttpStatus.OK);
    }
    @PostMapping(
            path = "",
            consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
            produces = MediaType.APPLICATION_JSON_VALUE
    )
    public ResponseEntity<Todo> saveTodo(@RequestParam("title") String title,
                                         @RequestParam("description") String description,
                                         @RequestParam("file") MultipartFile file) {
        return new ResponseEntity<>(service.saveTodo(title, description, file), HttpStatus.OK);
    }
    @GetMapping(value = "{id}/image/download")
    public byte[] downloadTodoImage(@PathVariable("id") Long id) {
        return service.downloadTodoImage(id);
    }
}

Testing our uploads and downloads from the S3 bucket

Conclusion

Congratulations! Now that you learned how to upload and download files from Amazon S3, go ahead and implement the logic for uploading multiple files to Amazon S3.

Find the source code here.

Happy Coding!

Python Generators

Odhiambo Paul — Wed, 18 Nov 2020 03:30:02 +0000

Generators are functions that return an iterable set of items that can be iterated through one at a time. They were introduced with Python Enhancement Proposal 255 (PEP 255). Python Enhancement Proposal 255 is a design document that provides information to the Python developer community describing the concept of generators in Python.

Python generators are objects that can be looped over similar to that of a list. Unlike lists, lazy iterator contents are not stored in the memory. The efficient way to iterate through large datasets is through the use of generators.

Prerequisites

You'll need to have a basic understanding of Python to get started. This will help guide you through each of the functions and the keywords. All the functions and code used can be found on repl.it.

To execute various functions, click on RUN.

You will now be able to access the variables and the function return values from the Python interpreter.

A generator function is defined like a normal function, but rather than returning a value from the return keyword, it produces and returns a value via the yield keyword. If we find the keyword yield in the function, it becomes a generator function and returns a generator object.

Generator Object uses lazy evaluations to yield sequences. To get values from the generator object, call the next() method on the generator object or loop through the generator object.

A yield statement pauses the execution of the function and returns the value to the caller, but maintains the status of the function to allow the function to continue execution from where it is left off. When resumed, the function continues execution immediately after the last run of the yield. This generates a set of values over time, rather than computing them at once and sending them back as a list.

Generator Expressions are a memory-efficient and high-performance generalization of list comprehensions and generators as discussed later in the article.

Generator Expressions versus List Comprehensions

list_comprehension = ['List Comprehension' for n in range(4)]
generator_expression = ('Generator expression' for n in range(4))

Unlike list comprehensions, generator expressions don’t construct list-objects, they generate values “just in time” as a generator function or class-based iterator would. Class-based iterator and generator functions implement the iterators which makes it possible to construct a list object by calling the next() object.

You need to call next() or loop through the generator object to access the values produced by the generator expression. When there isn't the next value in the generator object, a StopIteration exception is thrown.

>>> next(generator_expression)
'List Comprehension'
>>> next(generator_expression)
'List Comprehension'
>>> next(generator_expression)
'List Comprehension'
>>> next(generator_expression)
'List Comprehension'
>>> next(generator_expression)
StopIteration

A for loop can be used to iterate the generator object.

for expression in generator_expression:
    print(expression)

Given a range of n numbers to filter out odd numbers. We could solve the problem using three different methods listed below.

Using function.

def even_integers_function(n):
    result = []
    for i in range(n):
        if i%2==0:
            result.append(i)
    return result

>>>even_integers_function(10)
[0,2,4,6,8,10]

Using Generator function. The generator function below yields a generator object that we can iterate through to get a list of even numbers.

def even_integer_generator(n):
    for i in range(n):
        if i % 2 == 0:
            yield i

A generator function returns a generator object that can be iterated to get the values.

>>>even_integer_generator(10)
<generator object even_integer_generator at 0x1036c3200>

>>>list(even_integer_generator(10))
[0,2,4,6,8,10]

Using Generator expression.

>>> even_integers = (i for i in range(10) if i%2==0)
>>> list(even_integers)
[0,2,4,6,8]

Given a list of names, we can convert each name in the list to uppercase using the following code:

name_list = ["Adam","Eve","John","Doe","Peter","Paul","Kevin"]

Use of functions.

def names_to_uppercase(names):
    uppercase_names = []
    for name in names:
        name.upper()
        uppercase_names.append(name)
    return uppercase_names

Use of generator functions.

def names_to_uppercase_generator(names):
    for name in names:
        name.upper()
        yield name

Use of generator expressions.

>>> uppercase_names = (name.upper() for name in name_list)
>>> list(uppercase_names)
['ADAM', 'EVE', 'JOHN', 'DOE', 'PETER', 'PAUL', 'KEVIN']

Profiling Generator Performance

>>> import sys
# Square root of numbers in a range using list comprehension.
>>> nums_squared_list = [n * 2 for n in range(10000)]
>>> sys.getsizeof(nums_squared_list)
87724
# Square root of numbers in a range using generator functions.
>>> nums_squared_generator = (n ** 2 for n in range(10000))
>>> print(sys.getsizeof(nums_squared_generator))
125

List comprehension uses 87724 bytes of memory while the generator function uses only 125 bytes of memory. By using generators we save memory when compared to list comprehension where a lot of memory is used.

The getsizeof() object returns the amount of memory that holds the nums_squared_list list compared to len() object that would return the total number of items with the nums_sqaured_list list.

>>>len(nums_squared_list)
10000

Generating the Fibonacci sequence using generator functions.
The Fibonacci sequence is a series of numbers where the next number is identified by adding the two previous numbers.

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...

def fibonacci_gen():
    trailing,leading = 0,1
    while True:
        yield leading
        trailing,leading = leading, trailing+leading

To get the first value of the Fibonacci sequence that we generated using generator function above we call the fibonacci_gen() function and assign it to variable fib, we then call next() on the fib variable, fib.next() to get the next number in the sequence.

>>>fib = fibonacci_gen()
>>>fib.next()

To get the list of the first 10 values for the fibonacci_gen() function we call the fib.next() inside a for loop with a range of 10:

for _ in range(10):
    fib.next()

Building Generator pipelines

Data pipelines allow you to string together code to process large datasets or streams of data without maxing out your machine’s memory.

When working with pipelines:

Several pipes can be linked together.
Items flow one by one through the entire pipeline.
We can package pipeline functionality into a callable function.

To get the longest name from the text file names.txt, we create two generator methods.

The first generator method full_names yields names line by line stripping off the new line from each line.
The second generator method lengths yield a tuple with a name and an integer representing its length.
To get the longest name we use max and pass in the lengths generator object and pass a key for the lengths.

def get_longest_name():
    full_names = (name.strip() for name in open("text/names.txt"))
    lengths = ((name,len(name)) for name in full_names)
    longest = max(lengths,key=lambda x:x[1])

Conclusions

In this article we went over the concepts of generators in Python. They are useful when working with complex datasets and performing memory-intensive tasks. You can find more on Python generators from the Python wiki. These will help you create better applications with cleaner code and offer alternatives to creating nested for loops and working with list comprehensions that are memory intensive.

Kotlin RxJava Retrofit tutorial

Odhiambo Paul — Sun, 01 Mar 2020 15:03:36 +0000

Liquid syntax error: 'raw' tag was never closed

Kotlin and retrofit network calls

Odhiambo Paul — Sun, 01 Mar 2020 04:06:35 +0000

Liquid syntax error: 'raw' tag was never closed

Deploying React Webapp to github pages

Odhiambo Paul — Thu, 28 Nov 2019 15:34:36 +0000

Deploying a react app to github pages

Did you build React app and want to deploy it, following these simple steps you able to deploy and showing the world your amazing app.

I will show how to create and deploy React App using create-react-app and GitHub Pages

Prerequisites :

GitHub Account.

Install Git in your machine and Set up Git.

Make sure you have Node.js and Npm installed in your machine.

Install Node.js

Install Npm

Install Github desktop app to easen working with git

Notice You’ll need to have Node 8.10.0 or later on your local machine.

Procedure :

1- Clone this repo to follow along.

2- We need to install GitHub Pages package as a dev-dependency.

cd ReactPortfolio

run npm install to install node modules

npm install gh-pages --save-dev

3- Add properties to package.json file.

The first property we need to add at the top level homepage
second we will define this as a string and the value will be
"http://{username}.github.io/{repo-name}" {username} is your GitHub username,
and {repo-name} is the name of the GitHub repository you created it
will look like this :

"homepage": "http://paulodhiambo.github.io/ReactPortfolio"

Second in the existing scripts property we to need to add predeploy and deploy.

"scripts": {
//...
"predeploy": "npm run build",
"deploy": "gh-pages -d build"
}

{
  "homepage": "https://paulodhiambo.github.io/ReactPortfolio", #New
  "name": "tuto",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "react": "^16.12.0",
    "react-dom": "^16.12.0",
    "react-scripts": "3.2.0"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject",
    "predeploy": "yarn run build", #New
    "deploy": "gh-pages -d build" #New
  },
  "eslintConfig": {
    "extends": "react-app"
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "gh-pages": "^2.1.1"
  }
}

```

`


4- Add the repository to the github desktop app as a local repository.

5- Publish the repository to github

6- Now deploy it to GitHub Pages.
just run the following command :
*yarn run deploy* or <br/>
*npm run deploy*

This command will create a branch named gh-pages this branch 
host your app, and homepage property you created in package.json 
file hold your link for a live preview, or you can open the branch 
setting scroll down to GitHub Pages section you will find this:

[Visit deployed app](https://paulodhiambo.github.io/ReactApp)

7- Boom!! you site is live on github pages.