Forem: Abderrahmane Mustapha

A short story about web apis

Abderrahmane Mustapha — Fri, 09 Apr 2021 15:46:48 +0000

Developing a WEB API !!. Initially, I can say I know how to do this or at least I know the basics, but this time I wanted to know some history, who first used the WEB APIS and who gave this name to it, how it developed, and what drove these big companies To use WEB APIS

This story to my knowledge starts in 2000 with salesforce They wanted to change how people work on the Internet. In addition, they wanted to enter another interface that machines understand and communicate with as well.

It was not more than a year Ebay say's I'm here, They launched their own API under the name EBay application programming interface, but it was not available to everyone

Ebay and Salesforce is not alone here , a two years later Amazon and flicker did the same

But What About Facebook and Twitter !! , they have launched their own web apis because of the people who use the web crawler and web scrapper on their pages or those who create a rogue web apis,
same thing happened to google map apis.

Amazon was not the first, but it made a step that will lead us later to the world of cloud computing, Amazon launched the S3, through this service everyone will be able to store or extract any amount of data.

WEb Apis now is a big business , in mars 2010 twitter was serving
70,000 call Every Second, and in mars 2017 , 1000 million users access their banking accounts from mobiles and of course, these phones or applications connect to a web api somewhere

After reading some history, I began to ask about how to successfully develop the API and what are the stages that I must perform, I found some blogs they were talking about what they call the API lifecycle and this is what I will talk about in the next post :)

For Further Exploration :

I want to write with a scientific paper don't know how to start !!!

Abderrahmane Mustapha — Tue, 30 Mar 2021 22:27:00 +0000

I like to work in these fields (devops methodology, api development, ci / cd, function as a service), and I want to contribute with a scientific paper. Is it possible for anyone here to tell me the latest findings of researchers in one of this fields or about anything I can work on in the areas I mentioned

A python website no framework needed part 2

Abderrahmane Mustapha — Sat, 03 Oct 2020 11:38:01 +0000

Its the second and the last part of how you can make a website with python using no framework
, the purpose of these two articles is to show to the new developers that its possible to create a website with python without using a framework or any other PyPI package

Show the code

First, we need to separate the views, templates, and logic,
we will have the following folder structure :

--- mainFolder
---- app.py
--------- views
----------- __init__.py
----------- views.py
--------- templates
----------- index.html
--------- utils
----------- __init__.py
----------- render.py

let's start with the render.py file where we are going to write the code to render the HTML templates

# mainFolder/utils/render.py
def render(path):
    with open(path, "r") as f:
        template= f.read()
        return  template.encode("utf-8")

import the render method in the init.py

 #mainFolder/utils/__init__.py
 from .render import render

now let's move to the HTML template add this code to it

<!--  mainFolder/templates/index.html -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Document</title>
  </head>
  <body>
    <header>
      <h1>Home page</h1>
    </header>
    <main>
      <p>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc ut
        pellentesque sapien. Donec id pellentesque sem. Praesent vel urna vitae
        odio hendrerit malesuada et nec enim. Nulla risus dui,
      </p>
      <form action="/" method="POST">
        <input type="text" name="text" value="azaze" />
        <button type="submit">Submit</button>
      </form>
    </main>
  </body>
</html>

let's add a view now

# mainFolder/views/index.py
from utils import render
def index(environ):
    return render("templates/index.html")

import the index in init.py file of the views folder

# mainFolder/views/__init__.py
from .index import index

last step lets move our main file app.py to where im going to send responses to the client-side and handle post requests

from views import index

def app(environ, start_response):
        data = index(environ)
        if environ['REQUEST_METHOD'] == "POST":
            print("inputs ----------------- ", environ['wsgi.input'].read())
        start_response("200 OK", [
            ("Content-Type", "text/html"),
            ("Content-Length", str(len(data)))
        ])
        return iter([data])


if __name__ == '__main__':
    from gevent.pywsgi import WSGIServer
    WSGIServer(('', 8000),app, spawn=None).serve_forever()

you can run

python app.py

you will see an HTML template with a header, input form, and a submit button

add a text to the input hit the submit button and check the console you will see your text

inputs -----------------  b'text=azaze'
::1 - - [2020-10-02 13:06:19] "POST / HTTP/1.1" 200 777 0.003350

For Further Exploration

Do it your self framework

WSGI handle post request

Social footer

By Toumi abderrahmane twitter . github

A python website no framework needed part 1

Abderrahmane Mustapha — Sat, 26 Sep 2020 11:28:28 +0000

Why would you want to build a website using python with no framework here is some reasons :

you want to understand how this flask , django works
want to build your own framework in the future
you like to build things from scratch

writing a simple python website without a framework is easy if you just want some basics features,
but if you want more advanced features like routing, database connection, validate forms, and add your own template engine then things will become difficult bit by bit

Requirements

requirements or what you need to know before you can start doing this

a basic knowledge about http protocol, this is some resources if you want to learn more about how http work :

MDN Basics of HTTP

W3School HTTP Request Methods
what WSGI is? , some resources if you want to know :
An Introduction to the Python Web Server Gateway Interface

What is WSGI
how to serve a python web app using something like gunicorn , gevent

A simple hello World

here a simple example of a hello web python application

def app(environ, start_fn):
    start_fn('200 OK', [('Content-Type', 'text/plain')])
    return ["Hello World!\n"]

but how to run this ? , first we need to install gunicorn or gevent or something similar, for me i choose gevent because gunicorn will not work with windows.

Installing genvent

pip install gevent

Running the app

we need to import the gevent wsgi class to serve the app

....

if __name__ == '__main__':
    from gevent.pywsgi import WSGIServer
    WSGIServer(('', 8000),app, spawn=None).serve_forever()

want to know why we use

if __name__ == '__main__':

check this stackoverflow answer by Jack

our code will look like this now

def app(environ, start_response):
        data = b"Hello, Web!\n"
        start_response("200 OK", [
            ("Content-Type", "text/plain"),
            ("Content-Length", str(len(data)))
        ])
        return iter([data])


if __name__ == '__main__':
    from gevent.pywsgi import WSGIServer
    WSGIServer(('', 8000),app, spawn=None).serve_forever()

now you can go to your terminal and run

python app.py

check localhost:8000 and you will see a hello web message in your favorite web browser

in the next part, we will write a code to render an html file, create multiple pages, and navigate between them.

For Further Exploration

How to write a Python web framework

Python Web Applications: The basics of WSGI

How to use Flask with gevent (uWSGI and Gunicorn editions)

The u wsgi project

Simple Python Framework from Scratch

Social footer

By Toumi abderrahmane twitter . github

Drag and drop images with Django and vanilla js

Abderrahmane Mustapha — Sun, 26 Jul 2020 19:40:15 +0000

hey yes you prepare yourself to create an image drag and drop and add it to your Django project using pure javascript and Django forms for validation

first of all, if you want to start where I'm, you need to download this basic app from this link

The Coding part💻💻

in this simple app im using Django==3.0.7, Pillow==7.1.2, don't forget to install them, make/run migrations and create a superuser

let's start with our template index.html
we first need to make look better And here comes the role of CSS
in the styles.css add this code

/*image/static/styles.css*/

   .file-input{
     display: grid;
     grid-template-columns: auto;
     grid-template-rows: auto;
     width: 10rem;
     max-width: 12rem;
     height: 10rem;
     border: 1px dashed #4f86c3;

   }

  .file-input input{
    width: 0.1px;
    height: 0.1px;
    opacity: 0;
    overflow: hidden;
    position: absolute;
    z-index: -1;
   }
   .file-input label{
    width: 100%;
    height: 100%;
    cursor: pointer;
   }

   .file-input:hover{
     background-color: beige;
   }
   .file-input:active{
    border: 1px solid #4f86c3;
    box-shadow: inset 0 0 0 #4f86c3, 0px 10px 20px -10px #4f86c3;
  }

explanation :

.file-input input: we added this to hide the main file input because we are going to create a new one

.file-input:active: to add some effect when the user clicks
.file-input label: this is for a label that we are going to add now it will take the full width and high of its container and display some text on it to tell the user that he can click or drag and drop an image here

its time to add the label now 🙄, so go to the index.html
and add this inside the file-input div class

 <label class=" text-muted text-center" for="id_image">                 
                choose an image or drag and drop a one here
  </label>

our HTML template now will look like this

....
<body>

<form method="POST" >
    <div class="form-group">
        {{form.image.label_tag}}
        <div class="file-input">
            <label class=" text-muted text-center" for="id_image">                 
                choose an image or drag and drop a one here
            </label>  
            {{form.image}}
        </div>

    </div>
</form>
...

but now I think that we need some font-awesome magic If you think the same add this right before the end of the body tag

....
<script src="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.14.0/js/all.min.js" integrity="sha512-YSdqvJoZr83hj76AIVdOcvLWYMWzy6sJyIMic2aQz5kh2bPTd9dzY3NtdeEAzPp/PhgZqr4aJObB3ym/vsItMg==" crossorigin="anonymous"></script>
</body>...

before we start the javascript part lets center the form and make the label bolder

<!-- this is how we can 
 center a form horizontally  using bootstrap-->
<div class="w-25 mx-auto mt-5"> 
    <form method="POST" >
        <div class="form-group">
            {{form.image.label_tag}}
            <div class="file-input">
....
</div>

now in our styles.css add this line of code

 label{
      font-weight: bold;
  }

JavaScript Part 😎

first, we need to prevent the dragover and the drop from happening because i don’t want to see a new window opening every time i try to drag and drop an image

window.addEventListener("dragover",function(e){
    e = e 
    e.preventDefault();
  },false);
  window.addEventListener("drop",function(e){
    e = e 
    e.preventDefault();
  },false);

here is the main part of our js code, its now when we will get the image and show it to the user in our index.js add this code

file_inputs_drop  = document.getElementsByClassName('file-input')
Array.prototype.forEach.call(file_inputs_drop, element=>{ #loop through the inputs
    input=  element.getElementsByTagName('input')[0]
    element.addEventListener('drop', event=>{
        handle_file(element,event.dataTransfer.files[0])
        element.getElementsByTagName('input')[0].files = event.dataTransfer.files

        console.log(element.getElementsByTagName('input')[0].files)
    })

    /// get input inside this element
    input.addEventListener('change', event=>{
        handle_file(element,event.target.files[0],)
    })
})

Explanation:

file_inputs_drop = document.getElementsByClassName('file-input'): we used this to get all the file inputes in case you want use a multi drag drops

  element.addEventListener('drop', event=>{
        handle_file(element,event.dataTransfer.files[0])
        element.getElementsByTagName('input')[0].files = event.dataTransfer.files

        console.log(element.getElementsByTagName('input')[0].files)
    })

⬆ in this part (above) what we did is we added a drop event to the image input, its like the input waiting for the user to drag and drop an image in the drag/drop section, to show it to him

 /// get input inside this element
    input.addEventListener('change', event=>{
        handle_file(element,event.target.files[0],)
    })

⬆and here (above) we add change event in the case when the user drag and drop another picture,to replace the old image

the handle_file function :

handle_file = (element,file)=>{
    const reader = new FileReader();
    reader.readAsDataURL(file)
    reader.onload = function fileReadCompleted() {     
            delete_el(element,'label')
            delete_el(element,'droped-image')
            const img = new Image();          // creates an <img> element
            img.src = reader.result;         // loads the data URL as the image source
            img.className = "droped-image"
            element.appendChild(img);   // adds the image to the body        

    };
}

⬆this function shows the dropped image in the drag/drop section

this is how our small app looks like now, we are done with client-side

we need to send our data to the server-side using ajax
add these lines of code
in the bottom of your index.js

//////// THE AJAX PART ////////////////////

  ///////////// get cookie /////////////////
function getCookie(name) {
  var cookieValue = null;
  if (document.cookie && document.cookie !== '') {
      var cookies = document.cookie.split(';');
      for (var i = 0; i < cookies.length; i++) {
          var cookie = cookies[i].trim();
          // Does this cookie string begin with the name we want?
          if (cookie.substring(0, name.length + 1) === (name + '=')) {
              cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
              break;
          }
      }
  }
  return cookieValue;
}
var csrftoken = getCookie('csrftoken');
button = document.getElementsByTagName('button')[0]

  button.addEventListener('click', event=>{
    event.preventDefault()
    var xhttp = new XMLHttpRequest();
    formData = new FormData()
    formData.append('image', document.getElementById('id_image').files[0])

    xhttp.open("POST", '/', true);
    xhttp.setRequestHeader("X-CSRFToken", csrftoken);


    xhttp.send(formData)
    xhttp.onreadystatechange = function() {
        if (this.readyState == 4 && this.status == 200) {
            data = JSON.parse(this.responseText)

              console.log(data)

        }
    }
  })

im using getCookie to generate a csrf token because
Imposes on us to use csrf token with post request

Django Part 🐍🐍 :

first of all, you need to add a submit button inside the image form, (look at the code below)

.....
..... 
                {{form.image}}
            </div>

        </div>
        <button type="submit">submit</button>
    </form>
....

now go to your settings.py and add

#settings.py
# Base url to serve media files
MEDIA_URL = '/media/'
# Path where media is stored
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

in your main url file add this

from django.conf import settings
from django.conf.urls.static import static

## server media files when debug false
if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL,
                          document_root=settings.MEDIA_ROOT)

now we are done and we can save the image, but let's inform the user about errors and success

go to the views.py and replace the data variable with an HTTP response


#add this imports
from django.http import JsonResponse
from .models import ImageModel

def image(request):
    form = ImageForm()
    if request.method== "POST":
        form = ImageForm(request.POST, request.FILES)
        if form.is_valid():
            ImageModel.objects.create(image=form.cleaned_data.get('image'))#new
            return JsonResponse( {'details': "image saved  successfully"})#new
        else :
            data = {'details': form.errors}
            return JsonResponse(data)#new

    return render(request, 'image/index.html', {'form':form})
....

💥you can check your admin page you will find the images there

if you want to see the full code check this link

The "Basics" of adding PayPal payments to Django, server and client-side

Abderrahmane Mustapha — Sat, 11 Jul 2020 22:28:27 +0000

in this article will show a basic way on how to implement the Paypal payments using the checkout API on your Django project, for this I will create an e-learning website where the user can purchase courses

first clone this codebase from GitHub click here, because I want to focus on Paypal payments setup in this post

installing the requirements

go to your command line and run

pip install Django==3.0.7
pip install Pillow==7.1.2
pip install paypal-checkout-serversdk
pip install django-environ==0.4.5

create a sandbox account

first of all, click here and create a sandbox account
after creating this account successfully you will be redirected to your dashboard and click on My Apps & Cardntials

after this click on default application or you can create a new one

this will redirect you to this page where you will see a public key, the client key is public so you know anyone can see it

but don't let anyone see the private/secret key
here is the page of the client and secret key

update settings

now you can copy these two keys and past them, in line 26, and 27 in the settings.py file

#paypallpay/settings.py

#replace  this 
# 26 PAYPAL_CLIENT_ID  = env('PAYPAL_CLIENT_ID')
# 27 PAYPAL_SECRET_ID  =  env('PAYPAL_SECRET_ID')
#with this
PAYPAL_CLIENT_ID  = "client id here"
PAYPAL_SECRET_ID  =  "secret id here"

get client id from server-side

now let's add Paypal js sdk to our pay.html template, but the first thing that we need to do is to add a client_add variable to our pay in the views.py and assign the Paypal client id to it

#learn/views.py

def pay(request, pk):
    #line 10
    client_id = settings.PAYPAL_CLIENT_ID
    course = get_object_or_404(Course, pk=pk)
    return render(request, 'pay.html', {'course':course, 'client_id':client_id })

go to the learn/templates/pay.html, and scroll to the bottom and replace SB_CLIENT_ID with

{{client_id}}

, here is how you do it

{% block extrascript %}
#line 23
    <script src="https://www.paypal.com/sdk/js?client-id={{client_id}}&disable-funding=credit,card" > </script> 
    <script src="{% static  'learn/js/main.js' %}" ></script>
{% endblock %}

I added "&disable-funding=credit,card" because I just wan work with Paypal button , so I need to disable credit card button

client side setup

go to paypal server side smart buttons
you will see something like this

copy

<!-- Set up a container element for the button -->
    <div id="paypal-button-container"></div>

got to learn/tempaltes/pay.html and replace pay button with ""

<!-- delete this  -->
 <a href="{% url 'pay' course.pk %}" class="btn btn-warning 
font-weight-bold btn-lg col-12">Pay</a>

<!-- and replace it with this -->
    <div id="paypal-button-container"></div>

now go to the main.js file write or copy/paste ✨ this code

  paypal.Buttons({

            // Call your server to set up the transaction
            createOrder: function(data, actions) {
                return fetch('/paypal/create/'+course_id+'/', {
                    method: 'post',
                    headers: {"X-CSRFToken": csrftoken}
                }).then(function(res) {
                    return res.json();
                }).then(function(orderData) {
                    return orderData.id;
                });
            },

            // Call your server to finalize the transaction
            onApprove: function(data, actions) {
                return fetch('/paypal/' + data.orderID + '/capture/'+course_id+'/', {
                    method: 'post',
                    headers: {"X-CSRFToken": csrftoken}
                }).then(function(res) {
                    return res.json();
                }).then(function(orderData) {
                    // Three cases to handle:
                    //   (1) Recoverable INSTRUMENT_DECLINED -> call actions.restart()
                    //   (2) Other non-recoverable errors -> Show a failure message
                    //   (3) Successful transaction -> Show a success / thank you message

                    // Your server defines the structure of 'orderData', which may differ
                    var errorDetail = Array.isArray(orderData.details) && orderData.details[0];

                    if (errorDetail && errorDetail.issue === 'INSTRUMENT_DECLINED') {
                        // Recoverable state, see: "Handle Funding Failures"
                        // https://developer.paypal.com/docs/checkout/integration-features/funding-failure/
                        return actions.restart();
                    }

                    if (errorDetail) {
                        var msg = 'Sorry, your transaction could not be processed.';
                        if (errorDetail.description) msg += '\n\n' + errorDetail.description;
                        if (orderData.debug_id) msg += ' (' + orderData.debug_id + ')';
                        // Show a failure message
                        return alert(msg);
                    }

                    // Show a success message to the buyer
                    alert('Transaction completed by ' + orderData.payer.name.given_name);
                });
            }


        }).render('#paypal-button-container');

on the top of main.js add this lines of code

function getCookie(name) {
    let cookieValue = null;
    if (document.cookie && document.cookie !== '') {
        const cookies = document.cookie.split(';');
        for (let i = 0; i < cookies.length; i++) {
            const cookie = cookies[i].trim();
            // Does this cookie string begin with the name we want?
            if (cookie.substring(0, name.length + 1) === (name + '=')) {
                cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
                break;
            }
        }
    }
    return cookieValue;
}
const csrftoken = getCookie('csrftoken');

why we add this because csrftokens is important to secure our requests against csrf attacks

but before we execute this we need to edit the learn/urls.py, here is how you do it replace the old URLs patterns with this one

#learn/urls.py
urlpatterns = [
    path('', index, name="index"),
    path('paypal/pay/<pk>/', pay, name="pay"),
    path('paypal/create/<id>/', create, name="paypal-create"),
    path('paypal/<order_id>/capture/<id>/', capture, name="paypal-capture"), #changed #
     path('paypal/client-id/', getClientId , name="client-id")
]

we need to update our capture and create views to match our urls, go to learn/views.py

#learn/views.py
#line 18
def capture(request,order_id,id):
......

server side create an order

we start by importing create and capture request from python Paypal checkout SDK, add these lines on the top of learn/views.py

from paypalcheckoutsdk.orders import OrdersCreateRequest
from paypalcheckoutsdk.orders import OrdersCaptureRequest
from paypalcheckoutsdk.core import SandboxEnvironment

in our views , lets go to the create view, and setup the sandbox environment

#learn/views.py
def create(request):
    environment = SandboxEnvironment(client_id=settings.PAYPAL_CLIENT_ID, client_secret=settings.PAYPAL_SECRET_ID)
    client = PayPalHttpClient(environment)
.......

its time to create an order add an item and get the response from the Paypal api

 #order            
    create_order.request_body (
        {
            "intent": "CAPTURE",
            "purchase_units": [
                {
                    "amount": {
                        "currency_code": "USD",
                        "value": course.price,
                        "breakdown": {
                            "item_total": {
                                "currency_code": "USD",
                                "value": course.price
                            }
                            },
                        },                               


                }
            ]
        }
    )


#get the response
    response = client.execute(create_order)

#get the data dictionary  from the response  
    data = response.result.__dict__['_dict']

now we can return a json response to our client-side in the main js
our create view will look like this now

def create(request,id):
    if request.method =="POST":
        environment = SandboxEnvironment(client_id=settings.PAYPAL_CLIENT_ID, client_secret=settings.PAYPAL_SECRET_ID)
        client = PayPalHttpClient(environment)

        course = Course.objects.get(pk=id)
        create_order = OrdersCreateRequest()

        #order            
        create_order.request_body (
            {
                "intent": "CAPTURE",
                "purchase_units": [
                    {
                        "amount": {
                            "currency_code": "USD",
                            "value": course.price,
                            "breakdown": {
                                "item_total": {
                                    "currency_code": "USD",
                                    "value": course.price
                                }
                                },
                            },                               


                    }
                ]
            }
        )

        response = client.execute(create_order)
        data = response.result.__dict__['_dict']
        return JsonResponse(data)
    else:
        return JsonResponse({'details': "invalide request"})

before we can run this we need to add a course id so go to the pay.html and create a hidden input and add course id to it

#learn/templates/pay.html
<div class="card-footer bg-white"> 
   <input id="course-id"  value={{course.id}} hidden/>         
   <div id="paypal-button-container"></div>
</div>

and in the top of main.js let's add a client_id const , which contain the value of the course id

#learn/static/learn/js/main.js
const course_id = document.getElementById('course-id').value

Server-side capture the order id and finish the payment

it's simple we just need to send a request containing the order id to the paypal API to check if this order exists and then we get a response and send it to the client-side when everything is working fine the javascript code will send an alert to our end-user that the payment completed successfully, our capture view will look like this now

#learn/views
def capture(request,order_id,id):
    if request.method =="POST":
        capture_order = OrdersCaptureRequest(order_id)
        environment = SandboxEnvironment(client_id=settings.PAYPAL_CLIENT_ID, client_secret=settings.PAYPAL_SECRET_ID)
        client = PayPalHttpClient(environment)

        response = client.execute(capture_order)
        data = response.result.__dict__['_dict']

        return JsonResponse(data)
    else:
        return JsonResponse({'details': "invalide request"})

everything is ready now go to you sandbox dashboard and go to accounts u will see two email addresses the first one is personal and the other one is for business, personal for payer and business email for the receiver

💀Note: this two email addresses and keys is for testing if you want real keys and emails you need to upgrade your account upgrade

if you want to run this app in your localhost check this GitHub repository click here
you just need to replace

SECRET_KEY = 'SECRET_KEY' #your secret key here
PAYPAL_CLIENT_ID = 'PAYPAL_CLIENT_ID' #paypal client id here
PAYPAL_SECRET_ID = 'PAYPAL_SECRET_ID' #paypal secret id here

for further explorations

check the Paypal API here
and the Paypal checkout repository it helped me a lot to understand how things work here

Build your own COVID 19 Tracker with Dash, plotly , python part-2 : add cities info

Abderrahmane Mustapha — Tue, 07 Jul 2020 19:48:15 +0000

🤩full code on github🤩

🙄in the previous post! we talked about how to represent our data in a choropleth

🤠in this post, we talk about how can we add a sidebar which contains the city's data(total cases, recovered, deaths) for each country

Coding part 💻

now lets get and save the citys data add this lines of code to data.py


#covid19Dash/data.py
### get countries data
get_api_data(
        'https://www.trackcorona.live/api/cities', # API link
        'data', #field name
        'cities', #file name
)

by running python data.py our cities.csv file will look like this one below

,location,country_code,latitude,longitude,confirmed,dead,recovered,updated
0,Winterswijk,nl,51.977038,6.7454136,6,,,2020-05-19 21:01:51.948949+00:00
1,"Marion County, Tennessee",us,35.0928512,-85.64348700000001,33,1.0,,2020-05-19 21:00:19.712447+00:00
2,Juruá,br,-3.5811865,-66.4271499,3,0.0,,2020-05-19 21:02:48.021462+00:00
3,"Franklin County, Alabama",us,34.3967457,-87.6186379,344,4.0,,2020-05-19 21:00:19.712447+00:00
4,Capim Grosso,br,-11.3801768,-40.0068953,12,3.0,,2020-05-19 21:02:40.089219+00:00
5,Catende,br,-8.6679698,-35.7218229,17,6.0,,2020-05-19 21:02:40.738953+00:00
6,"Hale County, Alabama",us,32.7859102,-87.6186379,91,2.0,,2020-05-19 21:00:19.712447+00:00
7,Serra do Ramalho,br,-13.5413338,-43.5774778,3,0.0,,2020-05-19 21:02:58.150643+00:00
8,Cordeirópolis,br,-22.4818598,-47.4593897,29,0.0,,2020-05-19 21:02:41.588760+00:00
9,Lagoa Seca,br,-7.1549934,-35.8537249,10,1.0,,2020-05-19 21:02:48.193946+00:00
10,Ponta Porã,br,-22.523123,-55.7242526,6,0.0,,2020-05-19 21:02:53.805951+00:00
11,Tiaret,dz,35.3673553,1.3220322,148,148.0,9.0,2020-05-19 21:01:46.336022+00:00
12,Serra do Salitre,br,-19.110579,-46.6793595,1,0.0,,2020-05-19 21:02:58.158068+00:00
13,Serra,br,-20.1228954,-40.3054099,1381,82.0,,2020-05-19 21:02:58.164867+00:00

👀now let's display this data on a sidebar like this one

the first thing that we need to do here is to create a function which is going to find a list of cities by country you can put this function on the data.py file so now add this lines of code to the data.py file

def search_country_csv(data,query):
    data = data.set_index('country_code')
    return data.loc[tuple([query]), :].reset_index()


def country(name="dz"):

    cities_list  = search_country_csv(open_csv('data/cities.csv'), name.lower())
    return cities_list.sort_values(by='confirmed', ascending=False)

we have a list of cities sorted by confirmed cases😁

let's go to the app.py file, create our list and add the data to it
first you should replace the the app.py code from the last post by this one (the code start with app.layout = html.Div(children=[
dcc.Graph(......)

# earth graph
app.layout = html.Div(children=[
    html.Div(children=[

        #earth graph
        dcc.Graph(
            id='earth-graph',
            className="col-md-8",
            figure = earth() ,
            config={
        'displayModeBar': False
                }                       
        ),
        #end of earth graph

        #cities for each country
         html.Div(

            className="col-md-3 row",
            children=[
                #cities list
            html.H4(children="Algeria Citys", id="citys-title", className="col-12 text-center"),
                html.Ul(children=[
                    html.Li(children=[
                        #add info to the side bar
                        city['location'],
                        #order last
                        html.Div(children=[
                        # data containers
                            html.Div(children=[ 
                                html.P(children="Total",className="small-text"),
                                html.Span(children=city['confirmed'], className="badge badge-warning"),
                            ],  className="ml-1 mr-1 d-flex flex-column align-items-center"),
                            html.Div(children=[
                                html.P(children="Deaths",className="small-text"),
                                html.Span(children=city['dead'], className="badge badge-danger"),
                                ],className="ml-1 mr-1 d-flex flex-column align-items-center"),
                            html.Div(children=[
                                html.P(children="Recovered",className="small-text"), 
                                html.Span(children=city['recovered'], className="badge badge-success")
                            ],className="ml-1 mr-1 d-flex flex-column align-items-center"),
                        #end of data containers                    
                    ], className="d-flex flex-row justify-content-between align-items-center")
                ], className="list-group-item d-flex flex-column justify-content-center align-items-center") for _ , city in country().iterrows()] # the country function get data from csv for a specific country , here we loop over the cities and create a rows of all the citys each row contains the number of confirmed , deaths and recovered cases  and the name of the city of course

                ,className="list-group", id="cities-data")
                #end of cities list
            ]
         )
        # end of cities for each country

    ], className="row")  , 
     # end of earth graph  
], className="container-fluid")

in order to make things work perfectly, we need to add bootstrap in the top of app.py file so here is how we can do this

external_stylesheets = [
    {
        'href': 'https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css',
        'rel': 'stylesheet',
        'integrity': 'key here', # add the key here
        'crossorigin': 'anonymous'
    }
]

and do not forget to import the function in the data.py that we created to extract cities from the CSV file, add these lines of code in the of app.py file

from data import earth_data,country_data, country

🌎🌏last this that we need to do is to update the sidebar when we click on a specific country on the map, to do that let's add a callback, check this if you want to know more about flask callbacks

here is the code that you need to add

################# CALL BACKS °°°°°°°°°°°°°°°°°°°°°°°°°°

#update sidbar onclick

#side bar call back start
@app.callback(
    Output('cities-data', 'children'),
    [Input('earth-graph', 'clickData')])
def display_hover_data(clickData):
    name = clickData['points'][0]['customdata'][0] #get country name
    return  [html.Li(children=[
             #add info to the side bar
              city['location'],
              #order last
              html.Div(children=[

                    # data containers
                    html.Div(children=[ 
                        html.P(children="Total", className="small-text"),
                        html.Span(children=city['confirmed'], className="badge badge-warning"),
                    ],className="ml-1 mr-1 d-flex flex-column align-items-center" ),
                    html.Div(children=[
                        html.P(children="Deaths", className="small-text"),
                        html.Span(children=city['dead'], className="badge badge-danger"),
                        ],className="ml-1 mr-1 d-flex flex-column align-items-center"),
                    html.Div(children=[
                        html.P(children="Recovered", className="small-text"), 
                        html.Span(children=city['recovered'], className="badge badge-success")
                    ],className="ml-1 mr-1 d-flex flex-column align-items-center"),
                    #end of data containers

                ], className="d-flex flex-row justify-content-between align-items-center")


            ], className="list-group-item d-flex flex-column justify-content-center align-items-center") for _ , city in country(name).iterrows()]
#side bar call back end

finally, this is just a simple application to share what I learned about python dash framework

in the end run

python app.py

and you will see this

For further exploration

im using csv files to store my data if you want something better check omniscidb

if you have some problems with this you can check the full code on github

Build your own Covid 19 Tracker with Dash , plotly , python part-1 : represent data in choropleth

Abderrahmane Mustapha — Sun, 31 May 2020 17:43:38 +0000

I would greatly appreciate it if you kindly give me some feedback on this
part 2 is here 💥💥

in this post will show how to create a simple dashboard to follow the progress of the covid19🚑 virus and its spread

our app will look like this in the end 😀😀

Dash :

Dash is Python web framework for building analytical web application , wich use flask on the server side , react for the front end ,and plotly.js for the data visualisation
if you want learn more about dash
check the this post on data camp

and you know Truth can only be found in one place: the code so lets start coding

Coding part 💻

setup our dash app 🚩

install dash and plotly and pondas by typing

pip install dash dash-renderer dash-html-components dash-core-components plotly pandas

then lets create a directory for our project and name it ** covid19Dash **

 mkdir covid19Dash && cd covid19Dash

inside this directory create app.py file

 touch app.py

and add this code to it

#covid19Dash/app.py
#dash imports
import dash
import dash_core_components as dcc
import dash_html_components as html
#end of dash imports

##  HTML code ## 
app.layout = html.Div(children=[ "hello there"])
### end of HTML ##

#init the app and server
app = dash.Dash()
server = app.server

now run

python app.py

and check this link http://127.0.0.1:8050/ if everything is running fine lets move to the next part

🧭 we need to create another python file with name of data.py this files will container logic about how to extract and use our data

touch data.py

inside the data.py file lets define a function to extract data from APIs with three arguments api_url , field and the name of the csv file file_name that contains the data

#covid19Dash/data.py

# don't forget to import this
import pandas as pd
import json
import requests
######


def get_api_data(api_url, field_name, file_name):
    data = requests.get(api_url)
    results = pd.DataFrame(data.json()[field_name])

    #save the result in the data folder
    results.to_csv('data/'+file_name+'.csv', sep=',' )

    # return API data
    return results

to get the data we are using 2 APIs

the first one covid19api containe data about the countries total cases , new cases, total deaths , .... and more

the second one trackcorona containe data about the citys in each country (total cases, total deaths , total recovered)

now lets get and save the data add this lines of code to data.py


#covid19Dash/data.py
### get countries data
get_api_data(
        'https://api.covid19api.com/summary', # API link
        'Countries', #field name
        'countries', #file name
)

by running python data.py our countries.csv file will look like this one below

😎 lets visualize the data we need to create a new file charts.py , add this lines to it

#covid19Dash/charts.py
#librarys
import json
#plotly and dash
import plotly.express as px

define a viz_earth_data function which is going to get countries data from our csv file and plot it on choropleth

#covid19Dash/charts.py
def viz_earth_data(df=open_csv("data/countries.csv")):

    fig = px.choropleth( df, locations="Country",
                        color="TotalConfirmed", 
                        hover_name="Country", 
                        hover_data=[ 'CountryCode',
                            'NewConfirmed','TotalConfirmed',
                            'NewDeaths','TotalDeaths','NewRecovered',
                            'TotalRecovered','Date', 'Slug'
                                   ], 
                        locationmode="country names",
                        projection="orthographic",
                        color_continuous_scale=px.colors.sequential.dense,
                        height=600

                       ).update_layout(clickmode='event+select')


    return fig

now lets go back to our app.py and import the viz_earth_data function

from charts import viz_earth_data

after this add the graph to our HTML layout

app.layout = html.Div(children=[
     dcc.Graph(
            id='earth-graph', #graph id
            figure = viz_earth_data() , #this will show the earth graph 
            config={
              'displayModeBar': False
            }                       
        ),
])

now run python app.py and open http://127.0.0.1:8050/ in your browser , you will see something like this

Part 2 is here 💥💥

Create a live multistreaming app

Abderrahmane Mustapha — Sun, 24 May 2020 20:50:31 +0000

what i want do in this app is similar to streamyard, restream.io, manycam.com

I want to use one of this technologies django-channels | tornado or nodejs
I need help, some open source apps, some suggestions, posts, youtube videos, or tutorials

Thank you

Send emails with Django and Gmail , a better way

Abderrahmane Mustapha — Sun, 24 May 2020 06:49:56 +0000

In this post I want to talk about how can we send emails using Django and Gmail, I read a lot of articles about this but none of them is working for me, this is my way of doing this

Setup Django 💚

first let's install django



 pip install django

start a django project



django-admin startproject send_gmail

start a django app



cd send_gmail
python manage.py  startapp send

add the app to settings.py installed app



# send_gmail/settings.py
INSTALLED_APPS = [
...
send,
]

migrate and create a super user



python manage.py  migrate
python manage.py createsuperuser

check this post if you want to know more about how to create a virtual environment , or how to install Django and start your project

The Gmail part ✉

now you need to create a Gmail account and then click on Manage your google account

now click on the Security tab

make sure to enable two steps verification

now click on App passwords

you have to type your password again

click on select app choose *** other (Custome Name) *** and give a name to you app

the last step click on generate and Gmail will generate a key or an app password make sure to copy this key or save it in a text file

Edit your settings.py file



#gmail_send/settings.py
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_USER = 'yoorusername@gmail.com'
EMAIL_HOST_PASSWORD = 'key' #past the key or password app here
EMAIL_PORT = 587
EMAIL_USE_TLS = True
DEFAULT_FROM_EMAIL = 'default from email'

🎉🎉 now you are ready to send emails with Django and Gmail in production🎉🎉

For further exploration

you can use SendGrid, Mailgun, Sendinblue... for your apps too
don't forget to put your key in a .env file

How to add Arabic Text on PDF with reportlab

Abderrahmane Mustapha — Sun, 17 May 2020 18:24:30 +0000

ReportLab is a tool for creating complex data-driven PDF documents and custom vector graphics
but this tool does not support arabic, so when you want add an arabic text you will see something like

so how can we fix this
in this article i will show you an example how to add an arabic text with reportlab using reportlab.platypus
let's start ...

Install ReportLab

if you have both pip and python installed so its easy just do this

pip install reportlab

Let's import the reportlab packages that we are going to use

   from reportlab.lib.styles import getSampleStyleSheet
   from reportlab.lib.pagesizes import letter
   from reportlab.platypus import Paragraph, SimpleDocTemplate, Spacer,PageBreak
   from reportlab.lib.styles import ParagraphStyle
   from reportlab.pdfbase import pdfmetrics
   import reportlab
   from reportlab.pdfbase.ttfonts import TTFont

Install arabic-reshaper and python-bidi

we will need this two packages to reshape our text so just go to your command line and type

pip install arabic_reshaper
pip install python-bidi

now we can import this two packages by adding this two lines of code

import arabic_reshaper
from bidi.algorithm import get_display

Using a custom font

here we need to use a custom font because the default font in reportlab does not support arabic im using this font 29ltbukraregular !
now let's add the font and give it the name of Arabic

pdfmetrics.registerFont(TTFont('Arabic', 'path to the font directory/29ltbukraregular.ttf'))

Styling the text

first let's initialize our stylesheet

#init the style sheet
styles = getSampleStyleSheet()

now let's start styling our text

arabic_text_style = ParagraphStyle(
    'border', # border on
    parent = styles['Normal'] , # Normal is a defaul style  in  getSampleStyleSheet
    borderColor= '#333333',
    borderWidth =  1,
    borderPadding  =  2,
    fontName="Arabic" #previously we named our custom font "Arabic"
)

Add the text

most of the time you will want to add more than a one paragraph to your file so we can use an array which contain our paragraphs let's name it storys

storys = []

now let's add some paragraphs to our pdf document we will use ** Spacer ** to set the space between paragraphs


arabic_text =""" عندما يريد العالم أن ‪يتكلّم ‬ ، فهو يتحدّث بلغة
 يونيكود. تسجّل الآن لحضور المؤتمر الدولي العاشر ليونيكود (Unicode Conference)، الذي سيعقد في 10-12 آذار 1997 بمدينة مَايِنْتْس، ألمانيا. و سيجمع المؤتمر بين خبراء
  من كافة قطاعات الصناعة على الشبكة العالمية انترنيت ويونيكود، حيث ستتم، على الصعيدين الدولي والمحلي على حد سواء مناقشة سبل استخدام يونكود في النظم القائمة وفيما يخص التطبيقات الحاسوبية، الخطوط، تصميم النصوص والحوسبة متعددة اللغات."""

# reshape the text 
rehaped_text = arabic_reshaper.reshape(arabic_text)
bidi_text = get_display(rehaped_text)

# add the text to pdf
 ## dont forget to add the style arabic_text_style
storys.append(Paragraph(bidi_text,arabic_text_style))
storys.append(Spacer(1,8)) # set the space here
storys.append(Paragraph(bidi_text,arabic_text_style))

finally we will use *** SimpleDocTemlate *** to create the document and save it

doc = SimpleDocTemplate('mydoc.pdf',pagesize = letter)
## add the storys array to the pdf document
doc.build(storys)

and this is our final resul perfect ...!

But why we need to use arabic-reshaper and python-bidi !

if you use just the arabic-reshaper

rehaped_text = arabic_reshaper.reshape(arabic_text)
storys.append(Paragraph(rehaped_text,arabic_text_style))
storys.append(Spacer(1,8))
storys.append(Paragraph(rehaped_text,arabic_text_style))

and this is the result

and if we only use bidi

bidi_text = get_display(arabic-text)

storys.append(Paragraph(bidi_text,arabic_text_style))
storys.append(Spacer(1,8))
storys.append(Paragraph(bidi_text,arabic_text_style))

this is the result

For Further exploration

reportlab can be used with djago to generate pdf files and reports and this is a link if you want learn more click here

if you want know more about reportlab check the repotlab user guide