Forem: khalil snoussi

what does it take to a business to be ready for Industry 4.0

khalil snoussi — Thu, 24 Sep 2020 22:10:46 +0000

in this article i'm going to show you the basic knowledge you'll need to deploy your first IoT device and connect it to a Cloud IoT Service provider, i'm using Microsoft Azure due to its clear, user-friendly documentation.
i believe doing it yourself and manually will give you insights about what a business needs to make a successful shift into 4.0 industry.

What is Industry 4.0 :

the first industrial revolution happened in the late 1700's, and now we are living its fourth revolution : the digitalization of the manufacturing process and making them self-aware with deep-learning models.

Why Industry 4.0 is happening now ?

to understand why the industrial internet is happening now while its technologies have been arround for decades we have to look at the traditional industry ineffeciencies which are mainly :

the complexity of industrial systems has finally outpaced the human operator's ability to detect ineffeciencies, the outcome of this would be a large amount of machines that perform far below their capabilities, this factor alone is giving business owners a need to shift to 4.0
a big fall in the costs of compute, bandwidth, storage and sensors which means IT systems can now support industrial machine monitoring and analytics which is by the way the core of 4.0 industry and this fall of the costs is due to the Cloud technology offered by google, oracle and of course Amazon Web Services. The Cloud is now mature and make available powerful analytics tools.

Sensors are at the core of the 4.0 Industry due simply to the fact that they are our source of data in every industry, sensors types i know so far are not that much and very basic:

Temperature
Humidity
Pressure
Proximity
Level
Accelerometers
Gyroscope
Gas Sensors

Connect a Sensor to the cloud and monitor a machine state:

in the following i will use as a cloud Service provider Microsoft Azure because it is very well documented but the basic architecture is the same as Amazon's AWS.
also, the main goal of this section is to show you how to get data from your sensors, if you don't have any sensors there is a way of connection your cpu's Temperature sensor instead of buying one.

you'll find here every single Azure's IoT provided service, however in this example i'm using Azure IoT Hub and Azure Stream Analytics and maybe if you want to change your TensorFlow environement with Azure Machine learning

this application architecture looks like this :

also because i don't have a raspberry or arduino processors, we will simulate one, and in order to do that we will create 2 Azure CLI sessions in the local machine one for the simulated device or even your cpu's Temperature sensor, and the other one is for monitoring the machine state and sending messages.

launch the cloud shell in your account and add this command

az extension add --name azure-iot

the az extension add command adds the Microsoft Azure IoT Extension for Azure CLI to your CLI shell. The IOT Extension adds IoT Hub, IoT Edge, and IoT Device Provisioning Service specific commands to Azure CLI.

next step : connecting to IoT hub :

Run the az group create command to create a resource group. The following command creates a resource group named MyResourceGroup in the eastus location.

az group create --name MyResourceGroup --location eastus

Run the az iot hub create command to create an IoT hub. It might take a few minutes to create an IoT hub.

YourIotHubName. Replace this placeholder below with the name you chose for your IoT hub. An IoT hub name must be globally unique in Azure. This placeholder is used in the rest of this quickstart to represent your IoT hub name.

az iot hub create --resource-group MyResourceGroup --name {YourIoTHubName}

Run the az iot hub device-identity create command in the first CLI session. This creates the simulated device identity. YourIotHubName. Replace this placeholder below with the name you chose for your IoT hub. simDevice. You can use this name directly for the simulated device in the rest of this quickstart. Optionally, use a different name.

az iot hub device-identity create --device-id simDevice --hub-name {YourIoTHubName}

az iot device simulate -d simDevice -n {YourIoTHubName}

Optionally, you can send cloud-to-device messages by using the Azure portal. To do this, browse to the overview page for your IoT Hub, select IoT Devices, select the simulated device, and select Message to Device.

az iot device c2d-message send -d simDevice --data "Hello World" --props "key0=value0;key1=value1" -n {YourIoTHubName}

you can confirm that your simulated device has received the message "hello world" thus, everything is working a connection is made.

now that our (simulated) device is connected to IoT hub, we will run a python script that sends Humidity and Temperature data to IoT hub.

# Copyright (c) Microsoft. All rights reserved.
# Licensed under the MIT license. See LICENSE file in the project root for full license information.

import random
import time

# Using the Python Device SDK for IoT Hub:
#   https://github.com/Azure/azure-iot-sdk-python
# The sample connects to a device-specific MQTT endpoint on your IoT Hub.
from azure.iot.device import IoTHubDeviceClient, Message

# The device connection string to authenticate the device with your IoT hub.
# Using the Azure CLI:
# az iot hub device-identity show-connection-string --hub-name {YourIoTHubName} --device-id MyNodeDevice --output table
CONNECTION_STRING = "{Your IoT hub device connection string}"

# Define the JSON message to send to IoT Hub.
TEMPERATURE = 20.0
HUMIDITY = 60
MSG_TXT = '{{"temperature": {temperature},"humidity": {humidity}}}'

def iothub_client_init():
    # Create an IoT Hub client
    client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
    return client

def iothub_client_telemetry_sample_run():

    try:
        client = iothub_client_init()
        print ( "IoT Hub device sending periodic messages, press Ctrl-C to exit" )

        while True:
            # Build the message with simulated telemetry values.
            temperature = TEMPERATURE + (random.random() * 15)
            humidity = HUMIDITY + (random.random() * 20)
            msg_txt_formatted = MSG_TXT.format(temperature=temperature, humidity=humidity)
            message = Message(msg_txt_formatted)

            # Add a custom application property to the message.
            # An IoT hub can filter on these properties without access to the message body.
            if temperature > 30:
              message.custom_properties["temperatureAlert"] = "true"
            else:
              message.custom_properties["temperatureAlert"] = "false"

            # Send the message.
            print( "Sending message: {}".format(message) )
            client.send_message(message)
            print ( "Message successfully sent" )
            time.sleep(1)

    except KeyboardInterrupt:
        print ( "IoTHubClient sample stopped" )

if __name__ == '__main__':
    print ( "IoT Hub Quickstart #1 - Simulated device" )
    print ( "Press Ctrl-C to exit" )
    iothub_client_telemetry_sample_run()

now that we have Temperature Data we can do machine learning on it :

once the script is ran on the simulated device we will have the follwing output:

we can model the temperature data with python's scikit-learn for Time Series. here you find some of the fancy plots i got:

Temperature Trend

Temperature Seasonality

Conclusion:

to summerize this humble work, IIoT or Industry 4.0 is a great technology that aims to improve manufacturing process and making it self-aware and smarter in order to catch up to machine speed and thus making the production process more efficient.

However to make a successful shift to this digital Area a business owner need to improve his staff cloud skills and recruit a whole team of data anaylysts and data scientists as well as software engineers and developers.

Also, with sensors price drop, one is able to get as many as possible sensors because as we saw above more sensors means more data and more data means better precision of forcasting models.

did you ever ask yourself how file compression works ? the maths behind greedy algos

khalil snoussi — Mon, 21 Sep 2020 02:51:06 +0000

A little bit of background:

Yes, the foundation pillar of computer file compression is -spoiler alert- Maths, for images and videos which are basically just sequence of frames, it is slightly different because they support Lossy Compression techniques.
there are 2 kinds of compression techniques, Lossy and Lossless.

Lossy techniques as the name itself explains it, accept losing some of the original data after compression because we can still see and understand what happens in a video or a picture, these techniques cannot be applied to text because the meaning of the message will be lost, the receiver of your text won't understand the joke you wanted to tell him in the message for example.

Losseless techniques don't lose any valuable data after compression they are best suited for data that don't support losing some of its original information before compression like texts.

Lossless techniques for texts came to life because computers in the early 80's like Micro BBC (8 bits) had only a storage of max 800kb .. so can we please don't use a tremendous amount of bits ? as each character in the ascii table is coded in 8 bits, (1 bit = 0 or 1) so a text of 10 000 letters would weight 80 000 bits = 80 kb which is 10% of our storage disk. Our storage disk can only store a max size of 10 tiny text files ... do you feel the pain ?

However, a Mathematician called David Huffman found a way of compressing files which is called -spoiler alert- Huffman Coding, huffman coding is an algorithm of file compression which instead of representing each character (symbole in his research paper, 1995) in 8 bits, finds a way for representing each symbole in 1 bit 2 bits ,3 bits or even at max 8 bits or even 9 bits but those higher bits are for the least frequent symbols in the text so they don't really take that much space. the most frequent symbols in texts which are basically either 'e' or empty space ' ' are represented in 1 bit only.
Also, there is another problem in the normal 8 bits representation of each symbol in the ascii table, let's suppose 'a' representation is 0100001 and 'b' is 01000010
so 'ab' is 010000101000010 and this is unreadable the computer wouldn't know who is who in the binary sequence.

so much trouble ... jesus.

Algorithm:

let's first suppose we have an imaginary text right? and that we have a program that calculates the frequency of each character in the text and returns a dictionnary (Map) of each character and its frequency. -you might want to write this program for learning purpose-. and this program returns this.
inputs:
inputs = (a1,a2, ...,ap) p characters
weights= (w1,w2,...,wp) their p respective weights in the text
let's suppose we have inputs = ('a','b','c','d') and w =(10,15,30,16)

inputs = {'d':16,'c':30,'a':10,'b':15}

huffman algorithm outputs p respective codes, these are the new light representations of our characters.

output = (c1,c2,...,cp).

the algorithm finds output = (c1,c2,...,cp). by sorting the inputs based on their weights and by generating a sorted binary tree like the following:

and finally we generate a code table by traveling through the tree to find each character, if we go left you add 0, right you add 1 to your code until you find the character.

code = "" if (root.left): code = code + "0" else: code = code + "1"

char	frequency	code	size
a	10	110	3 bits
b	15	111	3 bits
c	30	0	1 bit
d	16	10	2 bits

As i told you earlier the most frequent characters are encoded in 1 bit in this example 'c' is 0 which makes it a genius idea, and the least frequent characters are encoded in bigger number of bits ('a': 110). the compression effect here won't be very efficient because the file itself is very tiny it contains only 10+15+30+16 = 71 characters so before the compression the file would weight 71*8 = 568 bits, arround 0.55 kilobits, after the compression the file size is :
10*3+15*3+30*1+16*2 = 137 bits which means the file lost
568 - 137 = 431 bits arround 76% of its original size without even losing data worth it don't you think ?

here ends my first ever article on the internet, follow me on twitter for more, next time we will do the same to pictures and videos which is a little bit more complicated but fun ! that's why we are passionate about Coding, Also i will share my python implementation of this algorithm here, and on
Github repo

aslo, let's get connected and share more knowledge on:

Twitter
or:
Linkedin

def sort_dict(inputs):
    frequencies_list = list(inputs.keys())
    frequencies_list.sort()
    temp = []
    for  item in frequencies_list:
        temp.append(inputs[item])
    new_dict = dict(zip(frequencies_list,temp))
    return new_dict



def huffmanTree(inputs):
    temp = sort_dict(inputs)#temporary sorted dict
    condition_arg =  list(temp.keys())
    condition = len(condition_arg) > 1
    while condition:
        temp = sort_dict(temp)
        char = list(temp.values())
        freq = list(temp.keys())
        if len(char) > 1 and len(freq) > 1:
            freq0 = freq.pop(0)
            freq1 = freq.pop(0)
            value0 = char.pop(0)
            value1 = char.pop(0)
        else:
            break
        temp = {freq0+freq1:(value0,value1)}
        remain_temp = dict(zip(freq,char)) 
        temp.update(remain_temp)
    return temp


def huffman_encode(char,inputs):
    temp = huffmanTree(inputs)
    temp = list(temp.values())
    temp = temp[0]
    print(temp)
    search(temp,char,"")

def search(root,char,code):
    code = code
    if(len(root)>1):
        search(root[0],char,code+"0")
        search(root[1],char,code+"1")
    else:
        if (root == char):
            print(code)





inputs1 = {12:"c",16:"e",9:"b",13:"d",5:"a",16:"e",45:"f"}
input2 = {10:'a',15:'b',30:'c',16:'d',29:'e'}

huffman_encode('a',inputs1) #should print 010