Forem: Siddhesh Shankar

Learn MongoDB + Python basics in 5 minutes !

Siddhesh Shankar — Sat, 17 Oct 2020 13:31:31 +0000

What is MongoDB?

MongoDB is a NoSQL database – a NoSQL database is one where you don’t query the database with SQL. Other than that NoSQL really means nothing to define a database. So let’s have another go at defining MongoDB. MongoDB is a JSON document datastore. It allows you to store and query JSON style documents with a few smarts on top. This means that you can store objects with nested data all in one collection (collections are like tables for MongoDB).

What is MongoDB Compass?

MongoDB Compass is the GUI for MongoDB. Compass allows you to analyze and understand the contents of your data without formal knowledge of MongoDB query syntax. Click here to download and watch this video to install compass.

Let's Code

Install pymongo. Just open the command prompt and type:

pip install pymongo

Open Jupyter Notebook. Let's create a database first.

DEFAULT_CONNECTION_URL = "mongodb://localhost:27017/"
database_name = "Indian_States_DB"

# Establish a connection with mongodb
client = pymongo.MongoClient(DEFAULT_CONNECTION_URL)

## Creating a Database
database = client[database_name]

Create a collection. Collection in NoSQL can also be called a table. We have created a empty table below.

# Creating a collection
COLLECTION_NAME = 'Gross_Domestic_Product'
# Adding the collection to our database
collection = database[COLLECTION_NAME]

Let's check if our database exists or not.

client.list_database_names()

> ['admin', 'config', 'crawlerDB', 'demoDB', 'local']

The database name does not appear in the above list because there is not data inside Gross_Domestic_Product.

Inserting Records

# Inserting one record
record = {
        "Rank": "1",
        "State": "Maharastra",
        "Nominal GDP ($ Billion)": "400",
        "Data Year": "2019-2020",
        "Comparable Country": "Philippines"
        }
collection.insert_one(record)

# Inserting multiple records
records = [{
        "Rank": "2",
        "State": "Tamil Nadu",
        "Nominal GDP ($ Billion)": "260",
        "Data Year": "2019-2020",
        "Comparable Country": "Vietnam"
        },
        {
        "Rank": "3",
        "State": "Uttar Pradesh",
        "Nominal GDP ($ Billion)": "250",
        "Data Year": "2019-2020",
        "Comparable Country": "Romania"
        },
        {
        "Rank": "4",
        "State": "Karnataka",
        "Nominal GDP ($ Billion)": "240",
        "Data Year": "2019-2020",
        "Comparable Country": "Portugal"
        }]
collection.insert_many(records)

Again check if the database exists or not.

client.list_database_names()

> ['Indian_States_DB', 'admin', 'config', 'crawlerDB', 'demoDB', 'local']

Find Records

The find() method returns all occurrences in the selection. Select rank and states from the collection.

for i in collection.find({},{"_id":0, "Rank":1, "State":1}):
    print(i)

{'Rank': '1', 'State': 'Maharastra'}
{'Rank': '2', 'State': 'Tamil Nadu'}
{'Rank': '3', 'State': 'Uttar Pradesh'}
{'Rank': '4', 'State': 'Karnataka'}

You are not allowed to specify both 0 and 1 values in the same object (except if one of the fields is the _id field). If you specify a field with the value 0, all other fields get the value 1, and vice versa. For selecting all the records, simply execute find().

Sort

Use the sort() method to sort the result in ascending or descending order. Use the value -1 as the second parameter to sort descending.

for i in collection.find({},{"_id":0,"State":1, "Nominal GDP ($ Billion)":1}).sort("State", -1):
    print(i)

{'State': 'Uttar Pradesh', 'Nominal GDP ($ Billion)': '250'}
{'State': 'Tamil Nadu', 'Nominal GDP ($ Billion)': '260'}
{'State': 'Maharastra', 'Nominal GDP ($ Billion)': '400'}
{'State': 'Karnataka', 'Nominal GDP ($ Billion)': '240'}

Limit the Result

To limit the result in MongoDB, we use the limit() method.

for i in collection.find({},{"_id":0, "Rank":1, "State":1, "Comparable Country":1}).limit(2):
    print(i)

{'Rank': '1', 'State': 'Maharastra', 'Comparable Country': 'Philippines'}
{'Rank': '2', 'State': 'Tamil Nadu', 'Comparable Country': 'Vietnam'}

Delete Records

To delete one document, we use the delete_one() method. To delete more than one document, use the delete_many() method.

collection.delete_one({"State": "Maharastra"})

for i in collection.find():
    print(i)

{'_id': ObjectId('5f8ae0347d1ce4be5cea8007'), 'Rank': '2', 'State': 'Tamil Nadu', 'Nominal GDP ($ Billion)': '260', 'Data Year': '2019-2020', 'Comparable Country': 'Vietnam'}
{'_id': ObjectId('5f8ae0347d1ce4be5cea8008'), 'Rank': '3', 'State': 'Uttar Pradesh', 'Nominal GDP ($ Billion)': '250', 'Data Year': '2019-2020', 'Comparable Country': 'Romania'}
{'_id': ObjectId('5f8ae0347d1ce4be5cea8009'), 'Rank': '4', 'State': 'Karnataka', 'Nominal GDP ($ Billion)': '240', 'Data Year': '2019-2020', 'Comparable Country': 'Portugal'}

This was all about basics of MongoDB. Hope you like my explanation. Furthermore, if you have any query, feel free to ask in a comment section.

Python 3.9 is here ! Top 3 cool features for you to try.

Siddhesh Shankar — Sun, 11 Oct 2020 05:54:23 +0000

Python 3.9 is out! Volunteers from all over the world have been working on improvements to Python, While beta versions have been available for some time, the first official version of Python 3.9 was released on October 5, 2020.
It’s clear that this version marks a breaking point from the old route of Python’s evolution, onto a new path. We’ll cover:

Merging Dictionaries
Removing prefix and suffix
Time zone support

Merging Dictionaries

Dictionaries are one of the fundamental data structures in Python. They’re used everywhere in the language and have gotten quite optimized over time.
"|" can be used to merge dictionaries in python.

>>> dict1 = {'Asia': ['India', 'Singapore'], 
             'Europe': ['Austria', 'Belgium']}
>>> dict2 = {'Africa': ['Egypt', 'Kenya'], 
             'North America': ['USA', 'Canada']}
>>> dict1|dict2
{'Asia': ['India', 'Singapore'], 
'Europe': ['Austria', 'Belgium'], 
'Africa': ['Egypt', 'Kenya'], 
'North America': ['USA', 'Canada']}

Removing prefix and suffix from a string

We have two new methods for removing string prefixes and suffixes. .removeprefix() and .removesuffix() remove at most one copy of the affix. If you want to be sure to remove all of them, then you can use a while loop

>>> str1 = "Mr. Donald Trump"
>>> str1.removeprefix("Mr. ")
'Donald Trump'

>>> str2 = "Martin Luther King Jr."
>>> str2.removesuffix(" Jr.")
'Martin Luther King

Proper Time Zone Support

Python has extensive support for working with dates and times through the datetime module in the standard library. However, support for working with time zones has been somewhat lacking. Here is zoneinfo module. zoneinfo uses an IANA time zone database residing on your local computer.
A Python implementation of the IANA Time Zone Database is available on PyPI as tzdata. You can install it with pip:

$ python -m pip install tzdata

Accessing Time Zones

Get a set containing all the valid keys for IANA time zones available anywhere on the time zone path. This is recalculated on every call to the function.

>>> from datetime import datetime, timezone
>>> import ZoneInfo

>>> zoneinfo.available_timezones()
>>> {'Asia/Shanghai', 'Asia/Karachi', 'Europe/Madrid', 'Pacific/Saipan', 'Europe/Zaporozhye', 'Asia/Harbin', 'Portugal', 'Asia/Pyongyang', 'Europe/Stockholm', 'Pacific/Funafuti', 'Europe/Tirane', 'Australia/Queensland', 'America/Rankin_Inlet', 'Europe/Gibraltar', 'America/Coral_Harbour', 'Africa/Bissau', 'Asia/Choibalsan', 'US/Michigan', 'Australia/Perth', 'America/Sao_Paulo', 'America/Mexico_City', 'Asia/Kabul', 'America/Mazatlan', 'Etc/GMT+2', 'Asia/Kolkata', 'America/Fort_Wayne', 'CET', 'Etc/Zulu', 'US/Samoa', 'Europe/Kaliningrad', 'Etc/GMT-11', 'Brazil/West', 'Europe/Vatican', 'EET', 'Asia/Sakhalin', 'Antarctica/Rothera', 'Pacific/Kosrae', 'Etc/GMT-7', 'America/Santa_Isabel', 'Pacific/Wallis', 'Africa/Monrovia', 'Australia/Sydney', 'Asia/Tokyo', 'Africa/Ceuta', 'America/Santiago', 'Indian/Kerguelen', 'Atlantic/Azores', 'Pacific/Yap', 'Antarctica/Casey', 'Australia/Broken_Hill', 'America/Guadeloupe', 'Canada/Pacific', 'Europe/Budapest', 'America/Scoresbysund', 'America/Moncton', 'Africa/Lagos', 'America/Guatemala', 'America/Adak', 'Europe/Guernsey', 'Etc/GMT-4', 'Asia/Ulan_Bator', 'Indian/Maldives', 'America/Winnipeg', 'US/Central', 'Pacific/Kiritimati', 'Antarctica/DumontDUrville', 'Europe/Simferopol', 'America/Barbados', 'Africa/Sao_Tome', 'America/Whitehorse', 'America/Goose_Bay', 'America/North_Dakota/Beulah', 'Etc/GMT-3', 'Asia/Bishkek', 'Australia/LHI', 'Mexico/BajaNorte', 'Navajo', 'Australia/Yancowinna', 'America/Atikokan', 'Libya', 'PST8PDT', 'Asia/Baku', 'Asia/Chongqing', 'MET', 'Australia/Lindeman', 'America/Denver', 'America/Halifax', 'Australia/Brisbane', 'Europe/Mariehamn', 'GMT', 'Etc/GMT-10', 'Europe/Oslo', 'America/Detroit', 'Pacific/Pitcairn', 'Iran', 'Europe/Malta', 'Africa/Addis_Ababa', 'Asia/Seoul', 'Asia/Dushanbe', 'America/Costa_Rica', 'Brazil/Acre', 'America/Punta_Arenas', 'Africa/Maputo', 'America/Thule', 'Africa/Bujumbura', 'Europe/Astrakhan', 'ROK', 'America/Argentina/Ushuaia', 'EST5EDT', 'Antarctica/Mawson', 'America/Tegucigalpa', 'America/North_Dakota/New_Salem', 'Europe/Kiev', 'America/Curacao', 'America/Porto_Velho', 'America/Yakutat', 'Asia/Ashkhabad', 'Europe/Dublin', 'America/Buenos_Aires', 'Pacific/Enderbury', 'Europe/Helsinki', 'America/Nassau', 'America/Campo_Grande', 'America/Eirunepe', 'America/Jujuy', 'America/Boise', 'Asia/Ulaanbaatar', 'Poland', 'Africa/Ouagadougou', 'Pacific/Fiji', 'America/Chicago', 'America/Catamarca', 'Africa/Maseru', 'America/Cayman', 'Etc/GMT-9', 'America/Pangnirtung', 'Asia/Gaza', 'Asia/Qyzylorda', 'Europe/Prague', 'Etc/UCT', 'Asia/Pontianak', 'America/Lima', 'US/Eastern', 'Etc/GMT+4', 'Antarctica/Syowa', 'Africa/Kigali', 'America/Juneau', 'America/Miquelon', 'Asia/Singapore', 'America/Sitka', 'Etc/GMT+5', 'Europe/Bucharest', 'America/Cancun', 'America/St_Thomas', 'Pacific/Palau', 'America/Paramaribo', 'Atlantic/Bermuda', 'America/Shiprock', 'Africa/Libreville', 'Indian/Comoro', 'Etc/GMT+1', 'Asia/Qatar', 'Etc/GMT', 'Asia/Kuching', 'Etc/GMT-2', 'Asia/Istanbul', 'Etc/GMT+10', 'America/Montevideo', 'Asia/Kuwait', 'US/Arizona', 'Africa/Niamey', 'Asia/Novosibirsk', 'Pacific/Kwajalein', 'Antarctica/Davis', 'America/Dominica', 'PRC', 'Africa/Lubumbashi', 'Africa/Dakar', 'America/Puerto_Rico', 'Europe/Vaduz', 'Asia/Thimphu', 'US/Aleutian', 'America/Martinique', 'Africa/Luanda', 'Pacific/Nauru', 'Europe/Ljubljana', 'Pacific/Chuuk', 'Africa/Mogadishu', 'America/Rio_Branco', 'Asia/Aqtobe', 'EST', 'Kwajalein', 'Atlantic/Faroe', 'Europe/Zagreb', 'Etc/GMT+6', 'Africa/Kampala', 'Asia/Calcutta', 'GMT+0', 'America/New_York', 'America/Menominee', 'Antarctica/Macquarie', 'America/Antigua', 'Pacific/Bougainville', 'America/Creston', 'Asia/Almaty', 'America/Toronto', 'Asia/Macau', 'Africa/Gaborone', 'America/Indiana/Marengo', 'Europe/Nicosia', 'America/Indiana/Petersburg', 'America/Ojinaga', 'Asia/Barnaul', 'Universal', 'Antarctica/Vostok', 'Asia/Yerevan', 'Canada/Saskatchewan', 'America/El_Salvador', 'Europe/Podgorica', 'America/Blanc-Sablon', 'Atlantic/Canary', 'GB-Eire', 'Pacific/Chatham', 'Europe/Luxembourg', 'America/Rosario', 'Europe/Istanbul', 'Europe/Isle_of_Man', 'Etc/GMT-13', 'Europe/Chisinau', 'Europe/Paris', 'Asia/Yakutsk', 'Asia/Ho_Chi_Minh', 'Asia/Ust-Nera', 'Etc/GMT-1', 'Africa/Juba', 'Europe/Rome', 'Africa/Banjul', 'America/Ensenada', 'America/Vancouver', 'Etc/GMT+3', 'Pacific/Rarotonga', 'Asia/Kuala_Lumpur', 'Pacific/Port_Moresby', 'Pacific/Guadalcanal', 'Asia/Riyadh', 'America/Santo_Domingo', 'Africa/Tunis', 'Asia/Anadyr', 'Asia/Chungking', 'Asia/Nicosia', 'America/Hermosillo', 'America/Dawson_Creek', 'Asia/Jerusalem', 'Atlantic/Faeroe', 'America/Argentina/Salta', 'Asia/Tbilisi', 'Asia/Dacca', 'Asia/Irkutsk', 'US/East-Indiana', 'Europe/Berlin', 'Etc/GMT+11', 'Antarctica/Palmer', 'Africa/Casablanca', 'Indian/Cocos', 'America/Indiana/Vincennes', 'America/Porto_Acre', 'America/St_Vincent', 'Europe/Bratislava', 'Pacific/Samoa', 'Asia/Rangoon', 'America/Panama', 'America/Argentina/ComodRivadavia', 'Etc/Universal', 'America/Godthab', 'Europe/Skopje', 'Asia/Vladivostok', 'America/Montreal', 'America/Monterrey', 'Pacific/Noumea', 'UCT', 'Pacific/Gambier', 'Africa/Brazzaville', 'Africa/Mbabane', 'Asia/Omsk', 'America/Matamoros', 'Etc/GMT+7', 'Antarctica/South_Pole', 'Mexico/General', 'Europe/Uzhgorod', 'Africa/Windhoek', 'America/Anguilla', 'America/Resolute', 'MST', 'Pacific/Auckland', 'Pacific/Guam', 'Europe/Moscow', 'Indian/Christmas', 'America/St_Barthelemy', 'Europe/Warsaw', 'Atlantic/Jan_Mayen', 'America/Los_Angeles', 'Jamaica', 'Pacific/Tongatapu', 'Africa/Harare', 'Asia/Tashkent', 'America/Iqaluit', 'Asia/Amman', 'W-SU', 'Asia/Baghdad', 'America/Tortola', 'Europe/Andorra', 'America/Nome', 'NZ', 'America/Cuiaba', 'Pacific/Pohnpei', 'Asia/Hong_Kong', 'America/Boa_Vista', 'Africa/Asmara', 'US/Hawaii', 'MST7MDT', 'America/Bahia_Banderas', 'Brazil/DeNoronha', 'Etc/GMT-0', 'US/Alaska', 'America/Maceio', 'America/Chihuahua', 'Asia/Tel_Aviv', 'America/Argentina/Buenos_Aires', 'Europe/Tiraspol', 'Australia/Melbourne', 'US/Mountain', 'America/Inuvik', 'Egypt', 'America/Indiana/Knox', 'Pacific/Galapagos', 'Asia/Krasnoyarsk', 'Africa/Ndjamena', 'Africa/Bangui', 'Factory', 'Asia/Kathmandu', 'America/Indiana/Winamac', 'Antarctica/McMurdo', 'US/Indiana-Starke', 'Asia/Bangkok', 'America/North_Dakota/Center', 'Africa/Conakry', 'Europe/Tallinn', 'Atlantic/Stanley', 'Asia/Aden', 'America/Recife', 'Europe/Samara', 'Africa/Khartoum', 'Asia/Yangon', 'America/Port-au-Prince', 'America/Argentina/La_Rioja', 'Zulu', 'Etc/GMT-5', 'Etc/GMT+12', 'Europe/Sarajevo', 'Europe/Athens', 'Pacific/Truk', 'Etc/GMT+8', 'Cuba', 'America/Marigot', 'Asia/Saigon', 'Asia/Taipei', 'Etc/GMT+9', 'Africa/Bamako', 'Singapore', 'Africa/Dar_es_Salaam', 'Asia/Makassar', 'Europe/Copenhagen', 'Eire', 'Europe/London', 'America/Nipigon', 'Chile/Continental', 'Asia/Atyrau', 'Asia/Tehran', 'Australia/Tasmania', 'Canada/Newfoundland', 'Pacific/Efate', 'America/Fort_Nelson', 'Africa/Cairo', 'Europe/Lisbon', 'America/Port_of_Spain', 'America/St_Lucia', 'Asia/Hovd', 'ROC', 'Asia/Jakarta', 'Turkey', 'Africa/Tripoli', 'Pacific/Norfolk', 'Pacific/Ponape', 'Australia/Eucla', 'Antarctica/Troll', 'Africa/Douala', 'America/Fortaleza', 'Canada/Atlantic', 'America/Asuncion', 'US/Pacific', 'America/Glace_Bay', 'America/Atka', 'Australia/Victoria', 'Etc/GMT0', 'Africa/Johannesburg', 'Africa/Nouakchott', 'America/Argentina/San_Juan', 'Iceland', 'Africa/Asmera', 'Brazil/East', 'Indian/Mayotte', 'Asia/Qostanay', 'Asia/Damascus', 'America/Aruba', 'Hongkong', 'Australia/ACT', 'Australia/Canberra', 'Indian/Antananarivo', 'Pacific/Honolulu', 'America/Metlakatla', 'Africa/Abidjan', 'Asia/Dhaka', 'Asia/Magadan', 'America/Caracas', 'Europe/Amsterdam', 'America/Guyana', 'America/Cambridge_Bay', 'America/Thunder_Bay', 'Pacific/Tarawa', 'Europe/Zurich', 'Asia/Kamchatka', 'America/Indiana/Indianapolis', 'Asia/Vientiane', 'CST6CDT', 'Asia/Aqtau', 'Europe/Belgrade', 'America/Kralendijk', 'Asia/Ashgabat', 'Mexico/BajaSur', 'Asia/Samarkand', 'Africa/Lusaka', 'Africa/Algiers', 'America/Yellowknife', 'Australia/Darwin', 'America/Manaus', 'Pacific/Easter', 'Indian/Mauritius', 'Canada/Mountain', 'Africa/Lome', 'Europe/Brussels', 'America/Argentina/Mendoza', 'Indian/Chagos', 'America/Rainy_River', 'Atlantic/South_Georgia', 'Asia/Bahrain', 'Australia/Adelaide', 'America/La_Paz', 'Europe/Ulyanovsk', 'America/Montserrat', 'Europe/Jersey', 'America/Anchorage', 'America/Merida', 'Etc/GMT-14', 'Asia/Kashgar', 'Asia/Tomsk', 'America/Cordoba', 'GMT-0', 'America/Bogota', 'Asia/Manila', 'Asia/Dubai', 'America/Tijuana', 'America/Edmonton', 'Europe/Vienna', 'America/St_Kitts', 'Europe/Minsk', 'America/Indianapolis', 'Atlantic/Madeira', 'America/Argentina/Rio_Gallegos', 'Europe/Saratov', 'America/Grenada', 'Canada/Eastern', 'Pacific/Majuro', 'Asia/Brunei', 'Africa/Blantyre', 'Africa/Porto-Novo', 'Australia/Hobart', 'America/Guayaquil', 'Africa/Nairobi', 'America/Cayenne', 'Europe/Belfast', 'Pacific/Midway', 'GMT0', 'Indian/Mahe', 'Pacific/Pago_Pago', 'Pacific/Tahiti', 'Asia/Phnom_Penh', 'Pacific/Johnston', 'Asia/Jayapura', 'Africa/Timbuktu', 'America/Bahia', 'Africa/Djibouti', 'Canada/Yukon', 'America/Jamaica', 'Europe/Volgograd', 'Asia/Famagusta', 'Pacific/Wake', 'NZ-CHAT', 'Europe/Sofia', 'Africa/El_Aaiun', 'Europe/Vilnius', 'America/Swift_Current', 'Greenwich', 'Europe/Monaco', 'Asia/Khandyga', 'Etc/GMT-12', 'America/Louisville', 'Atlantic/St_Helena', 'Pacific/Apia', 'Australia/North', 'America/Havana', 'Asia/Macao', 'America/Virgin', 'America/Noronha', 'Asia/Dili', 'Atlantic/Reykjavik', 'Asia/Yekaterinburg', 'America/Belem', 'Australia/Currie', 'Africa/Kinshasa', 'Australia/NSW', 'Pacific/Fakaofo', 'America/Knox_IN', 'Etc/GMT-6', 'Asia/Oral', 'America/Managua', 'America/Argentina/Catamarca', 'America/Santarem', 'Asia/Chita', 'America/Kentucky/Louisville', 'HST', 'Australia/West', 'America/Belize', 'America/Argentina/Tucuman', 'America/Nuuk', 'Asia/Novokuznetsk', 'Europe/Kirov', 'America/Phoenix', 'Etc/GMT+0', 'GB', 'Asia/Urumqi', 'Arctic/Longyearbyen', 'Australia/Lord_Howe', 'Etc/Greenwich', 'Australia/South', 'Japan', 'Atlantic/Cape_Verde', 'America/Dawson', 'Africa/Freetown', 'America/Danmarkshavn', 'America/St_Johns', 'Asia/Srednekolymsk', 'America/Indiana/Vevay', 'America/Argentina/Jujuy', 'Africa/Accra', 'America/Indiana/Tell_City', 'WET', 'Asia/Ujung_Pandang', 'Indian/Reunion', 'Asia/Thimbu', 'Chile/EasterIsland', 'Pacific/Niue', 'America/Lower_Princes', 'Asia/Colombo', 'Pacific/Marquesas', 'America/Mendoza', 'Israel', 'Asia/Katmandu', 'Canada/Central', 'Europe/San_Marino', 'UTC', 'Africa/Malabo', 'Etc/UTC', 'Asia/Muscat', 'America/Argentina/Cordoba', 'Europe/Riga', 'America/Araguaina', 'America/Argentina/San_Luis', 'America/Grand_Turk', 'America/Kentucky/Monticello', 'Asia/Beirut', 'Etc/GMT-8', 'America/Regina', 'Europe/Busingen', 'Asia/Hebron'}

>>> datetime.now(tz=ZoneInfo("Asia/Kolkata"))
datetime.datetime(2020, 10, 11, 11, 21, 46, 31856, tzinfo=zoneinfo.ZoneInfo(key='Asia/Kolkata'))

>>> datetime.now(tz=ZoneInfo("Hongkong"))
datetime.datetime(2020, 10, 11, 13, 52, 33, 186045, tzinfo=zoneinfo.ZoneInfo(key='Hongkong'))

The release of a new Python version is a big milestone for the community. You may not be able to start using the cool new features immediately, but in a few years Python 3.9 will be as widespread as Python 3.6 is today.

Build a Text to Speech Service with Python Flask Framework

Siddhesh Shankar — Sat, 26 Sep 2020 17:15:23 +0000

Text-to-speech technology reads aloud digital text. It can take words on computers, smartphones, tablets etc and convert them into audio. Python comes with a lot of handy and easily accessible libraries and I am going to show you how we can deliver text-to-speech with Python using pyttsx3. pyttsx3 is a text-to-speech conversion library in Python. Unlike alternative libraries, it works offline, and is compatible with both Python 2 and 3. It is easy to use the library which converts the text entered, into an audio.

Checkout the video

Steps:

Application invokes the pyttsx3.init() factory function to get a reference to a pyttsx3.Engine instance.
getProperty('rate') method helps you to set the current speaking rate.
Similarly, getProperty('volume') helps you to set the volume (min=0 and max=1)
voice is string identifier of the active voice.
runAndWait() blocks while processing all currently queued commands. Invokes callbacks for engine notifications appropriately. Returns when all commands queued before this call are emptied from the queue. ## Code for converting Text to Speech ```Javascript

import pyttsx3

def text_to_speech(text, gender):
"""
Function to convert text to speech
:param text: text
:param gender: gender
:return: None
"""
voice_dict = {'Male': 0, 'Female': 1}
code = voice_dict[gender]

engine = pyttsx3.init()

# Setting up voice rate
engine.setProperty('rate', 125)

# Setting up volume level  between 0 and 1
engine.setProperty('volume', 0.8)

# Change voices: 0 for male and 1 for female
voices = engine.getProperty('voices')
engine.setProperty('voice', voices[code].id)

engine.say(text)
engine.runAndWait()

1. Create a text that you want to convert into audio.
2. Select voice assistant.
```Javascript


text = 'Hello ! My name is Siddhesh.'
gender = 'Male'  # Voice assistant 
text_to_speech(text, gender)

Building a Web Service

This is a basic flask application with the default route. We also need to import specific flask libraries and a few other libraries as shown below:



# Importing the necessary Libraries
from flask_cors import cross_origin
from flask import Flask, render_template, request
from main import text_to_speech

app = Flask(__name__)


@app.route('/', methods=['POST', 'GET'])
@cross_origin()
def homepage():
    if request.method == 'POST':
        text = request.form['speech']
        gender = request.form['voices']
        text_to_speech(text, gender)
        return render_template('frontend.html')
    else:
        return render_template('frontend.html')


if __name__ == "__main__":
    app.run(port=8000, debug=True)

HTML Template Code



<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>Browser voices</title>
    <style type="text/css">
                            * {
                      box-sizing: border-box;
                    }
                    html,
                    body {
                      min-height: 100vh;
                      margin: 0;
                      padding: 0;
                    }
                    body {
                      font-family: Helvetica, Arial, sans-serif;
                      color: #0d122b;
                      display: flex;
                      flex-direction: column;
                      padding-left: 1em;
                      padding-right: 1em;
                    }
                    h1 {
                      text-align: center;
                      font-weight: 100;
                    }
                    header {
                      border-bottom: 1px solid #0d122b;
                      margin-bottom: 2em;
                    }
                    main {
                      flex-grow: 2;
                      display: flex;
                      justify-content: space-around;
                      align-items: center;
                      background-color: #fff;
                      border-radius: 12px;
                      margin-bottom: 2em;
                    }
                    @keyframes bg-pulse {
                      0% {
                        background-color: #fff;
                      }

                      50% {
                        background-color: #c7ecee;
                      }

                      100% {
                        backgrouond-color: #fff;
                      }
                    }
                    main.speaking {
                      animation: bg-pulse 1.5s alternate ease-in-out infinite;
                    }
                    .input {
                      text-align: center;
                      width: 100%;
                    }
                    label {
                      display: block;
                      font-size: 18px;
                      margin-bottom: 1em;
                    }
                    .field {
                      margin-bottom: 2em;
                    }
                    input {
                      font-size: 24px;
                      padding: 0.5em;
                      border: 1px solid rgba(13, 18, 43, 0.25);
                      border-radius: 6px;
                      width: 75%;
                      display: inline-block;
                      transition: border-color 0.25s;
                      text-align: center;
                    }
                    input:focus,
                    select:focus {
                      border-color: rgba(13, 18, 43, 1);
                    }
                    select {
                      width: 75%;
                      font-size: 24px;
                      padding: 0.5em;
                      border: 1px solid rgba(13, 18, 43, 0.25);
                      border-radius: 6px;
                      transition: border-color 0.25s;
                    }
                    button {
                      font-size: 18px;
                      font-weight: 200;
                      padding: 1em;
                      width: 200px;
                      background: transparent;
                      border: 4px solid #f22f46;
                      border-radius: 4px;
                      transition: all 0.4s ease 0s;
                      cursor: pointer;
                      color: #f22f46;
                      margin-bottom: 2em;
                    }
                    button:hover,
                    button:focus {
                      background: #f22f46;
                      color: #fff;
                    }

                    a {
                      color: #0d122b;
                    }
                    .error {
                      color: #f22f46;
                      text-align: center;
                    }
                    footer {
                      border-top: 1px solid #0d122b;
                      text-align: center;
                    }
    </style>
  </head>
      <body>

          <header>
              <h1>Text to Speech</h1>
          </header>

        <main>
          <form class="input" id="voice-form" method="post">
                <div class="field">
                    <label for="speech">Type Something</label>
                    <input type="text" name="speech" id="speech" required />
                </div>
                <div class="field">
                    <label for="voices">Choose a Voice Assistant</label>
                      <select name="voices" id="voices">
                          <option value="Male">Male</option>
                          <option value="Female">Female</option>
                      </select>
                </div>
              <button>
                    Say it!
              </button>
          </form>
        </main>

        <footer>
            <p>
                Built by <a href=<BLOG LINK>><YOUR NAME></a>
            </p>
        </footer>

      </body>
</html>

There are several APIs available to convert text to speech in python. gTTS can also be used for doing the same task. It is python library and CLI tool to interface with Google translate's text-to-speech API.

I hope you found this post useful. This will help end-users to build a web application using flask and convert text to speech.

Octocat will take you to my GitHub repository...

Deploying a Flask application on Heroku

Siddhesh Shankar — Sat, 12 Sep 2020 13:16:30 +0000

I my previous post I showed how to deploy a simple Machine learning model using flask into heroku. In this post I will show you how to deploy a web-scrapper flask app. I have created a simple python-flask API for fetching financial statement of a company by searching the ticker.

What is a Ticker ?

A ticker symbol is an abbreviation generally represented as a collection of letters that is used to identify a publicly traded security. Ticker symbols vary depending on what stock market they are traded on. They can consist of letters, numbers, or a combination of both. Sometimes they are also called stock symbols.

Steps involved:

Create a web scrapper.
Create a web app using flask.
Heroku hosting.

Web Scrapping

I am scrapping the data from Yahoo Finance. The python script will then send a HTTP request and will search for the given ticker in the website, the output will be a simple HTML response. This can be achieved using Beautiful Soup which is a Python library for getting data out of HTML, XML, and other markup languages. It downloads that page’s source code, just as a browser would, but instead of displaying the page visually, it filters through the page looking for HTML elements.

import pandas as pd
from bs4 import BeautifulSoup
from urllib.request import urlopen, Request


def search_company(search_string):
    url = 'https://finance.yahoo.com/quote/' + search_string + '/financials?p=' + search_string
    client = Request(url)
    response = urlopen(client).read()
    html = BeautifulSoup(response, "html.parser")
    return html


def company_financials(string_search):
    headers, temp_list, label_list, final = ([] for i in range(4))
    index = 0

    html_content = search_company(string_search)
    features = html_content.find_all('div', class_='D(tbr)')

    for item in features[0].find_all('div', class_='D(ib)'):
        headers.append(item.text)

    while index <= len(features) - 1:
        temp = features[index].find_all('div', class_='D(tbc)')
        for line in temp:
            temp_list.append(line.text)
        final.append(temp_list)
        temp_list = []
        index += 1
    df = pd.DataFrame(final[1:])
    df.columns = headers
    df.rename(columns={'ttm': 'Trailing 12 months'}, inplace=True)
    df.style.set_properties(subset=["Breakdown"], **{'text-align': 'left'})
    return df

Flask Application

I have the application code hosted on my Github account. I have added all the html templates for reference.
I have used pretty html table python package for creating HTML table. Please checkout my first blog on how to create HTML table using pretty html table package.

Convert a Dataframe into a pretty HTML table and send it over Email

Siddhesh Shankar ・ Jul 25 ・ 2 min read

#python #devops #html

from bs4 import BeautifulSoup
from flask_cors import cross_origin
from supp import company_financials
from pretty_html_table import build_table
from flask import Flask, render_template, request

app = Flask(__name__)


@app.route('/', methods=['POST', 'GET'])
@cross_origin()
def index():
    if request.method == 'POST':
        search_string = request.form['content'].replace(" ", "")
        content = company_financials(search_string)
        table = build_table(content, 'blue_light', font_size='small', font_family='Lucida Grande', text_align='right')
        financials = BeautifulSoup(table, 'html.parser')
        return render_template('main.html', tables=financials, company=search_string.upper())
    else:
        return render_template('index.html')


if __name__ == "__main__":
    app.run(port=8000, debug=True)

In order for us to successfully deploy any application to Heroku, we must add a Procfile to that application. Heroku apps include a Procfile that specifies the commands that are executed by the app on startup.

First install Gunicorn

pip install gunicorn

Update requirements.txt

pip freeze > requirements.txt

beautifulsoup4==4.9.1
bs4==0.0.1
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
Flask==1.1.2
Flask-Cors==3.0.9
gunicorn==20.0.4
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
numpy==1.19.2
pandas==1.1.2
pretty-html-table==0.9.dev0
pymongo==3.11.0
python-dateutil==2.8.1
pytz==2020.1
requests==2.24.0
six==1.15.0
soupsieve==2.0.1
urllib3==1.25.10
Werkzeug==1.0.1

Create a Procfile

web: gunicorn app:app

Commit your code GitHub and connect Heroku to GitHub

After you connect, there are 2 ways to deploy your app. You could either choose automatic deploy or manual deploy. Automatic deployment will take place whenever you commit anything into your github repository. Automatically the build will start. I have deployed it using manual deploy.
Just by selecting the branch and clicking on deploy, build will start. After successful deployment, deployment tab should look as shown below:

Heroku Hosting

After successful deployment, app will be created. Click on the view button and your app should open. An API will be created.

Checkout my app: https://get-company-financials.herokuapp.com/

I hope you found this post useful. This will help end-users to create a web scrapper and deploy it using flask.

Octocat will take you to my GitHub repository...

TextHero: New library for handling text data.

Siddhesh Shankar — Thu, 20 Aug 2020 02:55:19 +0000

Texthero is a python toolkit to work with text-based dataset quickly and effortlessly. Texthero is very simple to learn and designed to be used on top of Pandas. Texthero has the same expressiveness and power of Pandas and is extensively documented. Texthero is modern and conceived for programmers of the 2020 decade with little knowledge if any in linguistic.
Tutorial link: https://analyticsindiamag.com/texthero-guide-a-python-toolkit-for-text-processing/?utm_source=rss&utm_medium=rss&utm_campaign=texthero-guide-a-python-toolkit-for-text-processing

Deploy Machine Learning models using Flask into Heroku

Siddhesh Shankar — Sat, 08 Aug 2020 14:49:46 +0000

This post will help you understand how to deploy a machine learning model on the web using Flask. I have used heroku to deploy the ML model.

What is Heroku ?

Heroku is a multi-language cloud application platform that enables developers to deploy, scale, and manage their applications. Heroku is elegant, flexible, and easy to use, offering developers the simplest path to getting their apps to market. Salesforce acquired Heroku in 2010, spending $212 million on the cloud computing outfit.

Follow the steps mentioned below:

Train ML model
Create a web app using Flask
Commit the code to GitHub
Connect GitHub to Heroku
Deploy the model

Train the Machine Learning model

I am using logistic regression as the classification algorithm. I have used Framingham Heart study dataset to predict the risk of developing a heart disease. The objective of the Framingham Study is to study the incidence and prevalence of cardiovascular disease (CVD) and its risk factors.

import pandas as pd
import numpy as np
import pickle
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Reading dataframe and dropping rows with na values
data = pd.read_csv("framingham.csv")
data.dropna(inplace=True)  # Consists 3658 records

# Computing Correlation
corr_matrix = data.corr().abs()
high_corr_var = np.where(corr_matrix > 0.35)
high_corr_var = [(corr_matrix.index[x], corr_matrix.columns[y]) for x, y in zip(*high_corr_var) if x != y and x < y]

"""
Variables to consider
age: Age of a person (Input a number)
smoker: Yes or No
Cigs per day: (Input a number)
diabaties: Yes or No
bmi: weight(Kg) and height(meters) calculate
BP: input a number
"""


def bmi(weight, height):
    return round(float(weight) / (float(height) * float(height)), 2)


X_cols = ['male', 'age', 'currentSmoker', 'cigsPerDay', 'diabetes',
          'sysBP', 'BMI']
Y_col = ['TenYearCHD']

X_vars = data[X_cols]
Y_var = data[Y_col]

# Renaming Columns
X_vars.columns = ['Gender', 'Age', 'Smoker', 'Cigarettes_Per_Day',
                  'Diabetic', 'BP', 'BMI']
Y_var.columns = ['Chances_of_hear_disease']

# Splitting data
X_train, X_test, y_train, y_test = train_test_split(X_vars, Y_var, test_size=0.25, random_state=0)

# Initiate the Model
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X_train, y_train)

pickle.dump(logreg, open('model.pkl', 'wb'))

I have used pickle library, which basically helps you to dump your model in the form character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script. This will create model.pkl file in write bytes mode which will later be used to deploy the model.

Create a web app using Flask

Below I have defined the app routes and completing the app.py file. I have created a Index.html which will serve as the home page, where we will have all the field required to run the model.
First step is to setup a secret key which is needed to keep the client-side sessions secure. When working with sessions in Flask you need to create a secret_key comprised of random bytes represented in hexadecimal.

# Set the secret key to some random bytes. 
# Keep this really secret!
app.secret_key = b'_5#y2L"F4Q8z\n\xec]/'

Dataform class is defining the structure of the page comprising of radio buttons, checkboxes and labels. This will help us to capture the data in proper manner. The last step is to render the index.html template and take inputs from the user. When the user submits the inputs, index function will be triggered and the app will start computing the final output.

import os
import pickle
from flask_wtf import FlaskForm
from flask import Flask, render_template
from wtforms.validators import DataRequired
from wtforms import IntegerField, BooleanField, FloatField, SubmitField, RadioField


class Config(object):
    SECRET_KEY = os.environ.get('SECRET_KEY') or 'you-will-never-guess'
    APP_PATH = os.path.dirname(__file__)


app = Flask(__name__)
app.config.from_object(Config)


class DataForm(FlaskForm):
    gender = RadioField('Gender', validators=[DataRequired()], choices=['Male', 'Female'])
    age = IntegerField('Age', validators=[DataRequired()])
    diabetic = BooleanField('Are you Diabetic?')
    smoker = BooleanField('Do you smoke? ')
    cig_count = FloatField('How many cigarettes do you smoke per day ?', default=0)
    weight = FloatField('Weight (Kg)', validators=[DataRequired()])
    height = FloatField('Height (cm)', validators=[DataRequired()])
    bp = IntegerField('Blood Pressure (mmHg)  Normal range : (90-120)/(60-80)', validators=[DataRequired()])
    submit = SubmitField('Submit')


@app.route('/')
@app.route('/index', methods=['GET', 'POST'])
def index():
    form = DataForm()
    if form.validate_on_submit():
        li = ['age', 'gender', 'cig_count', 'diabetic', 'height', 'weight', 'smoker', 'bp']
        data = {}
        for ele in li:
            data[ele] = eval('form.' + ele + '.data')

        gender = 1 if data['gender'] == 'Male' else 0
        smoker = 1 if data['smoker'] else 0
        diabetic = 1 if data['diabetic'] else 0
        filename = os.path.join(app.config['APP_PATH'], 'model.pkl')
        with open(filename, 'rb') as f:
            model = pickle.load(f)
            value = model.predict([[gender, data['age'], smoker, data['cig_count'],
                                    diabetic, data['bp'], bmi(data['weight'], data['height'])]])
            data = ['done', value[0]]
        return render_template('index.html', value=data, form=form, data=data)
    return render_template('/index.html', form=form)


def bmi(weight, height):
    return round(float(weight) / (float(height / 100) * float(height / 100)), 2)


if __name__ == '__main__':
    app.run(debug=True)

Most important thing is to create a Procfile and requirement.txt, which handles the configuration part in order to deploy the model into heroku server. web: gunicorn is the fixed command, after that the first parameter is app.py file i.e the file which will be executed first. Provide the first parameter without the file extension. Second parameter is the flask app name.
Requirements consists of all the libraries that has to get installed in heroku environment.

web: gunicorn app:app

click==7.1.2
Flask==1.1.2
Flask-WTF==0.14.3
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.2
joblib==0.16.0
MarkupSafe==1.1.1
numpy==1.19.1
pandas==1.1.0
python-dateutil==2.8.1
python-dotenv==0.14.0
pytz==2020.1
scikit-learn==0.23.2
scipy==1.5.2
six==1.15.0
sklearn==0.0
threadpoolctl==2.1.0
Werkzeug==1.0.1
WTForms==2.3.3

Commit your code GitHub and connect Heroku to GitHub

Create a app name in heroku and connect your github repository as shown below.

Final app

After successful deployment, app will be created. Click on the view and your app should open. A new url will be created.

Checkout my app: https://heart-disease-predictor-api.herokuapp.com/

I hope you found this post useful. This will help end-users to interact with our ML models without any external dependency.

Done!

Octocat will take you to my GitHub repository...

Convert a Dataframe into a pretty HTML table and send it over Email

Siddhesh Shankar — Sat, 25 Jul 2020 13:39:34 +0000

My first post in dev.to.
I have learnt how to automate the creation of nice html tables using pretty-html-table package. This package is embedding very nicely with other packages used to send emails.
In this post I am going to cover:

Sending Email using SMTP
Convert pandas dataframe to pretty HTML table
Sending the email with HTML content

Setup SMTP to send Email

Python comes with the built-in smtplib module for sending emails using the Simple Mail Transfer Protocol (SMTP). I have used Gmail SMTP server to send emails, but the same principles apply to other email services. Google’s Gmail SMTP server is a free SMTP service which anyone who has a Gmail account can use to send emails.

Gmail SMTP server address: smtp.gmail.com
Gmail SMTP username: Your Gmail address
Gmail SMTP password: Your Gmail password (Unique 12 letter password). Click here to set your password
Gmail SMTP port (TLS): 587 ```

from smtplib import SMTP
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

def send_mail(body):

message = MIMEMultipart()
message['Subject'] = 'Top 5 Economies of the World!'
message['From'] = '<sender>@gmail.com'
message['To'] = '<receiver>@gmail.com'

body_content = body
message.attach(MIMEText(body_content, "html"))
msg_body = message.as_string()

server = SMTP('smtp.gmail.com', 587)
server.starttls()
server.login(message['From'], 'Unique password')
server.sendmail(message['From'], message['To'], msg_body)
server.quit()

### <u>Create a Dataframe</u>

import pandas as pd

def get_gdp_data():
"""
GDP data
:return:
"""
gdp_dict = {'Country': ['United States', 'China', 'Japan', 'Germany', 'India'],
'GDP': ['$21.44 trillion', '$14.14 trillion', '$5.15 trillion', '$3.86 trillion', '$2.94 trillion']}
data = pd.DataFrame(gdp_dict)
return data

### <u>Convert pandas dataframe to HTML table</u>
Pretty HTML table package integrates very well with other python packages used to send emails. Just add the result of this package to the body of the email.

from send_email import send_mail
from get_data import get_gdp_data
from pretty_html_table import build_table

def send_country_list():
gdp_data = get_gdp_data()
output = build_table(gdp_data, 'blue_light')
send_mail(output)
return "Mail sent successfully."

send_country_list()

Done!
Octocat will take you to my GitHub repository...
[![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9vq4d02xo1f1m7jwi8gh.png)](https://github.com/siddheshshankar/pyHTML)