Forem: James Shah

OpportuniTrack - Job Application Tracker powered by Nylas

James Shah — Sun, 01 Sep 2024 17:04:39 +0000

This is a submission for the Nylas Challenge: Galaxy Brain.

What I Built and Why

As someone who has been through the job hunt process myself, I know how exhausting it is. Especially in this market, candidates are applying to hundreds of jobs every day and it gets very overwhelming to keep track of each application. That's when I decided to build an app to automate this process!

Presenting to you OpportuniTrack - Track your next big opportunity, effortlessly! (Join the waitlist now to get early access!)

OpportuniTrack automatically tracks job applications by analyzing the user's emails and extracting information such as Company Name, Job Position, and Status of application (Applied, Rejected, Offered, etc.) from the job-related emails(predicted by the model)

Unlike other job tracking tools, OpportuniTrack uniquely combines real-time email analysis with instant notifications, ensuring you stay on top of every opportunity without lifting a finger.

Okay, enough talk. Let's see it in action!

Demo

Code

jamesshah / opportunitrack

OpportuniTrack is an automated job application tracker app

OpportuniTrack - Track Your Next Opportunity, Effortlessly.

OpportuniTrack is an automated job application tracker using email.

Job hunt is exhausting (especially in this market!), and it's essential to keep track of hundreds of applications you submit everyday.

OpportuniTrack saves hours of manual work of keeping track of job applications everyday by automating it using your email

Features

Track applications automatically by connecting your email
Get notified whenever there is an update on your application
Add or edit applications manually
More features coming soon!

Architecture

Note

OpportuniTrack is powered by Nylas API to securely analyze your emails
We only access emails that is likely to be a job application related email (predicted by our Machine Learning Model)
We don't store your email in our database, we only extract job application related information and store it

View on GitHub

My Journey

Building OpportuniTrack was not as easy as I initially thought. As I've never developed a mobile app before, my first intuition was to build a web app. But with web app, I would not be able to give real-time update to user which is essential for this app since a job application update is a time-sensitive notification for users.

Now that I had decided to build a mobile app, the next challenge was to access the user's emails easily and securely. Nylas really shined here and with it's unified API for all different providers, I didn't have to worry about going through API documentation and setup process of various different providers.

With these tools and a lot of excitement, I started building OpportuniTrack.

I started with setting up Nylas and started exploring the APIs and features. Below are three 3 features that I used in this app, and I am planning to add more in future!

Auth API to get access to user emails
Email API to access user's past emails
Webhooks to get notified whenever user receives an email

Setting up a Node.js server and getting emails through Email API and Webhooks was really easy with Nylas.

Now the next task was to implement the brain of the application. The email analysis engine which would find the job application related email, extract information from it and finding the status of the application.

Initially, I thought of using a Large Language Model to do this, but soon realized that it was an over-kill and not cost-efficient at all. So, I decided to use traditional AI solutions - Text Classifier and Named Entity Recognition models.

I used a fine-tuned BART based model for email classification and GLiNER for extracting information from the email. These models are much smaller compared to LLMs and give decent accuracy with small fine-tuning.

With these two components ready, I had to enable communication between these two, since I needed asynchronous processing of the emails (Webhooks expect a response within 10 seconds and it can take more than 10 seconds to completely process the email and send notifications to the user). That's when I decided to use redis for queue-based communication between the server and the processing engine.

Finally, it was time to build the UI and connect all these pieces together to get it up and running. The complete architecture of the application came out to be as below:

Things I learned and am proud of

This is my first iOS app, and getting it together within a few weeks is an achievement for myself
I learned more about distributed systems and the overhead that comes with it
Choosing Redis for queue-based communication was pivotal in ensuring the app's scalability and responsiveness. This choice allowed me to handle asynchronous processing seamlessly, addressing the challenge of webhook response time limitations effectively
Keeping the engineer ego aside and not using LLM just because it is trending and going with the simple solution that would work best for the problem at hand was a decision that will always stay with me

Thank you to @nylas_team and @thepracticaldev for providing the platform to bring OpportuniTrack to life. This challenge has been an incredible journey, and I’m excited about the potential of OpportuniTrack to empower job seekers globally!

Please feel free to ask any questions you have in the comments!

OpportuniTrack - Job Application Tracker Powered by Nylas

James Shah — Sun, 01 Sep 2024 16:52:49 +0000

This is a submission for the Nylas Challenge: AI Expedition.

What I Built and Why

Presenting to you OpportuniTrack - Track your next big opportunity, effortlessly! (Join the waitlist now to get early access!)

It also notifies the users in real-time whenever there is a new job application or an update to an existing job application, so that users never lose any opportunity in a large pile of emails!

Okay, enough talk. Let's see it in action!

Demo

Code

jamesshah / opportunitrack

OpportuniTrack is an automated job application tracker app

OpportuniTrack - Track Your Next Opportunity, Effortlessly.

OpportuniTrack is an automated job application tracker using email.

Job hunt is exhausting (especially in this market!), and it's essential to keep track of hundreds of applications you submit everyday.

OpportuniTrack saves hours of manual work of keeping track of job applications everyday by automating it using your email

Features

Track applications automatically by connecting your email
Get notified whenever there is an update on your application
Add or edit applications manually
More features coming soon!

Architecture

Note

OpportuniTrack is powered by Nylas API to securely analyze your emails
We only access emails that is likely to be a job application related email (predicted by our Machine Learning Model)
We don't store your email in our database, we only extract job application related information and store it

View on GitHub

My Journey

With these tools and a lot of excitement, I started building OpportuniTrack.

I started with setting up Nylas and started exploring the APIs and features. Below are three 3 features that I used in this app, and I am planning to add more in future!

Auth API to get access to user emails
Email API to access user's past emails
Webhooks to get notified whenever user receives an email

Setting up a Node.js server and getting emails through Email API and Webhooks was really easy with Nylas.

Finally, it was time to build the UI and connect all these pieces together to get it up and running. The complete architecture of the application came out to be as below:

Things I learned and am proud of

This is my first iOS app, and getting it together within a few weeks is an achievement for myself
I learned more about distributed systems and the overhead that comes with it
Keeping the engineer ego aside and not using LLM just because it is trending and going with the simple solution that would work best for the problem at hand was a decision that will always stay with me

Finally, I would like to thank @nylas_team and @thepracticaldev for this challenge. It was a catalyst in building this app, and Nylas made this app see a day of light within a few weeks!

Please feel free to ask any questions you have in the comments!

Learning Data Analysis With Python - Pandas DataFrame

James Shah — Fri, 04 Sep 2020 13:33:07 +0000

⚠️ Note - This post is a part of Learning data analysis with python series. If you haven't read the first post, some of the content won't make sense. Check it out here.

In the previous article, we talked about Pandas Series, working with real world data and handling missing values in data. Although series are very useful but most real world datasets contain multiple rows and columns and that's why Dataframes are used much more than series. In this post, we'll talk about dataframe and some operations that we can do on dataframe objects.

What is a DataFrame?

As we looked in the previous post, A Series is a container of scalars,A DataFrame is a container for Series. It's a dictionary like data structure for Series. A DataFrame is similar to a two-dimensional hetrogeneous tabular data(SQL table). A DataFrame is created using many different types of data such as dictionary of Series, dictionary of ndarrays/lists, a list of dictionary, etc. We'll look at some of these methods to create a DataFrame object and then we'll see some operations that we can apply on a DataFrame object to manipulate the data.

DataFrame using dictionary of Series

In[1]:
    d = {
    'col1' : pd.Series([1,2,3], index = ["row1", "row2", "row3"]),
    'col2' : pd.Series([4,5,6], index = ["row1", "row2", "row3"])
    }

    df = pd.DataFrame(d)

Out[1]:
        col1    col2
row1     1       4
row2     2       5
row3     3       6

As shown in above code, the keys of dict of Series becomes column names of the DataFrame and the index of the Series becomes the row name and all the data gets mapped by the row name i.e.,order of the index in the Series doesn't matter.

DataFrame using ndarrays/lists

In[2]:
    d = {
        'one' : [1.,2.,3.],
        'two' : [4.,5.,6.]
    }

    df = pd.DataFrame(d)

Out[2]:
    one  two
0   1.0  4.0
1   2.0  5.0
2   3.0  6.0

As shown in the above code, when we use ndarrays/lists, if we don't pass the index then the range(n) becomes the index of the DataFrame.And while using the ndarray to create a DataFrame, the length of these arrays must be same and if we pass an explicit index then the length of this index must also be of same length as the length of the arrays.

DataFrame using list of dictionaries

In[3]:
    d = [
        {'one': 1, 'two' : 2, 'three': 3},
        {'one': 10, 'two': 20, 'three': 30, 'four': 40}
    ]

    df = pd.DataFrame(d)

Out[3]:
    one two three four
0    1   2   3     NaN
1    10  20  30    40.0


In[4]:
    df = pd.DataFrame(d, index=["first", "second"])

Out[4]:

        one two three four
first    1   2   3     NaN
second   10  20  30    40.0

And finally, as described above we can create a DataFrame object using a list of dictionary and we can provide an explicit index in this method,too.

Although learning to create a DataFrame object using these methods is necessary but in real world, we won't be using these methods to create a DataFrame but we'll be using external data files to load data and manipulate that data. So, let's take a look how to load a csv file and create a DataFrame.In the previous post, we worked with the Nifty50 data to demonstrate how Series works and similarly in this post, we'll load Nifty50 2018 data, but in this dataset we have data of Open, Close, High and Low value of Nifty50. First let's see what this dataset looks like and then we'll load it into a DataFrame.

In[5]:
    df = pd.read_csv('NIFTY50_2018.csv')

Out[5]:
        Date      Open        High        Low        Close
0   31 Dec 2018  10913.20   10923.55    10853.20    10862.55
1   28 Dec 2018  10820.95   10893.60    10817.15    10859.90
2   27 Dec 2018  10817.90   10834.20    10764.45    10779.80
3   26 Dec 2018  10635.45   10747.50    10534.55    10729.85
4   24 Dec 2018  10780.90   10782.30    10649.25    10663.50
...     ...         ...        ...         ...         ...
241 05 Jan 2018  10534.25   10566.10    10520.10    10558.85
242 04 Jan 2018  10469.40   10513.00    10441.45    10504.80
243 03 Jan 2018  10482.65   10503.60    10429.55    10443.20
244 02 Jan 2018  10477.55   10495.20    10404.65    10442.20
245 01 Jan 2018  10531.70   10537.85    10423.10    10435.55

In[6]:
    df = pd.read_csv('NIFTY50_2018.csv', index_col=0)

Out[6]:
                  Open        High        Low        Close
    Date
31 Dec 2018     10913.20    10923.55    10853.20    10862.55
28 Dec 2018     10820.95    10893.60    10817.15    10859.90
27 Dec 2018     10817.90    10834.20    10764.45    10779.80
26 Dec 2018     10635.45    10747.50    10534.55    10729.85
24 Dec 2018     10780.90    10782.30    10649.25    10663.50
     ...           ...        ...         ...          ...
05 Jan 2018     10534.25    10566.10    10520.10    10558.85
04 Jan 2018     10469.40    10513.00    10441.45    10504.80
03 Jan 2018     10482.65    10503.60    10429.55    10443.20
02 Jan 2018     10477.55    10495.20    10404.65    10442.20
01 Jan 2018     10531.70    10537.85    10423.10    10435.55

As shown above, we have loaded the dataset and created a DataFrame called df and looking at the data, we can see that we can set the index of our DataFrame to the Date column and in the second cell we did that by providing the index_col parameter in the read_csv method.

There are many more parameters available in the read_csv method such as usecols using which we can deliberately ask the pandas to only load provided columns, na_values to provide explicit values that pandas should identify as null values and so on and so forth. Read more about all the parameters in pandas documentation.

Now, let's look at some of the basic operations that we can perform on the dataframe object in order to learn more about our data.

In[7]:
    # Shape(Number of rows and columns) of the DataFrame
    df.shape

Out[7]:
    (246,4)

In[8]:
    # List of index
    df.index

Out[8]:
    Index(['31 Dec 2018', '28 Dec 2018', '27 Dec 2018', '26 Dec 2018',
       '24 Dec 2018', '21 Dec 2018', '20 Dec 2018', '19 Dec 2018',
       '18 Dec 2018', '17 Dec 2018',
       ...
       '12 Jan 2018', '11 Jan 2018', '10 Jan 2018', '09 Jan 2018',
       '08 Jan 2018', '05 Jan 2018', '04 Jan 2018', '03 Jan 2018',
       '02 Jan 2018', '01 Jan 2018'],
      dtype='object', name='Date', length=246)

In[9]:
    # List of columns
    df.columns

Out[9]:
    Index(['Open', 'High', 'Low', 'Close'], dtype='object')

In[10]:
    # Check if a DataFrame is empty or not
    df.empty

Out[10]:
    False

It's very crucial to know data types of all the columns because sometimes due to corrupt data or missing data, pandas may identify numeric data as 'object' data-type which isn't desired as numeric operations on the 'object' type of data is costlier in terms of time than on float64 or int64 i.e numeric datatypes.

In[11]:
    # Datatypes of all the columns
    df.dtypes

Out[11]:
    Open     float64
    High     float64
    Low      float64
    Close    float64
    dtype: object

We can use iloc and loc to index and get the particular data from our dataframe.

In[12]:
    # Indexing using implicit index
    df.iloc[0]

Out[12]:
    Open     10913.20
    High     10923.55
    Low      10853.20
    Close    10862.55
    Name: 31 Dec 2018, dtype: float64

In[13]:
    # Indexing using explicit index
    df.loc["01 Jan 2018"]

Out[13]:
    Open     10531.70
    High     10537.85
    Low      10423.10
    Close    10435.55
    Name: 01 Jan 2018, dtype: float64

We can also use both row and column to index and get specific cell from our dataframe.

In[14]:
    # Indexing using both the axes(rows and columns)
    df.loc["01 Jan 2018", "High"]

Out[14]:
    10537.85

We can also perform all the math operations on a dataframe object same as we did on series.

In[15]:
    # Basic math operations
    df.add(10)

Out[15]:
                      Open        High        Low        Close
    Date
31 Dec 2018     10923.20    10933.55    10863.20    10872.55
28 Dec 2018     10830.95    10903.60    10827.15    10869.90
27 Dec 2018     10827.90    10844.20    10774.45    10789.80
26 Dec 2018     10645.45    10757.50    10544.55    10739.85
24 Dec 2018     10790.90    10792.30    10659.25    10673.50
     ...           ...        ...         ...          ...
05 Jan 2018     10544.25    10576.10    10530.10    10568.85
04 Jan 2018     10479.40    10523.00    10451.45    10514.80
03 Jan 2018     10492.65    10513.60    10439.55    10453.20
02 Jan 2018     10487.55    10505.20    10414.65    10452.20
01 Jan 2018     10541.70    10547.85    10433.10    10445.55

We can also aggregate the data using the agg method. For instance, we can get the mean and median values from all the columns in our data using this method as show below.

In[16]:
    # Aggregate one or more operations
    df.agg(["mean", "median"])

Out[16]:
            Open            High            Low            Close
mean    10758.260366    10801.753252    10695.351423    10749.392276
median  10704.100000    10749.850000    10638.100000    10693.000000

However, pandas provide a more convenient method to get a lot more than just minimum and maximum values across all the columns in our data. And that method is describe. As the name suggests, it describes our dataframe by applying mathematical and statistical operations across all the columns.

In[17]:
    df.describe()

Out[17]:
            Open           High            Low             Close
count   246.000000      246.000000      246.000000      246.000000
mean    10758.260366    10801.753252    10695.351423    10749.392276
std     388.216617      379.159873      387.680138      382.632569
min     9968.800000     10027.700000    9951.900000     9998.050000
25%     10515.125000    10558.650000    10442.687500    10498.912500
50%     10704.100000    10749.850000    10638.100000    10693.000000
75%     10943.100000    10988.075000    10878.262500    10950.850000
max     11751.800000    11760.200000    11710.500000    11738.500000

And to get the name, data types and number of non-null values in each columns, pandas provide info method.

In[18]:
    df.info()

Out[18]:
    <class 'pandas.core.frame.DataFrame'>
    Index: 246 entries, 31 Dec 2018 to 01 Jan 2018
    Data columns (total 4 columns):
    #   Column  Non-Null Count  Dtype
    ---  ------  --------------  -----
    0   Open    246 non-null    float64
    1   High    246 non-null    float64
    2   Low     246 non-null    float64
    3   Close   246 non-null    float64
    dtypes: float64(4)
    memory usage: 19.6+ KB

We are working with a small data with less than 300 rows and thus, we can work with all the rows but when we have tens or hundreds of thousand rows in our data, it's very difficult to work with such huge number of data. In statistics, 'sampling' is a technique that solves this problem. Sampling means to choose a small amount of data from the whole dataset such that the sampling dataset contains somewhat similar features in terms of diversity as that of the whole dataset. Now, it's almost impossible to manually select such peculiar rows but as always, pandas comes to our rescue with the sample method.

In[19]:
    # Data Sampling -  Get random n examples from the data.
    df.sample(5)

Out[19]:
              Open        High        Low         Close
Date
04 Jul 2018 10715.00    10777.15    10677.75    10769.90
22 Jun 2018 10742.70    10837.00    10710.45    10821.85
14 Mar 2018 10393.05    10420.35    10336.30    10410.90
09 Jan 2018 10645.10    10659.15    10603.60    10637.00
27 Apr 2018 10651.65    10719.80    10647.55    10692.30

But, executing this method produces different results everytime and that may be unacceptable in some cases. But that can be solved by providing random_state parameter in the sample method to reproduce same result everytime.

As shown above, we can perform many operations on the DataFrame object to get information of the DataFrame and from the DataFrame. These are just basic operations that we can perform on the DataFrame object, there are many more interesting methods and operations that we can perform on the DataFrame object such as pivot , merge , join and many more. Also, in this given dataset, we have time as the index of our DataFrame i.e this is the TimeSeries dataset and pandas also provide many methods to manipulate the TimeSeries data such as rolling_window.

That will be all for this post. In the next post we'll look at some of these methods and we'll perform 5 analysis tasks using these methods. Till then, you can take a look at the pandas documentation and find more information about DataFrame objects and the methods that can be applied on the DataFrame object.

Thank you for reading

Learning Data Analysis With Python - Pandas Series, Missing Values and Real World Data

James Shah — Thu, 16 Jul 2020 10:58:55 +0000

⚠️ This post is a part of Learning data analysis with python series. If you haven't read the first post, some of the content won't make sense. Check it out here.

In the previous article, we talked about pandas library in python, Series data structure, different ways to create a Series object using a list,a dictionary and a numpy array. and how we can apply some functions on it to get some statistical data from it. In this post, we'll talk about how you can create a Series object from an external file(csv), what is loc and iloc and some tasks that we can achieve using Series. Also, we'll see an introduction to the DataFrame object as in real world data, almost in every case we come across tabular data(data with multiple columns and rows) rather than a single column data and thus, we'll be using DataFrame more frequently than Series.

What is loc and iloc?

loc and iloc are probably the most asked topic in the data science interviews and many newcomers and data science aspirants get confused initially about loc and iloc. However, it's not rocket science and if you understand loc and iloc in Series objects, it will be easier for you to understand how to use them for indexing the DataFrame object.

In the last post, we saw that we can give index of our choice for the convenience of accessing data using that index but sometimes we may want to use the default integer indexing that pandas provide (0,1,2,...) to access some data. For example, let's say, we have data consisting 100 rows and you want to get the first 50 rows, we can do this using slicing but now that we have custom indexing, we must know the first and 50th index to get the data. This is just an oversimplified case of such scenarios, as in real world data analysis tasks you may be working on a dataset with thousands or millions of rows. In that case, it is not feasible to find custom index and then access data.
And that's when loc and iloc come-in handy. loc and iloc are builtin pandas method for indexing data from a pandas Series or DataFrame. In plain english loc is label based indexing which means that use name(custom index) of row and column whereas iloc is integer index-based which means we use integer index(default pandas index) of row and column to access the data. Let's look at the code and it will make things more clear.

In[1]:
    s = pd.Series([0.0, 1, 1, 2, 3, 5, 8], index=[1, 2, 3, 4, 5, 6, 7])

Out[1]:
    1    0.0
    2    1.0
    3    1.0
    4    2.0
    5    3.0
    6    5.0
    7    8.0
    dtype: float64

In[2]:
    s.loc[3] 

Out[2]:
    1.0

In[3]:
    s.iloc[3] 

Out[3]:
    2.0

As you can see, in our Series object s, we have given custom index starting from 1 to 7. And then, when we use the loc method on this Series and pass 3 as index, we get the data corresponding to the 3 in our index column. And when we use the iloc method on this Series and again pass 3 as index, we get the data corresponding to the 3 in the default pandas index and as the dafault pandas indexing starts from 0, we get the data from 4th row. We can use the reset_index method to reset the index and get the default interger indexing in our Series. Looking at both the indexing side by side will make things more clear.

In[4]:
    s.reset_index()

Out[4]:

        index   0
    0     1    0.0
    1     2    1.0
    2     3    1.0
    3     4    2.0
    4     5    3.0
    5     6    5.0
    6     7    8.0

As you can see, the first unnamed column represents the default pandas indexing, second column named as 'index' represents the custom index that we provided and the third column named as '0' represents corresponding values. I hope now you get the idea of what is loc and iloc and what's the difference between them. We'll talk more about it in the DataFrame objects article.

Missing values or NaNs

Missing values or NaN(Not A Number) values are very common in real world datasets and before using such datasets we have to handle these missing values and NaNs in our dataset in the data cleaning phase. There are many different ways in which we can handle missing values such as dropping rows that contain missing value or replacing missing values with some other values such as mean of that column or 0 or any other value. It's a subjective matter that how you handle the missing values but anyways, Pandas provide very easy way to find missing values using isnan(), isna(), notna() in our data and using it's advance indexing methods, changing those missing values become very easy. Now let's look at the very basic example of handling missing values by changing it to the mean of that column.


In[5]:
    '''Series of Mass of Planets, where index is the name of the planet and the corresponding value is the mass of that planet.'''
    mass = pd.Series([0.33, 4.87, 5.97, 0.642, 1898, 568, 86.8, 102, 0.0146],
                    index=['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Saturn', 'Uranus', 'Neptune', 'Pluto'])
    print(mass)

Out[5]:
    Mercury       0.3300
    Venus         4.8700
    Earth         5.9700
    Mars          0.6420
    Jupiter    1898.0000
    Saturn      568.0000
    Uranus       86.8000
    Neptune     102.0000
    Pluto         0.0146
    dtype: float64

In[6]:
    '''Series of Diameter of Planets, where index is the name of the planet and the corresponding value is the diameter of that planet.'''
    diameter = pd.Series([4879, 12104, 12756, 3475, 6792, 142984, 120536, 51118, 49528, 2370],
                        index=['Mercury', 'Venus', 'Earth', 'Moon', 'Mars', 'Jupiter', 'Saturn',  'Uranus', 'Neptune', 'Pluto'])

    print(diameter)

Out[6]:
    Mercury      4879
    Venus       12104
    Earth       12756
    Moon         3475
    Mars         6792
    Jupiter    142984
    Saturn     120536
    Uranus      51118
    Neptune     49528
    Pluto        2370
    dtype: int64

In[7]:
    # Finding density using the mass and diameter series.
    density = mass / (np.pi * np.power(diameter, 3) / 6)
    print(density)

Out[7]:
    Earth      5.493286e-12
    Jupiter    1.240039e-12
    Mars       3.913302e-12
    Mercury    5.426538e-12
    Moon                NaN
    Neptune    1.603427e-12
    Pluto      2.094639e-12
    Saturn     6.194402e-13
    Uranus     1.241079e-12
    Venus      5.244977e-12
    dtype: float64

In[8]:
    pd.isna(density)

Out[8]:
    Earth      False
    Jupiter    False
    Mars       False
    Mercury    False
    Moon        True
    Neptune    False
    Pluto      False
    Saturn     False
    Uranus     False
    Venus      False
    dtype: bool

In[9]:
    # Get the index of data in density where the value is NaN and change it to the mean of density.
    density[pd.isna(density)] = np.mean(density)
    print(density)

Out[9]:
    Earth      5.493286e-12
    Jupiter    1.240039e-12
    Mars       3.913302e-12
    Mercury    5.426538e-12
    Moon       2.986303e-12
    Neptune    1.603427e-12
    Pluto      2.094639e-12
    Saturn     6.194402e-13
    Uranus     1.241079e-12
    Venus      5.244977e-12
    dtype: float64

If that was a lot to take in, let me walk you through it. First I've created two Series objects named mass and diameter containing mass and diameter of the planets respectively. The only change in these two is that diameter Series contains 'Moon' in it but mass doesn't. And thus, when we find density using these two Series we get the NaN value for Moon as mass doesn't contain it. I infer you thinking "How did this happen?". Let me explain, when we merge two Series or create a new Series by applying some functions on two different Series, what pandas does is that it matches both the Series by their indices. Also, the order of the indices doesn't matter. Thus, when we created a new Series 'density' using 'mass' and 'diameter', pandas did the calculations and merged these two Series by their indices but in the case 'Moon' it couldn't find the 'mass' and thus, it added NaN.

After creating this new 'density' Series, we see that we have NaN value in our data and to change it we have to find it's index. Now, in our example for the sake of simplicity, we had a single NaN value and also our Series had only 10 rows so we could have changed the NaN value manually using the index of 'Moon', but with large data and many more NaN values we can't go through all the rows and find the indices of each NaN values, therefore pandas has a builtin function to find all the values in which has NaN values. And using the indexing techniques that pandas offer we changed the value of Moon's diameter in one line, otherwise we would have to go through all the data in the Series using a for loop and change the NaN values with the mean value.

As you can see, pandas package provides builtin methods for all the major tasks that we have to do while working with data and it uses numpy library that makes these operations fast as numpy use vectorization methods for all the computer heavy tasks and that's the reason why numpy and pandas are the most popular python packages for data science.

Reading Data From External File & Working with Real World Data

Till now, we have created and worked with the data that we created using list and dictionaries but we don't create data by ourself in real world tasks, we use the data from external files such as csv(comma seperated values) files, text files, excels files, etc. And pandas provides many builtin methods to load the data from external files and make Series and DataFrame objects from that data very easily. Let's look at the example of loading 'csv' file and creating a Series object from it.

We have a real world data of closing value of Nifty index from the 1st January 2019 to 31st December 2019. Nifty is a benchmark Indian stock market index that represents the weighted average of 50 of the largest Indian companies listed on NSE. Let's see what this data looks like.

Now let's see how to load this data in our code and create a Series object. Pandas has a method named read_csv() which takes file path and some other parameters such asseperator,skip_rows,header, index_col and many more and creates a DataFrame object from it. But we want to create a Series object, thus we'll use iloc and get only the first column from the DataFrame that padas creates.


In[10]:
    # Using first column as index column create a Series from 'nifty_data.csv
    nifty = pd.read_csv("nifty_data.csv", index_col = 0).iloc[:,0]
    print(nifty.head(5)) # Print first 5 rows

Out[10]:
    Date
    01-Jan-2019    10910.10
    02-Jan-2019    10792.50
    03-Jan-2019    10672.25
    04-Jan-2019    10727.35
    07-Jan-2019    10771.80

Now, let's look at some basic operations on this data such as mean, median, max and other statistical data. But instead of doing all these operations one by on the Series, we can use a builtin pandas method describe() to get all these data together using a single function.


In[11]:
    nifty.describe()

Out[11]:
    count      245.000000
    mean     11432.632245
    std        454.214611
    min      10604.350000
    25%      11003.050000
    50%      11512.400000
    75%      11841.550000
    max      12271.800000
    Name: Close, dtype: float64

And just like that, we can easily get all the statistics information from the data with a single line. Here 25%,50% and 75% represents percentile of the data meaning that 25% of values in our data is below 11003.050000 and so on. You may ask that here we dont' have median of the data but that's because the median is equal to 50th percentile ,thus 50% also represents median of the data. From a little statistics knowledge I can infer that there are not many significant outliers as there isn't much difference between mean and median.

Now remember I told you, when we apply some operations on two Series and get a new Series, pandas match them using their indices. Let's see that with another example and why keeping that mind is important. Let's say you want to find the total days on which the closing value was higher than the previous day's closing value. Well that's pretty simple, we can just take two slices from this Series, first being the data without the first column and second being the data without the last column and we can just subtract and see if the result is greater than zero or not.Right? Let's see what happens when we do that.

In[12]:
    nifty[1:]

Out[12]:
    Date
    02-Jan-2019    10792.50
    03-Jan-2019    10672.25
    04-Jan-2019    10727.35
    07-Jan-2019    10771.80
    08-Jan-2019    10802.15
        ...          ...
    24-Dec-2019    12214.55
    26-Dec-2019    12126.55
    27-Dec-2019    12245.80
    30-Dec-2019    12255.85
    31-Dec-2019    12168.45
    Name: Close, Length: 244, dtype: float64

In[13]:
    nifty[:-1]

Out[13]:
    Date
    01-Jan-2019    10910.10
    02-Jan-2019    10792.50
    03-Jan-2019    10672.25
    04-Jan-2019    10727.35
    07-Jan-2019    10771.80
        ...          ...
    23-Dec-2019    12262.75
    24-Dec-2019    12214.55
    26-Dec-2019    12126.55
    27-Dec-2019    12245.80
    30-Dec-2019    12255.85
    Name: Close, Length: 244, dtype: float64

In[14]:
    nifty[1:] - nifty[:-1]

Out[14]:
    Date
    01-Apr-2019    0.0
    01-Aug-2019    0.0
    01-Feb-2019    0.0
    01-Jan-2019    NaN
    01-Jul-2019    0.0
                ...
    31-Dec-2019    NaN
    31-Jan-2019    0.0
    31-Jul-2019    0.0
    31-May-2019    0.0
    31-Oct-2019    0.0
    Name: Close, Length: 245, dtype: float64

Ooops!! That's not what we expected it to be right? Yes and that's because pandas matched indices of both the slices and subtracted their corresponding values and for the values that it couldn't find matching indices (01-Jan-2019 and 31-Dec-2019), it addded NaN values. Thus, when we want to subtract two series by element by element and don't want the index to be matched, we can use values() method on these slices and get the numpy array of values for both the slices and then subtract them.

In[15]:
    (nifty.values[1:] - nifty.values[:-1]) > 0

Out[15]:
    array([False, False,  True,  True,  True,  True, False, False, False,........,False])

In[16]:
    np.sum((nifty.values[1:] - nifty.values[:-1]) > 0)

Out[16]:
    129

Well that's pretty much it for this post and Series object. There are many more things that you can do with the pandas Series object but including each and everything in my posts is pretty much impossible. Look at the pandas documentation and see what you can do with the Series object. In the next post, We'll talk about DataFrame objects and all the good things we can do with it. Thank you for reading.

Get the full code notebook here : Github

Get the Nifty 50 data here : Data

Learning Data Analysis with Python - Introduction to Pandas

James Shah — Sun, 05 Jul 2020 13:42:57 +0000

What is Pandas and Why do we need it

Pandas is python library that provides rich data structures and functions that makes working with relational and structured data easy, fast and convenient. It is developed on top of the Numpy package for the high performance computing and it gives flexible data manipulation techniques of relational databases. Okay, now why do we need panadas at all? If we want to mainpulate high dimensional data, we have numpy, so why do we need it?
Well, indeed numpy is there for the high dimensional data manipulation but what it doesn't provide is a way to index or label our data like this.

And, although we can do so many interesting things to our data using numpy functions such as finding statistics like mean,median,mode and standard deviation but we can't fill missing values in our data. Numpy doesn't provide any built-in method to handle missing values in our data and in real world problems, we don't always come around the perfect datasets.

There are other limitations to numpy such as no way to group data and no way to pivot data, and thus, Pandas was created to help programmers and data scientists with the data analysis task in Python.

Data Structures in Pandas

Well as we've mentioned earlier, pandas provides rich data structures and functions, let's start by looking at the data structures and then we'll look at the functions and methods that we can apply on these data structures to crunch data.

Pandas provide two major data structures namely "Series" and "DataFrames". In plain english, Series is a one-dimensional labeled array, whereas DataFrames is two-dimensional labeled array with columns of potentially different types. Although, Series are useful in many tasks, DataFrames are the most commonly used pandas object. You can think of DataFrame as an SQL table.

Let's look at the example for both of these data structures.

Let's start by looking at the different ways of creating a Series object.But, Before we get into coding, we need to install pandas as python package. This can be done using pip install pandas or you can install it using conda install pandas if you are using Anaconda Distribution for Python. If you don't have python installed on your machine, the best way to get started without all the hustle of installing it is Google Colab. Learn more about it here. I'm going to use jupyter notebooks which you can find here.

Series

As mentioned above, Series is a one-dimensional array-like object containing an array of data and an array of labels associated with the data. The data labels are called 'index'.
We can create a Series from a list or a dictionary or a numpy array.
Let's start with a list.

Series using a list

In[1]:
    import pandas as pd
    import numpy as np

    friends = ["Monica", "Chandler", "Joey", "Pheobe", "Ross", "Rachel"]
    series = pd.Series(friends)
    series

Out[1]:
    0      Monica
    1    Chandler
    2        Joey
    3      Pheobe
    4        Ross
    5      Rachel
    dtype: object

As we can see, we get the Series object with the data that we provided with the associated index of for each row. We can access our data from this series using the index just like we do with the list.

In[2]:
    series[0]

Out[2]:
    'Monica'

Or we can get all the values in a list using .value attribute on series object.

In[3]:
    series.values

Out[3]:
    array(['Monica', 'Chandler', 'Joey', 'Pheobe', 'Ross', 'Rachel'],
      dtype=object)

Similarly, we can get all the indices in a list using .index attribute.

In[4]:
    series.index

Out[4]:
    RangeIndex(start=0, stop=6, step=1)

Note something different? Right, it doesn't return a list of indices as it did in values, that's because the implicit index that pandas automatically creates for us is just a list of consecutive numbers starting from 0. A more intiuative approach would be to provide index identifying each data point while creating a Series.

In[5]:
    nick_names = ["Monana", "Officer Bing", "Ken Adams", "Regina Filange", "Dinosaur Guy", "Fun Aunt Rachel"]
    series = pd.Series(data = friends, index = nick_names)
    series

Out[6]:
    Monana               Monica
    Officer Bing       Chandler
    Ken Adams              Joey
    Regina Filange       Pheobe
    Dinosaur Guy           Ross
    Fun Aunt Rachel      Rachel
    dtype: object

And now, we can access our data using the index that we provided like this.

In[7]:
    series["Dinosaur Guy"]

Out[7]:
    'Ross'

It's clearly a bad example of indexing but you got the idea, right?

Series using a dictionary

Now, let's instantiate Series from a dictionary. When creating a Series from dictionaries, we don't pass index, the Series gets ordered by dictionary's insertion order and the keys of dictionary becomes indices of Series.

In[8]:
    d = {"a" : 1, "b" : 2, "c" : 3}
    series = pd.Series(d)
    series

Out[8]:
    a    1
    b    2
    c    3
    dtype: int64

Series object also works like dictionaries. You can get or set the values in the Series by index label just like dictionaries.

In[9]:
    series["c"]

Out[9]:
    3

In[10]:
    series["d"] = 4
    series

Out[10]:
    a    1
    b    2
    c    3
    d    4
    dtype: int64

Series using a numpy array

And finally, we can create a Series from a numpy array.

In[11]:
    d = np.random.randn(5) # Create a numpy array of 5 random numbers
    series = pd.Series(d)
    series

Out[11]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    3   -0.495840
    4    0.065357
    dtype: float64

Series is also similar to numpy array in functionalities. We can index data from a Series object as we do from a numpy array.

In[12]
    series[:3] # Returns first 3 elements

Out[12]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    dtype: float64

In[13]:
    series[series > 0] # Returns elements whose value is greater than 0

Out[13]:
    2    0.526216
    4    0.065357
    dtype: float64

Now, once we created a Series, we can apply many different methods on it to achieve different tasks. A few of them are as below. Read more about all the methods available on Series object at pandas official documentation.

In[14]:
    series.size # Returns size of the series.
Out[14]:
    5

In[15]:
    series.shape # Returns shape of the series.
Out[15]:
    (5,)

In[16]:
    series.add(1) # Returns series with elements' value increased by 1
Out[16]:
    0    0.492757
    1    0.884269
    2    1.526216
    3    0.504160
    4    1.065357
    dtype: float64

In[17]:
    series.sub(1) # Returns series with elements' value decreased by 1
Out[17]:
    0   -1.507243
    1   -1.115731
    2   -0.473784
    3   -1.495840
    4   -0.934643
    dtype: float64

In[18]:
    series.mul(10) # Returns series with elements' value multiplied by 10
Out[18]:
    0   -5.072431
    1   -1.157307
    2    5.262160
    3   -4.958401
    4    0.653565
    dtype: float64

In[19]:
    series.div(10) # Returns series with elements' value divided by 10
Out[19]:
    0   -0.050724
    1   -0.011573
    2    0.052622
    3   -0.049584
    4    0.006536
    dtype: float64

In[20]:
    series.abs() # Returns series with absolute value of elements
Out[20]:
    0    0.507243
    1    0.115731
    2    0.526216
    3    0.495840
    4    0.065357
    dtype: float64

In[21]:
    series.max() # Returns maximum of all the elements.
Out[21]:
    0.5262159987471329

In[22]:
    series.min() # Returns minimum of all the elements.
Out[22]:
    -0.5072430551642492

In[23]:
    series.sum() # Returns sum of all the elements.
Out[23]:
    -0.5272413324692681

In[24]:
    series.mean() # Returns mean of the series.
Out[24]:
    -0.10544826649385361

In[25]:
    series.median() # Returns median of the series.
Out[25]:
    -0.1157306666767307

In[26]:
    series.std() # Returns standard deviation of the series.
Out[26]:
    0.43073327047221427

In[27]:
    series.drop(4) # Returns series with specified index removed.
Out[27]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    3   -0.495840
    dtype: float64

In[28]:
    series
Out[28]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    3   -0.495840
    4    0.065357
    dtype: float64

Look at the last code when we print the original series object, we get the elements same as we created, because all the operations that we applied on the series object gives us the view of the series and doesn't affect the original object.
Okay, so this is just a beginning, there are many more interesting things that we can do with pandas such as dropping null values or replacing them with some other values from data, visualize data using so many plots and charts, combine and merge two or more Series, create and manipulate Time Series, etc.

That's it for this post.In the next couple of posts, I'll be playing with some real world datasets and make some analysis on those data using Pandas.

Get full code notebook here : Pandas-1 Notebook

Thank you for reading. Happy coding.👨‍💻👩‍💻

Six Elements Of Machine Learning - A Beginner's Guide

James Shah — Wed, 03 Jun 2020 10:53:08 +0000

Note : Content of this article including some images are taken from the Deep Learning Course by One Fourth Labs

Before introducing the 6 elements of Machine Learning, let's understand Why do we need Machine Learning? Why not the old school expert systems where we used to come up with a solution and then code it in one of our favourite programming languages for the computer to understand it and then feeding it some input to get the desired result? A simple example of such expert systems is "Whether a patient has dengue or not?" Now, how do doctors decide that? They ask the patient "Do they have high fever and cold? Do thay have headache or rashes?" i.e. Whether the patient has symptoms of Dengues or not? And from the past experience doctor decides "If the patient has Dengue or not?"

Now, How do we solve this problem using a computer? We can simply write some if-else conditions to get the output as shown below.

if HIGH_FEVER and COLD:
    return no_dengue
elif HIGH_FEVER and VOMITING and not RASH:
    return no_dengue
elif HIGH_FEVER and VOMITING and RASH:
    return dengue

And now we can simply run this program and feed it with the symptoms of a new patient and can decide "Whether he/she has dengue or not?"
Cool, right? So Why do we need Machine Learning?

The answer is the limitations of the expert systems that we talked about. Predicting dengue or not was somewhat easy task as we know about it's symptoms but let's say we want to predict whether a patient has Ebola or not? In this case, the symptoms(rules to check) are unknown and thus we can't feed the computer the rules that we don't know.

Another limitations is that sometimes we have lots of data. For example, "Should a bank give you a loan or not?" Now, giving loan depends on lots of factors including your salary, your credit score, size of your family and many more. Now in this case, we have so much data to make sense from that writing if-else blocks is not an easy and effective task.

And sometimes, there are rules that we can't express to the computer to understand. For example, sometimes your salary may be a little less than the required salary to get loan but you've been an honest and a regular customer to the bank and the bank manager knows that though your salary is not enough to meet the minimum requirements but they believe you and your honesty and gives you loan anyway.How do you feed such rule to the computer? How can a computer understand honesty?

These are the limitations that needs to be conquered and Machine Learning Does exactly that.
Using Machine Learning, we give some data as input and some matching output for that data and then find the relationship between the input data and output data.
The 6 elements of the the Machine Learning are:

Data
Task
Model
Loss Function
Learning Algorithm
Evaluation

Now, let's dive deep into these elements one by one and see how we can see Machine Learning through the lens of these elements.

1. Data

Data simply means information. All types and formats of information. Today,there is enormous amount of data produced every second and this data can be used to answer so many questions. There is text data as well as audio-video data, there is structured data as well as unstructured data.

Data is the fuel of Machine Learning.

When starting a Machine Learning project, we need some answers to the questions like,
"Do we have the required data? If no, how do we get the data and where from? And if yes, is it in a structured form or unstructured and raw?"

These questions lead us to different paths in the project. If we don't have the data needed, then Data Curation comes in picture. There are many ways to get the data.

If you are lucky enough, there are many resources where you can get datasets that are publically available. Few examples of such resources are Google Dataset Search, Kaggle Datasets, Indian Government Data.

If you are rich enough, there are many websites on which you can assign some people to curate and collect data for your needs. For example, let's say you have images of animals and you want these images to be labelled with the name of animal, then you can assign such task to people and you can get the data easily and in a short span of time. Few examples of such crowdsourcing marketplaces are
Amazon mturk, Dataturks.

And If you are smart enough, you can conduct surveys and get data from people or using the data scrapping techniques you can collect data from different websites.

One important thing to remember is that it doesn't matter in which format you get the data, at the end all the data needs to be encoded as numbers before feeding it to the computers. For example, images can be converted into matrix of pixel values.

2. Task

Now, you have data. The next question you should ask is "What task do I need to achieve? Or What task can I achieve using the data that I have?"

Generally, Machine Learning tasks are divided into two parts: Supervised Learning And Unsupervised Learning. Supervised Learning is when you have some input data as well as some output data corresponding to the input data. Then you can use Machine Learning to get the relationship between the input and output data and then use that relationship to predict output for the new input. And Unsupervised Learning is when you have only input data.

Supervised learning is further divided into two sub sections as Regression and Classification where Regression means you want to predict some continous value for example House Price Prediction and Classification means you want a discrete answer such as yes or no. An example of Classification is, given an image you want to detect whether there is a dog or not in that image.

Unsupervised learning is often used for Generation and Clustering. For example, the image given below shows that given the tweets of US President Donald Trump as an input, machine generates more such tweets.

Although Unsupervised learning is useful in some cases, most of the real world problems are solved using Supervised learning as it gives more accurate results than Unsupervised learning.

"Supervised learning has created 99% of economic value in AI." - Andrew NG, Co-founder of Coursera.

3. Model

A model is nothing but a mathematical function which defines relationship between input data and output data.

A model can be a simple Linear Function such as y = mx + c or it can be a complex function such as y = sin(cos(ax)) + tan(sin(bx^2 + cx)) .

The example that we talked about before, given an image(data), predict if it has a dog or not(task), we have to build a system that answers this question and such system is called a model.

There are many models available that are created over the years and each one has it's own pros and cons and each one is suitable for a unique task. For a human, it is almost impossible to come up with a function that defines relationship between the input and output, given that the input data has over thousand different features.

4. Loss Function

Although, it's not possible to come up with a model just by looking at the given data, but let's assume, you have come up with a model that you think correctly defines the relationship between input and output. But, two of your friends are solving the same problem and they have come up with a slightly different model, then how do you decide, which model is better? Loss function gives the answer to that question.

As shown in the image above, three different values predicted by the models that you and your friends have written are f1(x), f2(x) and f3(x), respectively. And the true value is y. Now, to know which model is best suited for our data, we compute the difference between the true value and predicted value for all the data using a loss function. One such loss function is Square Error Loss.

And counting loss function value for all three models, we get to know that the best model for our data is the one with the minimum value of loss function which is 1.38.

This way using a loss function, we can get the best model that defines the relationship between our input data and output data. More such examples of Loss Function are Cross Entropy Loss and KL Divergence.

5. Learning Algorithm

Now, we assumed a hypothetical situation where you and your friends came up with a model for our data and task but that won't happen in reality as there will be hundred and thousands of features in the data and you can't just look at it and propose a model that satisfies the relationship. Instead, you can propose a function such as y = ax^3 + bx^2 + cx and say that I think this should define the relationship between the input x and output y, but you don't know the paramteres a, b and c. So, to identify these parameters for a model, we need a learning algorithm.

As shown in the picture above, we have a model with three parameters a,b and c and we need values of a,b and c at which the loss function's value is minimum. In calculus, there are many optimization strategies available that we can use to find the best values of parameters which minimizes the loss.
Some examples of such optimization solvers are Gradient Descent, Adagrad, RMSProp, etc.

Learning algorithm is one task that computer does, other than that from data collection to deciding task and then proposing a suitable model is done by humans but then finding parameters of the proposed model is done by computer.

6. Evaluation

Okay,let's say using the above 5 elements, you've created an ML model to detect the animal in the image. And you want to sell it to some company or just show off to your friends. How do you know that your model is detecting the right animal? You guessed it right! you have to test your model. That's what evaluation is. Testing your model by feeding it some test data and then checking if it predicts the correct output in all the cases? If not, then how accurate your model is?

Note: You evaluate your model using test data and not the training data, but why so? that's beyond the scope of this article!

There are many performance matrices available for evaluation, one such matrix is accuracy. We find the accuracy of a model diving "Number of correct predictions" by "Total numbers of predictions" , as shown in the image below.

Thank you for reading! I Hope you liked it. #HappyLearning

Originally published at : bytetales.co

Learning Data Structures And Algorithms

James Shah — Fri, 15 May 2020 14:53:43 +0000

Data Structures and Algorithms are like backbone of a well designed and efficient code. For a software developer, Data Structures and Algorithms are the most important assets for programming, but Algorithms are not only limited to software engineering but in any field, Algorithms are a great way to define instructions to achieve some specific task. As a software development student, I've always tried to learn Data Structures and Algorithms, and I know basics and some intermediate level stuff, but I've always wanted to learn more about it and I tried so many times but I lacked consistency. But in this quarantine period, I had all the time and decided to start learning Data Structures And Algorithms.

What is Data Structures and Algorithms?

Data Structure is a way to store and organize the data in order to facilitate access and modification to the data. There many different data structures such as Stacks , Linked List, Queues, Trees, Graphs, Hash Tables, etc.
And there are some programming language specific data structures,too. For example, Python provides Dictionaries, Tuples, etc.

Algorithms have many different definitions all around the world. In general terms, am Algorithm is a collection of explicit, precise and unambiguous instructions, implemented in defined sequence to accomplish a certain task. And thus, Algorithms are not limited to software development but it has many other uses, too. There are so many problems that can be solved using specific algorithms such as simple Sorting, Searching and also some complex problems such as Identifying all the genes in a human DNA, Predictions and Recommendation Systems.

Motivation To Learn

Well as we all know, Data Structures and Algorithms are very important to crack coding interviews but that isn't the only thing where learning Data Structures and Algorithms are useful. Data Structures and Algorithms Knowledge give us the ability to improve our solution to the problem and the ability to write much better and efficient code.

But most importantly, it helps to build problem solving mindset. And programming is almost all about problem solving. We can adapt any programming language syntax but the problem solving skill can't be adapted overnight. Thus, learning Data Structures and Algorithms can be a major learning curve for any computer science student.

What I Have To Learn?

Algorithm Design
Algorithm Analysis
Algorithm Techniques
How to improve the time and space complexity
Advanced Data Structures (Trees, Graphs,etc.)

My Resources & Plan To Learn Data Structures and Algorithms

There are so many books, courses, blogs and videos to learn Data Structures and Algorithms. And every DSA learner has this question that "Which course/book is the best to learn DSA?" And I am no exception. I had the same question as well, but over the past 3 years, I've learned many programming languages and frameworks using online courses and books. And what I've realized is that You don't have to find the best resources. Just take a book or a course and start learning.

While learning anything, I start with an online course and then refer to some book. So I started a Data Structures and Algorithms Specialization by University Of California, San Diego on coursera. It consists total of 6 courses from Basic Data Structures and Algorithms to Advance Algorithms and Complexity. And in the last course they have a project "Genome Assembly Programming Challenge" . They also have a book to refer during the course which can be of great use if you prefer text based content over videos. I'm almost halfway through the first course of the specialization and What I like the most about this specialization is that all the courses have a lots of programming challenges as assignments to solve and so, you don't just learn theory but also, implement it in code.

Along with this course, I'm using Introduction To Algorithms book (aka CLRS). It's a great book for someone who knows basics of Data Structures and Algorithms but for a total beginner, it can be overwhelming. In my opinion, if you are starting from scratch you should refer to GeeksForGeeks for basic Data Structures and try to implement them in a programming language of your choice.

Also, Mathematics plays very important role in learning Data Structures and Algorithms. I have a pretty good background in Mathematics but still, I have to go through some topics of Algebra and Calculus to refresh the dead cells of my brain.😅

Other Resources :

Courses :

Books :

Algorithms By Robert Sedgewick
Data Structures and Algorithms Made Easy by Narasimha Karumanchi

Youtube Channels

How to be consistent ?

Learning Data Structures and Algorithms can be overwhelming sometimes but consistency is very important. The skills will fade away if you stop learning. I'll tell you what worked for me to be consistent. If you are stuck at some problem for a long time and by long time I mean hours. Then stop trying to solve it at that time because your mind won't be able to think about the solution at that time and eventually you will get frustrated and disappointed. But don't skip the problem, if you are not able to solve it after many attempts and you feel like skipping it, go to google and search for the solution and analyze your mistake and your thinking process and then close that solution and implement it by yourself. But, don't make it a habit. Be patient. Data Structures and Algorithms is a vast ocean and you can't learn it all in a day. Keep learning, keep making mistakes, and keep solving them.
😉

What's Next ?

Obviously, one cannot master Data Structures and Algorithms with just one book and a specialization course. It's a huge field and this is just a beginning. After this course, I'll be practicing my Data Structures And Algorithms skills on platforms like Leetcode and Hackerrank. Also, some of my friends have recommended me this book called Crack Coding Interview by Gayle Laakmann McDowell, so I'll solve some problems from that book, too.

Also, if you have any suggestions, please let me know in the comments down below.

Originally published at: bytetales.co

Thank You For Reading. Happy Coding👩‍💻 👨‍💻

Housie(Tambola) Ticket Generator using Python

James Shah — Thu, 30 Apr 2020 09:32:43 +0000

Announcement: I recently developed my own blog at bytetales.co 😍😍 Please check it out.

Housie aka Tambola aka Bingo (British Version) is probably the best game to play in Lockdown to get our minds off from Corona and all the negative news from all around the world and to have fun with friends and family. I've been playing it a lot, lately.
For those who don't know what Housie is, let me explain.

What is Housie?

Housie is basically a game of probability in which players mark off numbers on cards as the numbers are drawn randomly by a caller. So at the beginning of the game, all the players get a ticket [3x9 matrix] in which there are total of 27 boxes and 15 of them are filled with a distinct random number from 1 to 90. Here is a sample housie ticket.

And then a caller calls a random number from 1 to 90 and the winner being the first person to mark off all their numbers. So, It's a pretty simple yet interesting and gripping game that one can play with friends and family.

Housie Ticket Rules

Now there are some rules to generate housie tickets and you can't just randomly put 15 numbers in any boxes of your choice. So, let's see the rules and then create a Python script that will generate such a housie ticket for us.

RULE #1 - Each row cannot have more than 5 numbers.

RULE #2 - Each column is assigned a range of numbers only:

(ex. 1-10 can appear only in column 1)

RULE #3 - In a specific column, numbers must be arranged in ascending order from top to bottom.

Now, of course it's a cumbersome task(and a little biased, too) for a human to generate such tickets, and for the record, we are talking about at least 20-25 tickets per game. So yeah it's definitely not a fun task. So, I thought of creating a python script that would take the number of tickets that we want to generate as input and will return the tickets as an output.

I know, it's not at all a very fancy or difficult script to write but anyway I am free in this lockdown period so I thought why not?

Let's see how we are gonna build this script.

Step 1 : Create a 3x9 matrix (array) of zeros



import numpy as np

# Create a 2D array[3x9] of 0s
ticket_array = np.zeros((3, 9), dtype=int)

Here I've used numpy library of Python to create the ticket_array because while creating a single matrix, it may not be effective but when we want to create a very large number of lists, NumPy creates it very fast and therefore it is efficient to use numpy to create the array.

Step 2: Randomly Generate 15 indices to fill values in a ticket.

Now, in numpy we can use tuple for indexing of an array. i.e. To get the first element of first row and first column from ticket_array , we can use ticket_array[(0,0)].

And so, first we'll generate an array total_indices which contains all the indices of 3x9 array as tuples (i.e. (0,0), (0,1)), (0,2), ..., (2,8))

and then, we'll randomly generate 5 indices from each row.



total_indices = [(i, j) for i in range(3) for j in range(9)]

random_indices = []


first_row = random.sample(total_indices[:9], 5)
second_row = random.sample(total_indices[9:18], 5)
third_row = random.sample(total_indices[-9:], 5)

for i in first_row:
    random_indices.append(i)

for i in second_row:
    random_indices.append(i)

for i in third_row:
    random_indices.append(i)

Step 3: Now Let's fill these indices with random values

So let's create a list ranging from 1 to 90. Then to satisfy the Rule #2, we'll check the column number from index and will fill the according value.



for num in random_indices:
    if num[1] == 0:
        number = random.choice(total_numbers[:10])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 1:
        number = random.choice(total_numbers[10:20])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 2:
        number = random.choice(total_numbers[20:30])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 3:
        number = random.choice(total_numbers[30:40])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 4:
        number = random.choice(total_numbers[40:50])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 5:
        number = random.choice(total_numbers[50:60])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 6:
        number = random.choice(total_numbers[60:70])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 7:
        number = random.choice(total_numbers[70:80])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 8:
        number = random.choice(total_numbers[80:89])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0

Step 4: Now let's sort the ticket_array column-wise



for col in range(9):

    # if all the rows are filled with a random number

    if(ticket_array[0][col] != 0 and ticket_array[1][col] != 0 and ticket_array[2][col] != 0):
        for row in range(2):
            if ticket_array[row][col] > ticket_array[row+1][col]:
                temp = ticket_array[row][col]
                ticket_array[row][col] = ticket_array[row+1][col]
                ticket_array[row+1][col] = temp


    # if 1st and 2nd row are filled by random number

    elif(ticket_array[0][col] != 0 and ticket_array[1][col] != 0 and ticket_array[2][col] == 0):
        if ticket_array[0][col] > ticket_array[1][col]:
            temp = ticket_array[0][col]
            ticket_array[0][col] = ticket_array[1][col]
            ticket_array[1][col] = temp


    # if 1st and 3rd row are filled by random number

    elif(ticket_array[0][col] != 0 and ticket_array[2][col] != 0 and ticket_array[1][col] == 0):
        if ticket_array[0][col] > ticket_array[2][col]:
            temp = ticket_array[0][col]
            ticket_array[0][col] = ticket_array[2][col]
            ticket_array[2][col] = temp

    # if 2nd and 3rd rows are filled with random numbers

    elif(ticket_array[0][col] == 0 and ticket_array[1][col] != 0 and ticket_array[2][col] != 0):
        if ticket_array[1][col] > ticket_array[2][col]:
            temp = ticket_array[1][col]
            ticket_array[1][col] = ticket_array[2][col]
            ticket_array[2][col] = temp

Step 5: Wrap it up in a function and print the tickets.

To print the tickets I used tabulate package of python using which we can present array and matrix in a neat way. Learn more about Tabulate Here.



if __name__ == "__main__":

    # Take number of tickets from user as system argument
    numberOfTickets = sys.argv[1]
    tickets = []

    for i in range(int(numberOfTickets)):
        ticket = getTickets()
        tickets.append(ticket)

    for ticket in tickets:
        print(tabulate(ticket, tablefmt="fancy_grid", numalign="center"))

And Done!! Now, Let's check by running the script from terminal.



$ python ticket-generator.py 2

Output:

You can check out the full code on Github :

jamesshah / housie-tickets-generator

Housie Tickets Generator Using Python

Originally Published At: bytetales.co

Thank You For Reading!

HappyCoding👨‍💻👩‍💻

Covid19 India Update | Twilio x Dev Hackathon Submission

James Shah — Thu, 23 Apr 2020 05:00:33 +0000

What I built

A covid19 case updates webapp. Basically, user registers on this website with their name, phone number and their state(limited to Indian states only) and then they receive covid19 case updates daily twice a day. This webapp helps user to be aware of total cases in their state as well as new cases of the day. I've used an external API to get the data of covid19 cases and using Twilio Programmable SMS API, it sends the daily updates to the users.

Category Submission:

Interesting Integration: I integrated an external api to get the data of covid19 cases in India.

Other Use Cases: GRE Words Of The Day(Sending 5 words daily to user to learn new GRE Words),
Appointment Reminder Using Google Calendar API integrated with Twilio SMS API.

Demo Link

Covid 19 India Updates

Here is the sample message that user receives when they register successfully on the app.

Here is a sample message of case updates that user receives daily.

Link to Code

Here is a github repo of the project.

jamesshah / covid19-updates

Simple web app to send daily covid-19 cases updates to users.

How I built it (what's the stack? did I run into issues or discover something new along the way?)

Node.JS
Express Framework
MongoDB Databse
Twilio Programmable SMS API

I wanted to take this hackathon as opportunity to learn something new and that's what I did. I learned Node.JS & Express to build the website and also learned MongoDB(NoSQL) for the Databse. Although I've been using Javascript for sometime but building a fully functional web app taught many many things that the tutorial and videos didn't.

Additional Resources/Info

I've documented the whole journey of building this project from scratch in 4 post blog series. You can check it out here.

Building A Node.js - Express App (covid19-updates)|Part-1

James Shah ・ Apr 9 '20 ・ 6 min read

#twiliohackathon #node #javascript #showdevs

It's been a great experience participating in this hackathon and building a project from an idea to the application. Hope you like it.

HappyCoding

Building A Node.js - Express App (covid19-updates)|Part-4

James Shah — Tue, 21 Apr 2020 08:29:29 +0000

If you haven't read the last post, I recommend you to check it out here, else many things in this post will not make sense.

In the last post we finally completed the database creation and user registration and also scheduled twilio sms api to send the case updates sms to the users daily and now in this post we will receive an sms from the user, in case they want to stop the service and we'll deactivate their account to stop sending messages to them anymore and will send them a reply back. And after that, we'll deploy our webapp on Heroku. Although, there are many cloud services available to serve Node.JS application online but I chose heroku as it's makes deployment very easy using github integration. Let's start with receiving an SMS from the user.

Receiving and Replying to SMS using Twilio

Twilio can send our web application an HTTP request when certain events happen, such as an incoming text message to our Twilio phone number.So, we'll create a route that twilio will call whenever a user sends a message on our Twilio phone number to stop or restart the service.
Now, We need to configure our Twilio phone number so that it can send an HTTP request to the route in our application, whenever our it receieves an incoming message. But, we'll need our websites URL for that,which we don't have for now. So, let's skip this and create the route in our application and after we deploy our application on heroku, we'll add the URL in the Twilio console.

Create a new file in routes directory called sms.js and add the following code to it.
Twilio provides code snippet for almost every programming language to achieve this task.

const express = require("express");
const router = express.Router();
const MessagingResponse = require("twilio").twiml.MessagingResponse;
const User = require("../models/User");

router.post("/", async (req, res) => {
  const twiml = new MessagingResponse();
  phn_number = req.body.From.slice(3);

  if (req.body.Body === "STOP") {
    await User.update(
      { number: phn_number },
      {
        $set: {
          active: false,
        },
      }
    );
    twiml.message(
      "You will not receive updates any more. To restart the service reply with 'START'\n Thank You."
    );
  } else if (req.body.Body === "START") {
    await User.update(
      { number: phn_number },
      {
        $set: {
          active: true,
        },
      }
    );
    twiml.message("Thank you for your interest. You will now receive updates.");
  } else {
    twiml.message("Sorry, you've send an invalid response.");
  }
  res.writeHead(200, { "Content-Type": "text/xml" });
  res.end(twiml.toString());
});

module.exports = router;

So, now when a user sends an sms to our Twilio phone number with 'STOP' or 'START' in the message body, Twilio will call the /sms route in our application and our application will handle the sms according to the message body and will reply to the user.
We'll need to add the following code in app.js to add this route in our application.

app.use("/sms", require("./routes/sms"));

And we are done with the coding part. Now, let's deploy the application on Heroku and set it live for the users to use.I will be deploying the app on Heroku using Github repository, however we can also make use of heroku-cli to deploy. You can learn more about Heroku CLI at their website, here.

Creating a Github Repository And Deploying using Heroku

Before creating a Github repository, we'll add a .gitignore file in our project directory to specify what parts of our application we want to ignore(i.e, node modules, .env file) while uploading it to the Github repository. We can manually write the .gitignore file but for the best practice we can get the specific file for specific programming language or framework that we use in our application. Github itself provides a repository that contains the collection of such files that we can use in our application and I used one for this application as well. You can find it here.

Now, let's create a new Github Repository and use our local project directory to upload.If you don't know how to create a github repository, you can learn how to do it from their documentation.

Now, let's go to Heroku and sign in to your account, create a new account,if you don't have one by simplying filling out a form for free.

Now, on the dashboard, let's creat a new app and then connect your Github account on the next page and select the Project Repository that we've just created and connect.

Now your app has been created but to tell Heroku that this is a Node.JS app, we'll need to add a buildpack. Go to settings and select "Add Buildpack" tab and select nodejs option from the list.

Remember, we used .env file to hide our secrets but we've not uploaded it to our Github repository for the very obvious reason. So let's add those secrets in Heroku Config Variables. In settings tab select Reveal Config Var and then add all the keys and values one by one.

And that's it, Now we can visit the url by clicking Open App from dashboard and we have our app live on server!!

Now remember, we skipped a step above, configuring the Twilio console to add the URL of our app. Let's finish it by going into console page of our Twilio Account. Then click on your Twilio phone number. Find the Messaging section and the "A MESSAGE COMES IN" option and add the url to the application and add /sms as it is the route that handles incoming messages in our application.

And Finally, We've completed all our project from the scratch to find an idea, learning tech stack, creating and then deploying the application.

Full code can be found here:

jamesshah / covid19-updates

Simple web app to send daily covid-19 cases updates to users.

Thank You For Reading. #HappyCoding

Building A Node.js - Express App (covid19-updates)|Part-3

James Shah — Wed, 15 Apr 2020 13:38:36 +0000

If you haven't read the last post, I recommend you to check it out here, else many things in this post will not make sense.

In the previous post, we created database and user registration for our application and now in this post we'll add Twilio API to send SMS to users whenever they register successfully and we'll also add external API call to get the coronavirus data to send to our users. Lots of work to do!! So, let's get started.

Using Twilio API

Twilio provides many services ranging from SMS and Chat to AI Bots, Programmable SMS API being one of them. Using the Programmable Twilio SMS API we can add SMS service in our web application and that's exactly what we'll doing.

To use the Twilio SMS API,we need to create an account on Twilio website and buy a twilio number with SMS support. Twilio provides free number with some limitations. But luckily, As a part of #DevTwilioHackathon they are providing Free 20\$ credits using the promo code DEVHACK20. So I went ahead and bought a number with this promo code. And now if we go to the console on the twilio website, we can get 'ACCOUNT SID' and 'AUTH TOKEN' which will be used to connect to the Twilio API in future. Copy that and paste it into the .env file in our project directory as we don't want anyone to be able to see it.

To use the twilio API in our application, we'll need Twilio Helper Library and we can download it using the following code in the terminal.

$ npm install twilio

Now, Twilio provides code snippets to connect to it's various APIs in many different programming language, And we'll use it to connect to the twilio api in our app.Let's go to the users.js file in our project directory and add some code to connect to the API.

const accountSid = process.env.ACCOUNT_SID;
const authToken = process.env.AUTH_TOKEN;
const client = require('twilio')(accountSid, authToken);

client.messages
  .create({
     body: 'Thank you for your registration. You will receive Corona-Virus updates everyday.To stop the service reply with "STOP"',
     from: process.env.PHN_NUM,
     to: // user's mobile number (with country code).
   })
  .then(message => console.log(message.sid));

We'll add this snippet in the /users/add_user route, so that whenever a new user gets added to the database, we'll send the SMS to the user. Here is the sample SMS that our application sends to the user.

Getting Data From The External API

Now, we want to send corona virus case updates to our users but where do we get the data from? And there comes the External API. There are so many Corona Virus datasets available publically online. I've used the api.covid19india.org to get the data as it is crowdsourced and it is based on the data declared by the State Governments so it's a reliable source of data and also, it gets updated every 5 minutes, so we'll get latest updates.And also, this api requires no authentication at all.

In node.js, we cannot use fetch as we do in the client-side javascript, but there is an npm package to our rescue called node-fetch. So let's install it.

$ npm install node-fetch

Now, let's add a new file in our project directory called script.js and let's add some code in it to get the data from the api and use it.

const fetch = require("node-fetch");

const get_data = async () => {
  cases = {};
  await fetch("https://api.covid19india.org/data.json")
    .then((res) => {
      return res.json();
    })
    .then((data) => {
      data_array = data.statewise;
      total_obj = data_array.filter((data) => data.state === "Total")[0];
      gj_obj = data_array.filter((data) => data.state === "Gujarat")[0];
      mh_obj = data_array.filter((data) => data.state === "Maharashtra")[0];

      cases.total_cases = total_obj.confirmed;
      cases.total_new = total_obj.deltaconfirmed;
      cases.mh_total = mh_obj.confirmed;
      cases.mh_new = mh_obj.deltaconfirmed;
      cases.gj_total = gj_obj.confirmed;
      cases.gj_new = gj_obj.deltaconfirmed;
    })
    .then();
  return cases;
};

Okay, let me explain it a little bit. Here, we are fetching the state wise data from the api and it returns array of objects in which there are many properties like, total cases, new cases, total deaths, new deaths, total recovered,etc. We want to find the total cases and new cases in states. Here i've shown only 2 states (Gujarat & Maharashtra) but we can do the same for any state and can get the same data for all the states of India. So i've created an object called 'cases' and in this object I've added 6 keys namely, 'total_cases', 'total_new', 'mh_total', 'mh_new', 'gj_total', 'gj_new' and their values gives the number of 'Total Cases in India', 'New Cases in India', 'Total Cases in Maharashtra','New Cases in Maharashtra', 'Total Cases in Gujarat', 'New Cases in Gujarat' respectively.

Sending Message To The Users According To Their State

We want to send a distinct message to the user according to his/her state.i.e, if the user registered with the state 'Gujarat' then we want to send updates of cases in Gujarat and in India and for users of Maharashtra, we want to send updates of cases in Maharashtra and in India. So to do that we'll create two different messages to send to the user and for all the users in our database we'll check the state and send the according message to them.Let's create a new function get_users to get the users mobile numbers and divide them into different arrays according to their state.

const User = require("../models/User"); // To retrieve users from the Database.

const get_users = async () => {
  let numbers = {};

  // Getting all the active users from the Database
  const active_users = await User.find({
    active: true,
  });

  // Getting an array of numbers of Active Users in Gujarat
  guj_users_num = active_users
    .filter((user) => user.state === "Gujarat")
    .map((user) => user.number);

  // Getting an array of numbers of Active Users in Gujarat
  mh_users_num = active_users
    .filter((user) => user.state === "Maharashtra")
    .map((user) => user.number);

  numbers.MH = mh_users_num;
  numbers.GJ = guj_users_num;

  return numbers;
};

So, as we can see we first get all the active users from our database and insert their mobile numbers in different arrays and add that arrays to the 'numbers' object with the corresponding key.

Now, let's create a new function send_msg() to create and send message to our users using the Twilio API.

const send_msg = async () => {
  // Getting Users' Mobile Numbers And Data From API.
  const users_num = await get_users();
  const cases_data = await get_data();

  // Message For Gujarat Users
  const gj_msg = `Total Cases In India: ${cases_data.total_cases} \n Total Cases In Gujarat: ${cases_data.gj_total} \n New Cases In India: ${cases_data.total_new} \n New Cases In Gujarat: ${cases_data.gj_new} \n #StayHome #StaySafe`;

  // Message For Maharashtra Users
  const mh_msg = `Total Cases In India: ${cases_data.total_cases} \n Total Cases In Maharashtra: ${cases_data.mh_total} \n New Cases In India: ${cases_data.total_new} \n New Cases In Maharashtra: ${cases_data.mh_new} \n #StayHome #StaySafe`;

  // Sending Messages To Users In Gujarat
  users_num.GJ.forEach((user) => {
    client.messages
      .create({
        body: gj_msg,
        from: process.env.PHN_NUM,
        to: "+91" + user,
      })
      .then((msg) => console.log(msg.sid))
      .catch((err) => console.log(err));
  });

  // Sending Messages To Users In Maharashtra
  users_num.MH.forEach((user) => {
    client.messages
      .create({
        body: mh_msg,
        from: process.env.PHN_NUM,
        to: "+91" + user,
      })
      .then((msg) => console.log(msg.sid))
      .catch((err) => console.log(err));
  });
};

Scheduling Automatic SMS

We have our message ready with us to send to users but how do we schedule it to send automatically at a specific time? Cron Jobs! The node-cron package helps us schedule the job to perform at a specific time.So, let's install this npm package and use it to schedule our send_msg function to send SMS automatically everyday.

$ npm install node-cron

const cron = require("node-cron");

const tasks = cron.schedule("0 9,20 * * *", () => {
  send_msg();

  modules.exports = tasks;
});

The schedule method of cron module takes two arguments, first the pattern of time that we want to schedule and second is the function that we want to schedule. I've set time to 9a.m. and 8p.m. Everyday.
Read More About Cron Patterns Here.

And now we'll have to call the method tasks.start() in our app.js file when the server starts like this.

const script = require("./script");

app.listen(process.env.PORT || 3000, () => {
  script.tasks.start();
});

And that's it.Now when we deploy our application on server, it will send sms to the users automatically everyday at 9 a.m. and 8 p.m. using the data retrieved by the external api. Here is what this message looks like.

And Bingo! We've created a fully functional application which registers users and sends them cases updates daily automatically. Now in the next post, we'll add functionality to receive sms to users in case they want to stop the service and then we'll deploy our application on heroku server.

Full Code Can Be Found Here:

jamesshah / covid19-updates

Simple web app to send daily covid-19 cases updates to users.

Thank You For Reading.

HappyCoding👩‍💻 👨‍💻

Building A Node.js - Express App (covid19-updates)|Part-2

James Shah — Sat, 11 Apr 2020 09:20:32 +0000

If you haven't read the last post, I recommend you to check it out here, else many things in this post will not make sense.

In the last post we've created a simple index('/') route with HTML file as response using EJS template engine.
Now, it's time to look into middlewares and how can we use them to serve our routes with the same prefix easily and create user registration.

Middlewares

Middleware functions are functions that have access to the request object (req), the response object (res), and the next function in the application’s request-response cycle.The next function is a function in the Express router which, when invoked, executes the middleware succeeding the current middleware.

For example, let's say you have a route ('user/login') in your express app to login the user. Now you want to check the credentials given by users with the credentials stored in the database and authorize that user to the account.And when do you check this credentials? After the user sends request and before your application sends the response. And that's when the middlewares comes in handy. You can write your logic to authenticate the user in the middleware function and response to the user according the return value of the middleware function.Other than authentication, middlewares have many uses as well. Read More About Middleware Here.

To use the middleware function, we need to add app.use(middlewareFunctionName); in our code. And that will add the defined middleware function to all the routes that we define in our application. It is called the 'application level middleware'. But we can also add middlewares to particular routes of our choice by providing that route as the first argument in the app.use function. i.e : app.use('/', middlewareFunctionName);

Now, let's add middlewares in our code to serve the routes from routes directory.

app.use('/', require('./routes/index'));

Now let's create the user registration page in our app. But that will require MongoDB database. So, I created an account on MongoDB Atlas, they provide 500MB of free database hosting. And I got the database URL from that which we'll be using to connect to our database. the url seems something like this.
"mongodb+srv://<username>:<password>@cluster0-kmdpn.mongodb.net/test?retryWrites=true&w=majority

Creating Database Model and Connecting to The Database

First of all, let's use mongoose and the database URL and see if we can connect to our databse.We will need mongoose package and we'll use it's connect method to connect to our database. Let's add the below code to our app.js file.

const mongoose = require("mongoose");

const DATABASE_URL = "mongodb+srv://<username>:<password>@cluster0-kmdpn.mongodb.net/test?retryWrites=true&w=majority";

mongoose.connect(DATABASE_URL, (err) => {
    if (err) console.log(err);
    else console.log("Connected to DB.");
})

Now we can start the server and see if we've connected to the database or not but we are currently working on a localhost server and this code is in our machine only but when we deploy our application online on cloud, we don't want everyone to see the DATABASE_URL as it contains our username and password of the database. So, to protect it we can use .env file to store our secret credentials.And while uploading it online we won't be adding this .env file. To use it let's install the 'dotenv' npm package.

$ npm install dotenv

Then,create a .env file in the project directory and add DATABASE_URL to this file. And to use the value from this file in our application we can make following changes.

require("dotenv").config();

// To connect to MongoDB Database
mongoose.connect(
  process.env.DB_CONN,  
  (err) => {
    if (err) console.log(err);
    else console.log("Connected to DB!");
  }
);

Okay!! So now we have our application connected to our database.
Now,let's create a database model which is basically a schema of our user's information. So first let's create a new directory 'Models' in our app directory and create a new file User.js. In this file, we'll write the below code to create and export our model.
We'll need 4 field in our User model. Name, Contact Number, State, and Active. In which, Name,Number and State fields will be String type and Active field will be Boolean type.

const mongoose = require("mongoose");
const UserSchema = mongoose.Schema({
  name: {
    type: String,
    required: true,
  },
  number: {
    type: String,
    required: true,
  },
  state: {
    type: String,
    required: true,
  },
  active: {
    type: Boolean,
    required: true,
    default: true,
  },
});

module.exports = mongoose.model("User", UserSchema);

Now, we have our User model ready with us. Now, we we'll create routes and create HTML page to register a user. Let's start by creating ('/users/register') route. Create a new file users.js in routes directory and add the below code.

const express = require("express");
const router = express.Router();

router.get('/register', (req,res) => {
    res.render('register');
});

module.exports = router;

Creating User Registration Form and Registering Users

Here we've used an in-built express middleware Router. Using router instance, we can easily serve routes with same prefix which is 'user'. We'll be creating more routes with this 'user' prefix and therefore we can take advantage of the Router middleware.Now, let's create a form for users to register in the register view that we are rendering to the register route.

<div class="jumbotron">
  <form action="/users/add_user" method="post">
    <fieldset>
      <legend
        style="text-align: center; text-decoration: underline;"
        class="mt-3"
      >
        Register Now
      </legend>
      <div class="form-group">
        <label for="name">Name</label>
        <input
          type="text"
          name="name"
          class="form-control"
          id="name"
          required="true"
          placeholder="Enter Your Name"
        />
      </div>
      <div class="form-group">
        <label for="phone">Phone Number</label>
        <input
          name="number"
          type="text"
          class="form-control"
          id="number"
          required="true"
          aria-describedby="phoneHelp"
          placeholder="Enter Your Number"
        />
        <small id="phoneHelp" class="form-text text-muted"
          >We'll never share your number with anyone else. Promise!</small
        >
      </div>
      <div class="form-group">
        <label for="state">State</label>
        <select
          class="form-control"
          id="state"
          name="state"
          style="cursor: pointer;"
          required="true"
        >
          <option
            selected="selected"
            id="state_default"
            value=""
            style="display: none;"
            >Select State</option
          >
          <option value="Gujarat">Gujarat</option>
          <option value="Maharashtra">Maharashtra</option>
        </select>
      </div>
      <button type="submit" class="btn btn-primary">Count Me In!</button>
    </fieldset>
  </form>
</div>

It should look something like this,

As we've added the path /users/add_user in the action attribute of the form tag, let's create that route in our users.js file to register user to our database.

In this, first we'll check if the phone number entered by the user already exist in our database, if it exists we'll redirect the user to the already_registerd page and if it doesn't exist already, we'll add it our database and redirect the user to the register/successful page. This is going to be a little bit complex, let's start coding.

const User = require("../models/User"); // To access the User Model

router.post("/add_user", async (req, res) => {
  try {     
    const { name, number, state } = req.body; // Getting Form Fields From The Request Body.    
    const new_user = await User.aggregate([{ $match: { number: number } }]); // Checking if the number already exists.
    if (new_user.length != 0) {
      res.render("already_registered",
        {
          message: "It Seems Like, This number is already registered with us.",          
        }
      );
    } else {
        // Create a new user object to add to the DB.
      const user = new User({
        name: name,
        number: number,
        state: state,
      });
      const addedUser = await user.save(); // Saving the user to the database.
      res.redirect("/users/register/successful");
    }
  } catch (error) {
    console.log(error);
    res.json({ message: error });
    }
};

And Done!! Really?? No, This code will generate an error and that's because we can't directly use the request body in our app. We'll have to convert it to JSON to be able to use it in our application. And to do that we'll use body-parser package. Let's add that in app.js and we are done with the registration!!

$ npm install body-parser;

const bodyParser = require("body-parser");

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

And now we are finally done!! Now in the next post we'll add the most exciting part of this application - the Twilio API and we'll send the users an SMS when they register to let them know they have registered successfully and also we'll create a route to receive the message from the users, if they want to stop our service. Fun things are coming up!!

Full code can be found at this repository :

jamesshah / covid19-updates

Simple web app to send daily covid-19 cases updates to users.

covid19-updates (India)

Simple web app to send daily covid-19 cases updates to users using Node, Express.js, Twilio Api and some other packages.

About

This application uses Twilio API to send daily covid-19 cases updates to the registered users.
Data Source - Covid 19 India API (Unofficial)
This web app is a part of Dev-Twilio Hackathon on dev.to

How it works

Users register with their Name, Phone Number and State (currently only Gujarat and Maharashtra is available) and they will get daily updates of cases in their state and in India, twice a day(9 am & 8 pm) via SMS (Twilio SMS Api)
User can choose to stop the service by sending "STOP" to the number(Twilio Number).

Features

Node.js web server using Express.js
Basic web user interface using EJS for templating and Bootstrap for UI
User interface to register
Project Specific environment variables using .env files.

Usage

Requirements

Node.js
A Twilio…

View on GitHub

Thank you for reading!!