Forem: Revanth Reddy

TAMU DATATHON

Revanth Reddy — Sun, 17 Oct 2021 13:51:19 +0000

DaHack - Hackathons Dataset!

GitHub Link: https://github.com/revanth-reddy/tamudatathon

Hackathon's Dataset in Hackathon

This Dataset is built as a part of TAMU Datathon 2021. It contains the details of hackathons which include Title, Location, Start Date, End Date, Prize Money, Number of Participants, Host of Hackathon, Themes in the Hackathon

As a part of Tamu Datathon 2021 challenges, we chose TD Data Synthesis Challenge. We approached the problem of synthesis of Hackathons Dataset throughout the world.

Construction of Dataset

The data for this dataset is obtained by Web Scraping using Python and Selenium.

Size

We are able to generate a dataset of 6,236 hackathons due to time constraints. However this size can be tremendously increased.

Uniqueness

This is the first Hackathon's Dataset with 8 unique features that are the major datapoints of a Hackathon

Usefulness

To Observe the trends:
- Number of Hackathons per year.
- Hackathons Prize Money Trend over years.
- Rise of Themes through the years.
- Participation in the Hackathon across years/themes.
- And many more ...
To Build ML models in order :
- to predict Prize Money based on other features like participation, themes, location.
- to estimate user participation based on location, prize money, themes, etc.
- to analyze relationship between the above features.

Visualization

Interactive Visualization: https://datastudio.google.com/reporting/269afbc3-e982-4fef-aee5-08010b427070 (Open to interactive with visualization playground)
User Interactive graphs and charts to analyze the hackathon's dataset

Team's Approach

To build a dataset, we need to find a potential and authentic source, for which we used devpost as the source.
To get the data from Website, we used Web Scraping tools which helps in getting the raw data
We have used Selenium to scrape the website.
We used a Chrome Web Driver to create the automation of scrolling and scraping the data.
After Scraping, we got the raw data which is processed further and cleaned.
The clean data is stored as a dataset in a CSV file.
Unique datapoints like Number of participants, Start Date, End Date, Location(Online/Offline location), Prize Money, Themes
After obtaining a decent dataset, we used Google Studio to provide insights of the dataset and relations between the variables in the dataset.

Reflection

### Problems
- While scraping the website, the content inside website is not rendered as static content.
- As the content is dynamic one, it is hard to scrape.
- The website has a pagination effect based on page scrolling. So, initially x items are loaded in the website and on scrolling another x items are loaded and so on.
- Unless we scroll we can't scrape the entire dataset.
### Future Scope
- The Dataset can be used to build a Machine Learning Model (to help in picking a theme/prize money/participant estimation and also in analyzing the trends of Hackathons)
- Automation of Scraping using Job Schedulers(CI/CD) to scrape data periodically
- Large data can be scraped from various sources
- Scraping process can be further optimized by using mutlithreading. ## Team DaHack

Build Notifications @ Slack using GitHub Notifications

Revanth Reddy — Sat, 15 Aug 2020 15:46:44 +0000

If you have a team and if there are many people in the team pushing to GitHub then it might be hard sometimes to check the build status of an active project. The slack notification GitHub action will notify the team regarding the build status.

My Workflow

To enable the slack notifications, I searched for the marketplace for one such action and found this:

act10ns / slack

Slack messages for GitHub Action workflows, jobs and steps

Slack messages for GitHub Actions workflows, jobs and steps

A simple and flexible Slack integration with GitHub Actions.

Features

Advanced users can use a configuration file and Handlebars templates to configure every aspect of the Slack message.
In addition to "legacy" attachments, rich messages can be created using layout blocks for flexible message visualisation and interactivity.

Configuration

Environment Variables (`env`)

`SLACK_WEBHOOK_URL` (required)

Create a Slack Webhook URL using either the Incoming Webhooks App (preferred) or by attaching an incoming webhook to an existing Slack App (beware, channel override not possible when using a Slack App):

env
  SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Input Parameters (`with`)

`status` (required)

The status must be defined. It can either be the current job status using:

  with: 
    status: ${{ job.status }}

or a hardcoded custom status such as "starting" or "in progress":

  with: 
    status: in progress

`steps` (optional)

The individual status of job…

View on GitHub

I used this action in my django project by adding some lines of code in .github/workflows/django.yml.

name: Django CI
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      max-parallel: 4
      matrix:
        python-version: [3.6]
    env:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
    steps:
    - uses: act10ns/slack@v1
      with:
        status: starting
        channel: '#build-notifications'
      if: always()
    - name: Checkout
      uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run Tests
      run: |
        python manage.py test shop -v 2
    - name: Slack Notification
      uses: act10ns/slack@v1
      with:
        status: ${{ job.status }}
        steps: ${{ toJson(steps) }}
        channel: '#build-notifications'
      if: always()

This is my .yml workflow file. You can add these lines to your workflow file and get the job done.

Observe that I have used act10ns/slack@v1 which is the github action responsible for all this.

Here secret is similar to an environment variable that can be added by going to settings > secrets >. Add key as SLACK_WEBHOOK_URL and pair as a webhook URL that can be obtained from Incoming Webhooks App to generate slack messages

Submission Category:

DIY Deployments

Yaml File

Link to Yaml File

Link to Code

revanth-reddy / Raitubazaar

Platform between fruits&vegetable vendors(farmers) and end-users

RaituBazaar

Follow me @

View on GitHub

Additional Resources

Slack GitHub Actions
Slack GitHub Actions Repo

PS: I have implemented the Slack Notification GitHub Action in one of my existing projects. You can add it to your existing projects too.

Keep Exploring !!!

Introducing GraphQL ft. Django

Revanth Reddy — Fri, 14 Aug 2020 06:55:33 +0000

Hey There 👋!

Hope everyone is safe and healthy. It's been 4 months since I was stuck at home because of the CoVid19 pandemic. So, during these times, while I was working on a REST Api for one of my Application, I encountered an article regarding GraphQL and that really impressed me and led me to write this post. In traditional REST APIs, we create URLs for all the data objects that are needed to be accessed. If we wanted to create a Question-Answer Forum - we'll need URLs for all Questions, URL for each question and URL for all answers of that specific question. So we are to do a lot of requests for such related data.

And what if the user is experiencing a slow internet connection?

Here comes GraphQL into the play. GraphQL isn't an API architecture like REST. It is a Query language to retrieve and manipulate related data in an easier way.

History

Having its origin from Facebook, GraphQL has finally moved into the hands of GraphQL Foundation.

Let's create an application for Question-Answer using GraphQL, Django.

`Let's Go 🚀`

Schema

Creating Types

Types depict the data that the data object possess.

Consider:

type Answer {
  id: ID!
  answer: String!
}

type Question {
  id: ID!
  question: String!
  answers: [Answer]
}

The ID type tells us that the field is the unique identifier for that type of data(also known as primary key).

Note: The exclamation mark ! signifies that the field is required.

Here we have primitive type String (others being Int, Char)
and also the custom type Answer

The list type is enclosed in square brackets([Answer])

Queries

A Query is a question that we ask in to retrieve the data:

type Query {
  question(id: ID!): Question
  answer(id: ID!): Answer
  answers: [Answer]
  questions: [Question]
}

The above query will retrieve Question and Answer using their ID, and also Question list, Answer list without any filter

Mutations

Mutation tells about the operations that can be done to change the data.
Mutations need two things:

Inputs: It is used to send an entire object rather than the individual fields.

input AnswerInput {
  id: ID
  answer: String!
}

input QuestionInput {
  id: ID
  question: String!
  answers: [AnswerInput]
}

Payloads: These depict the output of a mutation(just like a response from an API Endpoint)

type AnswerPayload {
  ok: Boolean
  answer: Answer
}

type QuestionPayload {
  ok: Boolean
  question: Question
}

Here ok field is similar to status in an API response.

And Mutation binds inputs & payloads together:

type Mutation {
  createAnswer(input: AnswerInput) : AnswerPayload
  createQuestion(input: QuestionInput) : QuestionPayload
  updateAnswer(id: ID!, input: AnswerInput) : AnswerPayload
  updateQuestion(id: ID!, input: QuestionInput) : QuestionPayload
}

Schema

Finally, we build the schema. It contains both the queries and mutations

schema {
    query: Query
    mutation: Mutation
}

Python & GraphQL

GraphQL is platform-independent, and its users are free to use any language(Python, Java, PHP, Go ...), frameworks(Node, Rails, Django).

In Python we need not use GraphQL's syntax for creating the schema, we can directly use python but for that, we need a library called Graphene.

`Let's get into Action 👨‍💻`

It's a good practice to maintain a virtual environment, requirements file and gitignore file.

Let's create a folder and then initialize a virtual environment myvenv(you can use your virtual environment name) in it.

mkdir quorum && cd quorum
python3 -m venv myvenv

Now let's activate the virtual environment myvenv.

source myvenv/bin/activate

Note: To exit the virtual environment, type deactivate in the terminal.

Installing packages

Inside the virtual environment, we use pip to install the required libraries:

pip install django
pip install graphene_django

And we create a Django project:

django-admin startproject qproj .

A django project can have many applications. Apps are like the components within a project which can be reused.

Let's create an app inside this project.

python manage.py startapp qapp

After creating the app we need to tell Django that it should use it. We do that in the file qproj/settings.py --open this folder in your code editor and open settings.py file.

We need to find INSTALLED_APPS and add graphene_django & qapp
so finally it should look like this:

...
...
INSTALLED_APPS = [
    'graphene_django',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'qapp',
]
...
...

Creating Models

In the qapp/models.py file we define all objects called Models – this is a place in which we will define our Question and Answer models.

Let's open qapp/models.py in the code editor, remove everything from it, and write code like this:

from django.db import models

class Question(models.Model):
    question = models.CharField(max_length=100)

    def __str__(self):
        return self.question

class Answer(models.Model):
    answer = models.CharField(max_length=100)
    question = models.ForeignKey(Question, on_delete=models.CASCADE)

    def __str__(self):
        return self.answer

Now we have the blueprint of our models but we need to imprint that in our database. In django this thing is carried out by making migrations and migrating them.

python manage.py makemigrations
python manage.py migrate

Now add a superuser(admin) who can handle the admin panel using:

python manage.py createsuperuser

In qapp/admin.py we register our models so that admin can access these models:

from django.contrib import admin
from .models import Question, Answer

admin.site.register(Question)
admin.site.register(Answer)

Now run server using python manage.py runserver and visit http://127.0.0.1:8000/admin or http://localhost:8000/admin to see the admin dashboard.
Add the relevant data to questions and answers.

Schema with Graphene

Queries

In qapp folder, create a new file called schema.py and define it as follows:

import graphene
from graphene_django.types import DjangoObjectType, ObjectType
from .models import Question, Answer

# Create a GraphQL type for the Answer model
class AnswerType(DjangoObjectType):
    class Meta:
        model = Answer

# Create a GraphQL type for the Question model
class QuestionType(DjangoObjectType):
    class Meta:
        model = Question

In the same file add the following Queries:

# Create a Query
class Query(ObjectType):
    question = graphene.Field(QuestionType, id=graphene.Int())
    answer = graphene.Field(AnswerType, id=graphene.Int())
    questions = graphene.List(ActorType)
    answers= graphene.List(AnswerType)

    def resolve_answer(self, info, **kwargs):
        id = kwargs.get('id')
        if id is not None:
            return Answer.objects.get(pk=id)
        return None

    def resolve_question(self, info, **kwargs):
        id = kwargs.get('id')
        if id is not None:
            return Question.objects.get(pk=id)
        return None

    def resolve_answers(self, info, **kwargs):
        return Answer.objects.all()

    def resolve_questions(self, info, **kwargs):
        return Question.objects.all()

Here we have written how the query works: as a single object using an ID and as a list.

Schema

Add this to the end of qapp/schema.py

schema = graphene.Schema(query=Query)

Registering the schema

To make this API available through the project we need to register and for that we need to import it in qproj.
So create a new schema.py in qproj folder and the code:

import graphene
import qapp.schema

class Query(qapp.schema.Query, graphene.ObjectType):
    # This class will inherit from multiple Queries
    # as we begin to add more apps to our project
    pass

schema = graphene.Schema(query=Query)

In settings.py add the GRAPHENE schema. You can add it anywhere inside it. I'm adding it between INSTALLED_APPS and MIDDLEWARE.

GRAPHENE = {
    'SCHEMA':'qproj.schema.schema'
}

Unlike naive APIs, data can be accessed using only one endpoint in GraphQL. Let's create that url endpoint as graphql/ in urls.py

Open qproj/urls.py and modify the code to:

from django.contrib import admin
from django.urls import path
from graphene_django.views import GraphQLView
from .schema import schema

urlpatterns = [
    path('admin/', admin.site.urls),
    path('graphql/', GraphQLView.as_view(graphiql=True)),
]

Test Test Test

To check the API, we need to run the project. In terminal run the following code:

python manage.py runserver

If there are no errors you can see a server running at port 8000 as follows

Once the server is ready goto http://127.0.0.1:8000/graphql/. You'll find a GraphQl editor. In this playground you can write queries, mutations(we didn't create mutations yet).

Building Queries

In the admin panel I have added some relevant Questions and answers

Now write some queries in the graphql editor at http://127.0.0.1:8000/graphql/

Run the query by clicking on the run button to see the output.

Till now, we have are doing the querying part in graphql ide but what if you want to do it as a POST request
An application communicating with your API would send POST requests to the /graphql endpoint. Before we can make POST requests from outside the Django project, we need to change urls.py as:

from django.contrib import admin
from django.urls import path
from graphene_django.views import GraphQLView
from .schema import schema
from django.views.decorators.csrf import csrf_exempt # New library

urlpatterns = [
    path('admin/', admin.site.urls),
    path('graphql/', csrf_exempt(GraphQLView.as_view(graphiql=True))),
]

You can turn off the graphql editor(Graphiql) by replacing graphiql=True with graphiql=False

CSRF means (Cross-Site Request Forgery)protection- an inbuilt feature of Django to prevent unauthenticated/ improperly authenticated users from doing evil things 👻.
Though it's useful it would prevent external applications from interacting with the API. To counter the scarf exemption you might consider authentication or a more secured way to access the API.

Now enter the following curl request in a new terminal keeping the server active

curl -X POST -H "Content-Type: application/json" --data '{ "query": "{ questions { question } }" }' http://127.0.0.1:8000/graphql/

You'll get a result something like this:

Last Word

Here we didn't work on changing the data(Mutations), you are to explore that part 😜.

Code References:

GitHub Repo

My First Post

Revanth Reddy — Thu, 21 May 2020 02:58:21 +0000

This is my first blog post ever. Hopefully, I come up with a better post next time.

Hello World! I am Revanth Reddy, a final year Comp. Science undergrad from National Institute of Technology Durgapur.

Long back, probably in my first year of college, I was introduced to Git and GitHub. It is really cool and interesting. And the curiosity led me to obtain my GitHub Student Education Pack. Using it I got my first domain http://revanth.tech/ . I learned React and using my React knowledge and sources from GitHub I developed my PortFolio and hosted it using GitHub pages and linked it to my domain.

GitHub Education pack is really cool with many benefits for students. It helped me in learning Cloud services like AWS, Azure. I even told my friends and helped them in obtaining educational packs!

In case if you don't have a pack then get one now! https://education.github.com/pack

Forem: Revanth Reddy

TAMU DATATHON

DaHack - Hackathons Dataset!

Construction of Dataset

Size

Uniqueness

Usefulness

Visualization

Team's Approach

Reflection

Build Notifications @ Slack using GitHub Notifications

My Workflow

act10ns / slack

Slack messages for GitHub Action workflows, jobs and steps

Slack messages for GitHub Actions workflows, jobs and steps

Features

Configuration

Environment Variables (env)

SLACK_WEBHOOK_URL (required)

Input Parameters (with)

status (required)

steps (optional)

Submission Category:

DIY Deployments

Yaml File

Link to Code

revanth-reddy / Raitubazaar

Platform between fruits&vegetable vendors(farmers) and end-users

RaituBazaar

Additional Resources

Introducing GraphQL ft. Django

History

Let's Go 🚀

Schema

Creating Types

Queries

Mutations

Schema

Python & GraphQL

Let's get into Action 👨‍💻

Installing packages

Creating Models

Schema with Graphene

Queries

Schema

Registering the schema

Test Test Test

Building Queries

Last Word

Code References:

My First Post

Environment Variables (`env`)

`SLACK_WEBHOOK_URL` (required)

Input Parameters (`with`)

`status` (required)

`steps` (optional)

`Let's Go 🚀`

`Let's get into Action 👨‍💻`