Forem: tinega

Dependency management in Python Using Poetry

tinega — Fri, 31 May 2024 14:43:28 +0000

Introduction

Dependency management in Python is an important component of software development because it entails managing the libraries and packages required by a project. Proper dependency management ensures that the necessary dependencies are installed, conflicts are avoided, and the project is reproducible across different environments.

Concepts

Dependencies
Dependencies are external libraries or modules that a project relies on to function. These can range from standard libraries included with Python to third-party packages available on repositories like PyPI (Python Package Index).
Dependency Management Tools
Include tools such as:
- pip: The default package installer for Python.Uses requirements.txt to list dependencies and their versions.
- virtualenv: Creates isolated Python environments to manage dependencies for different projects independently.
- venv: Provides similar functionality to virtualenv but is built into Python.
- conda: A package manager and environment management system that supports multiple languages.
- poetry: A modern tool for dependency management and packaging.Uses pyproject.toml to specify project configurations and dependencies.

Poetry

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution.

Pyproject.toml file

pyproject.toml is a configuration file used by packaging tools such as poetry.
example

[tool.poetry]
name = "poetry-test"
version = "0.2.0"
description = ""
authors = ["None"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
requests = "^2.30.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Installation

Poetry requires Python 3.8+

Linux,macOS

curl -sSL https://install.python-poetry.org | python3 -

Windows(Poweshell)

(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

Add Poetry to your PATH(Environment Variables)

Add the following paths to PATH variable depending on your system.
$HOME/.local/bin on Unix.
%APPDATA%\Python\Scripts on Windows.

poetry --version: verify that poetry is installed correctly and added to $PATH.

Usage

poetry new <package name>: starts a new Python project by creating a new directory with a standard Python project structure.

new-package
  new-package
      __init__.py
  tests
      __init__.py
  pyproject.toml
  README.md

poetry init: Initializes a directory by prompting you to provide details about your project and its dependencies interactively.
poetry config virtualenvs.in-project true: create virtual environments inside the project's directory.Poetry by default will create a virtual environment under {cache-dir}/virtualenvs.
poetry shell : activate the virtual environment created.
poetry add <django>: adds a dependency to your project.This will add django to your pyproject.toml file and install django.

# Allow >=2.0.0, <3.0.0 versions i.e minor version
poetry add requests^2.0.0

# Allow >=2.0.0, <2.1.0 versions i.e patch versions
poetry add requests~2.0.0

# Allow only a specific version
poetry add requests==2.0.0 | poetry add requests@2.0.0

poetry add --dev <pytest>: add dependencies only needed in development.Pytest will be added as a development dependency.
poetry remove <package name>: removes a package from the current list of installed packages.
poetry show: list all the available packages.
poetry show <package name>: list details about a specific package.
poetry version: shows the current version of the project.
poetry version <patch|minor|major>: bumps the version of the project and writes to pyproject.toml.
poetry list: displays all the available Poetry commands.
poetry install: reads the pyproject.toml file from the current project, resolves the dependencies, and installs them.

Commands
Importing existing requirements.txt file to poetry
pyproject file

A Docker Compose Example (FlaskAPP + PostgreSQL + Traefik).

tinega — Tue, 06 Feb 2024 13:27:49 +0000

Introduction

Docker Compose is a tool used for defining and running multi-container Docker applications. It allows developers to describe all the services that make up an application in a single YAML file, defining the relationships and configurations between them. Docker Compose then uses this YAML configuration to deploy and manage the application's containers as a single unit. Docker Compose is particularly useful for development environments, testing, and deploying multi-container applications in a consistent and reproducible manner. It simplifies the process of managing complex application architectures by providing a declarative configuration file and a set of commands for common operations.

Components of Docker compose

compose file

A YAML file is used to define the configuration of the entire application stack, including services, networks, and volumes. It specifies details such as the Docker images to use, container dependencies, environment variables etc.

The default path for a Compose file is compose.yaml or compose.yml. docker-compose.yml/docker-compose.yaml also supported for backwards compatibility.

Services

Each service in a Docker Compose file represents a containerized application component. Services can be built from a specified Dockerfile or use pre-built images from a registry. You can define the relationships and dependencies between services.

Volumes

Volumes can be defined to persist data outside the containers. This is crucial for storing data that needs to survive container restarts or updates.

Environment Variables

Docker Compose allows you to set environment variables for services.

Networks

Docker Compose allows you to define custom networks to connect containers and control communication between them. Services can be attached to specific networks to isolate or expose their communication.

A Sample compose file

The Docker Compose file below defines a set of services that work together to run a Flask-based todo application along with a PostgreSQL database and Traefik as a reverse proxy/load balancer.

services:
  todo:
    image: tinegagideon/flask-todo
    ports:
      - "5000:5000"
    environment:
      - DATABASE_URL=postgresql+psycopg2://postgres:postgres@todo-db:5432/postgres
    labels:
      # Explicitly tell Traefik to expose this container
      - "traefik.enable=true"
      # The domain the service will respond to
      - "traefik.http.routers.todo.rule=Host(`localhost`)"
    depends_on:
      - todo-db
      - traefik

  todo-db:
    image: postgres:14
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_DB=postgres
    volumes:
      - pgdata:/var/lib/postgresql/data

  traefik:
    image: traefik:v3.0
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

volumes:
  pgdata:

Compose file breakdown

The compose file is divided into two sections: services and volume, with three services.

Services Section

todo Service

It uses an image named tinegagideon/flask-todo for the Flask todo application.
Exposes port 5000 of the container to port 5000 of the host machine.
Sets environment variables for the Flask application, including the database URL.
Uses labels to configure Traefik to expose this service and defines the routing rule.
Depends on the todo-db service and the traefik service.

todo-db Service

Uses the official PostgreSQL image with version 14.
Exposes port 5432 of the container to port 5432 of the host machine.
Sets environment variables for the PostgreSQL database i.e the password, username, and database name.
Mounts a volume named pgdata to persist PostgreSQL data.

traefik Service

Uses the Traefik image with version 3.0.
Specifies command-line arguments for Traefik:

--api.insecure=true
enables insecure access to the Traefik API.

--providers.docker=true
enables Docker as a provider.

--providers.docker.exposedbydefault=false
specifies that services must be explicitly exposed.

--entrypoints.web.address=:80
defines the entry point for HTTP traffic.

Exposes port 80 and port 8080 of the container to the corresponding ports on the host machine.
Mounts the Docker socket to interact with Docker daemon inside the Traefik container.

Volumes Section

Defines a named volume named pgdata that is used by the todo-db service to persist PostgreSQL data.

Crafting a Docker Image for Your Flask Web App

tinega — Fri, 12 Jan 2024 19:20:28 +0000

Introduction🐋

Docker is a revolutionary containerization platform that streamlines software development and deployment. It encapsulates applications and their dependencies, ensuring consistency across different environments. Docker eliminates the "it works on my machine" problem, promoting collaboration and scalability. Its efficient resource utilization and isolation capabilities make it a vital tool for modern software development.

Prerequisites 🗒

Flask Application.
Docker Follow these steps to install docker to your computer.
Docker Hub Account create a free account.

I will assume that you have a basic understanding of python, especially the Flask framework and an understanding of networking ports.

Sample project 🏗️

Project Directory Structure:

hello-flask/
├── app/
│   ├── __init__.py
│   └── app.py
├── requirements.txt
└── Dockerfile

Sample code in app.py file:

from flask import Flask

app = Flask(__name__) # create an instance of our imported Flask class.


@app.route("/")
def home():
    return 'hello flask'

# allows the code to be executed directly as a script
if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

Official Flask Documentation

Dockerfile 📝

Docker automates image creation by interpreting instructions from a Dockerfile, a text file containing sequential commands required to build a specific image. A Docker image comprises immutable layers, each representing an instruction from the Dockerfile. These layers are arranged in a stack, with each one capturing the incremental changes made compared to the previous layer.
create a Dockerfile with the following instructions:

FROM python:alpine

RUN apk update 
RUN apk add -y python-pip; pip install --no-cache-dir --upgrade pip
WORKDIR /hello-flask
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY /app .
ENTRYPOINT [ "python" ]
CMD [ "app.py" ]

Each instruction break down

FROM python:alpine
It specifies the base image to use, in this case, python:alpine. This image is based on the Alpine Linux distribution and includes Python.

Note that while it's possible to use a Debian-based distro as the base image, it would require manual installation of Python within the image. A python image based on alpine is smaller size compared to other distributions, therefore contributing to a more lightweight and efficient container.

RUN apk update
This updates the package index on the Alpine system using the apk package manager.

RUN apk add -y python-pip; pip install --no-cache-dir --upgrade pip
This installs python-pip using apk and then upgrades pip to the latest version. The -y flag is used to automatically confirm the installation.

WORKDIR /hello-flask
Sets the working directory inside the container to /hello-flask.

COPY requirements.txt .
Copies the requirements.txt file from the build context (where the Dockerfile is located) into the /hello-flask directory in the container.

RUN pip install -r requirements.txt
Installs the Python packages listed in requirements.txt using pip. This is typically done in a separate step to take advantage of Docker layer caching.

COPY /app .
Copies the contents of the /app directory from the build context into the current working directory /hello-flask in the container.

ENTRYPOINT [ "python" ]
Specifies the default executable when the container starts. In this case, it sets the default executable to Python.

CMD [ "app.py" ]
Provides default arguments for the ENTRYPOINT. When the container is run, it will execute python app.py by default.

Building your docker image 🔨

The docker build command builds Docker images from a Dockerfile and a context.

docker image build -t myrepo/hello-flask .

This will build an image with the repository name as myrepo/hello-flask and a tag of latest is applied by default.
The -t option allows you to tag your Docker image with a specific name and optionally a tag.
Tags are typically appended to the image name by specifying them following a colon.

The dot at the end of the command specifies the build context. The build context is the path to the directory containing the Dockerfile and any files needed for the build process. In this case, it's set to the current directory

Running a container from your image 📦

The docker container run command runs a command in a new container, pulling the image if needed and starting the container.

docker container run --rm -p 5000:5000 myrepo/hello-flask

replace myrepo with your docker hub username or if you don't intend to upload to image just use name of the image.e.g

docker container run --rm -p 5000:5000 hello-flask

--rm
This flag removes the container automatically after it stops running. It helps to keep the system clean by avoiding the accumulation of unused containers.

-p 5000:5000
This flag maps the port 5000 from the host to the port 5000 on the container. It allows communication between the host machine and the running container through this specified port.

myrepo/hello-flask
This is the name of the Docker image used to create the container. It's assumed that you have an image named hello-flask in the myrepo repository.

another useful OPTION to add to the docker container run command is:
--detach or -d this runs the container in the background and prints its ID.

You should be able to view a webpage by visiting http://localhost:5000

Pushing your image to docker hub 🚀

Before trying to push your image make sure to login in the docker CLI using docker login with your docker hub credentials.

Use docker image push to upload your images to the Docker Hub registry or to a self-hosted one.

docker image push myrepo/hello-flask

Other Commands

docker image tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG] -- create an image tag
docker image ls -- lists images in your local machine.
docker container ls or docker ps [OPTIONS] -- lists containers.
docker container stop [container id | container name] -- stop a running container
docker container rm [container id | container name] -- remove container
docker image prune [OPTIONS] -- remove unused images.

Images Vs Containers

An image represents the application we intend to execute, while a container is an instance of that image running as a process.

Think of a Docker image as a recipe or a blueprint for creating a specific environment, and a Docker container as a ready-made dish prepared using the recipe (image).

Elevating PRs with the Magic of an AI-Agent for a Seamless Experience: PR-Agent Vs Copilot For PRs

tinega — Thu, 07 Dec 2023 10:14:29 +0000

Pull Request

A pull request (PR) is a fundamental concept in version control systems, particularly in distributed version control systems like Git. It is a mechanism used to propose changes to a codebase in a collaborative software development environment.

Benefits of a pull request workflow

Isolation of Changes: Each pull request is associated with a specific branch, allowing changes to be isolated until they are thoroughly reviewed and tested.
Collaborative Code Review: They facilitate collaborative code reviews, allowing multiple team members to review and discuss proposed changes before merging.
Continuous Integration: Integration with CI tools ensures that changes do not break existing functionality and meet project standards before being merged.
Improve code quality and performance: When creating a pull request, you're inviting a colleague to review your code, seeking constructive feedback. This collaborative process aims to enhance code quality through the insights and suggestions provided by the reviewer.

Pull requests play a pivotal role in version control systems. Tools like Copilot for PRs and CodiumAI PR-Agent serve the purpose of enhancing PR descriptions and facilitating faster review and merging processes, thereby contributing to improved collaboration within your team.

CodiumAI's PR-Agent

CodiumAI's PR-Agent is an open source AI agent tool designed to streamline and expedite the process of reviewing pull requests (PRs). This tool automatically analyzes both the commits and the PR itself, delivering various types of feedback to enhance the efficiency of the review process. CodiumAI's PR Agent features include:

PR Review: Feedback about the PR main theme, type, relevant tests, security issues, focused PR, and various suggestions for the PR content. 
Auto-Description: Automatically generating PR description - name, type, summary, and code walkthrough.
Question Answering: Answering free-text questions about the PR. 
Code Suggestion: Committable code suggestions for improving the PR.

Getting started with CodiumAI's PR-Agent

For public repositories

Just mention the name @CodiumAI-Agent in your git provider environment and ask to /review, /describe, /improve, or /ask anything e.g. "Is my PR documented enough?", and the agent will perform these tasks for you.

For private repositories and open-source projects:

You will first need to install the app on you private repo. Just follow this detailed step by step guide. Then summon your pull request helping agent easily, just write /review, /describe, /improve or /ask anything e.g. "is my PR documented enough?", and the agent will perform these tasks for you.

Try it it's Free

In any PR comment on your public GitHub repository, mention @CodiumAI-Agent with a desired command in any PR comment. The agent will generate a response based on your command.

PR-Agent Commands

Explore the links for in-depth details and examples related to the commands.

✅ /review: Scans the PR code changes, and automatically generates a PR review.

✅ /ask "...": Answers questions about the PR, based on the PR code changes.

✅ /improve: Scans the PR code changes, and automatically generates committable suggestions for improving the PR code.

✅ /describe: Scans the PR code changes, and automatically generates PR description - title, type, summary, code walkthrough and labels.

✅ /add_docs: Scans the PR code changes, and automatically suggests documentation for the undocumented code components (functions, classes, etc.).

✅ /similar_issue: Retrieves the most similar issues to the current issue.

✅ /update_changelog: Automatically updates the CHANGELOG.md file with the PR changes.

✅ /generate_labels: Scans the PR code changes, and given a list of labels and their descriptions, it automatically suggests labels that match the PR code changes.

Copilot for PRs

Copilot for PRs brings the power of Copilot to the PR experience, to help you write better PR descriptions, and to help your team review and merge PRs faster. Copilot for PRs features include:

Suggestions for your pull request description : allows developers to insert marker tags in their pull request description. When the description is saved, Copilot will expand the marker into a description of the changes in the pull request.
AI-powered review responses: auto-generate changes in response to requests to add documentation or even to add another test
AI-powered PR repair: Proposes a fix to forgotten documentation and unit test updates during the PR process.
AI-powered PR completion: Copilot's bot is being empowered with the capability to complete pull requests given a few indicative edits such as adding comments to all your functions, throughout all your code.
Gentest: Copilot's prototype uses AI to identify changes in your PR that may be lacking tests, and suggests tests for you to build on or use directly.
Resolving issues with AI: automatically describes how to solve an issue and even to automatically suggest the changes you need to make.

Getting started with Copilot for PRs

Sign in with GitHub. This dashboard will be used to manage your Copilot for PRs settings.
Install the GitHub App. You must have admin access to the repository or be an owner of the organization to install the GitHub App.
Add your repos to the waitlist. Copilot for PRs is available as a technical preview in order to identify if it brings value to our users. We're limited on capacity, so you'll need to add your repositories to our waitlist.
Include marker tags in your PR description. Copilot will automatically expand them for you.

Copilot for PRs Markers/commands

copilot:all: showcases all the different kinds of content in one go.
copilot:summary: expands to a one-paragraph summary of the changes in the pull request.
copilot:walkthrough: expands to a detailed list of changes, including links to the relevant pieces of code.
copilot:poem: expands to a poem about the changes in the pull request.

Comparison

Platforms 💻

CodiumAI's PR-Agent offers extensive compatibility with various Git providers such as GitHub, GitLab, Bitbucket, and CodeCommit. Its versatility is showcased through multiple usage options including CLI, GitHub Action, GitHub App, Docker, among others. While Copilot for PRs currently caters exclusively to GitHub, this platform-specific support seamlessly integrates with GitHub workflows. Additionally, PR-Agent provides users with the flexibility to choose from a diverse range of models, including GPT-4, GPT-3.5, Anthropic, Cohere, and Llama2, allowing for a customized experience based on specific project requirements or user preferences.

Source code 📜

CodiumAI's PR-Agent operates on an open-source basis, actively encouraging contributions from the community. In contrast, GitHub Copilot is not open source; it is a proprietary product developed by GitHub, a subsidiary of Microsoft. While GitHub Copilot utilizes OpenAI's Codex model as its underlying technology, the specific implementation and integration with GitHub remain proprietary.

Ease of Use 👍

When it comes to installation and use case scenarios, CodiumAi provides excellent documentation and clear, easy-to-follow explanations. As of right now, Copilot for PRs documentation is not as informative. In addition PR-Agent commands are concise and provide detailed feedback.

Data Privacy 🕵️‍♀️

CodiumAI empowers developers who prioritize privacy to independently host their PR-Agent instances, with a steadfast commitment: no user data is stored or used for training purposes. In contrast, GitHub Copilot Business gathers personal data across three categories: User Engagement Data, Prompts, and Suggestions.

Pricing 💰

CodiumAI's PR-Agent is available for free, in contrast to GitHub Copilot, which comes with a price tag of $10 USD per month or $100 USD per year. Nonetheless, GitHub Copilot does offer a complimentary tier for students and educators.

Conclusion

By automatically generating detailed pull request descriptions, providing intelligent code suggestions, and ensuring documentation accuracy, CodiumAI sets itself apart with unparalleled depth and customization. Unlike settling for less, users embrace a tool that evolves with them, thanks to its open-source nature. CodiumAI serves as a community-driven platform, incorporating diverse expertise to cater to the ever-changing needs of developers. Acting as a coding companion, it supports users in tasks ranging from intricate code reviews to generating test cases, answering queries, and streamlining changelog updates.

Python One Liners

tinega — Mon, 27 Nov 2023 08:32:01 +0000

In Python, a "one-liner" is a concise and frequently ingenious solution to a problem that can be stated in only one line of code. For readability and maintainability in bigger codebases, they are not always advised, despite their sometimes-elegant and potent nature. They are, nevertheless, frequently employed in scripting, where conciseness is valued. Here are a few sample lines of code for one-liner solutions.

Multiple variable assignments

name, age, occupation = "keisha", 30, "cook"
print(name, age, occupation)
# keisha 30 cook

HTTP server

python -m http.server

Swap variables

a = "a"
b = "b"
a, b = b, a
print(a, b)
# b a

Ternary operator

yob = 1997
print("hey kid(gen alpha)" if yob >= 2013 else "wassup gen z" if yob >= 1995 else"hello millennials" if yob >= 1980 else "hey gen x" if yob >= 1965 else "hello baby boomer")
# wassup gen z

Variable unpacking

users = ['kalama', 'kairetu','munga','mwanaisha']
print(*users)
#kalama kairetu munga mwanaisha

user1, user2 , *other_users= users
print(other_users)
#['munga', 'mwanaisha']

Reverse a list

z = [10,20,30,40]
print(z[::-1])
#[40, 30, 20, 10]

Sort a list

numbers = [1000000, -500000, 999999999, -1000000000, 123456789]
print(sorted(numbers))
# [-1000000000, -500000, 1000000, 123456789, 999999999]

Sort a list - reversed

numbers = [1000000, -500000, 999999999, -1000000000, 123456789]
print(sorted(numbers, reverse=True))
#[999999999, 123456789, 1000000, -500000, -1000000000]

Filter from a list

numbers = [1,2,3,4,5,6,7,8,9,10]
odd = list(filter(lambda x: x%2!=0, numbers))
print(odd)
#[1, 3, 5, 7, 9]

List comprehension

names = ['lizzy','bruce','wafula']
print([name.title() for name in names])
#['Lizzy', 'Bruce', 'Wafula']

Remove duplicates from a list


num = [5, 2, 8, 3, 5, 2, 1, 8, 9, 4, 1]
print(list(set(num)))
# [1, 2, 3, 4, 5, 8, 9]

Combine two lists to a dictionary

letters = ["a", "b", "c", "d"]
num = [1,2,3,4]
print(dict(zip(letters, num)))
# {'a': 1, 'b': 2, 'c': 3, 'd': 4}

Unpacking using * operator and zip

pair = [('jack', 50),('inaam', 80), ( 'Anna ', 70), ('John', 35)]
names, marks = zip(*pair)
print(names)
print(marks)
#('jack', 'inaam', 'Anna ', 'John')
#(50, 80, 70, 35)

Type conversion of items in an iterable using map function

strings = ("1","2","3","4","5","6","7","8")
print(tuple(map(int, strings)))
# (1, 2, 3, 4, 5, 6, 7, 8)

Alternative value for input statements


username = input('Name: ') or 'N/A'

#using ternary operator

name_input = input('Name: ')
username = name_input if name_input else 'N/A'

Creating a shadow copy of a list

mylist = [1,2,3,4,5]
copy_list = mylist[:]

Web Scraping using Selenium: A Guide for Beginners

tinega — Mon, 09 Oct 2023 13:07:48 +0000

Introduction

Selenium is a framework that many developers use to automate web browsers. It's an open source tool that offers compatibility, with web browsers and supports programming languages. While its commonly used for testing web applications it's also handy for tasks like web scraping and automating browser actions. WebDriver, which is a part of Selenium allows developers to control web browsers programmatically.

Using Selenium to scrape data

Task: Scraping the latest news posts from a popular blog post
Please note that to follow along you must have an understanding of basic HTML elements and CSS classes and id selectors as well as python

To use selenium we first need to install the package using pip.
pip install selenium
Using a code editor of your choice create a .py file and import selenium.webdriver module which provides all the WebDriver implementations.



from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://www.google.com')

To test if everything is working so far we need to instruct the Selenium WebDriver to navigate a given URL.
Running the .py file should launch a browser instance and close immediately the website loads.

Chrome() class also provides implementation for chromium based browsers e.g brave and not just the chrome.

To use Brave, a Chromium-based browser, with Selenium WebDriver in Python, you can set the binary location using the webdriver.ChromeOptions() class and then pass these options as a parameter when creating the WebDriver instance. Here's how you can do it:



from selenium import webdriver

BROWSER_PATH = r"C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe"
OPTIONS = webdriver.ChromeOptions()
OPTIONS.binary_location = BROWSER_PATH

driver = webdriver.Chrome(options=OPTIONS)
driver.get('https://www.google.com')

To prevent the browser instance from closing, add the following property with name 'detach' and set its value to 'True'
OPTIONS.add_experimental_option("detach", True)

Scraping the data

I will demonstrate using pulselive website. The following code should load the website on a browser instance.



from selenium import webdriver

BROWSER_PATH = r"C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe"
OPTIONS = webdriver.ChromeOptions()
OPTIONS.binary_location = BROWSER_PATH
OPTIONS.add_experimental_option("detach", True)

URL = 'https://www.pulselive.co.ke/news'

driver = webdriver.Chrome(options=OPTIONS)
driver.get(URL)

We now need find the HTML elements that may be of use to us.
To interact with web pages and find HTML elements using Developer Tools, you can follow these steps:

Open Brave browser.
Navigate to the web page you want to inspect.
To open Developer Tools:
- On Windows/Linux: Press Ctrl + Shift + I.
- On macOS: Press Option + ⌘ + I or right-click anywhere on the page and select "Inspect" from the context menu.
In the Developer Tools panel that opens, you will see various tabs such as "Elements," "Console," "Network," etc. You are interested in the "Elements" tab, which shows the HTML structure of the web page.
In the "Elements" tab, you can expand and collapse elements to explore the page's structure. Look for the <div> element with a class name of "article-list-items." It should be under the <ul> (unordered list) element, and each list item <li> should contain an <a> tag for the article link and an <h2> tag for the article title.

Locating elements using By

This is done by first importing By from from selenium.webdriver.common.by
from selenium.webdriver.common.by import By
next we instruct the webdriver instance using find_element method i.edriver.find_element(By.CLASS_NAME, value='article-list-items') then assign this to a variable(i named mine articles)
Note: div(class='article-list-items') is the parent element of the articles.

Executing this code returns a selenium webElement, To access the relevant text and links, we need to narrow down using css selectors to the specific element that we are targeting i.e getting all the links contained in the div element with a class name article-list-items

Remember the div with a class name article-list-items,it is an unordered list element containing list elements(the list elements also contain other divs and anchor tags)
find_elements(By.CSS_SELECTOR) value='li a' helps narrow down to the target element in our case, links in the anchor tags.
Since find_elements does not return a single element but all the elements matching our value, assign a variable to point to the results and iterate to get a hold of each item.
Finally for each item call the get_attribute method with attribute name 'href' to return the url contained in each anchor tag.

For the article titles the approach is similar.

To get the title contained in h2 tags use find elements by css selector value='li h2' and because h2 tags do not have attributes but instead contain texts in between. The text can be retrieved by using the property text.

The above code prints the articles with article titles and links separated by a semi-colon.

Final tips
Have you noticed how every time we ran the code a new browser application window is launched.using this option OPTIONS.add_argument("--headless=new") runs everything in the background.
To automatically shut down the chromiumdriver executable instance i.e browser instance add driver.quit() at the end of your code.

Conclusion

This is just a simple example use case on how automation can be used to simplify the process of acquiring information. The amount of applications is vast and i hope this demonstration provides you will a few insights and tips on how to get started with web scraping automation.

Check out the Unofficial documentation here.

List & Dictionary Comprehension in Python

tinega — Fri, 08 Sep 2023 10:12:42 +0000

List Comprehension

Python lists are a fundamental data structure for storing and managing data collections. Lists are adaptable, flexible, and may store a variety of data kinds, including integers, strings, and even other lists.
List comprehension lets you make a new list by applying an expression to each item in an existing iterable (such as a list, tuple, or range) and optionally filtering the items depending on a condition. List comprehensions are a more compact and readable means of creating lists than typical for loops.
As an example, suppose we want to create a new list of even numbers from a list of random numbers.

numbers = [84, 77, 50, 89, 33, 36, 59, 31, 76, 1, 20, 58, 59, 93, 70, 28, 51, 7, 48, 74]

Using a for loop

even_numbers = []

for number in numbers:
    if number % 2 == 0:
        even_numbers.append(number)

Using list comprehension

even_numbers = [num for num in numbers if num % 2 == 0]

the syntax for list comprehension is :

list = [new_item for item in some_list if conditional_statement]

Dictionary Comprehension

Dictionaries in Python are a versatile data structure that may be used to store and manage collections of key-value pairs.
Dictionary comprehension is a simple and effective method for creating dictionaries in Python. You can create dictionaries by specifying key-value pairs based on an expression and an iterable. Dictionary comprehensions are a logical extension of list comprehensions and a useful feature for easily generating dictionaries without the use of explicit loops.

suppose we have a dictionary of cars and year of manufacture, and we want to create a dictionary of recently manufactured ones

car_dictionary = {
    "Tesla Model 3": 2020,
    "Toyota Camry": 2022,
    "Ford Mustang": 2021,
    "Honda Civic": 2023,
    "Chevrolet Silverado": 2019
}

using a for loop

recently_manufactured = {}

for (car,year) in car_dictionary.items():
    if year > 2021:
        recently_manufactured[car] = year

using dictionary comprehension

recently_manufactured = {car:year for (car, year) in car_dictionary.items() if year > 2021}

the syntax for dictionary comprehension is:

dictionary = {new_key:new_value for (key, value) in some_dict.items() if conditional_statement}

Manually add an appimage software shortcut to application menu in linux

tinega — Tue, 23 Mar 2021 20:04:29 +0000

I am going demonstrate how you can add an appimage application shortcut to linux desktop system's application menu and also how to customize its icon.But first lets get to understand what an appimage software is.

What is an appimage application and what are its benefits

For starters,there are many ways an application can be installed in a linux system i.e deb packages,snaps,flatpaks and appimages.Our focus today is appimages.
An Appimage application is a single portable and executable software that can be run on linux without installation.

Benefits of using appimages

Compatibility with most linux distributions.
Simple to run i.e download,make it executable and run.since it is not installed no system libraries are altered.
Download and use trusted software directly from the original author(s).
Uninstalling is as simple as just deleting the file.

Running an app image

Lets download the fluent reader software(it is a modern rss reader) as an example.

Once download is completed,navigate to the downloads folder and right-click on the downloaded appimage file,click on Properties thereafter,Click on the permissions tab and check the execute option.

The software should be running by now without any problems.

Adding a shortcut with custom icon to the linux application menu

Note: super user permission is required for this process.

Open the terminal
Navigare to the /usr/share/applications directory.
Create a new file with a .desktop extension in the applications directory.
Open the file with gedit
Add the following template text to your empty file and replace with the location of your exec(i.e appimage location) and icon of your choice.eg
Save your file and exit the terminal.

You should be able to see your software shortcut in the menu and it should run when clicked if you followed the steps above correctly.

Thanks and have fun ☺️

Hosting Your App For Free

tinega — Fri, 10 Jul 2020 11:31:25 +0000

A weather application is always the first and easiest way for many developers to learn how to use APIs.Today we are going to go through the basics of creating and deploying your site build using Vue.js.

Getting Started

First we need to create a free open weather account to access their API.
To use vueCLI, we need to download and install Node.js.

Once the installation is completed run the following commands in the terminal.

npm install -g @vue/cli
# OR
yarn global add @vue/cli

creating the project

To create a project open your terminal and,run vue create weather

Navigate and select default using the arrow keys.

When completed a folder will be created with git initialized.
Open the folder using your code editor of choice.
It should show something similar to this.

Coding

Since it is a very small application we wont need the components folder and we are going to have to delete the boilerplate code already there in the App.vue file.

inside the div tag type the following code

<header>
      <h1>WeatherApp</h1>
      <input
        type="text"
        autocomplete="off"
        class="search-box"
        placeholder="Search for a city..."
        v-model="query"
        @keypress="fetchWeather"
      />
    </header>
    <main class="card" v-if="typeof weather.main != 'undefined'">
      <div class="location">{{weather.name}}, {{weather.sys.country}}</div>
      <div class="temp">{{ Math.round(weather.main.temp)}}°C</div>
      <div class="weather">{{weather.weather[0].main}}</div>
    </main>

V-model is a directive in vue.js that interacts with the DOM,we have bind the input to a variable data stored in query.
@keypress(which can also be expressed as v-on:keypress) is a directive that listens to DOM events and runs fetchWeather method when triggered.

The following code goes inside the script tag

export default {
  name: "App",
  data() {
    return {
      api_key: "<your api key>",
      url_base: "https://api.openweathermap.org/data/2.5/",
      query: "",
      weather: {}
    };
  },
  methods: {
    fetchWeather(e) {
      if (e.key === "Enter") {
        fetch(
          `${this.url_base}weather?q=${this.query}&units=metric&APPID=${this.api_key}`
        )
          .then(res => {
            return res.json();
          })
          .then(this.setResults);
      }
    },
    setResults(results) {
      this.weather = results;
    }
  }
};

Viewing the app in a local server open the terminal,run npm run server then click at the localhost:8080 link.

Deployment

There are various platforms from which we can deploy our app from.But we are going to focus on netlify for today.

We will need to sign up for an account.i recommend signing up with github.

Netlify allows you to drag and drop the files to be deployed but
alternatively you can deploy directly from github.

Using github makes it easy to update your site since changes are made as soon as they are pushed to your repository.To do so follow the steps below

On the sites menu select new site from git.
Connect to Git Provider
Search for your repo/configure the netlify app on Github.
Build and deploy

Always ensure the build command is set to npm run build and the publish directory set to dist

Stuck in choosing the right color for your website?

tinega — Fri, 12 Jun 2020 06:42:03 +0000

Choosing the right colors for your website can sometimes become exhausting,but with the help of certain tools the whole process can be simplified....
Here are some of the tools i use:

COLORSinspo

A simple and easy interface,just click on an image and paste the code to your css.Includes hundreds of color palettes and gradient colors.

color-hex

Color-hex gives information about colors including color models (RGB,HSL,HSV and CMYK), Triadic colors, monochromatic colors and analogous colors calculated in color page.

ColorPick Eyedropper
This is a chrome extension,color chooser that allows you to select colors from webpages.

In conclusion..

I hope you found this information useful.This is just a small list,feel free to share others in the comments below.