Forem: Alit Darma Putra

Building a Testnet Faucet Bot for Ethereum using Go

Alit Darma Putra — Sun, 11 May 2025 08:00:06 +0000

This article offers tutorials on how to build a Testnet Faucet Bot for Ethereum using Golang.

Introduction

Recently Web3 technology has developed rapidly. This new technology is considered the next evolution of the internet. Blockchain as the foundation of Web3, enables decentralized, transparent, and secure systems that eliminate the need for intermediaries. Many developers are trying to implement blockchain in various aspects of human life with the aim of increasing data security and integrity.

One of the most prominent platforms driving this innovation is Ethereum. Ethereum allows users to create a wide variety of programs within their network. Do you know that to develop on Ethereum, we need some test tokens? These tokens are essential for testing smart contracts, interacting with decentralized applications (dApps), and experimenting with transactions without spending real cryptocurrency. Then, to gain these tokens we need a faucet bot to automatically dripping tokens for the developers.

What is Faucet Bot

A faucet bot is an automated bot designed to provide small amounts of cryptocurrency or tokens to users, allowing users to conduct various experiments on technology. This bot is important in blockchain application development so that we no longer need to manually send tokens to developers.

In this article, we will discover how to build a Testnet Faucet Bot for Ethereum using Golang, with a user interface provided through Discord. The bot that we build is simple and only consists of token drip operations, which serve as the core engine of any faucet bot.

Additional functionality such as conditional dripping based on user criteria, rate limiting, or integration with external services is considered part of further development.

Prerequisites

To follow along, you should have:

Basic understanding of Blockchain
Golang (v1.23+) installed
Blockchain Network: A local or test network like Ethereum’s Rinkeby, Goerli, or a private Ethereum network for testing and development of the faucet functionality. In this article I will use hardhat to provide Ethereum development environment (testnet) in our local machine. For more information about how to setup and run hardhat you could refer to the quick start documentation.
A Go project already set up

If you don't have a project yet, create one with:

mkdir go-faucet
cd go-faucet
go mod init go-faucet

Installing Dependency

Before we begin, we need to install a few necessary Go modules that will help us interact with the Ethereum network and handle message from Discord application. To install them, run the following commands in your terminal:

go get github.com/bwmarrin/discordgo
go get github.com/ethereum/go-ethereum
go get github.com/joho/godotenv

Preparing Project Structure

After installing the dependencies, the next step is to prepare the project structure. A well-organized structure will make it easier for us to manage the code.

mkdir pkg
mkdir cmd
mkdir internal
mkdir config

pkg/: Contains reusable code and libraries that can be imported by other projects or services.
cmd/: Holds the entry points (main.go).
internal/: Stores project-internal code that cannot be accessed or imported by external projects.
config/: Stores configuration files and logic to manage settings and behavior of the application.

After executing those commands, our project structure will looks like this.

go-faucet
│   go.mod
│   go.sum
├───cmd
├───config
├───internal
└───pkg

Creating Configuration File

In the config directory, create a new file named config.go by running the following command:

touch config/config.go

The code in config.go will manage all operations related to application configuration, including getting env value and providing credentials for connecting to external systems.

// ./config/config.go

package config

import "os"

type Config struct {
    BotToken         string
    WalletPrivateKey string
    RpcUrl           string
}

func GetConfig() *Config {
    return &Config{
        BotToken:         os.Getenv("BOT_TOKEN"),
        WalletPrivateKey: os.Getenv("WALLET_PRIVATE_KEY"),
        RpcUrl:           os.Getenv("RPC_URL"),
    }
}

There are three environment variables that need to be provided for the faucet application. These values should be declared in the .env file as follows:

// .env
BOT_TOKEN= <your-bot-token-here>  # A string for the bot's authentication token.
WALLET_PRIVATE_KEY= <your-private-key-here>  # A string for the wallet's private key (used as the token source).
RPC_URL= <your-rpc-url-here>  # A string for the URL of the RPC (Remote Procedure Call) endpoint.

Details for each variable:

BOT_TOKEN: This value can be obtained from Discord Developer Portal.
WALLET_PRIVATE_KEY and RPC_URL: These values can be obtained from your blockchain network (e.g., Hardhat). If you are using hardhat, choose one of the twenty accounts provided.

Creating Transfer Service

Now it's time to build the core engine of our Faucet Bot. Each time a user triggers the bot command, the bot will send 0.01 ETH to the user's wallet. Before processing the transaction, we need to ensure that the bot's wallet has sufficient balance. If the balance is adequate and the transaction is valid, the bot will proceed to transfer the tokens to the user's address.

To begin, create a new directory named transfer inside the internal directory. Then, within the transfer directory, create a new file named transfer.service.go.

mkdir internal/transfer
touch internal/transfer/transfer.service.go

After that, we will write the transfer logic in transfer.service.go.

// internal/transfer/transfer.service.go

package transfer

import (
    "context"
    "fmt"
    "log"
    "math/big"

    "go-faucet/config"
    "go-faucet/pkg/ether"
)

type TransferService struct {
    config *config.Config
    ether  *ether.Ether
}

func NewTransferService(config *config.Config, ether *ether.Ether) *TransferService {
    return &TransferService{
        config: config,
        ether:  ether,
    }
}

func (t *TransferService) TransferService(ctx context.Context, publicKey string) string {
    // validate balance
    balance, err := t.ether.GetBalance(ctx, t.config.WalletPrivateKey)
    if err != nil {
        log.Println("error: Getting faucet account balance\n", err)
        return "Error sending token"
    }

    if balance.Cmp(big.NewInt(0)) == -1 { // balance < 0
        return "Faucet account out of balance, Please contact the customer support"
    }

    value := big.NewInt(10000000000000000) // in wei (0.01 eth)

    transactionHash, err := t.ether.Transfer(ctx, publicKey, t.config.WalletPrivateKey, value)
    if err != nil {
        log.Println("error: Sending token", err)
        return "Error sending token"
    }

    return fmt.Sprintf("Success sending token to `%s` with transaction hash: `%s`", publicKey, transactionHash)
}

At this point, we still haven't written any utility functions to interact with the Ethereum network. To make things easier, I've written some of the essential functions needed and wrapped them in a struct.

Create a new directory inside pkg called ether.

mkdir pkg/ether
touch pkg/ether/ether.go
touch pkg/ether/utils.go

Then place the following code in that directory.

// pkg/ether/ether.go

package ether

import (
    "context"
    "math/big"

    "github.com/ethereum/go-ethereum/common"
    "github.com/ethereum/go-ethereum/core/types"
    "github.com/ethereum/go-ethereum/crypto"
    "github.com/ethereum/go-ethereum/ethclient"
)

type Ether struct {
    client *ethclient.Client
}

func NewEther(client *ethclient.Client) *Ether {
    return &Ether{
        client: client,
    }
}

func (e *Ether) GetBalance(ctx context.Context, walletPrivateKey string) (*big.Int, error) {
    privateKeyECDSA, err := crypto.HexToECDSA(walletPrivateKey)
    if err != nil {
        return nil, err
    }

    account, err := GetAccountFromECDSAPrivateKey(privateKeyECDSA)
    if err != nil {
        return nil, err
    }
    balance, err := e.client.BalanceAt(ctx, *account, nil)
    if err != nil {
        return nil, err
    }

    return balance, nil
}

func (e *Ether) GetPendingNonce(ctx context.Context, account common.Address) (uint64, error) {
    nonce, err := e.client.PendingNonceAt(ctx, account)
    if err != nil {
        return 0, err
    }

    return nonce, nil
}

func (e *Ether) Transfer(ctx context.Context, toPublicKey string, walletPrivateKey string, value *big.Int) (string, error) {
    privateKeyECDSA, err := crypto.HexToECDSA(walletPrivateKey)
    if err != nil {
        return "", err
    }

    account, err := GetAccountFromECDSAPrivateKey(privateKeyECDSA)
    if err != nil {
        return "", err
    }

    nonce, err := e.GetPendingNonce(ctx, *account)
    if err != nil {
        return "", err
    }

    gasLimit := uint64(21000)
    gasPrice, err := e.client.SuggestGasPrice(ctx)
    if err != nil {
        return "", err
    }

    toAccount := common.HexToAddress(toPublicKey)
    var data []byte
    tx := types.NewTransaction(nonce, toAccount, value, gasLimit, gasPrice, data)

    chainID, err := e.client.NetworkID(ctx)
    if err != nil {
        return "", err
    }

    signedTx, err := types.SignTx(tx, types.NewEIP155Signer(chainID), privateKeyECDSA)
    if err != nil {
        return "", err
    }

    err = e.client.SendTransaction(ctx, signedTx)
    if err != nil {
        return "", err
    }

    return signedTx.Hash().Hex(), nil
}

// pkg/ether/utils.go

package ether

import (
    "crypto/ecdsa"
    "errors"

    "github.com/ethereum/go-ethereum/common"
    "github.com/ethereum/go-ethereum/crypto"
)

func GetAccountFromECDSAPrivateKey(privateKeyECDSA *ecdsa.PrivateKey) (*common.Address, error) {
    publicKey := privateKeyECDSA.Public()
    publicKeyECDSA, ok := publicKey.(*ecdsa.PublicKey)

    if !ok {
        return nil, errors.New("cannot assert type: publicKey is not of type *ecdsa.PublicKey")
    }

    account := crypto.PubkeyToAddress(*publicKeyECDSA)
    return &account, nil
}

Creating Transfer Controller

After completing the logic of the transfer service, next we will create a controller for this transfer feature. In this case, the controller will filter messages sent by the user and get the appropriate arguments for the public key of the transfer destination.

Create a new file named transfer.controller.go inside the internal/transfer directory.

touch internal/transfer/transfer.controller.go

// internal/transfer/transfer.controller.go

package transfer

import (
    "context"
    "log"
    "strings"

    "github.com/bwmarrin/discordgo"
)

type TransferHandler struct {
    transferService *TransferService
}

func NewTransferHandler(transferService *TransferService) *TransferHandler {
    return &TransferHandler{
        transferService: transferService,
    }
}

func (t *TransferHandler) Handler(s *discordgo.Session, m *discordgo.MessageCreate) {
    if strings.HasPrefix(m.Content, "!transfer") {
        log.Printf("info: Received transfer operation from %s - (Channel: %s)\n", m.Author.Username, m.ChannelID)
        var content string = ""

        args := strings.Split(m.Content, " ")

        if len(args) < 2 {
            content = "Please provide your wallet public key"
        } else {
            content = t.transferService.TransferService(context.Background(), args[1])
        }

        _, err := s.ChannelMessageSend(m.ChannelID, content)
        if err != nil {
            log.Println("error: sending discord message\n", err)
        }
        return
    }
}

Creating the Main Application File

We have completed almost the entire code of this system. The last step is we need to create a main function to be able to run the application and connect all components in the system, starting from loading the configuration file, connecting to the Ethereum network, controllers, to application services.

Inside the cmd directory, create a new file named main.go

touch cmd/main.go

Then write the following codes inside that file.

// cmd/main.go

package main

import (
    "log"
    "os"
    "os/signal"
    "syscall"

    "go-faucet/config"
    "go-faucet/internal/transfer"
    "go-faucet/pkg/ether"

    "github.com/bwmarrin/discordgo"
    "github.com/ethereum/go-ethereum/ethclient"
    "github.com/joho/godotenv"
)

func init() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error reading .env file")
    }
}

func main() {
    config := config.GetConfig()

    // Initialize eth client
    client, err := ethclient.Dial(config.RpcUrl)
    if err != nil {
        log.Fatalln("error: Error connecting to eth client", err)
        return
    }
    defer client.Close()
    log.Println("info: Success connecting to eth client")

    // Opening discord session
    dg, err := discordgo.New("Bot " + config.BotToken)
    if err != nil {
        log.Fatalln("error: Error creating discord session:\n", err)
        return
    }

    // Registering handler
    ether := ether.NewEther(client)
    transferService := transfer.NewTransferService(config, ether)

    dg.AddHandler(transfer.NewTransferHandler(transferService).Handler)
    log.Println("info: Success registering bot handler")

    dg.Identify.Intents = discordgo.IntentsGuildMessages

    err = dg.Open()
    if err != nil {
        log.Fatalln("error: Error opening discord session:\n", err)
        return
    }

    defer dg.Close()
    log.Println("info: Success opening discord session")

    // Wait until CTRL-C or other term signal is received
    log.Println("info: Go Faucet BOT is running...")
    sc := make(chan os.Signal, 1)
    signal.Notify(sc, syscall.SIGINT, syscall.SIGTERM, os.Interrupt)
    <-sc
}

Final Code Structure

To wrap things up, here’s the complete structure of the project as it stands after implementing all the discussed components. This directory layout should give you a clear overview of how everything fits together:

go-faucet
│   .env
│   go.mod
│   go.sum
├───cmd
│       main.go
├───config
│       config.go
├───internal
│   └───transfer
│           transfer.controller.go
│           transfer.service.go
└───pkg
    └───ether
            ether.go
            utils.go

Running and Testing the Application

Run the Faucet Bot by executing the following command:

go run cmd/main.go

Output:

Once the application is running, invite the bot to your Discord server. To receive tokens, send a message in any channel the bot has access to using the format:

!transfer <wallet_public_key>

Output:

The bot will automatically send 0.01 ETH to the specified wallet address.

Conclusion

We have successfully built our own faucet bot using Go. Next you can develop this system further such as implementing conditional dripping based on user criteria or trying to integrate it with other interfaces such as websites and various social media platforms.

You can check out the full source code on GitHub:

alitdarmaputra / go-faucet

Core of the faucet engine for dripping tokens written in go

Go Faucet

Faucet core engine written in go with discord bot interface

Table of Contents

⚡️ Quick start

Prerequisites

The project utilizes the following tools:

Go 1.23+: A statically typed, compiled programming language designed for simplicity, concurrency, and performance. Go is used for building the bot and handling interactions with the blockchain.
Go-Ethereum 1.14.12: A Go implementation of the Ethereum protocol, enabling the bot to interact with the Ethereum blockchain for cryptocurrency distribution.
Blockchain Network: A local or test network like Ethereum’s Rinkeby, Goerli, or a private Ethereum network for testing and development of the faucet functionality.
Make 4.3+ (optional): A build automation tool used for managing tasks. The Makefile provides predefined commands for setting up, building, and running the project, simplifying the development workflow.

Executing Program

Clone the repository

git clone git@github.com:alitdarmaputra/go-faucet.git
cd go-faucet

Install module

go mod

…

View on GitHub

Feel free to clone the repo, explore the implementation, or fork it for your own use. If you have any questions don't be hesitate to ask in the comments section. Thank you for reading this article.

Laid-Back Approaches to Clean Data

Alit Darma Putra — Wed, 10 Jan 2024 10:57:53 +0000

The importance of data in the output results of the model means that we need to carry out further handling of the input data. One way that can be done to improve data quality is to carry out data cleaning techniques.

What is Data Cleaning?

Data cleaning is a technique that aims to improve data quality by identifying and eliminating errors and inconsistencies in data.

Here I will share some data cleaning methods and providing simple implementation in how to do it.

There are 4 commonly methods in data cleaning:

Scalling Feature Value
Handling Extreme Outlier
Binning
Scrubbing

Scalling Feature Value

Feature scaling is the process of normalizing the range of features in a dataset. In real cases, the range of feature values varies greatly. If one of the features has a wide value, then that feature will greatly influence the calculations of the algorithm used. Therefore, the range of all features must be normalized so that each feature can provide a comparable contribution. There are several techniques that can be used to perform feature scaling, including:

Absolute Maximum Scaling

Absolute Maximum Scaling is a scaling technique that is carried out based on the absolute maximum value of each feature. The stages of this technique are:

Determine the maximum absolute value of the feature in the data set.
Divide all values in the column by the maximum value.

import matplotlib.pyplot as plt

def max_absolute_scaling(data):
    # Determines the maximum absolute value
    max_abs_value = max(map(abs, data))

    # Divide each value by the maximum absolute value
    scaled_data = []
    for x in data:
        scaled_data.append(x/max_abs_value)

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = max_absolute_scaling(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.5, -0.16666666666666666, 1.0, 0.3333333333333333, -0.6666666666666666]

Min-Max Scaling

Min-Max Scaling is a scaling technique that is carried out by reducing each value in the dataset by the minimum value and then dividing by the range of the dataset (maximum-minimum). By applying this technique, all feature values will be between 0 and 1. The weakness of this technique is also the same, namely that it is susceptible to outliers.

import matplotlib.pyplot as plt

def min_max_scaling(data):
    # Determine the maximum and minimum values
    max_value = max(data)
    min_value = min(data)

    # Reduces each value by the minimum value
    # then divided by the range of dataset values
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-min_value)/(max_value-min_value)
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = min_max_scaling(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.7, 0.3, 1.0, 0.6, 0.0]

Normalization

Normalization is a scaling technique that is similar to min-max scaling, but each feature value is reduced by the average value of the dataset. The results of the reduction are then divided by the range of dataset values.

import matplotlib.pyplot as plt
from statistics import mean

def normalization(data):
    # Determine maximum, minimum, average values
    max_value = max(data)
    min_value = min(data)
    mean_value = mean(data)

    # Subtract each value by the average value
    # then divided by the range of dataset values
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-mean_value)/(max_value-min_value)
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = normalization(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.18, -0.22000000000000003, 0.48, 0.08, -0.52]

Standardization (Z-score Normalization)

Standardization is a scaling technique that is carried out by reducing each feature value by the average and dividing by the standard deviation value or what is usually called the z-score. The result of this technique is data that is scaled so that it has features centered on the average and a standard deviation of 1. This technique is suitable if the features have a normal distribution such as salary or age

import matplotlib.pyplot as plt
from statistics import mean, stdev

def standardization(data):
    # Determine the average value, standard deviation
    mean_value = mean(data)
    stdev_value = stdev(data)

    # Subtract each value by the average value
    # then divided by the standard deviation
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-mean_value)/(stdev_value)
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = standardization(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.469476477861571, -0.5738045840530313, 1.2519372742975226, 0.20865621238292043, -1.3562653804889828]

Robust Scaling

In the Robust Scaling technique, each data is reduced by the median value and then divided by the Inter Quartile Range (IQR) value. IQR is the difference between the upper quartile (Q3) and the lower quartile (Q1).

import matplotlib.pyplot as plt
import pandas as pd

def robust_scaling(data):
    series = pd.Series(data)
    # Determine the median value, IQR
    q1, median, q3 = series.quantile([0.25, 0.5, 0.75])
    IQR = q3 - q1

    # Subtract each value by the median value
    # then divided by the IQR value
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-median)/IQR
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = robust_scaling(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.25, -0.75, 1.0, 0.0, -1.5]

Scaling to Vector Unit Length

Scaling to Vector Unit Length is a scaling technique that is carried out by transforming the components of a feature vector so that the transformed vector has a length of 1. In this technique, each feature value is divided by the vector length.

This technique can only be done if the value ||X||>0

import matplotlib.pyplot as plt
import numpy as np

def vector_normalization(data):
    vector = np.array(data)
    # Determines the length of the vector
    magnitued = np.linalg.norm(vector)

    # Normalize vectors to unit length
    scaled_data = vector/magnitued

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = vector_normalization(data)
print("Data:", data)
print("Scaled Data:", scaled_data.tolist())
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.3692744729379982, -0.12309149097933272, 0.7385489458759964, 0.24618298195866545, -0.4923659639173309]

Handling Extreme Outlier

Outliers are values that are much different from the majority of the data in the data set. Outliers may represent natural variation in the population. However, in most cases outliers are caused by an error in the data collection process such as entering incorrect data, equipment failure, or other measurement errors. If outliers are not handled, they can affect the results of statistical analysis and the accuracy of the model being developed.

Outlier Detection

Outlier Detection with Sorting Methods

The sorting method is the simplest method that can be used to detect outliers. Quantitative data can be sorted from low to high and manually detected for data with very low or very high values. In the Python language, sorting can be done using the sorted function.

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
sorted_data = sorted(data)
print("Data:", data)
print("Sorted Data:", sorted_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
Sorted Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 100]

A value of 100 is very high when compared to other data. So 100 can be detected as outliers. However, this method is less accurate because outliers are not determined using statistical calculations.

Outlier Detection using the Histogram Method

Histograms can be used to help visualize data and find out whether there are outlier values in a set of data. Data that is outside the data curve is detected as an outlier. The disadvantages of this method are the same as the average method where outliers are determined only from visual observation of the data.

import matplotlib.pyplot as plt

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
plt.hist(data)
plt.show()

Output:

In the histogram visualization above, it can be seen that the data values are more centered between the values 0 to 40. However, there is data that is separated from other data, namely 100 and this data can be categorized as an outlier.

Outlier Detection with Box-Plot

Box-Plot is a summary of the sample distribution presented graphically which can describe the shape of the data distribution (skewness), a measure of central tendency and a measure of the spread (diversity) of observational data. There are 5 statistical measures that can be read in the box plot, namely minimum, maximum, Q1, median, and Q3. Values outside the box and whisker can be categorized as outliers.

import matplotlib.pyplot as plt

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27] plt.boxplot(data, vert=False)
plt.title("Detecting outliers using Boxplot")
plt.xlabel('Data')
plt.show()

Output:

Value 100 that is outside the whisker can be categorized as an outlier.

Deteksi Outlier dengan Z-Score

The criteria for determining outliers with z-score is that every data point that has a z-score value that is outside the 3rd standard deviation is an outlier. The stages carried out using this technique include:

For all data points, calculate the z-score value using the formula (Xi-mean)/std.
Initialize the threshold value=3 and mark data points that have an absolute z-score value greater than the threshold as outliers.

import statistics as s

def detect_outliers_zscore(data):
    # Threshold initialization
    thres = 3

    # Determine the average value and standard deviation
    mean = s.mean(data)
    std = s.stdev(data)

    outliers = []
    # Determine the z-score value for each data
    for i in data:
        z_score = (i-mean)/std

        # Check whether the data is outliers
        if (abs(z_score) > thres):
            outliers.append(i)

    return outliers

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outliers = detect_outliers_zscore(data) 
print("Outliers:", outliers)

Output:

Outliers: [100]

Outlier Detection with Inter Quartile Range (IQR)

Based on the Inter Quartile Range (IQR) value, outliers can be detected if the data point is located 1.5 times the IQR above Q3 and below Q1. The stages in determining outliers with IQR are:

Sort asset data in ascending order
Calculate 1st and 3rd quartiles (Q1, Q3)
Calculate the value of IQR=Q3-Q1
Calculate the lower limit value = (Q1–1.5*IQR) and the upper limit = (Q3+1.5*IQR)
For all data in the data set, check whether any data is below the lower limit and above the upper limit. Then mark the data as an outlier.

import pandas as pd

def detect_outliers_iqr(data):
    series = pd.Series(data)

    # Determine Q1, Q3, IQR values
    Q1, Q3 = series.quantile([0.25, 0.75])
    IQR = Q3-Q1

    # Determine lower bound and upper bound
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR

    outliers = []
    # Determine the z-score value for each data
    for i in data:
        # Check whether the data is outliers
        if (i < lower_bound or i > upper_bound):
            outliers.append(i)

    return outliers

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outliers = detect_outliers_iqr(data) 
print("Outliers:", outliers)

Output:

Outliers: [100]

Outlier Handling

After successfully detecting outlier data in the dataset, the next stage is to handle the outlier data. There are several ways that can be done to handle outlier data that has been detected.

Trimming

Outlier data detected using this technique will be removed from the dataset. However, this method is not the best practice to do.

def trimming(data, outlier):
    new_data = []

    # remove data that includes outliers
    for i in outlier:
       new_data = [x for x in data if x != i ]

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outlier = [100]
new_data = trimming(data, outlier)
print("Data:", data)
print("New Data:", new_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27]

Quantile Based Flooring and Capping

Handling of outliers in this technique is carried out by limiting outliers to certain values above the 90th percentile value or placed in factors below the 10th percentile value.

import pandas as pd

def handle_quantile_outlier(data):
    series = pd.Series(data)

    # Determine the 10th and 90th percentiles
    P10 = series.quantile(0.1)
    P90 = series.quantile(0.9)
    new_data = []

    # Replace the data value with P10 for data < P10
    # and with P90 for data > P90
    for x in data:
        if x < P10:
            x = P10
        elif x > P90:
            x = P90
        new_data.append(x)

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
new_data = handle_quantile_outlier(data)
print("Data:", data)
print("New Data:", new_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [17.1, 17.1, 18, 19, 20, 21, 22, 23, 24, 26.9, 26, 26.9]

Mean/Median Imputation

The average value is greatly influenced by the presence of outliers, so it is recommended to replace these outliers with median values.

import statistics as s

def handle_median_outlier(data, outlier):
    # Determine median value
    median = s.median(data)

    # Change outlier with median value
    new_data = []
    for i in outlier:
        for x in data:
            if x == i:
                new_data.append(median)
            else:
                new_data.append(x)

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outlier = [100]
new_data = handle_median_outlier(data, outlier)
print("Data:", data)
print("New Data:", new_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 21.5, 26, 27]

Log Transformation

Log transformation is a common technique used to reduce the skew in a distribution and make it more symmetric. In this way, the occurrence of extreme values can be reduced and the data becomes more normally distributed.

import math
import matplotlib.pyplot as plt

def handle_log_outlier(data):
    new_data = []
    # Transformasi setiap nilai data dengan log
    for x in data:
        new_data.append(math.log(x, 10))

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
new_data = handle_log_outlier(data)
print("Data:", data)
print("New Data:", new_data)
plt.plot(data, 'red', label='Data')
plt.plot(new_data, 'blue', label='New Data')
plt.legend()
plt.show()

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [1.2041199826559246, 1.2304489213782739, 1.2552725051033058, 1.2787536009528289, 1.301029995663981, 1.322219294733919, 1.3424226808222062, 1.3617278360175928, 1.380211241711606, 2.0, 1.414973347970818, 1.4313637641589871]

Binning

Data binning is a method of separating or grouping continuous numerical values into discrete intervals called "bins" or "groups". Data grouping methods can be used to simplify data distribution and assist in statistical analysis and visualization. There are several techniques that are often used to group data, including:

Equal Width Binning

This technique groups data into intervals or bins with the same width and has been determined previously. Even though this method is simple, it cannot be applied to data with a skewed distribution.

import pandas as pd 

data = {'age': [16, 17, 18, 19, 20, 21, 22, 23, 24, 20, 26, 27]}

df = pd.DataFrame(data)

num_bins = 4

# Calculate the bin width
bin_width = (df['age'].max() - df['age'].min()) / num_bins

# Create bin limits
bin_edges = [df['age'].min() + i * bin_width for i in range(num_bins + 1)]

df['age_bins'] = pd.cut(df['usia'], bins=bin_edges, include_lowest=True, right=True)

print(df)

Output:

    age        age_bins
0     16  (15.999, 18.75]
1     17  (15.999, 18.75]
2     18  (15.999, 18.75]
3     19    (18.75, 21.5]
4     20    (18.75, 21.5]
5     21    (18.75, 21.5]
6     22    (21.5, 24.25]
7     23    (21.5, 24.25]
8     24    (21.5, 24.25]
9     20    (18.75, 21.5]
10    26    (24.25, 27.0]
11    27    (24.25, 27.0]

Equal Frequency Binning

In this technique, data is grouped into bins with each bin having approximately the same number of data points. This technique is useful when maintaining the same frequency or distribution across bins if it is important. This binning method can also effectively deal with outlier data and skewed data.

import pandas as pd 

data = {'age': [16, 17, 18, 19, 20, 21, 22, 23, 24, 20, 26, 27]}

df = pd.DataFrame(data) 
df['age_bins'] = pd.qcut(df['age'], q=3)

print(df)

Output:

    age          age_bins
0     16  (15.999, 19.667]
1     17  (15.999, 19.667]
2     18  (15.999, 19.667]
3     19  (15.999, 19.667]
4     20  (19.667, 22.333]
5     21  (19.667, 22.333]
6     22  (19.667, 22.333]
7     23    (22.333, 27.0]
8     24    (22.333, 27.0]
9     20  (19.667, 22.333]
10    26    (22.333, 27.0]
11    27    (22.333, 27.0]

Quantile Binning

In this technique, data is grouped based on percentile values. The limits of a bin are based on certain percentile values (e.g. 25th, 50th, and 75th percentiles).

import pandas as pd 
import numpy as np

data = {'age': [16, 17, 18, 19, 20, 21, 22, 23, 24, 20, 26, 27]}
df = pd.DataFrame(data) 

# Defines percentiles for bin boundaries
percentiles = [0, 25, 50, 75, 100] # In this case, quartiles are used

# Defines percentiles for bin boundaries
bin_edges = np.percentile(df['age'], percentiles)

df['age_bins'] = pd.cut(df['age'], bins=bin_edges, include_lowest=True)

print(df)

Output:

    age        age_bins
0     16  (15.999, 18.75]
1     17  (15.999, 18.75]
2     18  (15.999, 18.75]
3     19    (18.75, 20.5]
4     20    (18.75, 20.5]
5     21    (20.5, 23.25]
6     22    (20.5, 23.25]
7     23    (20.5, 23.25]
8     24    (23.25, 27.0]
9     20    (18.75, 20.5]
10    26    (23.25, 27.0]
11    27    (23.25, 27.0]

Scrubbing

Data scrubbing is a process for changing or deleting incomplete, incorrect, inaccurate, or repetitive data in a dataset. By carrying out this process, it can help improve data consistency, accuracy and reliability.

Deleting Repetitive Data

Deleting duplicate data events is one way to perform data scrubbing. Repeated data often appears if the dataset used comes from several different sources.

import pandas as pd

data = {'age': [15, 17, 23, 22, 17],
        'height': [155, 162, 165, 170, 162]}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Check if data is duplicated
duplicate_data = df[df.duplicated()]

print("Duplicate Data:")
print(duplicate_data)

# Delete duplicate data
df = df.drop_duplicates()
print("New Data:")
print(df)
Output:
Data:
   age  height
0    15     155
1    17     162
2    23     165
3    22     170
4    17     162
Duplicate Data:
   age  height
4    17     162
New Data:
   age  height
0    15     155
1    17     162
2    23     165
3    22     170

Handling Missing Data

In real cases, usually there is a lot of missing data in a data set. The causes of this data loss are very varied, ranging from data corruption to device failure when recording measurements.

Deleting Missing Data

Missing data can be resolved by deleting rows or columns of data that have NULL values.

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, np.nan, 17],
        'height': [155, 162, np.nan, 170, 162]}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Deletes rows of missing data
df.dropna(axis=0, inplace=True)

print("New Data:")
print(df)

Output:

Data:
   age  height
0  15.0   155.0
1  17.0   162.0
2  23.0     NaN
3   NaN   170.0
4  17.0   162.0
New Data:
   age  height
0  15.0   155.0
1  17.0   162.0
4  17.0   162.0

Pros:
• A model trained by removing all missing values will produce a robust model.
Cons:
• Losing a lot of information.
• Works poorly if the percentage of missing values is too large compared to the data set.

Fill in missing data with mean/median/mode values

Columns in a dataset that have numeric values can be replaced with the mean, median, or mode of other data in that column. This technique will prevent data loss like the previous method.

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, np.nan, 17],
        'height': [155, 162, 165, 170, 162]}
df = pd.DataFrame(data)

print("Data:")
print(df)

df_filled_mean = df.copy()
df_filled_median = df.copy()
df_filled_mode = df.copy()

# Determine mean/median/mode value
mean = df['age'].mean() 
median = df['age'].median()
mode = df['age'].mode().values[0]

# Fill in the data with the mean/median/mode value
df_filled_mean['age'].fillna(mean, inplace=True)
df_filled_median['age'].fillna(median, inplace=True)
df_filled_mode['age'].fillna(mode, inplace=True)

print("New Data:")
print("Filled mean:")
print(df_filled_mean)
print("Filled median:")
print(df_filled_median)
print("Filled mode:")
print(df_filled_mode)
Output:
Data:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3   NaN     170
4  17.0     162
New Data:
Filled mean:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3  18.0     170
4  17.0     162
Filled median:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3  17.0     170
4  17.0     162
Filled mode:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3  17.0     170
4  17.0     162

Pros:
• Prevent data loss resulting in deleted rows or columns
• Works well with small data sets and is easy to implement.
Cons:
• Only works with numeric continuous variables.
• May cause data leaks

Fill in Missing Data in Categorical Columns

When missing data is found in a categorical column of either character or number type, the missing data can be filled in with the highest frequency of the category. If there is a lot of missing data, the data is replaced with a new category.

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, 20, 17],
        'impression': ['good', 'fair', 'fair', 'very good', np.nan]}
df = pd.DataFrame(data)

print("Data:")
print(df)

most_category = df['impression'].mode().values[0]

# Fill in the data with the highest category frequency
df['impression'].fillna(most_category, inplace=True)

print("New Data:")
print(df)

Output:

Data:
   age        impression
0    15         good
1    17         fair
2    23         fair
3    20    very good
4    17          NaN
New Data:
   age        impression
0    15         good
1    17         fair
2    23         fair
3    20    very good
4    17         fair

Pros:
• Prevent data loss resulting in deleted rows or columns
• Works well with small data sets and is easy to implement.
• Eliminate data loss by adding unique categories
Cons:
• Only works with categorical variables.
• Adding new features to the model while coding may result in poor performance

Data Type Conversion

Most machine learning models cannot be run on categorical data. Therefore, categorical data needs to be converted into numerical data. One technique that can be used is one-hot-encoding. One-hot-encoding is a representation of categorical variables in binary vector form.

import numpy as np

# Categorical data to be converted
colors = ["red", "green", "yellow", "red", "blue"]

# Color list
total_colors = ["red", "green", "blue", "black", "yellow"]

# map each color to numeric
mapping = {}
for x in range(len(total_colors)):
  mapping[total_colors[x]] = x

one_hot_encode = []

# Convert the numeric value of each data
for c in colors:
  arr = list(np.zeros(len(total_colors), dtype = int))
  arr[mapping[c]] = 1
  one_hot_encode.append(arr)

print(one_hot_encode)

Output:

[[1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 0, 0, 1], [1, 0, 0, 0, 0], [0, 0, 1, 0, 0]]

Deleting Irrelevant Data

Data is said to be irrelevant when the data does not match the problem being researched.

import pandas as pd

data = {'age': [15, 17, 23],
        'email': ['ahmad@gmail.com', 'putra@yahoo.com', 'tegar@gmail.com']}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Remove irrelevant attributes
df = df.drop('email', axis=1)

print("New Data:")
print(df)

Output:
Data:
   age              email
0    15   ahmad@gmail.com
1    17   putra@yahoo.com
2    23   tegar@gmail.com
New Data:
   age
0    15
1    17
2    23

Avoiding Structural Errors

Structural errors include typos, incorrect naming conventions, incorrect use of capital letters, and so on. The following is an example of improvements to letter capitalization in categorical features:

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, 20, 17],
        'impression': ['good', 'Fair', 'fair', 'Very good', 'Good']}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Fixed letter capitalization
df['impression'] = df['impression'].str.lower()

print("New Data:")
print(df)

Output:

Data:
   age    impression
0    15         good
1    17         Fair
2    23         fair
3    20    Very good
4    17         Good
New Data:
   age    impression
0    15         good
1    17         fair
2    23         fair
3    20    very good
4    17         good

Closing

In conclusion, employing effective data cleaning methods not only enhances the reliability of your analyses but also paves the way for informed decision-making, ensuring that your data-driven journey is built on a solid foundation of accuracy and integrity