Forem: Ali Moradi

Simplifying Kafka Testing in Python: A Mockafka-py Tutorial

Ali Moradi — Thu, 30 Nov 2023 18:35:49 +0000

Mockafka-py is a versatile and user-friendly Python library designed specifically for simulating Kafka in a testing environment. It streamlines the process of testing applications that are tightly integrated with Kafka, offering a simplified, in-memory mock of Kafka's complex messaging system. This is especially valuable for projects with a strong coupling to Kafka, where testing against a real Kafka cluster can be cumbersome and resource-intensive.

Introducing Mockafka-py

Mockafka-py stands out because it replicates Kafka’s functionality in a lightweight, in-memory setup. This eliminates the need for a physical Kafka setup, reducing the overhead and complexity involved in writing and running tests. Its compatibility with confluent-kafka ensures that it can seamlessly integrate with your existing Kafka-based applications.

git repo :
https://github.com/alm0ra/mockafka-py

Key Advantages of Using Mockafka-py:

Simplified Testing: Eliminates the need for a real Kafka cluster, making it easier to set up and run tests.
Speed: In-memory simulation speeds up the testing process significantly.
Isolation: Tests can be run in an isolated environment, leading to more accurate and reliable results.
Ease of Use: Provides decorators and simple interfaces to mimic Kafka's behavior, making it accessible even for those who are not Kafka experts.

Scenario: Testing an E-Commerce Consumer for shipping

Imagine an e-commerce platform where various events (like order placement, shipment, and delivery) trigger notifications sent through Kafka. Testing this system can be challenging due to the complexities involved in integrating with Kafka.

Using Mockafka-py for Testing:

Setting up the Scenario: You need to test whether your application correctly produces and consumes messages related to order updates.
Mock Kafka Environment: Instead of setting up a Kafka cluster, use Mockafka-py to create a mock Kafka environment. This environment mimics the actual Kafka behavior for your tests.
Writing the Tests:
Certainly! Let's expand on the e-commerce notification system scenario and provide an example of how to write a test using Mockafka-py. This example will illustrate testing both the production and consumption of Kafka messages related to order updates.

Scenario: E-Commerce Order Update Notifications

Suppose your e-commerce application has a consumer:

order_shipped_consumer: Consume a message and produce it to a kafka topic . here just an example for testing functionallity

Step 1: Setting Up Mockafka-py

First, ensure Mockafka-py is installed:

pip install mockafka-py

Step 2: Writing the Tests

Scenario: A Consumer in E-Commerce System

from mockafka import FakeConsumer, FakeProducer

def order_shipped_consumer():
    # Note: Utilize Mockafka's producer and consumer in tests. Depending on your implementation, you can inject FakeProducer and FakeConsumer into your code.

    # Consuming the message
    consumer = FakeConsumer()
    consumer.subscribe(['orders'])
    message = consumer.poll()
    message.commit()

    if message.key() != 'order_shipped':
        return

    # Processing the message, which could entail various operations. Here, as an example, we're producing another message post-consumption.
    producer = FakeProducer()
    producer.produce(
        key='order_finished',
        value=message.value(),
        topic='orders',
        partition=0
    )

Test for Producing Messages (order_shipped_consumer):

from mockafka import produce, setup_kafka
from my_ecommerce_app import order_consumer

@setup_kafka(topics=[{"topic": "orders", "partition": 1}])
@produce(topic='orders', key='order_shipped', value={'customer': 'sample-1'}, partition=0)
def test_order_shipped_consumer():
    # Simulating the behavior of the order shipped consumer.
    order_consumer()

    # Based on the logic, we expect another message to be produced. Here, we can verify any related outcomes of the consumer's actions.
    consumer = FakeConsumer()
    consumer.subscribe(['orders'])
    message = consumer.poll()

    assert message.key() == 'order_finished'
    assert message.value() == {'customer': 'sample-1'}

Conclusion

Mockafka-py is highly beneficial for projects heavily reliant on Kafka. It simplifies the testing process, making it more efficient and less resource-intensive. Whether you’re developing a small service or a large-scale application, Mockafka-py ensures that your Kafka interactions are thoroughly and efficiently tested, leading to more robust and reliable applications.

https://github.com/alm0ra/mockafka-py

Backtesting is A piece of Cake(signal backtester Guide)

Ali Moradi — Sat, 05 Mar 2022 08:25:16 +0000

some time is hard to back test your strategy with back test library i made easy . with 3 lines of code you can back test and see result of your strategy.

https://github.com/xibalbas/signal_backtester/
support the project with your stars :))

what is Signal Backtester?

it is a tiny backtester Based on Backtesting Lib .
easiest way to backtest your generated signal. just need a csv file contain candleStick informations. OHLCV + signal

why??

some time writing good backtest for a strategy is not too easy . and you may have some challenge with backtest libraries.

so i decided to make a seprate repo for backtesting in easiest way.
what you need is a csv file contain signal column . for buy signal you should put 2, and for sell signal you put 1.

and good news is you did not need to write strategy for how trade we wrote it before you just choose yours and finish you did it :))
see Strategy guide.

First

prepare your data and prepare your signal .
you can write your strategy and generate signal like this

here is an example of generating signal for EMA cross strategy and show you how generate your signal .
our signal generate as new column signal in our data frame .

notice: your data set column should contain Date,Open, High, Low, Close, Volume,signal

import talib  # notice you can install talib manually
import pandas as pd


def cross_EMA_signals(df, fast_period, slow_period):
    """_summary_

    Args:
        df (_type_): _description_
        fast_period (_type_): _description_
        slow_period (_type_): _description_
    """
    signal = [0] * len(df)

    df["fast"] = talib.EMA(df.Close, timeperiod=fast_period)
    df["slow"] = talib.EMA(df.Close, timeperiod=slow_period)
    for idx in range(len(df)):
        if idx > slow_period:

            if (
                df.iloc[idx - 1].fast < df.iloc[idx - 1].slow
                and df.iloc[idx].fast > df.iloc[idx].slow
            ):
                # buy signal
                signal[idx] = 2

            if (
                df.iloc[idx - 1].fast > df.iloc[idx - 1].slow
                and df.iloc[idx].fast < df.iloc[idx].slow
            ):
                # sell signal
                signal[idx] = 1

    df["signal"] = signal
    df.to_csv("./final_dataset.csv")


df = pd.read_csv("./data.csv")

if __name__ == "__main__":
    cross_EMA_signals(df, 15, 30)

second (Backtest in 3 line)

installation

pip install signal-backtester

make a backtest

from signal_backtester import SignalBacktester


dataset_address = "./final_dataset.csv"

backtest = SignalBacktester(
    dataset=dataset_address,
    strategy="two_side_sl_tp_reversed",
    cash=100000,
    commission=0.0005,
    percent_of_portfolio=99,
    stop_loss=1,
    take_profit=2,
    trailing_stop=3,
    output_path="./result",  # path of result files
)

backtest.run()

and simply get result of Backtesting lib:

final_report.html

final_report.csv

order_report.csv

how become an expert trader with backtest your skill(strategies) in the past on trading view?(introducing PTV)

Ali Moradi — Fri, 14 Jan 2022 12:36:29 +0000

Demo accounts became widespread in the twenty-first century, along with online trading. They are marketed to customers as a way for a customer to test a platform’s user experience and features before committing their own money to these investments or paying the platform trading commissions.

Learn before trade

if you want backtest your skills in stock market what do you do?

if you were in the past what did you do ? what position did you enter?
you need to repeat the candles untill you can do a good trade.

introduce PTV (a trading view widget)

if you are familiar with trading view platform , you must know it has a feature call Bar Reply ,
what is bar reply?

for simulate demo trading in the past you can use PTV
trading view does not have this feature and i decided to implement this feature by my self .
for more information please checkout github repository of PTV
it's free and open source
check out this link:

https://github.com/xibalbas/paper_trading_view.git

don't forget to star the repo :))

How practice and improve trade skills with demo trading on trading view with bar reply mode feature? (open source project)

Ali Moradi — Fri, 14 Jan 2022 12:00:50 +0000

paper trading view repo on github
a few weeks ago when i was practicing price action method in MT4 i found out Trading View has a feature call Bar reply , you can cut the candlestick chart to the back , so i decided to buy a pro account .

but after a while i find out this feature does not prepare paper test . if you want to practice with Bar reply mode you should write your trade on paper and this was not a good idea.

Quick installation tutorial video

so i decided to search i found another package on Git hub they call tradingview-trainer but this did not help me because it was not complete .

it’s simple work with Selenium , tkinter

paper-trading-view Features

work with TradingView: For bar reply and getting price we use trading view.

Limit/Market orders: you can place a Limit or Market order in 2 side "long", "Short" .

SL/TP set: you can set stop loss and take profit for your market or limit order .

see Open orders: you can see list of open orders.

see Open positions: you can see list of open positions.

see trade history: you can see list of trade history.

Next/Play/Pause in bar reply: you can use this items in PTV and when price is changing open positions and open orders will be check.

initial setting: you can set your initial money / commission.

profit/loss of account: your total profit / loss calculate when your position close.

save report: you can save your trade history in a csv file.

paper-trading-view Github Source

https://github.com/xibalbas/paper_trading_view.git

The installation instructions can be found on project page! If you have any questions or comments, feel free to leave them down below! Good luck on your practice trading, hope this tool helps!

if you like it give a start :)))

designed with love