Forem: Hardal

122 Milyon Satır Analytics Datasını Nasıl Türkiye’ye Getirdik?

Mert Enercan — Tue, 25 Feb 2025 21:23:29 +0000

122 Milyon Satır Analytics Datasını Nasıl Türkiye’ye Getirdik?

Selam!

Yakın zaman içerisinde Google Cloud — europe sunucularından, Hardal’ın Türkiye sunucuları için kullandığı Huawei Cloud — Istanbul sunucularına transfer ettiğimiz 122 Milyon satır analytics datasının transfer sürecinden bahsedeceğim.

Özetle

Tablo yapılarımızı değerlendirerek iki seçenek arasından karar verdik ve sunucular arası transferi, ClickHouse’un yerel tablo transfer fonksiyonunu kullanarak gerçekleştirdik. Bu süreç yaklaşık 24 saat sürdü ve akıl sağlığımızdan bir miktar götürdü.

Transfer sürecinden önce mevcut tablo yapımızdan ve önümüzdeki seçeneklerden bahsetmem gerek:

Tablo Yapımız

Sunucularımızda veritabanı olarak ClickHouse’un MergeTree Engine tablolarını kullanıyoruz. Toplamda dört tablomuz bulunuyor: Analytics, Config, Telemetry ve iki adet optimize edilmiş Materialized View. ClickHouse’u tercih etmemizin en önemli nedenlerinden biri, aşağıda PostgreSQL ile yapılan karşılaştırmada da görebileceğiniz performans farkı. 👇Detaylar burada.

Tüm event verileri ham (raw) biçimde analytics tablosunda saklanıyor. Daha verimli okuma işlemleri için, bu ham veriyi iki ayrı tabloya aktardığımız bir optimizasyon sürecimiz var. Yani analytics tablosuna gelen her veri, otomatik olarak belirttiğim iki Materialized View tablosuna formatlanarak yazılıyor.

Transfer Süreci

Gelelim zurnanın zort dediği yere. Transfer süreci.

Aslında bu süreç genellikle sorunsuz ilerliyor, ancak konu 120 milyon gibi devasa bir veri hacmi olunca işler biraz karmaşık hale geliyor. Bu noktada önümüzde iki seçenek vardı:

1 — Mevcut dataları exportlayıp (CSV ya da native sql) yeni sunucu içine manual aktarmak

2 — Clickhouse’un native transfer fonksiyonlarını kullanıp otomatik bir şekilde sürecin tamamlanmasını beklemek

İlk olarak birinci yöntemi denedik. Ancak verinin büyüklüğü nedeniyle manuel dışa aktarma yönteminin sonucunu ancak çocuklarımızın görebileceğini fark edince, ikinci seçeneğe yönelmeye karar verdik.

Bu noktada tablo yapımızın büyük avantajını gördük. Materialized View tabloları, analytics tablosunu otomatik olarak inheritlediği için, aslında yalnızca analytics tablosunu transfer etmek, tüm tabloların otomatik olarak doldurulmasını sağlayacaktı.

Plan harikaydı, ve hemen aksiyon almaya başladık.

INSERT INTO remote(‘destination_ip:port’, ‘default.table_name’, ‘username’, ‘password’)
SELECT * FROM target_table_name ;

Bu yöntemi kullanarak işlemi başlattık. İşlemin %1’inin tamamlanması yaklaşık 5 dakika sürdü. Basit bir matematikle hesapladığımızda (5 × 100), işlemin yaklaşık 500 dakika (~8 saat) içinde tamamlanacağını öngördük.

Gece yatarken bırakırız, sabah olur işte.

Tabiki de öyle olmadı :D

Google Cloud’un tarayıcı tabanlı SSH bağlantısı bu kadar uzun süren işlemleri desteklemiyordu ve sık sık timeout hatası alıyorduk. Ancak asıl sorun bu değildi. Zamanla yarışıyorduk — her geçen saniye analytics tablosuna binlerce yeni satır ekleniyor ve bu da transfer sürecinin giderek uzaması anlamına geliyordu.

Üst üste yaşadığımız 3–4 başarısız denemenin (fazladan 24 saat kayıp) ardından artık daha sağlam bir yöntem deneme zamanı gelmişti. Bu yüzden, tarayıcı tabanlı SSH yerine doğrudan kendi terminalimizden SSH bağlantısı kurarak süreci yeniden başlattık.

Neden Kendi Terminalimizden SSH Bağlantısı Daha Sağlıklı?

Google Cloud’un tarayıcı tabanlı SSH bağlantıları, belirli bir süre işlem yapılmadığında otomatik olarak kapanır (timeout). Uzun süren işlemler için bu büyük bir dezavantajdır.
Lokal terminal üzerinden yapılan SSH bağlantıları, timeout sınırlarına takılmadan kesintisiz çalışabilir.
Tarayıcıdan yapılan SSH bağlantıları ağ kesintilerine ve sekme kapanmalarına karşı daha hassastır.
Lokal terminalden bağlandığımızda, ağ dalgalanmalarına rağmen bağlantıyı sürdürebiliriz.
tmux veya screen gibi araçlar kullanarak bağlantıyı kopsa bile işlemi arka planda çalıştırabiliriz.
Tarayıcı kapansa bile süreç devam eder.

12 saatin sonunda transferin %50’si tamamlanmıştı ve artık timeout problemi yaşamıyorduk. Süreç sorunsuz ilerliyordu.

Ancak, “Kaçırdığımız 24 saat içinde ne kadar veri eklenmiş olabilir ki?” sorusunu fazlasıyla hafife aldık.

Ve zafer! 🎉 Yaklaşık 24 saat sonunda, tüm verileri Google Cloud’un Avrupa sunucularından Huawei Cloud’un Türkiye sunucularına başarıyla taşıdık. Bu süreç boyunca yaşadığımız tüm zorluklara rağmen, sonunda transferi tamamlamış olmanın haklı gururunu yaşadık. 🚀

Bize de on numara bir anı kalmış oldu.

Daha optimize bir çözüm mümkün müydü? Tabii ki.
Sonuca ulaştık mı? Kesinlikle.

🎯 Mission completed. 🚀

Sevgiler ve saygılar

by.

How to use Hardal Signal Unity SDK

Mert Enercan — Wed, 12 Feb 2025 15:39:00 +0000

How to use Hardal Signal Unity SDK

Greetings, mighty internet!

Let’s talk about our fresh product: Hardal Signal Unity SDK

TL:DR

Hardal Signal Unity SDK is a lightweight and easy-to-use tool for Unity projects, allowing developers to track game events seamlessly. This guide will walk you through the installation, setup, and usage of the SDK.

How to use it

Let’s create a boilerplate Unity 2D game and implement Hardal Signal SDK to it step by step. From learning section, i’ll be selecting 2D Platformer Microgame.

Navigating to Window -> Package Manager

This will open a popup. All i need to do is copy Hardal Signal Unity SDK git repository URL and paste it to the input that stays in Install package from git URL option.

https://github.com/usehardal/unity-sdk.git

You’ll be able to see that Hardal package is successfully installed and ready to go!

Let’s send our first event. I’ll be triggering event every time when player dies. First, i need to initialize hardal sdk. Let’s do it by right click at the scene -> select Hardal -> Create Hardal Manager and replace placeholder domain at the Hardal Manager script with my own Hardal Signal domain.

HardalManager will automatically manage initializing the sdk

Now it’s time to implement our tracking code. It’ll be inside of PlayerDeath script file so lets do it. You can check technical documentation from repository itself!

And if you enable Debug option, you’ll start to see event flow from console!

Finally, i can confirm that events are successfully landing via Hardal Analytics Dashboard!

So with that being said, it’s time to go. I hope this one helps you to measure your Game Analytics.

Cya.

Send your analytics data wherever you want with Hardal

Mert Enercan — Sun, 12 Jan 2025 21:37:11 +0000

Send your analytics data wherever you want with Hardal

Greetings, mighty internet.

Today, I'd like to make a guidance about how you can transfer any kind of web events to literally any platform you want with Hardal Signal’s Custom Destinations feature that is published with version 1.0.4.

Concept

Basically, you can forward incoming request to any kind of service that is accepts http POST and GET requests. The craziest part is, you can manage payload format as well!

It all starts with clicking “Add Destination” button that is placed on your Signal’s Connection tab. You’ll encounter with this dialog below 👇

You may also use built-in templates!

Endpoint Label: Display Name of your connection
Endpoint ID: Unique identifier for your connection
Endpoint URL: Destination URL that data will sent
Request Method: HTTP Request method
Content Type (POST Method Only): You can select either application/json *or *application/x-www-form-urlencoded option for your Content-Type Header
Request Format (POST Method Only): This is the payload format that can be shaped with your destinations requirements. You can use build-in variables to form all kind of JSON format to use.
Query Parameters (GET Method Only): And this one stands for defining your query parameters within get requests. You can also use variables for this one!
Additional Headers: You can specify custom headers. This is the place that you want to take a look if your server has some kind of authentication mechanism.

Now i’ll be creating 3 custom destination to cover all the possible ways of transfer your data. Here goes my configs

And as you can see down below, i am successfully receiving my own analytics data with Hardal Signal SDK

Let us cook ️‍🔥

Peace.

Server-side GTM With Huawei

Mert Enercan — Thu, 03 Oct 2024 09:28:40 +0000

Hello everyone!

Let’s explore implementing server-side Google Tag Manager using Huawei Cloud Turkey, using Hardal for improved performance, better data accuracy, and enhanced user privacy.

TLDR

We utilized Cloud Container Instance (CCI) to deploy the server-side GTM container image, configured Elastic Load Balancer (ELB) and CCI Service for each workload, and created a DNS record using the ELB public IP.

The Plan

At the end of the day, we require a total of two running containers for the server-side Google Tag Manager.

One for Preview Server
One for Tagging Server

For this project, we chose to utilize the Cloud Container Instance (CCI) service on Huawei Cloud. This service is designed to run containerized applications, which perfectly aligns with our requirements. For more detailed information on how server-side Google Tag Manager functions, please refer to the official documentation.

Here is the full workflow:

1 — Import the Google Tag Manager image into Huawei Cloud’s Software Repository for Container (SWR).

2 — Set up a Preview Server CCI Workload using the imported SWR image.

3 — Configure Elastic Load Balancer (ELB) for the Preview Server workload

4 —Create a DNS record for the Preview Server.

5 — Deploy the Preview Server using CCI.

With these five steps, the preview server is successfully running on Huawei Cloud! We will repeat this process for our tagging server, with the only difference being that we will use the preview server’s URL as an environment variable in the tagging server’s image. Here’s the flowchart:

Managing SSL Certificates

Managing SSL certificates for the custom domain was the most challenging part of the entire process. Fortunately, with Certbot and Let’s Encrypt, we are able to generate free SSL certificates and store them in Huawei Cloud, allowing us to incorporate them into our CCI service configuration files.

In conclusion, the successful implementation of server-side Google Tag Manager on Huawei Cloud would not have been possible without the support of the Huawei team. Their assistance and expertise have been invaluable throughout this process. We are grateful for their guidance, which has significantly contributed to our project’s success. Thank you, Huawei team, for your continued support!

Until next time!

Server-side GTM With Azure App Service

Mert Enercan — Thu, 19 Sep 2024 21:18:38 +0000

Hello, everyone!

Explore implementing server-side Google Tag Manager using Azure App Service, using Hardal for improved performance, better data accuracy, and enhanced user privacy.

TL:DR
We create a container registry and a container app on Azure. After that, we map the container app to our desired custom domain. You can accomplish this with a single click on *Hardal.

The Plan

Create a server-side GTM and obtain a configuration key
Set up a server-side GTM container registry on Azure.
Create a container app on Azure that runs the server-side GTM registry.
Map a custom domain to the server-side GTM container app.

As always, let’s start with the basics of server-side GTM infrastructure. This diagram is from the official Google documentation. I won’t go into the details of how it works, but it’s essential to understand the framework of what we are doing.

Deployment Details

We need to focus on two important topics when deploying our container:

We need to set the Ingress value to "enabled" to access our Container App with a domain.
We need two container apps: one for the preview server and one for the tagging server. Environment values vary depending on the container type.

Preview Server Environment Variables

CONTAINER_CONFIG=config_string
RUN_AS_PREVIEW_SERVER=true

Tagging Server Environment Variables

CONTAINER_CONFIG=config string
PREVIEW_SERVER_URL=HTTPS preview server url

DNS Mapping

For the final step, we need to configure our domain to point to the container app. By selecting “Managed Certificate,” we need to add two DNS records to our domain.

Type: CNAME
Name: www or {subdomain}
Value: <DNS VALUE THAT AZURE PROVIDES>

Type: TXT
Name: asuid.www or asuid.{subdomain}
Value: <DNS VALUE THAT AZURE PROVIDES>

After a short delay, our container will be accessible via our custom domain!

With everything set up, you’re now ready to leverage server-side GTM on Microsoft Azure with your custom domain. This setup not only enhances your tracking capabilities but also ensures better performance and flexibility. Happy tracking!

Server-side GTM With Yandex Cloud

Mert Enercan — Thu, 19 Sep 2024 21:13:37 +0000

Server-side GTM With Yandex Cloud

Greetings, internet!
Since Server-Side GTM can operate as a simple Docker image, we’ve decided to expand our cloud provider options to include Yandex Cloud, in addition to Azure, GCP, and AWS. Today, I’d like to share how we made this happen!

First, let’s dive into how the Manual Setup works with Server-Side Google Tag Manager. I’ll be using some helper images from the official Google Docs after listing what we need in bullet points.

What we need

Server-side GTM Container, obviously xd (if you don’t have one, make it happen with Hardal for **free)
Custom Domain
Yandex Cloud account

With these ingredients and a bit of Hardal magic, we’re able to deploy our server-side Google Tag Manager on Yandex Cloud.

Structure of Server-side Google Tag Manager

This diagram essentially shows how it works. You’re already familiar with how Web GTM operates, but now it’s time to move it to the server that Hardal provides on Yandex Cloud.

Flowchart

Let’s start with the first step: creating a Serverless Container on Yandex Cloud that runs the Server-Side Google Tag Manager image. To accomplish this, we’ve created a Container Registry for the sGTM image. Now, we’re ready to create our Serverless Container.

Remember, we need Preview Server and Tagging Server

For the preview server, we ran the sGTM registry with the RUN_AS_PREVIEW_SERVER=true variable.

After successfully deploying the Preview Server image, we’re now ready to map a custom URL to the container. While Yandex Cloud doesn’t provide a direct way to manage domains for your containers, there’s no such thing as “impossible.” So, we found a workaround to make it happen.

Solution

The plan was to create an API Gateway and integrate it with the sGTM container. Instead of mapping the custom domain directly to the container, we mapped it to the API Gateway and connected the container to the gateway. After generating a Let’s Encrypt certificate for our custom domain, we were nearly at the final result. It was time to map our domain to the API Gateway.

The tricky part is that you can connect the domain **www.example.com, but not **example.com. This is due to how CNAME records are processed by DNS hosting. To resolve this, we created an ANAME DNS record on Yandex Cloud DNS.

With the configurations provided below, our API Gateway is now ready to use with a custom domain and serverless container. 🎊

/{proxy+}:
    x-yc-apigateway-any-method:
      x-yc-apigateway-integration:
        type: serverless_containers
        container_id: containerID
        service_account_id: serviceAccountID

Now we have the sGTM Preview Server with a custom domain working on Yandex Cloud. For the Tagging Server, we follow the same process, but instead of using the RUN_AS_PREVIEW_SERVER=true variable, we need to use the PREVIEW_SERVER_URL='' variable for the registry deployment.

Until next time.

Cookieless tracking in web2 and web3

Berkay — Sun, 27 Nov 2022 11:52:13 +0000

Cookieless tracking is basically a type of web analytics that does not require any cookies on your browser.

This type of marketing engineering has been around for a while but it has only recently become popular with GDPR regulations. This regulation states that if an organization collects data from users which using your website, web apps, or dApps. it must be able to delete this information at any time.

With cookieless tracking, companies can now keep their users' data private by deleting it from their servers straight away instead of keeping it there for years on end. Cookieless tracking is becoming more prevalent on the internet and so marketers are relying largely on it to keep their user's data private.

Cookieless tracking is a solution for the GDPR and cookie consent compliance problem. With the help of this technology, companies can track the user's activity on the website without collecting any personal data.

Using a server-side technology that doesn't use any cookies to identify or track visitors on your website. It does so by using other methods like IP address, device fingerprinting, and browser fingerprinting like 🌭Hardal for web2.

Web3 Analytics in marketing

In web3 Analytics allows for marketing engineers to track the user’s journey from the first visit to the last visit, without having to place a cookie on their device while they are using your dApps.

Integration this server-side technology that can identify and track the user events on a website which does not require any third-party cookies or scripts with on-chain wallet platforms like Metamask 🦊.

Hardal also provide us a gateway to web3 analytics with integrating wallets that using Ethereum and other platforms. It can be used to track all kinds of events on the blockchain, including user interactions with smart contracts and decentralized blockchain apps.

Marketing engineers are now able to create more better marketing campaigns, audience engineering and provide better customer experience due to this new tracking method.

Hardal also records user actions from blockchain and dApp. It helps marketers and business owners to create custom web3 audiences, generate targeted segments using custom events and activate them with automated marketing campaigns.

Why privacy is matter?

Privacy is an important issue for web3 Analytics, so they provide a cookieless solution that allows them to track users without having to store any of the data in a browser or add-ons.

Web developers and marketers need to respect the privacy of their users. However, with the advent of GDPR, this has become even more important.

This means that companies need ways to track their ads, offers, and website audiences without using cookies.

While cookieless tracking might sound like a challenging task for marketers, there are solutions that can help them with this by integrating it into their existing technology stack with ease.

Hardal web2 and web3 analytics provides an powerful solution for businesses who are using on-chain marketing tools and want to achieve compliance with GDPR regulations.

All of the analytics are done server-side, so there is no need to worry about cookies or third party integrations.