Forem: Iurii Panarin

Fitter - Open Source no-code tool for map-reduce data from different source and even more!

Iurii Panarin — Wed, 30 Oct 2024 18:46:10 +0000

Hello everyone!

I am Pxyup and today i want represent for you by open source project Fitter.

PxyUp / fitter

New way for collect information from the API's/Websites

Fitter + Fitter CLI

Fitter - new way for collect information from the API's/Websites

Fitter CLI - small cli command which provide result from Fitter for test/debug/home usage

Fitter Lib - library which provide functional of fitter CLI as a library

Way to collect information

Server - parsing response from some API's or http request(usage of http.Client)
Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information
Static - parsing static string as data

Format which can be parsed

JSON - parsing JSON to get specific information
XML - parsing xml tree to get specific information
HTML - parsing dom tree to get specific information
XPath - parsing dom tree to get specific information but by xpath

Use like a library

go get github.com/PxyUp/fitter

package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse

…

View on GitHub

How it was created

In 2023, I worked on an idea called Trip Searcher:

1.  You enter a budget.
2.  You specify a starting city or country.
3.  You set the trip duration and a range of possible start and end dates.

The Trip Searcher would monitor flights and return potential routes from the starting city, including total prices to various destinations, and send notifications to Telegram with:

1.  Flight costs (parsed from Google or Kiwi).
2.  Hotel prices (from Airbnb or Booking).
3.  Food costs (retrieved from Numbeo).

For this setup, I needed a list of countries, cities, and airport codes to plug into the sites mentioned. During development, I started thinking about how convenient it would be if all this information could be easily combined and parsed to streamline requests, which led to the idea for a project I call Fitter.

P.S.: This project was for personal use.

Fitter CLI

A no-code map-reducer that returns data in user-friendly (JSON) or custom formats, suitable for storage in a database or transmission via HTTP.

Features:

Supports parsing through HTML (query), JSON (gjson), XML, and xpath parsers.
Retrieves data as a browser would, using Docker, Playwright(+ stealth mode), HTTP Client, Cache, File, or propagated fields, with support for custom plugins.
Provides proxy support for Playwright and HTTP clients.
Can send or store information to a file, webhook, console, and more via plugins.
Handles all data types: int, float, bool, array, object, null, and string.
Combines (map-reduce) and transforms fields.
Utilizes the powerful expr library for template syntax, which is available across the application.
Offered as a standalone binary and Docker version.
Allows limits on request counts or instances for browser/Docker usage.

Examples

Static generation:

Here we will just generate static array from hardcoded

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_static_connector.json

[
        "PAGE: 1 INDEX: 0",
        "PAGE: 2 INDEX: 1",
        "PAGE: 3 INDEX: 2",
        "PAGE: 4 INDEX: 3",
        "PAGE: 5 INDEX: 4"
]

Get current time

Get information from the website and return to user.

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_current_time.json

"Current time is: 19:18:51"

Get current Steam Sales

That will create sales.md wile in provided directory

Get best news from HackerNews + Comment list for each

In this config we using template syntax for propagate result from the first request to next one.

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_cli.json

{
  "internal_url": "https://news.ycombinator.com/item?id=41975047",
  "content": {
    "kids": [
      {
        "response_id": "9db6ec73-3e35-4d06-b5eb-dbaa05a3a31d",
        "internal_url": "https://news.ycombinator.com/item?id=41975981",
        "content": {
          "text": "&amp;gt; We describe Flock as &amp;quot;Flutter+&amp;quot;. In other words, we do not want, or intend, to fork the Flutter community. Flock will remain constantly up to date with Flutter.&lt;p&gt;That was the first fear when I saw the title - splitting community and having two incompatible versions. Good to see it addressed in the post.&lt;p&gt;The second was just a fear of how it would complicate the development process, but it seems to be a drop-in replacement (just configuring FVM - Flutter Version Manager):&lt;p&gt;&lt;pre&gt;&lt;code&gt;   Configure .fvmrc to use Flock:   {     &amp;quot;flutter&amp;quot;: &amp;quot;master&amp;quot;,     &amp;quot;flutterUrl&amp;quot;: &amp;quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Flutter-Foundation&amp;#x2F;flutter.git&amp;quot;   }&lt;/code&gt;&lt;/pre&gt;Flutter is the best thing that happened to UI development since Qt. Most people don&amp;#x27;t realize how many apps written in Flutter they use daily, simply because it&amp;#x27;s impossible to tell. And the frustration described in the post is felt by many CTOs and developers. Especially those who use Flutter for desktop and web. Flutter provides an amazing experience for desktop apps, and precisely because of that, it feels so frustrating when you stumble upon some stupid bug that has been open for a year or two and never gets prioritized. Usually, it&amp;#x27;s nothing critical, but still requires workarounds and wasting time.&lt;p&gt;I don&amp;#x27;t know, the idea of Flock sounds good, the main question is engaging the community. Hopefully, the author (who seem to be an ex-Flutter team member himself) have a good grasp on the state of the community.&lt;p&gt;Wishing luck to the project and going to keep an eye on the progress.",
          "title": "comment"
        },
        "id": 41975981
      },
      {
        "content": {
          "text": "Back when I worked on GWT, we had trouble accepting outside contributions because the team had a mandate to support Googlers. That is, much like other libraries and tools at Google, changes could not break google3. This means &lt;i&gt;testing&lt;/i&gt; patches against google3 and either changing the patch, or fixing whatever code used it, and these are tasks that no outsider can do.&lt;p&gt;Shepherding these patches is no fun when you have your own changes to work on that are more important to the team.&lt;p&gt;We did something similar, by creating an external fork where changes could be tried out by the community, without necessarily being accepted into the internal version.&lt;p&gt;I think a fork &lt;i&gt;could&lt;/i&gt; work if there was enough external momentum, but even 20 people working full time would actually be pretty good for an open source project. How many developers will this fork attract? The fork would need to attract other businesses who can put people on it.&lt;p&gt;One downside is that the code isn&amp;#x27;t tested against google3. Sometimes you find actual bugs that way.&lt;p&gt;Edit: reading more closely, the complaint doesn&amp;#x27;t seem to be that patches weren&amp;#x27;t reviewed, but rather that bug reports weren&amp;#x27;t investigated. That&amp;#x27;s definitely something outside developers could do more of, and seems a lot easier than forking?",
          "title": "comment"
        },
        "response_id": "af15ca55-c45c-48a8-a18d-2fcbd7dd8f3b",
        "id": 41975765,
        "internal_url": "https://news.ycombinator.com/item?id=41975765"
      }
    ]
  }
}

Scrape all images from website and store them locally

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_image_multiple.json

[
        "/Users/pxyup/fitter/bin/1/basic-image.png",
        "/Users/pxyup/fitter/bin/2/alt-text.png",
        "/Users/pxyup/fitter/bin/3/no-size.png",
        "/Users/pxyup/fitter/bin/4/size.png",
        "/Users/pxyup/fitter/bin/5/image-with-title.png"
]

Fitter

Fitter it is extended version of the Fitter CLI which have:

Http server for trigger
Return response as telegram message/webhook
And that currently did not have documentation :)

Usage

This tools can be used in different purpose:

Web scrapper
Data scrapper with plugins
Produce specific load testing
Build chat bots - i use it for automate my telegram channel

For example this job every day send best Dev.to aritcles:

Plans

Add more browser tools. Like click/scroll(currently can be done only by JS injection)
Improve template syntax
Add custom template editor + config editor
May be will think about SASS for fitter CLI for run custom workflow for customers and return result to APP/Watch/etc.

Really wait for your feedback! Ask any question i will ask

Fitter - open-source instrument for amazing scraping.

Iurii Panarin — Tue, 30 Jan 2024 22:46:18 +0000

Hello everyone!

I want to introduce Fitter for you.

PxyUp / fitter

New way for collect information from the API's/Websites

Fitter + Fitter CLI

Fitter - new way for collect information from the API's/Websites

Fitter CLI - small cli command which provide result from Fitter for test/debug/home usage

Fitter Lib - library which provide functional of fitter CLI as a library

Way to collect information

Server - parsing response from some API's or http request(usage of http.Client)
Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information
Static - parsing static string as data

Format which can be parsed

JSON - parsing JSON to get specific information
XML - parsing xml tree to get specific information
HTML - parsing dom tree to get specific information
XPath - parsing dom tree to get specific information but by xpath

Use like a library

go get github.com/PxyUp/fitter

package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse

…

View on GitHub

This tool can do a lot of stuff:

Read content by XML/XPath/Json/HTML selectors
Emulate browser with docker/playwright/chromium
Support proxy
Global references
Fallback on no empty fields
Notification Telegram/Http/Redis/Console
File downloading
Full README.md
Can work as golang library
Support runtime golang plugins
Work as CLI
Works on Linux/Mac/Windows.
Support different type of fields
Support expression language for field and notification settings

......

N+1. Best and most important it is gluing of request

Repository have a multiple examples for CLI/Fitter

Real world example of the problem

Last week my friend need to monitor pool on some crypto stock. So here his problem:

First of all you need get all user positions https://docs.saucerswap.finance/v/developer/saucerswap-v2/liquidity-operations/get-user-positions
After that all stock pool https://docs.saucerswap.finance/v/developer/saucerswap-v2/liquidity-operations/fetch-all-pools
After that you should check if pool has user positions you should send notification.

Let's see how Fitter can solve this problem:

Let me explain what is here:

environment variables:

TELEGRAM_TOKEN - line 95

USER_ID - line 37

So my friend need just run Fitter and add UserIds for Telegram notification(line 95):

TELEGRAM_TOKEN=xxx USER_ID=xxx ./fitter --path=./config.json

What happens in config:

2.1 Creating Reference(line 3) on all pools and caching on 60 seconds(line 4). We take the response and create an array of the objects with two fields tokenA and tokenB.

2.2 Creating Reference(line 33) on AccountId will be just string from ENV.

2.3 We create items for monitoring every 10 seconds(line 88) which take values from the server via injecting AccountIdRef(line 52). Response it is array of objects with two field: first it is (line 66) - template string with injecting token name; second is boolean field which show if pair in pool(line 71)

Next we will inform if some of the values in array is true(line 93).

We will receive notification in Telegram:

[
 {
  "position": "tokenA=USDC tokenB=USDC[hts]",
  "value": true
 },
 {
  "value": false,
  "position": "tokenA=USDC tokenB=USDT"
 }
]

SafeStorageBot - keep’s your secrets in Telegram

Iurii Panarin — Fri, 20 May 2022 15:42:09 +0000

The Telegram team announced the Saved Messages tab quite a while ago. Over time, my friends and I noticed that some of our passwords from services began to appear there as well, and “Saved Messages” is already being used as a password repository.

It’s quite convenient, but there are some disadvantages:

Passwords are not encrypted.
It is hard to search without tags, because there is other information
It’s inconvenient to associate the service with the password
You have to select when copying (which is not always convenient when there are special characters)

Solve this problem securely for passwords with a bot, before the release of Telegram WebApp was impossible with one bot(you can encrypt separately and send the result):

No client side(code that can be verified)
Send unencrypted passwords to server is unsafe

With the advantage of WebApp this can be done (the main thing is to think up and remember the master-password):

Encrypt the password with a master password using Javascript (crypto API/crypto-JS) on the client side
Send the encrypted password to the server (there we encrypt the password a couple more times to secure the database)
When we need a password, we request it and decrypt it using the master password on the client side

That’s how this bot came about.

BOT LINK

Below I will show a short video of the bot:

How it works:

You add a password with the title
You enter the master password for encryption
Send title + encrypted password to the server (master-password is never sent)
On the server, encrypt the password and userId a second time with the server master-password
The entry is saved in the database ……
You request the password
The entry is retrieved from the database and decrypted with the server master-password
Sent to the client side
You enter your master password and decrypt on the client side using javascript
You see the decrypted password and can copy it

The client part of the application is posted in the repository:

SafeStorageBot / frontend_bot

Frontend for bot

TelegramPasswords

This project was generated with Angular CLI version 13.3.5.

Development server

Run ng serve for a dev server. Navigate to http://localhost:4200/. The application will automatically reload if you change any of the source files.

Code scaffolding

Build

Run ng build to build the project. The build artifacts will be stored in the dist/ directory.

Running unit tests

Run ng test to execute the unit tests via Karma.

Running end-to-end tests

Run ng e2e to execute the end-to-end tests via a platform of your choice. To use this command, you need to first add a package that implements end-to-end testing capabilities.

Further help

To get more help on the Angular CLI use ng help or go check out the Angular CLI Overview and Command Reference page.

View on GitHub

With the command /version you can always see the commit number of the client part of the application and see the code.

Frontend encryption module

I have long thought about monetisation and after consulting with friends I settled on the option 3 passwords can always be stored, and for a donation of 3 euros a month you can store any number

Thank you for your attention!

PS: Let me take advantage of the opportunity to be promoted again:
BOT LINK

How I built a super fast JS/TS framework

Iurii Panarin — Fri, 26 Apr 2019 14:52:46 +0000

How do other frameworks operate?

What if we didn't use Virtual DOM at all, instead work with real DOM instead? To answer this, we first need to understand how most frameworks work, to better illustrate this I drew a small example.

Violet are DOM/VDOM nodes, Blue = state, Red = diffing, Green = patching. I should explain what happens in this picture. So most of the frameworks that have state and DOM, act like this when something in the state is modified. In this example, state.bg has changed. So now we iterate through all of the nodes (those 5 steps) and renders them. If something is changed, then patching happens where rendered stuff goes to actual DOM (green action). Ok, that's cool, but why do we need to iterate every single node (note that here come those diffing algorithms that compete with each other and some does diff differently) when only state.bg changes?!..

So back to my original question “Why do we need Virtual DOM ?”, the answer is to iterate through all of DOM more quickly. Ok, so what if we didn't iterate?

Utopia way of doing this

But I want to show you the way that in my head it should work without diffing nor iterating, nor Vnodes for that matter as well.

So in this utopia way we changed the same state.bg and there is no iteration (so no need for Virtual DOM), only one render (that single red dot), only one single parameter/attribute/node modifies which is affected by state.bg changes. Much less unnecessary iteration, unnecessary memory usage for virtual nodes, much less rendering = much faster.

Motivation

1. Small bundle size
Many of the modern frameworks are very large. I honestly do not believe that the application “Hello World” should weigh more than 10kb.
To solve this problem, I decided to use a declarative way to describe the components. This has a positive effect on the final size of the application and allows you to use the tree-shaking directly in the component template.
Playground

2. Truly reactive
I really wanted to implement reactivity as it was done in VueJS. If you change the entity of something, then the data on the page also changes, and if you change the data, the entity changes.
Playground

3. Pure reusable components
Any framework should be able to create life-cycle-aware components. This removes duplication of code and allows the component to be reused in several places.
Playground

4. Conditional render and list render directives
I really like the ReactJS, very straight, the first acquaintance, it was just wonderful. However, if you need nested conditional rendering, you will have to either use component wrappers or make ternary expressions that are difficult to read.
Playground

Benchmark application

Playground

Another frameworks

React (Stack) — https://claudiopro.github.io/react-fiber-vs-stack-demo/stack.html
React (Fiber) — https://radi.js.org/fiber/Fiber%20Example.htm
Stencil.js — https://stencil-fiber-demo.firebaseapp.com/perf.html

Performance test

https://pxyup.github.io/Revact/ - try with throttling x6 times.
https://rawgit.com/krausest/js-framework-benchmark/master/webdriver-ts-results/table.html (old name faster-dom)

Link

https://github.com/PxyUp/Revact - source
https://pxyup.github.io/Revact/ - DEMO
https://github.com/RyuuGan/html2FastDom - Html transpiller

Lightweight replacement of React + MobX

Iurii Panarin — Mon, 25 Mar 2019 18:47:23 +0000

FastDom

Lightweight replacement of React + MobX (I hope in future Angular/Vue), which does not use the virtual DOM comparison, but the re-render of only the changed parts. Abandon the HTML template in favor of their interpretation in JS, give to us tree-shaking is components/templates and the speed of work increases since the time to parse the template is zero.

DEMO LINK
DEMO at StackBlitz

Features

Size - 4.1 kB or 1.38 kB gzipped.
The library rewrites only changes and only when it is necessary.
Performance - guarantee 60 fps.
Names of imported functions and classes are not finally and can be discussed.
There is a tree-shaking for components and templates !!!.

Vue conditional render based on breakpoint

Iurii Panarin — Tue, 23 Oct 2018 08:35:31 +0000

Good day all! I was create directive for conditionals renders based on screen size, support resize event.
https://github.com/PxyUp/vue-not-visible
https://www.npmjs.com/package/vue-not-visible
Demo: https://pxyup.github.io/vue-not-visible/index.html

Angular Virtual Table

Iurii Panarin — Mon, 22 Oct 2018 23:54:22 +0000

Good evening, I created virtual scroll table based on material virtual scroll. This table support filtering, sorting and you can easy customize that. Demo included

https://github.com/PxyUp/ng-virtual-table

Demo: https://pxyup.github.io/ng-virtual-table

In future planned so many options like drag and drop, more filter types and more like that. Thanks for any help :)