Forem: Dmitry Kalinin

TikTok Scraper

Dmitry Kalinin — Fri, 18 Dec 2020 05:07:03 +0000

This is not an official API support and etc. This is just a scraper that is using TikTok Web API to scrape media and related meta information.

Installation

tiktok-scraper requires Node.js v10+ to run.

Install from NPM

npm i -g tiktok-scraper

Install from YARN

yarn global add tiktok-scraper

In Terminal

$ tiktok-scraper --help
Usage: tiktok-scraper <command> [options]
Commands:
  tiktok-scraper user [id]     Scrape videos from username. Enter only username
  tiktok-scraper hashtag [id]  Scrape videos from hashtag. Enter hashtag without #
  tiktok-scraper trend         Scrape posts from current trends
  tiktok-scraper music [id]    Scrape posts from a music id number
  tiktok-scraper video [id]    Download single video without the watermark
  tiktok-scraper history       View previous download history
  tiktok-scraper from-file [file] [async]  Scrape users, hashtags, music, videos mentioned
                                in a file. 1 value per 1 line
Options:
  --version               Show version number                          [boolean]
  --timeout               Set timeout between requests. Timeout is in
                          Milliseconds: 1000 mls = 1 s              [default: 0]
  --number, -n            Number of posts to scrape. If you will set 0 then all
                          posts will be scraped                     [default: 0]
  --proxy, -p             Set single proxy                         [default: ""]
  --proxy-file            Use proxies from a file. Scraper will use random
                          proxies from the file per each request. 1 line 1
                          proxy.                                   [default: ""]
  --download, -d          Download video posts to the folder with the name input
                          [id]                        [boolean] [default: false]
  --asyncDownload, -a     Number of concurrent downloads            [default: 5]
  --hd                    Download video in HD. Video size will be x5-x10 times
                          larger and this will affect scraper execution speed.
                          This option only works in combination with -w flag
                                                      [boolean] [default: false]
  --zip, -z               ZIP all downloaded video posts
                                                      [boolean] [default: false]
  --filepath              File path to save all output files.
      [default: "/Users/blah/blah"]
  --filetype, --type, -t  Type of the output file where post information will
                          be saved. 'all' - save information about all posts to
                          the` 'json' and 'csv'
                               [choices: "csv", "json", "all", ""] [default: ""]
  --filename, -f          Set custom filename for the output files [default: ""]
  --noWaterMark, -w       Download video without the watermark. This option will
                          affect the execution speed  [boolean] [default: false]
  --store, -s             Scraper will save the progress in the OS TMP or Custom
                          folder and in the future usage will only download new
                          videos avoiding duplicates  [boolean] [default: false]
  --historypath           Set custom path where history file/files will be
                          stored
                   [default: "/var/folders/d5/fyh1_f2926q7c65g7skc0qh80000gn/T"]
  --remove, -r            Delete the history record by entering "TYPE:INPUT" or
                          "all" to clean all the history. For example: user:bob
                                                                   [default: ""]
  --webHookUrl       Set webhook url to receive scraper result as HTTP requests.
                     For example to your own API                   [default: ""]
  --method           Receive data to your webhook url as POST or GET request
                                      [choices: "GET", "POST"] [default: "POST"]
  --help             Show help                                         [boolean]
Examples:
  tiktok-scraper user USERNAME -d -n 100
  tiktok-scraper user USERNAME -d -n 100 -f customFileName
  tiktok-scraper hashtag HASHTAG_NAME -d -n 100
  tiktok-scraper trend -d -n 100
  tiktok-scraper music MUSICID -n 100
  tiktok-scraper music MUSIC_ID -d -n 50
  tiktok-scraper video https://www.tiktok.com/@tiktok/video/6807491984882765062
  tiktok-scraper history
  tiktok-scraper history -r user:bob
  tiktok-scraper history -r all
  tiktok-scraper from-file BATCH_FILE ASYNC_TASKS -d -n 25

Features

Download unlimited post metadata from the User, Hashtag, Trends, or Music-Id pages
Save post metadata to the JSON/CSV files
Download media with and without the watermark and save to the ZIP file
Download single video without the watermark from the CLI
Sign URL to make custom request to the TikTok API
Extract metadata from the User, Hashtag and Single Video pages
Save previous progress and download only new videos that weren’t downloaded before. This feature only works from the * * CLI and only if download flag is on.
View and manage previously downloaded posts history in the CLI
Scrape and download user, hashtag, music feeds and single videos specified in the file in batch mode

Recovers passwords from pixelized screenshots

Dmitry Kalinin — Wed, 09 Dec 2020 19:52:14 +0000

Depix is a tool for recovering passwords from pixelized screenshots.

This implementation works on pixelized images that were created with a linear box filter.

Usage

Cut out the pixelated blocks from the screenshot as a single rectangle.
Paste a De Bruijn sequence with expected characters in an editor with the same font settings (text size, font, color, hsl).
Make a screenshot of the sequence. If possible, use the same screenshot tool that was used to create the pixelized image.
Run pythonpython depix.py -p [pixelated rectangle image] -s [search sequence image] -o output.png

Algorithm

The algorithm uses the fact that the linear box filter processes every block separately. For every block it pixelizes all blocks in the search image to check for direct matches.
For most pixelized images Depix manages to find single-match results. It assumes these are correct. The matches of surrounding multi-match blocks are then compared to be geometrically at the same distance as in the pixelized image. * Matches are also treated as correct. This process is repeated a couple of times.
After correct blocks have no more geometrical matches, it will output all correct blocks directly. For multi-match blocks, it outputs the average of all matches.

Captcha solver extension for humans

Dmitry Kalinin — Mon, 16 Nov 2020 10:18:47 +0000

Buster is a browser extension which helps you to solve difficult captchas by completing reCAPTCHA audio challenges using speech recognition. Challenges are solved by clicking on the extension button at the bottom of the reCAPTCHA widget.

reCAPTCHA challenges remain a considerable burden on the web, delaying and often blocking our access to services and information depending on our physical and cognitive abilities, our social and cultural background, and the devices or networks we connect from.

The difficulty of captchas can be so out of balance, that sometimes they seem friendlier to bots than they are to humans.

The goal of this project is to improve our experience with captchas, by giving us easy access to solutions already utilized by automated systems.

Linting for your git commit messages

Dmitry Kalinin — Sat, 14 Nov 2020 20:08:58 +0000

Gitlint is a git commit message linter written in python: it checks your commit messages for style.

Features

Commit message hook
Easily integrated
Sane defaults
Easily configurable
Community contributed rules
User-defined rules
Full unicode support
Production-ready

Installation

pip

#### Pip is recommended to install the latest version
pip install gitlint

macOS

brew install gitlint
sudo port install gitlint  # alternative

Ubuntu

apt-get install gitlint

Docker

docker run --ulimit nofile=1024 -v $(pwd):/repo jorisroovers/gitlint

Usage

# Check the last commit message
gitlint
# Alternatively, pipe a commit message to gitlint:
cat examples/commit-message-1 | gitlint
# or
git log -1 --pretty=%B | gitlint
# Or read the commit-msg from a file, like so:
gitlint --msg-filename examples/commit-message-2
# Lint all commits in your repo
gitlint --commits HEAD

# To install a gitlint as a commit-msg git hook:
gitlint install-hook