Forem: krish

[Project Breakdown] Syncord: Encrypted File Storage via Discord

krish — Mon, 29 Dec 2025 22:08:35 +0000

In today’s episode of project breakdown (yes, I’m starting this just now), we’ll look at one of my recent projects: Syncord.

What Is Syncord?

Syncord is a CLI-based file storage tool that uses Discord as an encrypted file storage.

It allows you to upload files or entire directories to Discord, encrypt and partition them locally, and later download and reconstruct them securely; all through a simple command-line interface or an optional TUI dashboard.

So in short, it is more or less a google drive but instead of google it's discord and instead of drive it's encrypted partitions.

Why Even Create This Project?

I’ve seen similar projects before, but most of them relied on a web server (Flask, Express.js) and a browser-based workflow. I personally didn’t enjoy hosting a server just to upload or download files.

syncord was built to be purely command-line driven, something I could put in my PATH and just use instantly without opening a browser.

There are objectively better file storage solutions like Google Drive or One Drive, and I’d recommend those for most users. syncord exists primarily as a learning project and experiment, built during my college winter break.

People might question whether this violates Discord’s ToS. While syncord uses a user-owned bot, standard attachment uploads, and documented APIs. It does not exploit bugs. However, excessive or abusive usage could still result in rate limiting or action against the bot.

Why Discord?

Well, Discord provides up-to 8MB of upload limit per message and while Discord does not guarantee long-term retention, but in practice files persist unless removed or moderated (I still have photos sent to a friend back in 2019 on Discord). So why not use it?

I am thinking of adding telegram or other "free" storage providers too but that would most likely not happen anytime soon.

The Core Idea

Project structure:

syncord/
└── core/
    ├── ascii_hell.py
    ├── db_manager.py
    ├── discord_handler.py
    ├── encrypter.py
    ├── partition.py
    ├── setup.py
    └── tui.py
└── main.py
└── setup.yaml
└── syncord.db
└── syncord.exe

Let us talk about the 8MB upload limit first

files size <= 8MB

If a file is smaller than 8MB then you can upload it without having any issues from discord and would work as intended.

file size > 8MB

Problem starts when the file size is actually greater than 8MB. To tackle that in my project, I have created a partition system that chunks files into relevant size and a SQLite DB that tracks the partition number and partition ID.

What is this partition system? `core/partition.py`

When a file is intended to be uploaded via syncord, the file is broken into 5.5MB chunks and the metadata is stored in a database.

The files created have {number}.bin extension since I am essentially saying binary of the file itself.

DB Table: core/database_manager.py

CREATE TABLE if not exists syncord(parition_number integer, parition_uuid text, message_id text primary key, file_name text, folder_name text, file_size_bytes integer)

The DB includes: (yes typo was done in the DB itself which I never fixed)

parition_number: Which number partition this record represents. (e.g. 1.bin, 2.bin, etc)
parition_uuid: Which file are we talking about?
message_id: We get this after uploading to Discord.
file_name: Helps with knowing the original name along with it's extension.
folder_name: Since syncord support entire folder upload feature so this helps with knowing the exact tree of it.
file_size_bytes: Currently used for stats. Later could be used to checksum to further save the integrity of file.

What about privacy?

While uploading to discord sounds like a wonderful idea for some people. Some might also question how safe it is since anyone can join your discord server without much effort. For that, I have implemented an encryption system.

Encryption System `core/encrypter.py`

I used fernet (Fernet encryption is a symmetric, authenticated cryptography system from Python's cryptography library that uses a single secret key for both encryption and decryption, ensuring messages can't be read or tampered with unless you have the key.) to encrypt my bytes before making then .bin file which gets uploaded. The key is stored locally (setup.yaml) on the PC itself rather than uploading that on Discord.

Since fernet introduces it's own metadata overhead, padding and base64 encoding on top of the data, it increases the file size (approx ~33.33%) and that was why I chunked on 5.5MB and not 8MB itself.

How is it stored on discord?

I am using py-cord that is commonly used for creating a Discord BOT. A channel named syncord gets created in the provided guild ID. All the files are sent together via the API and their message IDs are stored in the DB to help retrieve the files again when downloaded.

setup:
  bot_token: "your_bot_token"
  encryption_key: "auto_generated_using_fernet"
  guild_id: 1234567890123456789

This pretty much sums up what exactly happens in the background.
Upload:

File is partitioned into 5.5MB chunks (if needed)
Each chunk is encrypted using Fernet
Chunks are uploaded via a Discord bot
Message IDs and metadata are stored locally in SQLite

Download:

Metadata is read from the local database
Chunks are fetched using stored message IDs
Data is decrypted locally
Original file is reconstructed and saved

Usage

Prebuilt Binary (Recommended)

Download syncord.exe
Place it in a directory included in your system PATH
Verify installation:

syncord --help

syncord can now be invoked from any directory.

Mandatory Initial Setup

Syncord requires setup before first use.

Both the Discord bot token and Guild ID are mandatory.

Required Configuration

Discord Bot Token
Discord Guild ID (server where files are uploaded)
Encryption Key (generated locally)

syncord setup --token YOUR_DISCORD_BOT_TOKEN --guild-id YOUR_GUILD_ID --encryption-key

Important Notes

The encryption key is critical.
Losing the key means permanent data loss.
Discord cannot decrypt or recover your files.
syncord will not function without a valid token and guild ID.

Uploading Files

Upload a Single File

syncord upload file.ext

Upload an Entire Directory

syncord upload folder_name

Behavior:

All files inside the directory are processed
Folder structure is logically preserved
Each directory becomes its own Syncord folder namespace

Downloading Files

syncord download folder_on_syncord/file

Syncord will:

Locate all required partitions
Download them from Discord
Decrypt locally
Reassemble the original file

If no folder is provided, Syncord assumes the default upload namespace default.

TUI Dashboard

syncord dashboard

Provides:

Stored file overview
Folder and file downloads
Storage usage
Interactive navigation

Usage Statistics

syncord stats

Displays:

Total files uploaded
Total downloads
Storage usage
Other tracked metrics

Command Summary

Command	Description
`syncord setup`	Mandatory initial configuration
`syncord upload`	Upload a file or directory
`syncord download`	Download and reconstruct a file
`syncord dashboard`	Launch terminal UI dashboard
`syncord stats`	Show usage statistics

Limitations & Tradeoffs

Discord is not a storage service and offers no durability guarantees.
Heavy usage may trigger rate limits or bot restrictions.
Loss of the encryption key results in permanent data loss.
Metadata (filenames, folder names) is stored locally but not encrypted.

Conclusion

This was my first time building a TUI-based application, coming from a background focused mainly on back-end API development. The project was fun.

Writing this blog also motivated me to start documenting my past and future projects. I hope the dev.to community enjoys these breakdowns. I plan to make them a regular thing.

This was also one of my first blog written specifically over a project. I would apologize if it was a little challenging to read or comprehend, I intend to improve over time and practice.

GitHub Link: https://github.com/krishsharma0413/syncord ⭐ if you like what you see.
personal rating: 8.5/10
Here is a video of the entire project.

Different Encoding Methods for your Dataset.

krish — Tue, 16 Jul 2024 04:27:32 +0000

Hey there, data enthusiasts! 🎀

In the exciting world of data science and machine learning, one of the first and most crucial steps is turning raw data into a format that our models can understand and learn from. This process, called data preprocessing, involves several important steps:

Data Cleaning: Removal of noise and inconsistent data. Let's say there was a feature with 80% null values. will you still keep it? What about 20% null values. Those can easily be filled with statistics like mean of all categorical data.
Data Integration: Combine multiple dataset sources for better predictions. Eg. combining driver's medical record with race and season data to predict their position in an F1 race. While the health wouldn't be much helpful but using that as a weight for previous race position will drastically increase its importance!
Data Selection: Selection important and useful data. Try doing feature engineering and get the best features for your model.
Data Transformation: Data are transformed and consolidated for mining by performing encodings and feature engineering. I consider this as the most important topic before data mining since without encoding, data mining is useless and unhelpful.
Data Mining: Intelligent methods are applied to extract data patterns OR Extraction of implicit, previously unknown and potentially useful information from data. Eg. using the race year and DOB of driver to find out the age of the driver to provide new insights while removing 2 columns from model.
Pattern Evaluation: Identify the truly fascinating pattern using various evaluation metrics.
Knowledge Presentation: Create graphs and stats like charts, heatmaps, and much more. Understand your data and improvise wherever needed using above steps.

Central to this preprocessing is the task of encoding. This blog delves into the various encoding methodologies, providing a comprehensive analysis of them.

Importance of Encoding

Encoding is a crucial step in the data preprocessing pipeline, especially when dealing with categorical data. Categorical variables, which represent data that can be divided into specific groups or categories, often need to be converted into a numerical format for machine learning algorithms to process them effectively. This conversion process is known as encoding. Machine learning models typically require numerical input because they are based on mathematical calculations that cannot interpret categorical data directly. By transforming categorical data into numerical values through various encoding techniques, we can ensure that our models can leverage all available information, leading to better performance and more accurate predictions. Encoding not only makes data suitable for analysis but also helps preserve the relationships and characteristics inherent in the original categorical variables.

Prerequisites

No sane person codes on paper, he who codes on paper has mastered the essence of coding or the truth behind the universe itself. - ME🎀

Install the following required Python libraries

pip install scikit-learn pandas category_encoders

Different datasets requires different encoding methods. Therefore, different examples might get used for each encoding methods.

Types of Encoding

While there are hundreds of encoding methods, we will focus on the most important and widely used ones.

Multi-Hot Encoding
Label Encoding
Ordinal Encoding
Binary Encoding
Target Encoding
Frequency Encoding

Multi-Hot Encoding

This method converts into binary-like data. Categorical values is mapped to a binary vector of length equal to the no. of categories. This method is usually used in classification models.

Example: Imagine you have a dataset of music tracks.

Name	Artist	Genre
Fly Me to the Moon	The Macarons Project	["slow", "acoustic", "pop"]
Mad at Disney	Salem ilese	["dance", "pop"]

Here, the genre is a feature we need to encode since providing array of multiple genre-names would be ineffective to the model.

from sklearn.preprocessing import MultiLabelBinarizer
import pandas as pd

# Creating the dataframe with list of genres per song
df = pd.DataFrame({
    "name": ["Fly Me to the Moon", "Mad at Disney"],
    "artist": ["The Macarons Project", "Salem ilese"],
    "genre": [["slow", "acoustic", "pop"], ["dance", "pop"]]
})

# Using MultiLabelBinarizer to handle the list of genres
mlb = MultiLabelBinarizer()
x_encoded = mlb.fit_transform(df["genre"])

# Creating the encoded dataframe
encoded_df = pd.DataFrame(x_encoded, columns=mlb.classes_)

# Concatenating the original columns with the encoded genres
df_final = pd.concat([df.drop(columns=["genre"]), encoded_df], axis=1)

print(df_final)

name	artist	acoustic	dance	pop	slow
Fly Me to the Moon	The Macarons Project	1	0	1	1
Mad at Disney	Salem ilese	0	1	1	0

The data is encoded with the genres where 1 means HOT (or present) and 0 means COLD (or absent). A similar approach can be taken with One-Hot Encoding but binary Encoding or Label Encoding is better in those cases most of the time.

Label Encoding

This method converts each categorical value into a numerical data.

Similar to multi-hot encoding in a way. The only key difference would be that Label Encoding might inadvertently introduce ordinal relationships where none exist, which can mislead some algorithms. multi-hot encoding avoids this by treating each category independently.

Example: A company sells shirt of different sizes and colours for X amount of price.

Colour	Size	Company	Price
red	L	Max	300
blue	S	ACM	230
red	XL	Zara	568
green	S	Gucci	927

where we need to use encoding for all 3 columns Colour, Size, and Company. We will use Label Encoding since that addition to bias can help model to predict with better accuracy.

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Creating the dataframe
df = pd.DataFrame({
    'Colour': ['red', 'blue', 'red', 'green'],
    'Size': ['L', 'S', 'XL', 'S'],
    'Company': ['Max', 'ACM', 'Zara', 'Gucci'],
    'Price': [300, 230, 568, 927]
})

# Label Encoding for 'Colour', 'Size', and 'Company'
label_encoder = LabelEncoder()
df['Colour_encoded'] = label_encoder.fit_transform(df['Colour'])
df['Size_encoded'] = label_encoder.fit_transform(df['Size'])
df['Company_encoded'] = label_encoder.fit_transform(df['Company'])

# Drop the original categorical columns after encoding
df_final = df.drop(columns=['Colour', 'Size', 'Company'])

print(df_final)

Price	Colour_encoded	Size_encoded	Company_encoded
300	2	0	2
230	0	1	0
568	2	2	3
927	1	1	1

The numerical value here is assigned by sorting (alphabetically or numerically) the categories by default but if we want to intentionally give a preference to this encoding then we should look into Ordinal Encoding

Ordinal Encoding

Similar to Label Encoding with the only difference that we ourselves provide a specific order of importance to the categories (unlink how label encoder sorted all categories to provide numbering to it).

Example: In the Label Encoding example, the company should be in your preference order since we know companies like Gucci or Zara will sell T-shirts at expensive prices.

Colour	Size	Company	Price
red	L	Max	300
blue	S	ACM	230
red	XL	Zara	568
green	S	Gucci	927

Let's use ["ACM", "Max", "Zara", "Gucci"] as our order of cheap to expensive T-shirts.

import pandas as pd
from sklearn.preprocessing import LabelEncoder, OrdinalEncoder

# Creating the dataframe
df = pd.DataFrame({
    'Colour': ['red', 'blue', 'red', 'green'],
    'Size': ['L', 'S', 'XL', 'S'],
    'Company': ['Max', 'ACM', 'Zara', 'Gucci'],
    'Price': [300, 230, 241, 927]
})

# Label Encoding for 'Colour' and 'Size'
label_encoder_colour = LabelEncoder()
label_encoder_size = LabelEncoder()

df['Colour_encoded'] = label_encoder_colour.fit_transform(df['Colour'])
df['Size_encoded'] = label_encoder_size.fit_transform(df['Size'])

# Ordinal Encoding for 'Company' with the specified reversed order
company_order = ["ACM", "Max", "Zara", "Gucci"]
ordinal_encoder = OrdinalEncoder(categories=[company_order])

df['Company_encoded'] = ordinal_encoder.fit_transform(df[['Company']])

# Drop the original categorical columns after encoding
df_final = df.drop(columns=['Colour', 'Size', 'Company'])

print(df_final)

Price	Colour_encoded	Size_encoded	Company_encoded
300	2	1	1
230	0	0	0
241	2	2	2
927	1	0	3

This adds bias to the model depending upon the company name.

Binary Encoding

This method converts each categorical value into binary digits (0s and 1s) then store them as separate columns. This is useful when you have many categories to encode and want to reduce dimensionality compared to multi-hot encoding.

Converts each category into binary code and then split the binary digits into separate columns. Results in log2(N) amount of columns while multi-hot encoding would provide (N) columns.

Example: Encoding just the Colours into something suitable.

Colour
Red
Green
Blue
Red

import pandas as pd
from category_encoders import BinaryEncoder

# Sample data
data = pd.DataFrame({'Colour': ['Red', 'Green', 'Blue', 'Red']})

# Create a BinaryEncoder object
encoder = BinaryEncoder(cols=['Colour'])

# Encode the categorical feature
encoded_data = encoder.fit_transform(data)

print(encoded_data)

Colour_0	Colour_1
0	1
1	0
1	1
0	1

Most of the time, if the categories are less. We should use multi-hot encoding or label encoding.

Target Encoding

Also known as Mean Encoding or Livelihood encoding. This method encodes the categorical values by replacing each category with statistics of the target variable in that category.

Highly recommended and very useful for handling high cardinality categorical variables. This captures relationship between the categorical variables and the target variable more effectively than one-hot encoding.

Formula:

\frac{(n \times Categorical Mean) + (m \times Global Mean)}{n + m}

here:

n: No. of samples.
m: smoothing parameter.

Example: In house prediction model, encoding neighborhood names wth mean of house price in those area would provide more insights than just normal label encoding.

House Number	Price	Neighborhood	Size (sq meter)
1	500000	Downtown	200
2	350000	Suburb	150
3	700000	City Center	300
4	450000	Suburb	180
5	600000	Downtown	250

import pandas as pd

# Original dataset
data = {
    'House Number': [1, 2, 3, 4, 5],
    'Price': [500000, 350000, 700000, 450000, 600000],
    'Neighborhood': ['Downtown', 'Suburb', 'City Center', 'Suburb', 'Downtown'],
    'Size (sq meter)': [200, 150, 300, 180, 250]
}

df = pd.DataFrame(data)

# Calculate mean price for each neighborhood
neighborhood_means = df.groupby('Neighborhood')['Price'].mean().to_dict()

# Map mean prices back to the original dataset
df['Neighborhood'] = df['Neighborhood'].map(neighborhood_means)

# Display the encoded dataset
print(df)

House Number	Price	Neighborhood	Size (sq meter)
1	500000	550000.0	200
2	350000	400000.0	150
3	700000	700000.0	300
4	450000	400000.0	180
5	600000	550000.0	250

Frequency Encoding

This method replaces each categorical value with its frequency or count within the training dataset.

Formula:

\text{Frequency}(\text{category}) = \frac{\text{Count}(\text{category})}{\text{Total observations}}

Example: Encoding cities based on the no. of times each city appears in the dataset.

Transaction ID	Amount	City	Product Category
1	100	New York	Electronics
2	200	Los Angeles	Clothing
3	150	Chicago	Electronics
4	300	New York	Groceries
5	250	Chicago	Clothing

import pandas as pd

# Example dataset with customer transactions
data = {
    'Transaction ID': [1, 2, 3, 4, 5],
    'Amount': [100, 200, 150, 300, 250],
    'City': ['New York', 'Los Angeles', 'Chicago', 'New York', 'Chicago'],
    'Product Category': ['Electronics', 'Clothing', 'Electronics', 'Groceries', 'Clothing']
}

df = pd.DataFrame(data)

# Using 'City' as a parameter (simple example)
selected_city = 'New York'

# Filter the dataset for the selected city
filtered_data = df[df['City'] == selected_city]

print(f"Data for transactions in {selected_city}:")
print(filtered_data)

# Applying frequency encoding to 'City'
city_frequency = df['City'].value_counts(normalize=True)
df['City'] = df['City'].map(city_frequency)

print(df)

Transaction ID	Amount	City	Product Category
1	100	0.4	Electronics
2	200	0.2	Clothing
3	150	0.4	Electronics
4	300	0.4	Groceries
5	250	0.4	Clothing

Conclusion

With this, all the important and necessary encoding methods are covered! Choosing the right encoding method can significantly impact the performance of your machine learning models.

If I could, THIS is how I will learn Python again 🐍

krish — Sun, 26 Nov 2023 11:09:45 +0000

tldr; official python docs

Unlocking the door to knowledge requires the right keys: curiosity, persistence, and a sprinkle of passion. Learning isn't a task, it's an adventure waiting to be embraced.

Python being one of the most used language in programming is surprisingly one of the easiest to learn as well.

Before we start, we need to set you up with your Python interpreter as well as the IDE you will be using.

Installation

Windows Users

Go to https://www.python.org/downloads/ and find the version that fits you (or is required for your project/company). If you are looking for the latest version then the Download Python 3.12.0 is all you need to get started.

After downloading the setup you need to install it on your PC.
I recommend checking the checkbox that asks you to add python to PATH variable.

Linux Users

Python is already installed in Linux but usually it's the Python2 instead of the new Python3
Although, you can use the terminal to install the latest version as well. on Ubuntu it is.

sudo apt-get update
sudo apt-get install python3.12

MAC Users

In general, the Mac OS x systems have already installed a version of python, which can be used.
Use this official website to download python.

Python Releases for macOS | Python.org

The official home of the Python Programming Language

python.org

If you want another, you can use brew.

brew install pyenv
pyenv install 3.12.0

NOTE: Most of the time you don't really need to use the terminal to install python and you can get away with it using the python.org website itself.

Which IDE?

IDK Just use whatever you like...
Personally, I use VSCode. Don't ask why. I just use it.
If you really are serious about Python then maybe it's better to choose a good IDE from the start and use that and learn its shortcuts since shortcuts help you speed up your productivity a lot than what you think it does.
Following are some of the IDEs out there.

Visual Studio Code

Visual Studio Code - The open source AI code editor

Visual Studio Code redefines AI-powered coding with GitHub Copilot for building and debugging modern web and cloud applications. Visual Studio Code is free and available on your favorite platform - Linux, macOS, and Windows.

code.visualstudio.com
Pycharm

PyCharm: The only Python IDE you need

Built for web, data, and AI/ML professionals. Supercharged with an AI-enhanced IDE experience.

jetbrains.com
Spyder

spyder-ide.org
Thonny

thonny.org

Reading Material

Official Python Documentation

Nothing is as good as the documentation of python itself. It is slow paced and teaches everything there is in python properly.

The Python Tutorial — Python 3.14.2 documentation

Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python’s elegant syntax an...

docs.python.org

Python Playground

Maybe you are not into just plain text reading and you want some practice while you learn. For that you can use the following link. It has a playground feature where you can practice with an online interpreter as well.
https://www.learnpython.org/
Liquid error: internal

Lecture Material

YouTube

YouTube is filled with multiple videos that teaches you python, Some being 12 hours long while some takes only 100 seconds. Personally i recommend looking into this

Paid Courses

One of the best paid python Specialization out there.

Python for Everybody | Coursera

Offered by University of Michigan. Learn to Program and Analyze Data with Python. Develop programs to gather, clean, analyze, and visualize ... Enroll for free.

coursera.org

Further Advancing in Python

Python is all about its library so learning how to use in-built libraries is a plus. I will recommend you to learn libraries like asyncio, math, datetime, time, hashlib, itertools, functools and unittest can greatly help you with your Competitive Programming or open source career.

Refer to the following link for official documentation on in-built libraries.

docs.python.org

Learn how to install external libraries using pip from here.

Installing Python Modules — Python 3.14.2 documentation

Email, distutils-sig@python.org,. As a popular open source development project, Python has an active supporting community of contributors and users that also make their software available for other...

docs.python.org

Something quite important when it comes to programming is learning how about Data Structures and Algorithm. Although, I won’t recommend python for CP.

Practice Practice Practice...

Without practice you won’t be able to remember what you learnt. You can look into GitHub for open-source projects to contribute into to improve your documentation and unit testing skills or you can look into the following websites to practice via problem questions.

Codewars: Good for both beginners as well as professionals. The website is filled with many questions and you can practice and get better in python using the website.

codewars.com

CodeBat: Code practice website. Quite basic but gets the job done.

codingbat.com

I personally don’t recommend leetcode or hackerrank until you have completed DSA.

After all this, the journey in python isn’t still completed. There is a lot to learn even now. Mainly domain specific libraries.

I am a python developer with over 4 years of experience. You can find me here

Forem: krish

[Project Breakdown] Syncord: Encrypted File Storage via Discord

What Is Syncord?

Why Even Create This Project?

Why Discord?

The Core Idea

files size <= 8MB

file size > 8MB

What is this partition system? core/partition.py

What about privacy?

Encryption System core/encrypter.py

How is it stored on discord?

Usage

Prebuilt Binary (Recommended)

Mandatory Initial Setup

Important Notes

Uploading Files

Downloading Files

TUI Dashboard

Usage Statistics

Command Summary

Limitations & Tradeoffs

Conclusion

Different Encoding Methods for your Dataset.

Importance of Encoding

Prerequisites

Types of Encoding

Multi-Hot Encoding

Label Encoding

Ordinal Encoding

Binary Encoding

Target Encoding

Frequency Encoding

Conclusion

If I could, THIS is how I will learn Python again 🐍

Installation

Windows Users

Linux Users

MAC Users

Python Releases for macOS | Python.org

Which IDE?

Visual Studio Code - The open source AI code editor

PyCharm: The only Python IDE you need

Reading Material

Official Python Documentation

The Python Tutorial — Python 3.14.2 documentation

Python Playground

Lecture Material

YouTube

Paid Courses

Python for Everybody | Coursera

Further Advancing in Python

Installing Python Modules — Python 3.14.2 documentation

Practice Practice Practice...

What is this partition system? `core/partition.py`

Encryption System `core/encrypter.py`