Forem: Roshan Acharya

Distillation Attacks : Risks and Controversies

Roshan Acharya — Sun, 01 Mar 2026 14:50:07 +0000

Do you know, You can replicate behaviour of large language model on a small model that has better inference time, less computation cost and competitive benchmark as LLM ?

This is possible using a technique called knowledge distillation, where a smaller model, commonly called "student" model learns to mimic larger model, "teacher" model.

Let's understand about this powerful concept and the issues that arises along with it.

What is Model Distillation ?

Model distillation is a technique where the basic idea is to make a smaller model (student) mimics the behaviour of a larger model (teacher).The goal is to get better generalization in the student model than training it from scratch. Here student model may learn from:

Soft probability distributions : Instead of predicting a single hard label, the model outputs a probability distribution over all possible classes or tokens.
Logits : Logits are the raw outputs of the neural network before softmax is applied.
Generated text Outputs : This is the final text produced by the model. Even if probabilities or softmax are hidden, attackers collect (prompt, generated_text) to perform distillation.

From Compression to Exploitation

Orginally model ditillation technique was designed as a compression technique, a way to transfer knowledge to small model. In this setting, both the teacher and student models belong to the same organization, and the goal is optimization.

However, the context changes when the teacher model is closed-source and accessible only through an API.

Distillation becomes an attack when:

The teacher model is proprietary and not publicly available
The student model is trained without permission
The attacker only interacts through API queries
The objective is not compression, but capability replication

In this scenario, the attacker does not need access to the model’s architecture, training data, or weights. Instead, they rely purely on the outputs generated by the system.

This process is commonly referred to as:

Model extraction
Model stealing
Black-box distillation

How Distillation Attack Works ?

Distillation attacks involves following steps :

Step 1: Querying
Here, Attacker generates random prompts on specific domain based on sysnthetic data. This collection of query is fed to Teacher model to generate output.

Step 2: Dataset Construction
In this step, The collected query is passed to model which generated output. Given input prompt along with model output is stored as a dataset.

Step 3: Student Model Training
In this final step, the attacker trains a smaller transformer model using the constructed dataset. The objective is simple: given the same input prompt, the student model tries to predict the teacher model’s output as closely as possible. This is typically done by minimizing cross-entropy loss, and in more refined setups, KL-divergence is used to match the probability distributions of the teacher’s predictions. Over time, the student does not just memorize responses, ut begins to mimmic teacher response.

Recent Controversies on Distillation Attack

Recently, Anthropic claimed that three chinese companies DeepSeek, Moonshot AI and MiniMax ran large scale distillation attacks on its claude models. Anthropic alleges that :

~24,000 fraudulent accounts were used to access Claude.
Over 16 million prompt–response exchanges were generated to extract capability data.
These interactions were then allegedly used to train rival AI systems.

Also Beyond formal reports, online communities have sparked their own controversies sometimes based on technical quirks rather than confirmed evidence.

One example that gained traction on Reddit was a post claiming that Anthropic’s Sonnet 4.6 model responded to certain Chinese‑language prompts with:

“I am DeepSeek‑V3, an AI assistant developed by DeepSeek.”

This led some users to speculate that :

Sonnet had been distilled from or trained on DeepSeek’s model outputs

Conclusion : Who Really Owns AI Knowledge ?

Large Language Models are trained on massive datasets scraped from internet. All of this contsnts are publicly available, yet its use in training is often debated legally or ethinically.

If training an AI on the internet is considered legal, some might ask "why is it illegal to replicate a model using its outputs through distillation ?"

What do you think? Drop your thoughts on comment.

Merge Resolver: Automatic Git Conflict Resolution Using AI

Roshan Acharya — Mon, 16 Feb 2026 12:58:21 +0000

This is a submission for the GitHub Copilot CLI Challenge

Merge conflicts are one of the most common pain points in software development.

When multiple developers work on the same codebase, resolving conflicts manually can be tedious, error-prone, and time-consuming.

What I Built

Git Merger AI is designed to make conflict resolution:

🤖 AI-Powered: Sends conflicting code blocks to Copilot for smart resolutions.
💾 Safe: Automatically backs up files before modifying them.
🔍 Preview Mode: Lets you see AI’s suggested changes before committing.
⚡ Cross-Platform: Works on Linux, macOS, and Windows.
📊 Conflict Summary: Shows exactly how many blocks were resolved per file.

The goal: let developers focus on building features, not spending hours resolving conflicts.

How It Works

Detect all conflicted files in the repo using Git commands.
Parse each file for conflict blocks using a robust regex parser.
Send each block to GitHub Copilot CLI with a concise prompt.
Replace the conflict block with the AI-generated solution.
Stage the file and optionally auto-commit the changes.
Provide a clean summary of all resolved conflicts.

Demo

Github Repository : https://github.com/roshan-acharya/Merge-Resolver

Screenshot

Merger-Resolver Preview Model

Merge-Resolver Resolving Merge

Merged File Output

My Experience with GitHub Copilot CLI

Using GitHub Copilot CLI was a game-changer:

It intelligently suggested merged code, combining multiple features automatically.
Allowed me to script the full workflow from detection → resolution → commit.
Helped me iterate faster and reduce manual conflict errors.

Extra effort I put in:

Backup system to prevent accidental loss
Preview mode for safe “dry-run” testing
Robust regex parser for multi-line and indented conflicts
Cross-platform support

This project demonstrates how AI can make DevTools smarter, reducing developer friction in real workflows.

Installation & Usage

Clone Repository and create CLI tool:

git clone https://github.com/roshan-acharya/Merge-Resolver
pip install -r requirements.txt
cd Merge-Resolver
pip install -e .

Preview Mode

git-merger --p

Run the resolver when conflict occur:

git-merger

Key Takeaways

Merge conflicts don’t have to be painful — AI can handle them.
GitHub Copilot CLI enables automation at the command line, not just IDEs.
Combining regex parsing + AI resolution + Git automation creates a powerful DevTool.

Team Member :

@loveless_bhartal_92b27cd9

Codify - MongoDB Atlas Hackathon 2022 on DEV

Roshan Acharya — Thu, 08 Dec 2022 17:00:13 +0000

What I built

Codify is a web app which allows you to write code , code snippets and store them on a cloud and access it . User can also share their code to other simple by providing link of the editor.

Category Submission:

Choose Your Own Adventure

App Link

Codify : https://codi-fy.netlify.app

Screenshots

Home Page

Code Collection Page

Editor Page

Description

Codify is an open source MIT licensed project that is built using MERN . It can be used as paste bin with many more features that a developers can use . Code Editor UI which provide user a better experience while using it.

Feature :

Write once access anywhere
Sharable to other people
Syntax highlighting and Line counter
Code Editor UI and many more

Link to Source Code

roshan-acharya / codify

A platform to write , access and share code

codify

Codify is a MIT licensed open source project for DEV X Mongodb hackathon.

View on GitHub

The application repository:

Permissive License

MIT

Background

The idea of making codify came when I saw some of my friends were using protected text to store their code at school's computer lab. They were using it because they can access they from anywhere whenever necessary. I have decided to make this project including some features that can be needed for a programmer. It includes features to write , share , access code snippets including code highlighting , auto-close brackets and many more for programmer from anywhere.

How I built it

In this hackathon project , I learned backend development which is one of the best achievement that I achieved . So I created application using

ReactJs
NodeJs
ExpressJS
MongoDB (Primary Database)

There are two schema. They are:

Users (For user info)
Codes (For users code)

I had learned a lot of things at this hackathon project . Thanks to Dev X MongoDb !! I will be participating in other hackathons to level up my skills.

Additional Resources/Info

Mongodb : https://www.mongodb.com/docs/
React : https://reactjs.org/docs/getting-started.html