Forem: lidianycs

How I Built a Tool to Detect AI-Generated Fake References

lidianycs — Mon, 05 Jan 2026 16:27:42 +0000

Large Language Models(LLMs) have become part of everyday academic and technical writing. But there is a problem the academic community has been flagging for a while, and many of us have encountered it firsthand: LLMs are very good at inventing citations. They may look plausible and almost match real papers. But they confidently cite work that does not exist at all. The academic community is calling them Ghost References.

Closing out my year with a journal editor shocker 🧵 Checking new manuscripts today I reviewed a paper attributing 2 papers to me I did not write. A daft thing for an author to do of course. But intrigued I web searched up one of the titles and that's when it got real weird...
— Ben Williamson (@benpatrickwill.bsky.social) 2025-12-19T17:20:04.127Z

As Professor Ben Williamson and Aaron Tay explained, the root problem is deep-seated:

"The ghost reference problem is a chronic condition that has become acute. The infection predates GenAI; the technology has simply lowered our immune response while accelerating transmission."

The issue is compounded because LLMs with general web search capabilities can fail to reliably verify references, as the web itself contains fake citations, creating a dangerous feedback loop. Despite being wrong, the sources are widely assumed to be authentic, the more they appear in published literature. For instance, one of the Ghost References to Prof. Williamson's work has accumulated 43 citations in Google Scholar.

Addressing the Reviewer's Burden

Peer reviewers are already stretched thin, and now, due to the proliferation of fake references, they have to manually copy-paste every single reference into a search engine to verify its existence.

This is a tedious, low-reward task often skipped in favor of focusing on the paper's actual content. But this "verification gap" is exactly where ghost references can slip through.

When It Happened to Me

That abstract concern turned into a concrete problem worth addressing when I discovered my own paper had been incorrectly cited in a published paper.

Seeing the flawed metadata published in a journal was a wake-up call that led me to build CERCA, an open-source tool designed to assist researchers, reviewers, and editors in quickly verifying the accuracy of references. It was developed to improve trust, transparency, and reliability academic writing.

What Is CERCA?

CERCA stands for Citation Extraction & Reference Checking Assistant.

Here's what it looks like in action:

In seconds, CERCA:

Scans a PDF and extracts the references
Queries OpenAlex, Crossref, and Zenodo
Flags potentially invalid citations with confidence scores
Shows you which metadata fields don't match

Instead of copy-pasting each reference manually, you get a verification report you can review in minutes.CERCA automates the tedious process of verifying whether the papers cited in a PDF file actually exist and if the metadata is accurate.

Development Insights

Building CERCA required solving a few interesting engineering challenges, particularly around fuzzy matching and bibliographic parsing.

Academic citations are messy. They come in dozens of formats (APA, MLA, IEEE, ACM, Vancouver, etc.). Creating a parser that could reliably extract these references without false positives was the first hurdle. I used Cermine, a Java library, to handle the heavy lifting of PDF parsing and metadata extraction.

The second was the verification logic. I used fuzzy matching to determine if a citation is close enough to be a typo or far enough to be a hallucination. Here's what the tool can detect:

Cerqueira, M.; Tavares, A.; Couto, C.; Maciel, R.; Santos, D.; Figueira, A. "Assessing software practitioners' work engagement and job satisfaction." [Example of Ghost Citation]

CERCA detects:
⚠️ Author list mismatch (6 fabricated, 9 omitted)
⚠️ Title incomplete
⚠️ First author name inconsistency

Correct paper reference:

Cerqueira, L., Nunes, L., Guerra, R., Malheiros, V., Freire, S., Carneiro, G., ... & Mendonça, M. (2025). Assessing Software Practitioners’ Work Engagement and Job Satisfaction in a Large Software Company—What We Have Learned. SN Computer Science, 6(3), 273.

🗃️ CERCA queries trusted repositories (OpenAlex, Crossref, Zenodo) and
uses fuzzy matching to catch these discrepancies, saving reviewers from
manually checking each citation.

🔍 Manual Fallback: If automatic search fails, you can right-click to
search for reference titles manually.

🔐 Due to reviewers' confidentiality, I put privacy first by design: PDFs are not uploaded and never leave your machine. All PDF parsing and reference extraction are performed locally.

Tech Stack

Java + JavaFX – Cross-platform desktop application
Cermine – PDF parsing and metadata extraction
OpenAlex, Crossref, Zenodo APIs – Reference verification
JavaWuzzy – Handles citation variations and typos

I chose this stack to build a Java desktop app using JavaFX for cross-platform compatibility (Windows, Mac, Linux).

Why Open Source?

Due to the tool's purpose, it must be transparent itself. Besides, this is a collective problem. By making CERCA open source, I'm inviting the community to audit the code, improve the parsers, and integrate more databases.

It is licensed under the GNU Affero General Public License (AGPL-3.0).

Who Can Use CERCA?

It is useful for anyone working on scholarly or technical writing. It is intended for:

Researchers performing final manuscript checks
Reviewers assessing reference consistency
Editors supporting editorial quality control
Meta-research and reproducibility workflows

Join the Effort

Ghost references are threatening scholarly trust. CERCA is a start, but it
needs your expertise:

Try it now:
📥 Download CERCA
(Windows | Mac | Linux)

Cerca does not solve the problem of ghost references, and it is not yet finished. It is a small, practical step. If it helps a researcher catch one incorrect reference, saves a reviewer time, or encourages more critical engagement with AI-generated text, then it is already serving its purpose. But you can help improve it:

🐛 Found an edge case?
💡 Have ideas?
🔧 Want to contribute?

👉🏾 Download the tool and explore the repository here

This project is a work in progress and an invitation to the research and developer communities to experiment, evaluate, and build better tools together.

Share your results: Did CERCA catch a ghost reference in your work? I'd love to hear about it in the comments.

What Does Empathy Really Mean in Software Development?

lidianycs — Tue, 15 Jul 2025 16:15:27 +0000

Empathy is often seen as a “nice-to-have” in tech, but what if it’s actually essential? In the first part of our articles about empathy in software engineering, we explore how developers define empathy, in their own words.

We just published a new study that explores what empathy looks like from the perspective of real software practitioners, using 55 blog posts from communities like DEV and Medium, plus insights from a follow-up survey with empathy experts.

The study was recently accepted at ACM Transactions on Software Engineering and Methodology (TOSEM), a leading peer-reviewed journal for high-quality research in software engineering. Read the preprint here:
👉 Exploring Empathy in Software Engineering

👀 Why Study Empathy?
Empathy helps us communicate, collaborate, and work better as teams. But in software engineering, it's often overlooked or misunderstood. We wanted to change that by listening to developers who’ve reflected publicly on their experiences.

🔍What We Found
Through qualitative content analysis, we identified how empathy shows up, where it breaks down, and what it can achieve when practiced well.

💡 5 Meanings of Empathy in SE

From these reflections, five key themes emerged:

Understanding – the most common definition: grasping how someone thinks or feels

“[Empathy is] the ability to understand how a person feels and what they might be thinking.” —P34
Perspective Taking – seeing a situation through another person’s eyes

“It’s the ability to see things as if from another’s perspective.” —P39

Embodiment – putting yourself in someone else's shoes (e.g., teammates, users)

“Empathy is the ability to put yourself in the other person’s shoes.” —P40

Compassion – caring about the people you work with

“Caring about the people you work with, not just the work you do.” —P48

Emotional Sharing – feeling what others feel, like mirroring stress or anxiety

“[Empathy] includes mirroring what that person is feeling.” —P54

We refer to each practitioner by an ID number, namely P1, P2, ... Pn.

🧠 A Multi-Faceted View

We grouped these meanings using a well-known psychological model:

🧠 Cognitive empathy: understanding, perspective taking, embodiment
💙 Compassionate empathy: caring about others’ well-being
💫 Emotional empathy: sharing someone’s emotional state

While other models in engineering highlight understanding and perspective taking, our study uniquely includes compassion, a crucial but often overlooked aspect.

✨ Why It Matters
This isn’t just about “being nice.” Empathy is a socio-technical skill that can improve software quality and well-being. By making empathy visible, naming its blockers, and offering strategies, our work helps teams reflect, adapt, and grow.

There’s no universal definition of empathy in software engineering. By gathering how real devs talk about it, we help build a clearer and more practical understanding that reflects the human side of coding, collaborating, and caring.

🙏 Thank You!
We’re deeply grateful to the software practitioners who shared their stories, whether through personal blog posts or by offering feedback in our follow-up survey. Your reflections gave this research meaning and depth. This study wouldn’t exist without your willingness to speak openly about the challenges and possibilities of empathy in software development.

🔭 What’s Next?
This is just one step in our journey! Next up: what gets in the way of empathy at work?

We’re also expanding this work through a mixed-methods study in software companies, combining survey data and qualitative insights to better understand how empathy is practiced in real-world teams.
As we continue, we’re also refining and evolving the conceptual framework based on this new data, so it can be even more actionable and relevant for devs, managers, and educators.
If you're passionate about empathy in tech or working to create more human-centered workplaces, let’s connect. And stay tuned, we’ll share more as this next phase unfolds.

🙌 Check It Out
📖 Full paper: https://arxiv.org/abs/2507.05325
🔍 Dataset for replication and reuse: https://doi.org/10.5281/zenodo.15800354

We’d love to hear your thoughts:

What does empathy look like in your day-to-day work?
Have you seen it foster (or falter) in your team?
What helps or hinders empathy in your org?

Exploring the DEV Community as a Data Source for Human Aspects in Software Engineering Research

lidianycs — Thu, 10 Jul 2025 18:42:38 +0000

The DEV community is a rich and underutilized data source for Software Engineering (SE) research. In my PhD, I've been using DEV articles since 2022 to build a conceptual framework of empathy in SE, capturing how empathy is perceived, practiced, and challenged across roles, tasks, and organizational settings. In this post, I’ll describe how to collect and analyze DEV.to content for research purposes, especially for those interested in the social dimensions of software work.

You’ll learn:

Why DEV.to is a valuable source for qualitative studies
How to extract articles using a Python script or Google Sheets
A brief overview of how to conduct a qualitative analysis
Links to open-access tools, scripts, and published studies

Whether you're a researcher, a student, or just curious about how developers’ voices can inform SE research, this post offers practical steps and reflections to help you get started.

The DEV platform, with its long-form posts written by developers, provides candid insights into how practitioners think, feel, collaborate, and grow. These reflections help us understand topics like empathy, well-being and mental health, inclusion, communication, and collaboration in real-world software practice.

DEV is not just a place for tutorials and code snippets! It's a vibrant community where developers openly share their experiences, struggles, and values, making it a valuable site for qualitative research.

Why DEV?

Unlike surveys or interviews, which may be shaped by researcher framing or social desirability bias, DEV posts represent voluntary, organic reflections. These narratives often cover deeply human experiences: burnout, team dynamics, empathy, mentorship, psychological safety, and more.

Grey Literature (GL), including blog posts, articles, and forum discussions, is a valuable source of data for SE researchers, especially in areas involving human and social dimensions.


The process to collect the web articles. Source

How to Retrieve Articles from DEV?

I've used two methods to collect articles tagged with the keyword empathy.

1. Python Scraper:
You can clone and run our Python script from Zenodo:

python dev_scraper.py

📦 The scraper and full replication package are publicly available on Zenodo:
👉 https://zenodo.org/records/15800354

2. Google Sheets + IMPORTJSON.gs:
If you prefer a low-code approach, use the IMPORTJSON.gs script:

Download the script here.
Open a new Google Sheets document.
Go to Extensions > Apps Script and paste the script code.
Use the code below to pull articles tagged with empathy:

=ImportJSON("https://dev.to/api/articles?tag=empathy&per_page=1000")

You can change the tag or increase the per_page value for broader queries.

Analyzing the Data

Once you have the articles, you can conduct a qualitative analysis process to explore themes, perceptions, and experiences, like this:

Clean and prepare the data (remove duplicates, unrelated posts).
Import into a qualitative tool (spreadsheets or tools like Atlas.ti, MAXQDA, or Taguette also work).
Inductive coding to identify emerging themes.
Synthesize themes to build conceptual categories (e.g., empathy practices, barriers, effects).
Triangulate or validate with experts or additional sources.

This approach was adopted to build a framework of empathy in software engineering based on practitioners' own voices. 👉 Check it here.

Want to Go Further?

If you’re interested in how DEV data can be used in real research, here are the studies using articles from the DEV community to explore empathy in SE:

📝 Published Studies Using DEV as Data Source

A Thematic Synthesis on Empathy in Software Engineering based on the Practitioners' Perspective, SBES 2023
🔗 https://doi.org/10.1145/3613372.3613407
Empathy and Its Effects on Software Practitioners’ Well-Being and Mental Health, IEEE Software, 2024
🔗 https://doi.org/10.1109/MS.2024.3377897
Exploring Empathy in Software Engineering: Insights from a Grey Literature Analysis of Practitioners' Perspectives, TOSEM, 2025
🔗 https://arxiv.org/abs/2507.05325 (preprint)

📖 Related Research

If you're looking to explore Grey Literature and online communities as research data in SE, you can start with those studies:

Mining DEV for social and technical insights about software development 🔗 https://doi.org/10.1109/MSR52588.2021.00053
What Evidence We Would Miss If We Do Not Use Grey Literature? 🔗 https://doi.org/10.1145/3475716.3475777

Final Thoughts

If you're interested in exploring human aspects of software work, from empathy to inclusion, communication to stress, the DEV community is an excellent place to start. It’s full of rich, first-person narratives that reflect the lived realities of developers worldwide.

🔍 Whether you're an SE researcher or a curious practitioner, DEV offers insights that go far beyond code!

🙏 Thank you, DEV community! I'd like to acknowledge the generosity of developers who openly share their experiences, reflections, and challenges here on DEV. To everyone who writes, reads, comments, and supports this community: thank you for contributing to knowledge exchange and inspiring meaningful, human-centered research in software engineering.

Finally, if you'd like to know more, check out the published studies or reach out. I’m happy to share experiences and support others on this journey!