Forem: Shafiq Jetha

Removing comments from code-based data source

Shafiq Jetha — Thu, 21 Dec 2023 15:06:16 +0000

Photo by Markus Spiske on Unsplash

Introduction

I am in the process of experimenting with training an LLM on a codebase. My goal is to build a foundational model that I can then create different generative AIs from that are more focused on a task, say Code Review or High-Level Documentation. I needed to start from a good known source even if it might be a little small for my ultimate goal so that I know whether or not I was going in the right direction. I started by setting up the training process for a BERT-style LLM. I chose BERT because I believe it is best for trying to build understanding of its source material.

Getting to work

Under the Microsoft username, there is a dataset called LCC_csharp. I started using this as the codebase I wanted to work with is also written in C#, but I quickly found a significant issue with the codebase.

// See the LICENSE file in the project root for more information. // // using System; using System.Runtime.InteropServices; using Windows.Foundation; #pragma warning disable 436 // Redefining types from Windows.Foundation namespace Windows.UI.Xaml.Media.Media3D

Where would you put the line breaks for the code snippet above?

The structured C# files had been compressed down to a single line. This isn't so much of an issue but it becomes hard to find where a comment ends and the code itself begins. I wasn't going to sit there and figure this out by hand, so I decided to program my way out of the problem.

The Execution

My plan was to use the Roslyn Analyzer developed by one of the .NET teams at Microsoft to do static analysis on C# code. The Analyzer has a concept known as Trivia, and each section of code is a different Trivia. Once the code is read into a Syntax Tree, it becomes a structured document and different parts of the file can be queried and modified easily.

var commentTrivia = from t in tree.GetRoot().DescendantTrivia()
    where t.IsKind(SyntaxKind.SingleLineCommentTrivia) ||
          t.IsKind(SyntaxKind.MultiLineCommentTrivia) ||
          t.IsKind(SyntaxKind.SingleLineDocumentationCommentTrivia)
    select t;

The above code block finds all of the major types of comment Trivia that exist in the document. The next thing I do is remove those items from the document and normalize the whitespace so that it takes on a more natural shape. I had intended to then save the document as is, with line breaks and all, but I quickly found out that this corrupted the Parquet format that the data was originally saved in. A lot of trial and error later, I settled on removing all of the line breaks, effectively putting all of the code back on one line, but this time, without the comments, the code could be read as a big line of code rather than a document where comments and code intermingled without a clear break between them all.

using System;using System.Runtime.InteropServices;using Windows.Foundation;#pragma warning disable 436 // Redefining types from Windows.Foundationnamespace Windows.UI.Xaml.Media.Media3D

Not perfect, but a lot better than before.

Conclusion

Now the codebase can be tokenized from scratch and should contain only meaningful code. Comments are incredibly important for us humans to fully understand a piece of code, but I feel that it's more important for a foundational LLM to be able to generate good code. From there I can reliably build Q&A, code-completion and documentation LLMs that fine-tune the base weights to be better at their individual tasks. Ideally I can then merge these all together in a Mixture of Experts model that can be good at a variety of tasks, and have all been trained or at least fine-tuned on the specific codebase.

Using GPT 3.5 Turbo to summarise specific YouTube comments

Shafiq Jetha — Mon, 30 Oct 2023 04:07:18 +0000

Photo by Jeswin Thomas on Unsplash

Generative AI has become quite the buzzword lately, but there aren’t a lot of projects out there that focus on improving our mental state with regards to the constant barrage we receive from social media.

My first attempt at helping to work towards this goal is to create a service that allows people to parse their YouTube comments and get summarise of comments that are classified as being below a certain threshold. As the criterion for this, I have chosen Psychological Safety, and so any comments that are classified as being of low psychological safety will get summarised by the Generative AI and the exact comment will never be displayed to the user. By doing this, we eliminate or at least significantly reduce the psychological load that social media can put on us.

Here is the output when running this video through the analyzer:

High Psychological Safety:

Great video as always but why is poor Jango posed like he needs to use the toilet

I like it, thanks for showing the comparison. Definitely be getting the Batman. As always, your builds and opinions are greatly appreciated.

You answered a question I was wondering about.

Hi again! Im a really big fan of your videos! You inspired me to build my own lego city and to buy many lego sets!!!!??????

I love this. I just wish lego would make some figures for other technic sets

I literally just got these two sets, Batman figure and his bike, yesterday! Now I know for sure they will look good enough to be displayed together. Thank you for the video.

Got both the Ducati, and Batcycle earlier in Summer. While both display great on their own, i too was curious how they would scale to the newest buildable figures. I was thinking the bike would be too small, but the scaling is much closer than initially thought. Balance looks to be a bit of an issue but cool to see the comparison shots. Definitely getting the Batman now, and maybe the Captain America as he looks like a good fit for the Ducati as both bikes are very similar in size.

Thanks for this video. Really answered the question that has been on my mind for some time. Nice job

Medium Psychological Safety:

Hi. Unrelated question. But when are we going to hear any news about the speed champions 2024?

Actually, combining construction figures with Technic motorcycles is not a new idea, and can be a fun way to waste some time. Your own Darth Vader Chopper inspired me to put some Star Wars buildable figures on some of the older Technic motorbikes and the results look pretty cool. I’ve put a New Order StormTrooper on an orange Dirtbike and a Pretorian Guard (the ones with the weird helmets and red robes) on a Streetracer and on the Ducati. The Batcycle is still waiting to be built, and I was thinking it would become Darth Vader’s bike with some small modifications. Now what vehicle would I put General Grievous on?

I wish Darth Vader had a motorbike of his own ??

I think it would have been pretty cool if they released both these products in a single set, which could have lowered the total price perhaps

Lol that split on two bikes frame xD

I like it might only get those sets only for that

Low Psychological Safety (paraphrased):

All of these lego figures are not as appealing as I’d like them to be. It would be great if they improved them in the future.

Those buildable figures have ugly design and the proportions are off too. Improvement is needed.

All of these lego figures are garbage. I wish they came up with a replacement for the old technic figures.

As you can see, all of the comments were reproduced as they were written aside from those that had been deemed as having low psychological safety, and were summarised to avoid unnecessary mental strain.

Combining Jest and Cypress code coverage reports in your Angular app

Shafiq Jetha — Sat, 31 Jul 2021 18:38:58 +0000

Photo by Isaac Smith on Unsplash

When writing front-end tests, code coverage is an important metric that helps you determine how many critical paths of your application are covered.

Cypress has their own tutorial and example repo which uses Babel, but in this post I’ll go through the process of doing this with an Angular application, without using Babel.

Instrumenting and reporting

Codecov has a feature whereby any code coverage metrics that are uploaded with the same branch name will be merged into one report. This is a useful feature if you have several different test runners and need them all to be combined into one report. For producing coverage reports for Jest and Cypress some features need to be added and enabled.

Jest comes with the istanbul instrumenter and coverage reporting tool built-in, and since this is now the standard for instrumenting code, this is a useful feature. To use it, you have to run jest with the --coverage flag added. If you already have this in your package.json file, you can make the following modification:

- "test": "ng test"
+ "test": "ng test -- --coverage"

This will produce a coverage report for your Jest tests, but this isn’t enough. We also need Jest to collect coverage for all of the files that have no test. In your jest.config.js file, you will need to make this modification:

+ rootDirs: ['<rootDir>/src'],
+ collectCoverage: true,
- coverageReporters: ['json', 'html'],
+ coverageReporters: ['lcov'],
+ collectCoverageFrom: ['**/*.ts'],
+ coverageDirectory: './coverage/jest',

This will create an lcov format coverage report, but will also give us a proper understanding of what is and isn’t covered across our entire application.

We’re also going to put out output in a separate directory so that when we run our Cypress tests, our Jest coverage report doesn’t get overwritten.

To get Cypress set up there are a few more steps involved. As mentioned in the readme on skylock/cypress-angular-coverage-example because the Cypress package doesn’t do instrumenting, we have to add the @cypress/code-coverage package and modify the build process. This will trigger instrumenting on our code so that we can produce a coverage report. This package uses Istanbul’s successor nyc, and as such a second step is required for us to generate an lcov format report. We also need to modify the webpack config for the development build so that the code gets instrumented correctly.

Begin by adding the @cypress/code-coverage package:

npm i -D @cypress/code-coverage

Then add the package to Cypress’ support/index.ts file:

+ import '@cypress/code-coverage/support

Now that we have Cypress set up, we can move onto modifying the build process. We achieve this by using the ngx-build-plus package which can be installed with the following command:

npm i -D @ngx-build-plus

In the cypress folder, a coverage.webpack.js needs to be added with the following content:

module.exports = {
  module: {
    rules: [
      {
        test: /\.(js|ts)$/,
        loader: 'istanbul-instrumenter-loader',
        options: { esModules: true },
        enforce: 'post',
        include: require('path').join(__dirname, '..', 'src'),
        exclude: [/\.(e2e|spec)\.ts$/, /node_modules/, /(ngfactory|ngstyle)\.js/],
      },
    ],
  },
};

This will allow us to instrument our code, but we also need to modify our angular.json file to use this webpack file:

- "builder": "[@angular](http://twitter.com/angular)-devkit/build-angular:dev-server",
+ "builder": "ngx-build-plus:dev-server",
+ "options": { "browserTarget": "ng-new-app:build", "extraWebpackConfig": "./cypress/coverage.webpack.js" }

Now we can test our project by running

ng e2e

npm run <project>:cypress-run

Once the test run has completed, you still need to covert the nyc output to lcov so that you can proceed with the next step. You can achieve this by running this command:

npx nyc report --reporter=lcov

This will transform your nyc output into lcov output, so that both reports are in the same format. You can now decide how you wish to track your code coverage.

Option 1: Codecov

When you upload multiple reports, Codecov will merge these reports together, so simply uploading the coverage directory’s content will combine these results and produce an accurate coverage report.

If you are using GitHub Actions, uploading the files is as simple as calling the relevant Action:

uses: codecov/codecov-action@v1

Once it is uploaded, you will get access to Codecov’s different coverage diagrams as well as a histogram of your coverage over time.

Coverage over time for an example project

Option 2: Manually merge the reports

If you simply want to manually merge the reports and feed them into your own code coverage tool, lcov reports can be appended to one another and still retain the same detail. After installing the lcov package in Ubuntu, you can generate a HTML report of the combined coverage by running the following command:

genhtml --prefix <directory> lcov.info --output-directory=./cov

An example of the genhtml output

If you’d like to have a ready-to-go project that has all of this included, and more, check out this project I helped out with: sardapv/angular-material-starter-template

Now go forth, and cover!

Why you need Wallaby.js for your Front-End Unit Tests

Shafiq Jetha — Sat, 03 Jul 2021 06:38:22 +0000

JJKPhoto by Jerin J on Unsplash

Wallaby.js is a test runner for front-end unit tests. It has been written so that tests are run as you type, giving you an incredible amount of insight into the code you write.

Wallaby.js integrates cleanly with the internet’s favourite IDEs, and can also be run from the command line, allowing for it to be run as part of a git pre-commit hook, or as part of a CI/CD pipeline.

Wallaby.js provides time-travel debugging, which allows you to step backwards and forwards through your tests to understand precisely what values are being set and when, so that you can build tests that are less flaky.

Integration into, for example, Visual Studio Code will makes it easy to see which tests pass, which ones fail, and provide gutter highlighting to show you which parts of your code have been covered, and where failures are occurring.

Wallaby’s strength comes from how quickly it can re-run tests and how quickly it shows errors across both the test file and all of the code that is touched by the test.

Wallaby.js in action

In contrast, using Jest’s watch feature, the same change takes more than 4 seconds to run the same test.

And if you’re like me and use Visual Studio Code for everything, there are various extensions that try to get close to Wallaby’s speed, but noting gives you that instant feedback and that wonderful feeling quite like Wallaby.