Forem: Navdeep Singh Rathore

Debugging problems in a deeply nested javascript dependency

Navdeep Singh Rathore — Sat, 22 Oct 2022 09:59:53 +0000

I was about to release a new patch version of the desktop app that I am working on and once the changes were ready, I decided to bump the npm version and do some final sanity checks
I first ran npm i and then npm start. And as I waited for the app launch, something felt off this time, and a quick look at the terminal showed what was wrong

keyv? This was not a package from my source code, and nor did I add any extra dependencies, so why now? what changed....

What changed in the peer dependency? (that I never knew I was using)

I still don't completely know what this package does but that isn't important

keyv is a package that is used by cacheable-request which is used by the famous HTTP client GOT. keyv now provides its own type definitions, so you don't need @types/keyv package as a dev dependency anymore.

Understandably, the DefinatelyTyped team, one that maintains the @type modules, decided to remove the types for keyv since everything had been stable for a while now.
The PR removes not just the keyv types but also it's reference inside the types of other packages that depended on it

But legacy projects that still either ran on node 14 (which does not easily provide support for the new ESM modules) or were just somehow using an older version of keyv as a nested dependency still needed @types/keyv

My fixation on this random "patch-fix" of a probably unheard package, started not because I was using an old version of GOT, but rather an old version of electron builder, which in turn was using an old version of GOT

(You see how deep this story goes!)

The reason for the bug is obvious now if you know how typescript references types for modules. If the package does not have an [filename].d.ts, the file's type definition are not available.
This was what DefinatelyTyped solved by providing the @types module.
So the packages that still referenced this old @types/keyv module broke once this PR was merged a few days back

What actually went wrong?

So first of all this could be easily considered a breaking change and should not have been released as a patch version,
as pointed out in this comment

I think cacheable-request, which again I did not know we depended on, had "*" as the version for keyv in its package.json

So everyone ended up upgrading to this package and now everything breaks

Solution: Pinning Dependency version

As mentioned in the referenced issue, you can pin the package to a particular version using overrides inside package.json

"overrides" : {
    "@types/keyv": "3.1.4"
  },

But the better thing would be to just update all your dependencies.

If you haven't done this for some time now, something like npm-check-upgrades. I find myself referencing this article from freecodecamp every time I have to do so - https://www.freecodecamp.org/news/how-to-update-npm-dependencies/

TIL: "*" as a dependency version is a disaster waiting to happen. If not for you, then for the people using your code.

Packets and Frames in libav transcoding pipeline

Navdeep Singh Rathore — Wed, 29 Dec 2021 07:19:12 +0000

I found it a bit confusing to identify when to use AVFrame and when to use AVPacket while working with libav.

TLDR; Packets contain encoded data. Frames contain decoded data

That is the only difference between packets and frames but it is the core reason for their differences in attributes and the stage in which they are produced in a transcoding pipeline.

Let us understand this with an example

Taking referrence from the famous libav tutorial on github

The AVFormatContext is the abstraction for the format of the media file, aka container (ex: MKV, MP4, Webm, TS). The AVStream represents each type of data for a given format (ex: audio, video, subtitle, metadata). The AVPacket is a slice of compressed data obtained from the AVStream that can be decoded by an AVCodec (ex: av1, h264, vp9, hevc) generating a raw data called AVFrame.

To implement this, we use the following functions (in this order)

avcodec_send_packet to send an encoded packet to the decoder
avcodec_receive_frame to recieve a raw frame from the decoder

We then perform any modifications such as scaling, resampling, cropping, format conversion, etc. on the raw frame.
Remember to keep the frames compatible for the encoder being used.

avcodec_send_frame to send a raw frame to the encoder
avcodec_receive_packet to recieve an encoded packet from the encoder

But this also has further intricacies like:

Multiple frames in single packet

A (encoded) packet could contain multiple (raw) frames while. This can occur separately while encoding and decoding and depends on the attributes of the codec (AVCodecContext).
Hence it is a good idea to use avcodec_receive_frame inside a loop and read frames till it does not return an error, specifically AVERROR(EAGAIN), that signals that the internal buffer (of the codec) is now empty.

`avcodec_decode_` and `avcodec_encode_` methods

It can be confusing to find these methods being used in other libav projects/scripts to build a complete transcoding workflow with fewer function calls.
But, as mentioned here the above mentioned workflow of packets and frames is meant to replace these legacy functions.

Setting network proxy on linux desktop

Navdeep Singh Rathore — Mon, 20 Dec 2021 18:44:15 +0000

I was trying to write a script to setup system wide proxy on my Ubuntu desktop. Turns out, Linux makes it way too difficult to configure a proxy across all apps.

App Specific Proxies
Gnome network settings
Using Environment Variables
Configure Firewall

0. Setting app specific proxies

Most web browsers (Firefox in particular) have their own internal proxy settings. There is abundant documentation for all of these on their respective websites.

1. Gnome network settings

If you are using Gnome, you can go to Settings > Network > Network Proxy. Select Manual and enter the proxy details.

This the most obvious method and it covers most general use cases like using a proxy for WiFi inside a corporate or university network.

Under the hood, this sets attributes in gconf(maybe dconf in modern distros). This can be done from the terminal as:



gsettings set org.gnome.system.proxy mode 'manual'
gsettings set org.gnome.system.proxy.http enabled true
gsettings set org.gnome.system.proxy.http host 'proxy.server.addr'
gsettings set org.gnome.system.proxy.http port proxy_port

2. Setting environment variables

The previous method does not always work for shell instances. A proven way for using a proxy while working on the terminal (either while doing a simple curl, or while launching an app) is by setting environment variables like http_proxy or HTTP_PROXY to set a proxy for all http request.

Note: you might want to set both the uppercase and the lowercase version. This is necessary because, without a proper standard, over the years apps and languages either support both (with a set of priorities) or any one of the two. This leads to some long, head scratching debugging sessions, like this one

You can either do this for each shell instance like:



http_proxy="username:password@proxy_address:port" 
curl http://www.example.com

# you can skip the authentication part for an open proxy

or be smart guy, and export this and other relevant variables inside .bashrc (or maybe .zshrc) as



export http_proxy="http://$username:$password@$proxy_address:$port"
export HTTP_PROXY="http://$username:$password@$proxy_address:$port"

By other relavant variables, I mean

https_proxy / HTTPS_PROXY
ftp_proxy / FTP_PROXY
no_proxy / NO_PROXY
rsync_proxy / RSYNC_PROXY

To check if these variables have been set properly, you can run the following:
set | grep -i proxy
Similarly, these can be unset individually when required with unset http_proxy

Bonus
If you feel like all this is too much, you can use a tool like (ProxyMan)[https://github.com/himanshub16/ProxyMan/] that is a lightweight shell script with a simple interface to both set and unset these variables.

This also does advanced steps like setting a proxy for commands like apt update, if you require.
This is done by adding the following to /etc/apt/apt.conf



Acquire::http::Proxy "http://username:password@proxy_address:port";

You can go even further with tools like proxychains that let you chain multiple proxies. This is used while setting up tor and comes pre-installed with pentesting distros like Kali and ParrotOS.

3. Using iptables

Most general use cases will be satisfied by the above 2 steps.
Even if you need to setup something like your own man-in-the-middle proxy like mitm or a transparent proxy like squid for simple logging, you could still use the previous step and launch apps from the terminal while setting the required variables to point to the proxy server
That does, mostly, work.

But what if you want to send all of your system's traffic through your proxy?
To setup a system wide proxy, most google searches will lead you to answers on stack-exchange that recommend some form of method #2. \
But, as stated in the Gitlab blog it does not always work, and is also a source of unwanted bugs.

Full disclaimer: I haven't tried this properly myself and am also fairly new to using iptables. But, it seems like the only valid approach out there that tries to solve this specific problem
I have kept my explanation; based on my rudimentary understanding; as short as possible
Please reference the post mentioned below for a better explanation

iptables is a tool provided by netfilter that is used to configure custom firewalls on your system.

For our use case, we can mark all our out going network packets from a certain user and forward it to the process running our proxy.

Make sure that the proxy is running under a separate user, so as to prevent the proxy from intercepting its own output.

Step-by-step instructions - https://serverfault.com/a/674389/946308

Some references online that helped me