Forem: Athan

Connect with the stdlib community on Zulip

Athan — Wed, 17 Dec 2025 05:45:04 +0000

As the stdlib community continues to grow and evolve, so has our need for new ways to connect and collaborate (see, for example, our announcement of office hours and a public events calendar). While Gitter's simple, single-channel interface worked well in the early days, it no longer scales with the range of conversations happening around the project. Today we're excited to announce our new Zulip chat, which provides a more full-featured, structured, and searchable space for us to interact.

Why Zulip?

Zulip is open source and generously supports open-source projects like ours with a free cloud plan. Its channel-and-topic model makes it easier to keep discussions focused, follow ongoing threads, and resurface past knowledge through powerful search features.

Anyone can browse the web-public channels of stdlib's Zulip without an account, and you can sign up at any time to join the conversation.

Join and get started

The stdlib Zulip chat is open to all. A welcome bot will greet you when you first join and share some tips specific to stdlib about how to participate effectively. If you're new to Zulip, their getting started guide is an invaluable resource. If you're already familiar with applications such as Slack or Discord, much of the experience will be familiar.

We encourage you to come say hello in the #introductions channel and take some time to explore other channels and topics that may be of interest to you. If you have any questions about Zulip itself, we've got a channel for that too (#zulip).

The stdlib team is active in the chat, and public messages are the best way to get timely help—no need for routine @-mentions. Asking questions in public is the fastest way to get a response, as more people can help, plus it's likely that someone else will benefit from finding out the answer to your question. The stdlib Code of Conduct applies to all community spaces, including stdlib's Zulip. Should you encounter an issue, Zulip's reporting tools and our moderation team are available.

See you there!

We're looking forward to seeing you in the stdlib Zulip instance! We welcome questions and suggestions as we continue shaping a space that is useful, inclusive, and genuinely supportive for everyone who wants to learn, build, or contribute.

stdlib is an open source software project dedicated to providing a comprehensive suite of robust, high-performance libraries to accelerate your project's development and give you peace of mind knowing that you're depending on expertly crafted, high-quality software.

If you've enjoyed this post, give us a star 🌟 on GitHub and consider financially supporting the project. Your contributions and continued support help ensure the project's long-term success and are greatly appreciated!

Google Summer of Code 2025

Athan — Thu, 27 Feb 2025 18:22:14 +0000

We're thrilled to announce that stdlib was accepted as a Google Summer of Code mentoring organization for 2025!

We are beyond excited to share that stdlib has once again been accepted as a mentoring organization for Google Summer of Code 2025! This marks our second consecutive year participating in this incredible program, and we cannot wait to work alongside aspiring open source contributors to push the boundaries of scientific computing on the web.

Google Summer of Code (GSoC) is a global initiative that introduces new contributors to open source software by offering mentorship and funding for meaningful, long-term projects. Over the years, GSoC has been instrumental in helping open source projects like stdlib grow, while also giving participants valuable real-world software development experience. With our acceptance into GSoC 2025, we are looking forward to welcoming a new wave of enthusiastic contributors who share our vision of making JavaScript and the extended ecosystem of TypeScript, Node.js, Deno, and other JavaScript runtimes first-class environments for numerical and scientific computing.

Reflecting on GSoC 2024: A Year of Growth

Last year marked our first time participating in GSoC, and we could not have asked for a better experience. We had the privilege of mentoring four incredibly talented contributors, each of whom made substantial contributions to the stdlib ecosystem.

From integrating BLAS bindings and optimizing special mathematical functions to enhancing support for boolean arrays and improving our interactive REPL experience, their work strengthened the foundation of stdlib and paved the way for even greater advancements. Beyond just code, their contributions sparked deeper engagement within our community, leading to over 2,000 pull requests from more than 100 contributors and 3,000+ new commits to stdlib since February 2024.

If you missed our retrospective on last year's program, be sure to check out our blog post: Reflecting on GSoC 2024.

What's in Store for GSoC 2025?

As we gear up for GSoC 2025, we have a range of exciting project ideas that we hope will inspire potential contributors. Whether you're passionate about numerical computing, statistical modeling, performance optimization, or developer tooling, there's something for you. Some areas we're particularly excited about include:

BLAS/LAPACK: continuing to expand stdlib's coverage of BLAS and LAPACK operations to provide a robust foundation for linear algebra and machine learning in JavaScript and Node.js.
WebAssembly: compiling BLAS and statistical kernels to WebAssembly with support for ergonomic inter-operation between WebAssembly and JavaScript.
ndarray kernels: implementing lower-level ndarray kernels for efficient element-wise iteration and reduction to improve performance.
Improving developer tooling: improving the stdlib development experience by creating better tools for automation, publishing, and managing the stdlib package ecosystem.
Expanding statistical distributions: building on previous efforts to provide C implementations for special mathematical functions, thus unlocking a wider range of probability distributions and making stdlib a comparable alternative to SciPy for statistical computing in JavaScript.

These ideas, however, are just the beginning. We believe that innovation comes from collaboration, and we welcome fresh ideas from prospective contributors. If you have a project concept that aligns with our mission and a clear plan for execution, we would love to hear about it. Our current list of ideas is available on our GSoC repository, but don't feel constrained by it—great ideas come from all directions!

How to Get Involved

If you're interested in contributing to stdlib for GSoC 2025, now is the perfect time to get started. Here's how you can begin your journey:

Explore stdlib: familiarize yourself with the project by browsing the project's GitHub repository and reading our documentation.
Join the conversation: engage with the stdlib community on Element to discuss project ideas, ask questions, and connect with mentors.
Review our guidelines: carefully read our GSoC Application Guidelines to understand what we're looking for in a proposal.
Start contributing: we strongly encourage all applicants to contribute to stdlib before submitting their application. This can be in the form of a bug fix, new feature, performance improvement, or some other enhancement to stdlib's capabilities.

The official GSoC timeline is as follows:

February 27 – March 24: prospective contributors discuss project ideas with mentoring organizations.
March 24 – April 8: application period (final deadline: April 8 at 18:00 UTC).
May 8: accepted proposals announced.
May 8 – June 1: community bonding period.
June 2 – September 1: standard 12-week coding period.

For the full timeline, visit the GSoC 2025 Timeline.

Looking Ahead

As we embark on another exciting GSoC season, we want to extend our deepest gratitude to Google for this opportunity. We are incredibly excited to meet new contributors, explore new ideas, and continue building an open source ecosystem where JavaScript thrives as a language for scientific computing.

If you're passionate about building high-quality software and eager to make an impact, we invite you to join us. We can't wait to see your ideas and begin working together to advance scientific computing in JavaScript. Let's make this year's GSoC program one to remember!

New ways to engage with the stdlib community!

Athan — Tue, 14 Jan 2025 00:47:48 +0000

Fostering a vibrant and inclusive community is crucial for ensuring the long-term success of open-source software, and stdlib is no exception. We believe that collaboration and open communication are key to driving innovation and making scientific computing on the web accessible to everyone. To that end, we're thrilled to announce two new initiatives designed to make it easier than ever for contributors, users, and maintainers to connect, collaborate, and grow together!

Weekly Office Hours

As part of our efforts to enhance transparency and collaboration, we're proud to announce weekly office hours! We've been running these informally for the past few months, and they've been a wonderful success, providing high-bandwidth opportunities to connect with project maintainers, users, and new and existing stdlib contributors.

To facilitate the coordination of office hours and other public project meetings, we've created a public GitHub repository to serve as a centralized hub where community members can propose agenda topics, review discussion points, and participate in shaping the direction of stdlib. Each week in advance of the next office hours, we'll create a new dedicated agenda issue, where you can link issues and pull requests you want to discuss, post questions in advance, and share any pre-reads. Thus far, agendas have run the gamut, from project overviews to live code reviews to discussions about the project roadmap to upcoming events and community announcements.

In short, if you have questions about stdlib or if you need help fixing a bug, figuring out what to do next, or are just looking for feedback, this is your time to shine! Please join our weekly office hours to connect with project maintainers, stay updated on the latest project news, and chat with other community members. This is a great opportunity to ask questions, share ideas, and engage directly with the stdlib team.

Everyone is welcome—drop in and say hello!

Public Community Calendar

Second, we're excited to introduce our new public community calendar, where you can stay up-to-date with all stdlib events, including office hours, project orientations, development meetings, and other important happenings.

With this calendar, you can:

Find the dates and times of upcoming office hours and meetings.
Add our events to your own calendar for easy reminders.
Stay informed about new opportunities to engage with the stdlib team and community.

How You Can Get Involved

Here are a few ways you can make the most of these new resources:

Bookmark the community calendar or add it to your own. Be on the lookout for upcoming events, and mark your calendar to join us.
Engage on GitHub. Visit our meetings repository to propose agenda topics or contribute to ongoing discussions.
Attend Office Hours. Whether you're stuck on a problem or curious about the latest project updates, office hours are an excellent opportunity to connect and learn.
Spread the Word. Help us grow the stdlib community by sharing these updates with anyone who might be interested.

Let's Build Together!

We're committed to creating a supportive and inspiring environment for everyone in the scientific computing ecosystem, and we're excited to see how these new initiatives will help our community thrive. Needless to say, we can't wait to connect with you at our next office hours!

Together, we're building the future of scientific computing on the web! 🚀

2024 Retrospective

Athan — Sat, 04 Jan 2025 21:16:29 +0000

A look back at 2024 and a preview of the year ahead.

2024 was a landmark year for stdlib, packed with progress, innovation, and community growth. Looking back, I am struck by the amount of time and effort members of the stdlib community spent refining existing APIs, crafting new functionality, and laying the groundwork for an exciting road ahead. I feel incredibly fortunate to be part of a community that is actively shaping the future of scientific computing on the web, and I am bullish on our continued success in the months to come.

In this post, I'll provide a recap of some key highlights and foreshadow what's in store for 2025. While I'll be making various shoutouts to individual contributors, none of what we accomplished this year could have happened without the entire stdlib community. The community was instrumental in doing the hard work necessary to make stdlib a success, from finding and patching bugs to reviewing pull requests and triaging issues to diving deep into the weeds of numerical algorithms and software design. If I don't mention you by name, please be sure to know that your efforts are recognized and greatly appreciated. A big thank you to everyone involved and to everyone who's helped out along the way, in ways both big and small. ❤️

TL;DR

This past year was transformative for stdlib, marked by significant growth, innovation, and community contributions. Some key highlights include:

Community Growth: 84 new contributors joined stdlib, tripling the size of our developer community and driving over 4,000 commits, 2,200 pull requests, and the release of 500+ new packages.
Google Summer of Code: four exceptional contributors helped advance critical projects, including enhanced REPL capabilities, expanded BLAS support, and new mathematical APIs.
Enhanced Developer Tools: major strides in automation included automated changelog generation, improved CI workflows, and better test coverage tracking.
Technical Milestones: significant progress was made in linear algebra (BLAS and LAPACK), fancy indexing, WebAssembly integrations, and C implementations of mathematical functions, all aimed at making JavaScript a first-class language for scientific computing.
Future Vision: looking ahead to 2025, we aim to expand our math libraries, improve REPL interactivity, explore WebGPU, and continue building tools to make scientific computing on the web more powerful and accessible.

With stdlib’s rapid growth and the collective efforts of our global community, we're shaping the future of scientific computing on the web. Join us as we take the next steps in this exciting journey!

Stats

To kick things off, some high-level year-end statistics. This year,

84 new contributors from across the world joined stdlib, tripling our developer community size and bringing new life and fresh perspectives to the project.
Together, we made over 4000 commits to the main development branch.
We opened nearly 2200 pull requests, with over 1600 of those pull requests merged.
And we shipped over 500 new packages in the project, ranging from new linear algebra routines to specialized math functions to foundational infrastructure for multi-dimensional arrays to APIs supporting WebAssembly and other accelerated environments.

These accomplishments reflect the hard work and dedication of our community. It was a busy year, and we were forced to think critically about how we can effectively scale the project and our community as both continue to grow. This meant investing in tooling and automation, improving our review and release processes, and figuring out ways to quickly identify and upskill new contributors.

Google Summer of Code

The one event which really set things in motion for stdlib in 2024 was our acceptance into Google Summer of Code (GSoC). We had previously applied in 2023, but were rejected. So when we applied in 2024, we didn't think we had much of a chance. Much to our surprise and delight, stdlib was accepted, thus setting off a mad dash to get our affairs together so that we could handle the influx of contributors to come.

GSoC ended up being a transformative experience for stdlib, bringing in talented contributors and pushing forward critical projects. As we detailed in our GSoC reflection, the road was bumpy, but we learned a lot and came out the other side. Needless to say, we were extremely lucky to have four truly excellent GSoC contributors: Aman Bhansali, Gunj Joshi, Jaysukh Makvana, and Snehil Shah. I'll have a bit more to say about their work in the sections below.

REPL

The Node.js read-eval-print loop (REPL) is often something of an afterthought in the JavaScript world, both underutilized and underappreciated. From stdlib's earliest days, we wanted to create a better REPL experience, with integrated support for stdlib's scientific computing and data processing functionality. Development of the stdlib REPL has come in fits and starts, but there's always been a goal of matching the power and feature set of Python's IPython in order to facilitate interactive exploratory data analysis in JavaScript. We were thus quite excited when Snehil Shah expressed interest in working on the stdlib REPL as part of GSoC.

Snehil already covered some of his work in a previous blog post on "Welcoming colors to the REPL!", but his and others' work covered so much more. A few highlights:

Preview completions: when typing characters matching a known symbol in the REPL, a completion preview is now displayed, helping facilitate auto-completion and saving developers precious keystrokes. Shoutout to Tudor Pagu, in particular, for adding this!
Multi-line editing: prior to adding support for multi-line editing, the REPL supported multi-line inputs, but did not support editing previously entered lines, often leading to a frustrating user experience. Now, the REPL supports multi-line editing within the terminal similar to dedicated editor applications.
Pagination of long outputs: a longstanding feature request has been to add support for something like less/more to the stdlib REPL. Previously, if a command generated a long output, a user could be confronted with a wall of text. This has now been addressed, with the hope of adding more advanced less-like search functionality in the months ahead.
Bracketed-paste: pasting multi-line input into the REPL used to execute the input line-by-line, instead of pasting it as a single prompt. While useful in some cases, this is often not the desired intent, especially when a user wishes to paste and edit multi-line input before execution.
Custom syntax-highlighting themes: developers who are used to developing in IDEs can often feel adrift when moving to a terminal lacking some of the niceties of their favorite editor. One of those niceties is syntax-highlighting. Accordingly, we worked to add support for custom theming, as detailed in Snehil's blog post.
Auto-pairing: another common IDE nicety is the automatic closing of brackets and quotation marks, helping save keystrokes and mitigate the dreaded missing bracket. Never one to shy away from a difficult task, Snehil implemented support for auto-pairing as one of his first pull requests leading up to GSoC.

Largely thanks to Snehil's work, we moved much closer to IPython parity in 2024, thus transforming the JavaScript experience for scientific computing. And we're not done yet. We still have pull requests working their way through the queue, and one thing I am particularly excited about is that we've recently started exploring adding support for the Jupyter protocol. Stay tuned for additional REPL news in 2025!

BLAS

Another area of focus has been the continued development of stdlib's BLAS (Basic Linear Algebra Subprograms) support, which provides fundamental APIs for common linear algebra operations, such as vector addition, scalar multiplication, dot products, linear combinations, and matrix multiplication. Coming into 2024, BLAS support in stdlib was rather incomplete, particularly in terms of its support for complex-valued floating-point data types. The tide began to change with Jaysukh Makvana's efforts to achieve feature parity of stdlib's Complex64Array and Complex128Array data structures with built-in JavaScript typed arrays.

These efforts subsequently paved the way for adding Level 1 BLAS support for complex-valued typed array data types and the work of Aman Bhansali, who set out to further Level 2 and Level 3 BLAS support in stdlib. After focusing initially on lower-level BLAS strided array interfaces, Aman expanded his scope by adding WebAssembly implementations and by adding support for applying BLAS operations to stacks of matrices and vectors via higher-level multi-dimensional array (a.k.a., ndarray) APIs.

In addition to conventional BLAS routines, stdlib includes BLAS-like routines which are not a part of reference BLAS. These routines include APIs for alternative scalar and cumulative summation algorithms, sorting strided arrays, filling and manipulating strided array elements, explicit handling of NaN values, and other operations which don't fall neatly under the banner of linear algebra, but are common when working with data.

During Aman's BLAS work, we cleaned up and refactored BLAS implementations, and Muhammad Haris volunteered to extend those efforts to our extended BLAS routines. His efforts entailed migrating Node.js native add-ons to pure C in order to reduce boilerplate and leverage our extensive collection of C macros for authoring of native add-ons and further entailed adding dedicated C APIs to facilitate interfacing with stdlib's ndarrays.

These developments ensure that stdlib continues to lead the way in linear algebra support for JavaScript developers, offering powerful tools for numerical computing. While much has been completed, more work remains, and BLAS will continue to be a focal point in 2025.

LAPACK

Building on the BLAS work as part of an internship at Quansight Labs, Pranav Goswami worked to lay the foundations for LAPACK (Linear Algebra Package) support in stdlib in order to provide higher order linear algebra routines for solving systems of linear equations, eigenvalue problems, matrix factorization, and singular value decomposition. Detailed more fully in his post-internship blog post, Pranav sought to establish an approach for testing and documentation of added implementations and to leverage the ideas of BLIS to create LAPACK interfaces which facilitated interfacing with stdlib's ndarrays and thus minimize data movement and storage requirements. While a good chunk of time was spent working out the kinks and iterating on API design, Pranav made significant headway in adding various implementation utilities and nearly 30 commonly used LAPACK routines. Given the enormity of LAPACK (~1700 routines), this work will continue into the foreseeable future, so be on the lookout for more updates in the months ahead!

As a quick aside, if you're interested in learning more about how stdlib approaches interfacing with Fortran libraries, many of which still form the bedrock of modern numerical computing, be sure to check out Pranav's blog post on calling Fortran routines from JavaScript using Node.js.

C implementations of special math functions

One of stdlib's longstanding priorities is continued development of its vectorized routines for common mathematical and statistical operations. While all scalar mathematical kernels (e.g., transcendental functions, such as sin, cos, erf, gamma, etc, and statistical distribution density functions) have JavaScript implementations, many of the kernels lacked corresponding C implementations, which are needed for unlocking faster performance in Node.js and other server-side JavaScript runtimes supporting native bindings.

Gunj Joshi and others sought to fill this gap and opened over 160 pull requests adding dedicated C implementations. At this point, only a few of the most heavily used double-precision transcendental functions remain (looking at you betainc!). Efforts have now turned to completing single-precision support and adding C implementations for statistical distribution functions. We expect this work to continue for the first half of 2025 before turning our attention to higher-level strided array and ndarray APIs, with implementations for both WebAssembly and Node.js native add-ons.

Fancy indexing

Another area where we made significant progress is in improving slicing and array manipulation ergonomics. Users of numerical programming languages, such as MATLAB and Julia, and dedicated numerical computing libraries, such as NumPy, have long enjoyed the benefit of concise syntax for expressing operations affecting only a subset of array elements. For example, the following snippet demonstrates setting every other element in an array to zero with NumPy.

import numpy as np

# Create an array of ones:
x = np.ones(10)

# Set every other element to zero:
x[::2] = 0.0

As a language, JavaScript does not provide such convenient syntax, forcing users to either use more verbose object methods or manual for loops. We thus sought to address this gap by leveraging Proxy objects to support "fancy indexing". While the use of Proxy objects does incur some performance overhead due to property indirection, you now need only install and import a single package to get all the benefits of Python-style slicing in JavaScript, thus obviating the need for verbose for loops and making array manipulation significantly more ergonomic.

import array2fancy from '@stdlib/array-to-fancy';

// Create a plain array:
const x = [ 1, 2, 3, 4, 5, 6, 7, 8 ];

// Turn the plain array into a "fancy" array:
const y = array2fancy( x );

// Select the first three elements:
const v = y[ ':3' ];
// returns [ 1, 2, 3 ]

// Select every other element, starting from the second element:
v = y[ '1::2' ];
// returns [ 2, 4, 6, 8 ]

// Select every other element, in reverse order, starting with the last element:
v = y[ '::-2' ];
// returns [ 8, 6, 4, 2 ]

// Set all elements to the same value:
y[ ':' ] = 9;

// Create a shallow copy by selecting all elements:
v = y[ ':' ];
// returns [ 9, 9, 9, 9, 9, 9, 9, 9 ]

In addition to slice semantics, Jaysukh added support to stdlib for boolean arrays, thus laying the groundwork for boolean array masking.

import BooleanArray from '@stdlib/array-bool';
import array2fancy from '@stdlib/array-to-fancy';

// Create a plain array:
const x = [ 1, 2, 3, 4, 5, 6, 7, 8 ];

// Turn the plain array into a "fancy" array:
const y = array2fancy( x );

// Create a shorthand alias for creating an array "index" object:
const idx = array2fancy.idx;

// Create a boolean mask array:
const mask = new BooleanArray( [ true, false, false, true, true, true, false, false ] );

// Retrieve elements according to the mask:
const z = x[ idx( mask ) ];
// returns [ 1, 4, 5, 6 ]

We subsequently applied our learnings when adding support for boolean array masking to add support for integer array indexing.

import Int32Array from '@stdlib/array-int32';
import array2fancy from '@stdlib/array-to-fancy';

// Create a plain array:
const x = [ 1, 2, 3, 4, 5, 6, 7, 8 ];

// Turn the plain array into a "fancy" array:
const y = array2fancy( x );

// Create a shorthand alias for creating an array "index" object:
const idx = array2fancy.idx;

// Create an integer array:
const indices = new Int32Array( [ 0, 3, 4, 5 ] );

// Retrieve selected elements:
const z = x[ idx( indices ) ];
// returns [ 1, 4, 5, 6 ]

While the above demonstrates fancy indexing with built-in JavaScript array objects, we've recently extended the concept of fancy indexing to stdlib ndarrays, a topic we'll have more to say about in a future blog post.

Needless to say, we are particularly excited about these developments because we believe they will significantly improve the user experience of interactive computing and exploratory data analysis in JavaScript.

Test and build

Lastly, 2024 was a year of automation, and I would be remiss if I didn't mention the efforts of Philipp Burckhardt. Philipp was instrumental in improving our CI build and test infrastructure and improving the overall scalability of the project. His work was prolific, but there are a few key highlights I want to bring to the fore.

Automatic changelog generation: Philipp shepherded the project toward using conventional commits, which is a standardized way for adding human and machine readable meaning to commit messages, and subsequently built a robust set of tools for performing automatic releases, generating comprehensive changelogs, and coordinating the publishing of stdlib's ever-growing ecosystem of over 4000 standalone packages. What was once a manual release process can now be done by running a single GitHub workflow.
stdlib bot: Philipp created a GitHub pull request bot for automating pull request review tasks, posting helpful messages, and improving the overall maintainer development experience. In the months ahead, we're particularly keen to extend the bot's functionality to help with new contributor onboarding and flagging common contribution issues.
Test coverage automation: with a project of stdlib's size, running the entire test suite on each commit and for each pull request is simply not possible. It can thus be challenging to stitch together individual package test coverage reports in order to obtain a global view of overall test coverage. Philipp worked to address this problem by creating an automation pipeline for uploading individual test coverage reports to a dedicated repository, with support for tracking coverage metrics over time and creating expected test coverage changes for each submitted pull request. Needless to say, this has drastically improved our visibility into test coverage metrics and helped improve our confidence in tests accompanying submitted pull requests.

While we've made considerable strides in our project automation tooling, we never seem to be short of ideas for further automation and tooling improvements. Expect more to come in 2025! 🤖

Look ahead

So what's in store for 2025?! Glad you asked!

We've already alluded to various initiatives in the sections above, but, at a high level, here's where we're planning to focus our efforts in the year ahead:

GSoC 2025: assuming Google runs its annual Google Summer of Code program and we're fortunate enough to be accepted again, we'd love to continue supporting the next generation of open source contributors.
Math and stats C implementations: expanding our library of scalar math and statistics kernels and ensuring double- and single-precision parity.
BLAS: completing our WebAssembly distribution and higher-level APIs for operating on stacks of matrices and vectors.
LAPACK: continuing to chip away at the ~1700 LAPACK routines (!).
FFTs: adding initial Fast Fourier Transform (FFT) support to stdlib to help unlock algorithms for signal processing.
Vectorized operations: automating package creation for vectorized operations over scalar math and statistics kernels.
ndarray API parity: expanding the usability and familiarity of ndarrays by achieving API parity with built-in JavaScript arrays and typed arrays.
REPL: adding Jupyter-protocol support and various user-ergonomics improvements.
WebGPU: while we haven't formally committed to any specific approach, we're keen on at least exploring support for WebGPU, an emerging web standard that enables webpages to use a device's graphics processing unit (GPU) efficiently, including for general-purpose GPU computation, in order to provide APIs for accelerated scientific computing on the web.
Project funding: exploring and hopefully securing project funding to accelerate development efforts and support the continued growth of the stdlib community.

That's definitely a lot, and it's going to take a village—a community of people dedicated to our mission of making the web a first-class platform for numerical and scientific computing. If you're ready to join us in building the future of scientific computing on the web, we'd love for you to join us. Check out our contributing guide to see how you can get involved.

A personal note

As we look ahead, I'd like to share a personal reflection on what this year has meant to me. Given our growth this year, I often felt like I was drinking from a fire hose. And, honestly, it can be hard not to get burned out when you wake up day-after-day to over 100 new notifications and messages from folks wanting guidance, answers to questions, and pull requests reviewed. But, when reflecting on this past year, I am awfully proud of what we've accomplished, and I am especially heartened when I see contributors new to open source grow and flourish, sometimes using the lessons they've learned contributing as a springboard to dream jobs and opportunities. Having the fortune to see that is a driving motivation and a privilege within the greater world of open source that I do my best to not take for granted.

And with that, this concludes the 2024 retrospective. Looking back on all we've achieved together, the future of scientific computing on the web has never been brighter! Thank you again to everyone involved who's helped out along the way. The road ahead is filled with exciting opportunities, and we can't wait to see what we will achieve together in 2025. Onward and upward! 🚀

The Accessor Protocol

Athan — Tue, 06 Aug 2024 20:39:01 +0000

In this post, I'll introduce you to the accessor protocol for generalized array-like object iteration. First, I'll provide an overview of built-in array-like objects, along with example usage. I'll then show you how you can create your own custom array-like objects using the same element access syntax. Next, we will explore why you might want to go beyond vanilla array-like objects to create more "exotic" variants to accommodate sparsity, deferred computation, and performance considerations. Following this, I'll introduce the accessor protocol and how it compares to possible alternatives. Finally, I'll showcase various example applications.

Sound good?! Great! Let's go!

TL;DR

The accessor protocol (also known as the get/set protocol) defines a standardized way for non-indexed collections to access element values. In order to be accessor protocol-compliant, an array-like object must implement two methods having the following signatures:

function get<T>( index: number ): T {...}
function set<T, U>( value: T, index: number ): U {...}

The protocol allows implementation-dependent behavior when an index is out-of-bounds and, similar to built-in array bracket notation, only requires that implementations support nonnegative index values. In short, the protocol prescribes a minimal set of behavior in order to support the widest possible set of use cases, including, but not limited to, sparse arrays, arrays supporting "lazy" (or deferred) materialization, shared memory views, and arrays which clamp, wrap, or constrain index values.

The following code sample provides an example class returning an array-like object implementing the accessor protocol and supporting strided access over a linear data buffer.

/**
* Class defining a strided array.
*/
class StridedArray {
    // Define private instance fields:
    #length; // array length
    #data;   // underlying data buffer
    #stride; // step size (i.e., the index increment between successive values)
    #offset; // index of the first indexed value in the data buffer

    /**
    * Returns a new StridedArray instance.
    *
    * @param {integer} N - number of indexed elements
    * @param {ArrayLikeObject} data - underlying data buffer
    * @param {number} stride - step size
    * @param {number} offset - index of the first indexed value in the data buffer
    * @returns {StridedArray} strided array instance
    */
    constructor( N, data, stride, offset ) {
        this.#length = N;
        this.#data = data;
        this.#stride = stride;
        this.#offset = offset;
    }

    /**
    * Returns the array length.
    *
    * @returns {number} array length
    */
    get length() {
        return this.#length;
    }

    /**
    * Returns the element located at a specified index.
    *
    * @param {number} index - element index
    * @returns {(void|*)} element value
    */
    get( index ) {
        return this.#data[ this.#offset + index*this.#stride ];
    }

    /**
    * Sets the value for an element located at a specified index.
    *
    * @param {*} value - value to set
    * @param {number} index - element index
    */
    set( value, index ) {
        this.#data[ this.#offset + index*this.#stride ] = value;
    }
}

// Define a data buffer:
const buf = new Float64Array( [ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 ] );

// Create a strided view over the data buffer:
const x1 = new StridedArray( 4, buf, 2, 1 );

// Retrieve the second element:
const v1 = x1.get( 1 );
// returns 4.0

// Mutate the second element:
x1.set( v1*10.0, 1 );

// Retrieve the second element:
const v2 = x1.get( 1 );
// returns 40.0

// Create a new strided view over the same data buffer, but reverse the elements:
const x2 = new StridedArray( 4, buf, -2, buf.length-1 );

// Retrieve the second element:
const v3 = x2.get( 1 );
// returns 6.0

// Mutate the second element:
x2.set( v3*10.0, 1 );

// Retrieve the second element:
const v4 = x2.get( 1 );
// returns 60.0

// Retrieve the third element from the first array view:
const v5 = x1.get( 2 );
// returns 60.0

As shown in the code sample above, a strided array is a powerful abstraction over built-in arrays and typed arrays, as it allows for arbitrary views having custom access patterns over a single buffer. In fact, strided arrays are the conceptual basis for multi-dimensional arrays, such as NumPy's ndarray and stdlib's ndarray, which are the fundamental building blocks of modern numerical computing. Needless to say, the example above speaks to the utility of going beyond built-in bracket syntax and providing APIs for generalized array-like object iteration.

To learn more about the accessor protocol and its use cases, continue reading the rest of the post below! 🚀

stdlib

A brief overview about stdlib. stdlib is a standard library for numerical and scientific computation for use in web browsers and in server-side runtimes supporting JavaScript. The library provides high-performance and rigorously tested APIs for data manipulation and transformation, mathematics, statistics, linear algebra, pseudorandom number generation, array programming, and a whole lot more.

We're on a mission to make JavaScript (and TypeScript!) a preferred language for numerical computation. If this sounds interesting to you, check out the project on GitHub, and be sure to give us a star 🌟!

Introduction

In JavaScript, we use bracket notation to access individual array elements. For example, in the following code sample, we use bracket notation to retrieve the second element in an array.

const x = [ 1, 2, 3 ];

// Retrieve the second element:
const v = x[ 1 ];
// returns 2

This works for both generic array and typed array instances. In the next code sample, we repeat the previous operation on a typed array.

const x = new Float64Array( [ 1, 2, 3 ] );
// returns <Float64Array>

// Retrieve the second element:
const v = x[ 1 ];
// returns 2

Similarly, one can use bracket notation for built-in array like objects, such as strings. In the next code sample, we retrieve the second UTF-16 code unit in a string.

const s = 'beep boop';

// Retrieve the second UTF-16 code unit:
const v = s[ 1 ];
// returns 'e'

In order to determine how many elements are in an array-like object, we can use the length property, as shown in the following code sample.

const x = [ 1, 2, 3 ];

const len = x.length;
// returns 3

Arrays and typed arrays are referred to as indexed collections, where elements are ordered according to their index value. An array-like object is thus an ordered list of values that one refers to using a variable name and index.

While JavaScript arrays and typed arrays have many methods (e.g., forEach, map, filter, sort, and more), the only required property that any array-like object (built-in or custom) must have is a length property. The length property tells us the maximum number of elements for which we can apply an operation. Without it, we'd never know when to stop iterating in a for loop!

Custom array-Like objects

We can create our own custom array-like objects using vanilla object literals. For example, in the following code sample, we create an object having numbered keys and a length property and retrieve the value associated with the key 1 (i.e., the second element).

const x = {
    'length': 3,
    '0': 1,
    '1': 2,
    '2': 3
};

// Retrieve the second element:
const v = x[ 1 ];
// returns 2

Notice that we're able to use numeric "indices". This is because, per the ECMAScript Standard, any non-symbol value used as a key is first converted to a string before performing property-value look-up. In which case, so long as downstream consumers don't assume the existence of specialized methods, but stick to only indexed iteration, downstream consumers can adopt array-like object neutrality.

For example, suppose we want to compute the sum of all elements in an array-like object. We could define the following function which accepts, as its sole argument, any object having a length property and supporting value access via numeric indices.

function sum( x ) {
    let total = 0;
    for ( let i = 0; i < x.length; i++ ) {
        total += x[ i ];
    }
    return total;
}

We can then provide all manner of array-like objects and sum is none-the-wiser, being capable of handling them all. In the following code sample, we separately provide a generic array, a typed array, and an array-like object, and, for each input value, the sum function readily computes the sum of all elements.

const x1 = [ 1, 2, 3 ];
const s1 = sum( x1 );
// returns 6

const x2 = new Int32Array( [ 1, 2, 3 ] );
const s2 = sum( x2 );
// returns 6

const x3 = {
    'length': 3,
    '0': 1,
    '1': 2,
    '2': 3
};
const s3 = sum( x3 );
// returns 6

This is great! So long as downstream consumers make minimal assumptions regarding the existence of prototype methods, preferably avoiding the use of methods entirely, we can create functional APIs capable of operating on any indexed collection.

But wait, what about those scenarios in which we want to use alternative data structures, such that property-value pairs are not so neatly aligned, or we want to leverage deferred computation, or create views on existing array-like objects? How can we handle those use cases?

Motivating use cases

Sparse arrays

Up until this point, we've concerned ourselves with "dense" arrays (i.e., arrays in which all elements can be stored sequentially in a contiguous block of memory). In JavaScript, in addition to dense arrays, we have the concept of "sparse" arrays. The following code sample demonstrates sparse array creation by setting an element located at an index which vastly exceeds the length of the target array.

const x = [];

// Convert `x` into a sparse array:
x[ 10000 ] = 3.14;

// Retrieve the second element:
const v1 = x[ 1 ];
// returns undefined

// Retrieve the last element:
const v10000 = x[ 10000 ];
// returns 3.14

// Retrieve the number of elements:
const len = x.length;
// returns 10001

Suffice it to say that, by not using the Array.prototype.push method and filling in values until element 10000, JavaScript engines responsible for compiling and optimizing your code treat the array as if it were a normal object, which is a reasonable optimization in order to avoid unnecessary memory allocation. Creating a sparse array in this fashion is often referred to as converting an array into "dictionary-mode", where an array is stored in a manner similar to a regular object instance. The above code sample is effectively equivalent to the following code sample where we explicitly define x to be an array-like object containing a single defined value at index 10000.

const x = {
    'length': 10001,
    '10000': 3.14
};

// Retrieve the second element:
const v1 = x[ 1 ];
// returns undefined

// Retrieve the last element:
const v10000 = x[ 10000 ];
// returns 3.14

Creating sparse arrays in this manner is fine for many use cases, but less than optimal in others. For example, in numerical computing, we'd prefer that the "holes" (i.e., undefined values) in our sparse array would be 0, rather than undefined. This way, the sum function we defined above could work on both sparse and dense arrays alike (setting aside, for the moment, any performance considerations).

Deferred computation

Next up, consider the case in which we want to avoid materializing array values until they are actually needed. For example, in the following snippet, we'd like the ability to define an array-like object without any pre-defined values and which supports "lazy" materialization such that values are materialized upon element access.

const x = {
    'length': 3
};

// Materialize the first element:
const v0 = x[ 0 ];
// returns 1

// Materialize the second element:
const v1 = x[ 1 ];
// returns 2

// Materialize the third element:
const v2 = x[ 2 ];
// returns 3

To implement lazy materialization in JavaScript, we could utilize the Iterator protocol; however, iterators are not directly "indexable" in a manner similar to array-like objects, and they don't generally have a length property indicating how many elements they contain. To know when they finish, we need to explicitly check the done property of the iterated value. While we can use the built-in for...of statement to iterate over Iterables, this requires either updating our sum implementation to use for...of, and thus require that all provided array-like objects also be Iterables, or introducing branching logic based on the type of value provided. Neither option is ideal, with both entailing increased complexity, constraints, performance-costs, or, more likely, some combination of the above.

Shared memory views

For our next motivating example, consider the case of creating arbitrary views over the same underlying block of memory. While typed arrays support creating contiguous views (e.g., by providing a shared ArrayBuffer to typed array constructors), situations may arise where we want to define non-contiguous views. In order to avoid unnecessary memory allocation, we'd like the ability to define arbitrary iteration patterns which allow accessing particular elements within an underlying linear data buffer.

In the following snippet, we illustrate the use case of an array-like object containing complex numbers which are stored in memory as interleaved real and imaginary components. To allow accessing and manipulating just the real components within the array, we'd like the ability to create a "view" atop the underlying data buffer which accesses every other element (i.e., just the real components). We could similarly create a "view" for only accessing the imaginary components.

// Define a data buffer of interleaved real and imaginary components:
const buf = [ 1.0, -2.0, 3.0, -4.0, 5.0, -6.0, 7.0, -8.0 ];

// Create a complex number array:
const x = new ComplexArray( buf );

// Retrieve the second element:
const z1 = x[ 1 ];
// returns Complex<3.0, -4.0>

// Create a view which only accesses real components:
const re = x.reals();

// Retrieve the real component of the second complex number:
const r = re[ 1 ];
// returns 3.0

// Mutate the real component:
re[ 1 ] = 10.0;

// Retrieve the second element of the complex number array:
const z2 = x[ 1 ];
// returns Complex<10.0, -4.0>

To implement such views, we'd need three pieces of information: (1) an underlying data buffer, (2) an array stride which defines the number of locations in memory between successive array elements, and (3) an offset which defines the location in memory of the first indexed element. For contiguous arrays, the array stride is unity, and the offset is zero. In the example above, for a real component view, the array stride is two, and the offset is zero; for an imaginary component view, the array stride is also two, but the offset is unity. Ideally, we could define a means for providing generalized access such that array-like objects which provide abstracted element indexing can also be provided to array-agnostic APIs, such as sum above.

Backing data structures

As a final example, consider the case where we'd like to compactly store an ordered sequence of boolean values. While we could use generic arrays (e.g., [true,false,...,true]) or Uint8Array typed arrays for this, doing so would not be the most memory efficient approach. Instead, a more memory efficient data structure would be a bit array comprised of a sequence of integer words in which, for each word of n bits, a 1-bit indicates a value of true and a 0-bit indicates a value of false.

The following code snippet provides a general idea of mapping a sequence of boolean values to bits, along with desired operations for setting and retrieving boolean elements.

const seq = [ true, false, true, ..., false, true, false ];
// bit array:    1      0     1  ...      0     1      0    => 101...010

const x = new BooleanBitArray( seq );

// Retrieve the first element:
const v0 = x[ 0 ];
// returns true

// Retrieve the second element:
const v1 = x[ 1 ];
// returns false

// Retrieve the third element:
const v2 = x[ 2 ];
// returns true

// Set the second element:
x[ 1 ] = true;

// Retrieve the second element:
const v3 = x[ 1 ];
// returns true

In JavaScript, we could attempt to subclass array or typed array built-ins in order to allow setting and getting elements via bracket notation; however, this approach would prove limiting as subclassing alone does not allow intercepting property access (e.g., x[i]), which would be needed in order to map an index to a specific bit. We could try and combine subclassing with Proxy objects, but this would come with a steep performance cost due to property accessor indirection--something which we'll revisit later in this post.

Accessor Protocol

To accommodate the above use cases and more, we'd like to introduce a conceptually simple, but very powerful, new protocol: the accessor protocol for generalized element access and iteration of array-like objects. The protocol doesn't require new syntax or built-ins. The protocol only defines a standard way to get and set element values.

Any array-like object can implement the accessor protocol (also known as the get/set protocol) by following two conventions.

Define a get method. A get method accepts a single argument: an integer value specifying the index of the element to return. Similar to bracket notation for built-in array and typed array objects, the protocol requires that the get method be defined for integer values that are nonnegative and within array bounds. Protocol-compliant implementations may choose to support negative index values, but that behavior should not be considered portable. Similarly, how implementations choose to handle out-of-bounds indices is implementation-dependent; implementations may return undefined, raise an exception, wrap, clamp, or some other behavior. By not placing restrictions on out-of-bounds behavior, the protocol can more readily accommodate a broader set of use cases.
Define a set method. A set method accepts two arguments: the value to set and an integer value specifying the index of the element to replace. Similar to the get method, the protocol requires that the set method be defined for integer indices that are nonnegative and within array bounds. And similarly, protocol-compliant implementations may choose to support negative index values, but that behavior should not be considered portable, and how implementations choose to handle out-of-bounds indices is implementation-dependent.

The following code sample demonstrates an accessor protocol-compliant array-like object.

// Define a data buffer:
const data = [ 1, 2, 3, 4, 5 ];

// Define a minimal array-like object supporting the accessor protocol:
const x = {
    'length': 5,
    'get': ( index ) => data[ index ],
    'set': ( value, index ) => data[ index ] = value
};

// Retrieve the third element:
const v1 = x.get( 2 );
// returns 3

// Set the third element:
x.set( 10, 2 );

// Retrieve the third element:
const v2 = x.get( 2 );
// returns 10

Three things to note about the above code sample.

The above example demonstrates another potential use case—namely, an array-like object which doesn't own the underlying data buffer and, instead, acts as a proxy for element access requests.
The signature for the set method may seem counter-intuitive, as one might expect the arguments to be reversed. The rationale for value being the first argument and index being the second argument is to be consistent with built-in typed array set method conventions, where the first argument is an array from which to copy values and the second argument is optional and specifies the offset at which to begin writing values from the first argument. While one could argue that set(v,i) is not ideal, given the argument order precedent found in built-ins, the protocol follows that precedent in order to avoid confusion.
In contrast to the built-in typed array set method which expects an array (or typed array) for the first argument, the accessor protocol only requires that protocol-compliant implementations support a single element value. Protocol-compliant implementations may choose to support first arguments which are array-like objects and do so in a manner emulating arrays and typed arrays; however, such behavior should not be considered portable.

In short, in order to be accessor protocol-compliant, an array-like object only needs to support single element retrieval and mutation via dedicated get and set methods, respectively.

While built-in typed arrays provide a set method, they are not accessor protocol-compliant, as they lack a dedicated get method, and built-in arrays are also not accessor protocol-compliant, as they lack both a get and a set method. Their lack of compliance is expected and, from the perspective of the protocol, by design in order to distinguish indexed collections from accessor protocol-compliant array-like objects.

Array-like objects implementing the accessor protocol should be expected to pay a small, but likely non-negligible, performance penalty relative to indexed collections using bracket notation for element access. As such, we expect that performance-conscious array-agnostic APIs will maintain two separate code paths: one for indexed collections and one for collections implementing the accessor protocol. Hence, the presence or absence of get and set methods provides a useful heuristic for determining which path takes priority. In general, for indexed collections which are also accessor protocol-compliant, the get and set methods should always take precedent over bracket notation.

The following code sample refactors the sum API defined above to accommodate array-like objects supporting the accessor protocol.

function isAccessorArray( x ) {
    return ( typeof x.get === 'function' && typeof x.set === 'function' );
}

function sum( x ) {
    let total = 0;

     // Handle accessor-protocol compliant collections...
    if ( isAccessorArray( x ) ) {
       for ( let i = 0; i < x.length; i++ ) {
            total += x.get( i );
        }
        return total;
    }
    // Handle indexed collections...
    for ( let i = 0; i < x.length; i++ ) {
        total += x[ i ];
    }
    return total;
}

For array-agnostic APIs which prefer brevity over performance optimization, one can refactor the previous code sample to use a small, reusable helper function which abstracts array element access and allows loop consolidation. A demonstration of this refactoring is shown in the following code sample.

function isAccessorArray( x ) {
    return ( typeof x.get === 'function' && typeof x.set === 'function' );
}

function array2accessor( x ) {
    if ( isAccessorArray( x ) ) {
        return x;
    }
    return {
        'length': x.length,
        'get': ( i ) => x[ i ],
        'set': ( v, i ) => x[ i ] = v
    };
}

function sum( x ) {
    let total = 0;

    x = array2accessor( x );
    for ( let i = 0; i < x.length; i++ ) {
        total += x.get( i );
    }
    return total;
}

As before, we can then provide all manner of array-like objects, including those supporting the accessor protocol, and sum is none-the-wiser, being capable of handling them all. In the following code sample, we separately provide a generic array, a typed array, an array-like object, and a "lazy" array implementing the accessor protocol, and, for each input value, the sum function readily computes the sum of all elements.

const x1 = [ 1, 2, 3 ];
const s1 = sum( x1 );
// returns 6

const x2 = new Int32Array( [ 1, 2, 3 ] );
const s2 = sum( x2 );
// returns 6

const x3 = {
    'length': 3,
    '0': 1,
    '1': 2,
    '2': 3
};
const s3 = sum( x3 );
// returns 6

const x4 = {
    'length': 3,
    'get': ( i ) => i + 1,
    'set': ( v, i ) => x4[ i ] = v
};
const s4 = sum( x4 );
// returns 6

Alternatives

At this point, you may be thinking that the accessor protocol seems useful, but why invent something new? Doesn't JavaScript already have mechanisms for inheriting indexed collection semantics (subclassing built-ins), supporting lazy materialization (iterators), proxying element access requests (Proxy objects), and accessing elements via a method (Array.prototype.at)?

Yes, JavaScript does have built-in mechanisms for supporting, at least partially, the use cases outlined above; however, each approach has limitations, which I'll discuss below.

Subclassing built-ins

In the early days of the web and prior to built-in subclassing support, third-party libraries would commonly add methods directly to the prototypes of built-in global objects in order to expose functionality missing from the JavaScript standard library—a practice which was, and still remains, frowned upon. After standardization of ECMAScript 2015, JavaScript gained support for subclassing built-ins, including arrays and typed arrays. By subclassing built-ins, we can create specialized indexed collections which not only extend built-in behavior, but also retain the semantics of bracket notation for indexed collections. Subclassing can be particularly beneficial when wanting to augment inherited classes with new properties and methods.

The following code sample demonstrates extending the built-in Array class to support in-place element-wise addition.

/**
* Class which subclasses the built-in Array class.
*/
class SpecialArray extends Array {
    /**
    * Performs in-place element-wise addition.
    *
    * @param {ArrayLikeObject} other - input array
    * @throws {RangeError} must have the same number of elements
    * @returns {SpecialArray} the mutated array
    */
    add( other ) {
        if ( other.length !== this.length ) {
            throw new RangeError( 'Must provide an array having the same length.' );
        }
        for ( let i = 0; i < this.length; i++ ) {
            this[ i ] += other[ i ];
        }
        return this;
    }
}

// Create a new SpecialArray instance:
const x = new SpecialArray( 10 );

// Call an inherited method to fill the array:
x.fill( 5 );

// Retrieve the second element:
const v1 = x[ 1 ];
// returns 5

// Create an array to add:
const y = [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ];

// Perform element-wise addition:
x.add( y );

// Retrieve the second element:
const v2 = x[ 1 ];
// returns 7

While powerful, subclassing built-ins has several limitations.

With respect to the use cases discussed above, subclassing built-ins only satisfies the desire for preserving built-in bracket notation semantics. Subclassing does not confer support for lazy materialization, separate backing data structures, or shared memory views.
Subclassing built-ins imposes a greater implementation burden on subclasses. Particularly for more "exotic" array types, such as read-only arrays, subclasses may be forced to override and re-implement parent methods in order to ensure consistent behavior (e.g., returning a collection having a desired instance type).
Subclassing built-ins imposes an ongoing maintenance burden. As the ECMAScript Standard evolves and built-in objects gain additional properties and methods, those properties and methods may need to be overridden and re-implemented in order to preserve desired semantics.
Subclassing built-ins influences downstream user expectations. If a subclass inherits from a Float64Array, users will likely expect that any subclass satisfying an instanceof check supports all inherited methods, some of which may not be possible to support (e.g., for a read-only array, methods supporting mutation). Distinct (i.e., non-coupled) classes which explicitly own the interface contract will likely be better positioned to manager user expectations.
While subclassing built-ins can encourage reuse, object-oriented programming design patterns can more generally lead to code bloat (read: increased bundle sizes in web applications), as the more methods are added or overridden, the less likely any one of those methods is actually used in a given application.

For the reasons listed above, inheriting from built-ins is generally discouraged in favor of composition due to non-negligible performance and security impacts. One of the principle aims of the accessor protocol is to provide the smallest API surface area necessary in order to facilitate generalized array-like object iteration. Subclassing built-ins is unable to fulfill that mandate.

Iterators

Objects implementing the iterator protocol can readily support deferred computation (i.e., "lazy" materialization), but, for several of the use cases outlined above, the iterator protocol has limited applicability. More broadly, relying on the iterator protocol has three limitations.

First, as alluded to earlier in this post, the iterator protocol does not require that objects have a length property, and, in fact, iterators are allowed to be infinite. As a consequence, for operations requiring fixed memory allocation (e.g., as might be the case when needing to materialize values before passing a typed array from JavaScript to C within a Node.js native add-on), the only way to know how much memory to allocate is by first materializing all iterator values. Doing so may require first filling a temporary array before values can be copied to a final destination. This process is likely to be inefficient.

Furthermore, operations involving multiple iterators can quickly become complex. For example, suppose I want to perform element-wise addition for two iterators X and Y (i.e., x0+y0, x1+y1, ..., xn+yn). This works fine if X and Y have the same "length", but what if they have different lengths? Should iteration stop once one of the iterator ends? Or should a fill value, such as zero, be used? Or maybe this is unexpected behavior, and we should raise an exception? Accordingly, generalized downstream APIs accepting iterators may require tailored options to support various edge cases which simply aren't as applicable when working with array-like objects.

We could, of course, define a protocol requiring that iterators have a length property, but that leads us to the next limitation: iterators do not support random access. In order to access the n-th iterated value, one must materialize the previous n-1 values. This is also likely to be inefficient.

Lastly, in general, code paths operating on iterators are significantly slower than equivalent code paths operating on indexed collections. While the accessor protocol does introduce overhead relative to using bracket notation due to explicitly needing to call a method, the overhead is less than the overhead introduced by iterators.

The following code sample defines three functions: one for computing the sum of an indexed collection, one for computing the sum of an array-like object implementing the accessor protocol, and a third for computing the sum of an iterator using JavaScript's built-in for...of syntax.

function indexedSum( x ) {
    let total = 0;
    for ( let i = 0; i < x.length; i++ ) {
        total += x[ i ];
    }
    return total;
}

function accessorSum( x ) {
    let total = 0;
    for ( let i = 0; i < x.length; i++ ) {
        total += x.get( i );
    }
    return total;
}

function iteratorSum( x ) {
    let total = 0;
    for ( const v of x ) {
        total += v;
    }
    return total;
}

To assess the performance of each function, I ran benchmarks on an Apple M1 Pro running MacOS and Node.js v20.9.0. For a set of array lengths ranging from ten elements to one million elements, I repeated benchmarks three times and used the maximum observed rate for subsequent analysis and chart display. The results are provided in the following grouped column chart.

In the chart above, columns along the x-axis are grouped according to input array/iterator length. Accordingly, the first group of columns corresponds to input arrays/iterators having 10 elements, the second group to input array/iterators having 100 elements, and so on and so forth. The y-axis corresponds to normalized rates relative to the performance observed for indexed collections. For example, if the maximum observed rate when summing over an indexed collection was 100 iterations per second and the maximum observed rate when summing over an iterator was 70 iterations per second, the normalized rate is 70/100, or 0.7. Hence, a rate equal to unity indicates an observed rate equal to that of indexed collections. Anything less than unity indicates an observed rate less than that of indexed collections (i.e., summation involving a given input was slower than using built-in bracket notation). Anything greater than unity indicates an observed rate greater than that of indexed collections (i.e., summation involving a given input was faster than using built-in bracket notation).

From the chart, we can observe that, for all array lengths, neither accessor protocol-compliant array-like objects nor iterators matched or exceeded the performance of indexed collections. Array-like objects implementing the accessor protocol were 15% slower than indexed collections, and iterators were 30% slower than indexed collections. These results confirm that the accessor protocol introduces an overhead relative to indexed collections, but not nearly as much as the overhead introduced by iterators.

In short, the accessor protocol is both more flexible and more performant than using iterators.

Computed properties

Another alternative to the accessor protocol is to use defined properties having property accessors. When implementing lazy materialization and proxied element access prior to ECMAScript standardization of Proxy objects and support for Array subclassing, property descriptors were the primary way to emulate the built-in bracket notation of indexed collections.

The following code sample shows an example class returning an array-like object emulating built-in bracket notation by explicitly defining property descriptors for all elements. Each property descriptor defines an accessor property with specialized get and set accessors.

/**
* Class emulating built-in bracket notation for lazy materialization without subclassing.
*/
class LazyArray {
    // Define private instance fields:
    #data; // memoized value cache

    /**
    * Returns a new fixed-length "lazy" array.
    *
    * @param {number} len - number of elements
    * @returns {LazyArray} lazy array instance
    */
    constructor( len ) {
        Object.defineProperty( this, 'length', {
            'configurable': false,
            'enumerable': false,
            'writable': false,
            'value': len
        });
        for ( let i = 0; i < len; i++ ) {
            Object.defineProperty( this, i, {
                'configurable': false,
                'enumerable': true,
                'get': this.#get( i ),
                'set': this.#set( i )
            });
        }
        this.#data = {};
    }

    /**
    * Returns a getter.
    *
    * @private
    * @param {number} index - index
    * @returns {Function} getter
    */
    #get( index ) {
        return get;

        /**
        * Returns an element.
        *
        * @private
        * @returns {*} element
        */
        function get() {
            const v = this.#data[ index ];
            if ( v === void 0 ) {
                // Perform "lazy" materialization:
                this.#data[ index ] = index; // note: toy example
                return index;
            }
            return v;
        }
    }

    /**
    * Returns a setter.
    *
    * @private
    * @param {number} index - index
    * @returns {Function} setter
    */
    #set( index ) {
        return set;

        /**
        * Sets an element value.
        *
        * @private
        * @param {*} value - value to set
        * @returns {boolean} boolean indicating whether a value was set
        */
        function set( value ) {
            this.#data[ index ] = value;
            return true;
        }
    }
}

// Create a new "lazy" array:
const x = new LazyArray( 10 );

// Print the list of elements:
for ( let i = 0; i < x.length; i++ ) {
    console.log( x[ i ] );
}

There are several issues with this approach:

Explicitly defining property descriptors is very expensive. Thus, especially for large arrays, instantiation can become prohibitively slow.
Creating separate accessors for each property requires significantly more memory than the accessor protocol. The latter only needs two methods to serve all elements. The former requires two methods for every element.
Element access is orders of magnitude slower than both built-in bracket notation and the accessor protocol.

In the following grouped column chart, I show benchmark results for computing the sum over an array-like object which emulates built-in bracket notation by using property accessors. The chart extends the previous grouped column chart by including the same column groups as the previous chart and adding a new column to each group corresponding to property accessor performance results. As can be observed, using property accessors is more than one hundred times slower than either indexed collection built-in bracket notation or the accessor protocol.

Proxies

The Proxy object allows you to create a proxy for another object. During its creation, the proxy can be configured to intercept and redefine fundamental object operations, including getting and setting properties. While proxy objects are commonly used for logging property accesses, validation, formatting, or sanitizing inputs, they enable novel and extremely powerful extensions to built-in behavior. One such extension—implementing Python-like indexing in JavaScript—will be the subject of a future post.

The following code sample defines a function for creating proxied array-like objects which intercept the operations for getting and setting property values. The proxy is created by providing two parameters:

target: the original object we want to proxy.
handler: an object defining which operations to intercept and how to redefine those operations.

/**
* Tests whether a string contains only integer values.
*
* @param {string} str - input string
* @returns {boolean} boolean indicating whether a string contains only integer values
*/
function isDigitString( str ) {
    return /^\d+$/.test( str );
}

/**
* Returns a proxied array-like object.
*
* @param {number} len - array length
* @returns {Proxy} proxy object
*/
function lazyArray( len ) {
    const target = {
        'length': len
    };
    return new Proxy( target, {
        'get': ( target, property ) => {
            if ( isDigitString( property ) ) {
                return parseInt( property, 10 ); // note: toy example
            }
            return target[ property ];
        },
        'set': ( target, property, value ) => {
            target[ property ] = value;
            return true;
        }
    });
}

// Create a new "lazy" array:
const x = lazyArray( 10 );

// Print the list of elements:
for ( let i = 0; i < x.length; i++ ) {
    console.log( x[ i ] );
}

While proxy objects avoid many of the issues described above for subclassing, iterators, and property accessors, including random access, instantiation costs, and general complexity, their primary limitation at the time of this blog post is performance.

The following group column chart builds on the previous column charts by adding a new column to each group corresponding to proxy object results. As can be observed, using proxy objects fares no better than the property accessor approach described above. Performance is on par with property accessors and more than one hundred times slower than either indexed collection built-in bracket notation or the accessor protocol.

Using "at" rather than "get"

The 2022 revision of the ECMAScript Standard added an at method to array and typed array prototypes which accepts a single integer argument and returns the element at that index, allowing for both positive and negative integers. Why, then, do we need another method for retrieving an array element as proposed in the accessor protocol? This question seems especially salient given that the protocol's get method only requires support for nonnegative integer arguments, making the get method seem less powerful.

There are a few reasons why the accessor protocol chooses to use get, rather than at.

The name get has symmetry with the name set.
The at method does not have a built-in method equivalent for setting element values. The set method is only present on typed arrays, not generic arrays, and does not support negative target offsets.
The at method does not match built-in bracket notation semantics. When using a negative integer within square brackets, the integer value is serialized to a string before property lookup (i.e., x[-1] is equivalent to x['-1']). Unless negative integer properties are explicitly defined, x[-1] will return undefined. In contrast, x.at(-1) is equivalent to x[x.length-1], which, for non-empty arrays, will return the last array element. Accordingly, the get method of the accessor protocol allows protocol-compliant implementations to match built-in bracket notation semantics exactly.
The accessor protocol does not specify the behavior of out-of-bounds index arguments. In contrast, when an index argument is negative, the at method normalizes a negative index value according to index + x.length. This, however, is not the only reasonable behavior, depending on the use case. For example, an array-like object implementation may want to clamp out-of-bounds index arguments, such that indices less than zero are clamped to zero (i.e., the first index) and indices greater than x.length-1 are clamped to x.length-1 (i.e., the last index). Alternatively, an array-like object implementation may want to wrap out-of-bounds index arguments using modulo arithmetic. Lastly, an array-like object implementation may want to raise an exception when an index is out-of-bounds. In short, the at method prescribes a particular mode of behavior, which may not be appropriate for all use cases.
By only requiring support for nonnegative integer arguments, the accessor protocol allows protocol-compliant implementations to minimize branching and ensure better performance. While convenient, support for negative indices is not necessary for generalized array-like object iteration.
As the EMCAScript Standard does not define a get method for arrays and typed arrays (at least not yet!), the presence or absence of a get method in combination with a set method and length property allows for distinguishing indexed collections from array-like objects implementing the accessor protocol. The combination of at, set, and length would not be sufficient for making such a distinction. This ability is important in order to allow downstream array-like object consumers to implement optimized code paths and ensure optimal performance.

For these reasons, an at method is not a suitable candidate for use in generalized array-like object iteration.

Examples

Now that we've considered the alternatives and established the motivation and need for the accessor protocol, what can we do with it?! Glad you asked! To answer this question, I provide several concrete implementations below.

Complex number arrays

Complex numbers have applications in many scientific domains, including signal processing, fluid dynamics, and quantum mechanics. We can extend the concept of typed arrays to the realm of complex numbers by storing real and imaginary components as interleaved values within a real-valued typed array. In the following code sample, I define a minimal immutable complex number constructor and a complex number array class implementing the accessor protocol.

/**
* Class defining a minimal immutable complex number.
*/
class Complex {
    // Define private instance fields:
    #re; // real component
    #im; // imaginary component

    /**
    * Returns a new complex number instance.
    *
    * @param {number} re - real component
    * @param {number} im - imaginary component
    * @returns {Complex} complex number instance
    */
    constructor( re, im ) {
        this.#re = re;
        this.#im = im;
    }

    /**
    * Returns the real component of a complex number.
    *
    * @returns {number} real component
    */
    get re() {
        return this.#re;
    }

    /**
    * Returns the imaginary component of a complex number.
    *
    * @returns {number} imaginary component
    */
    get im() {
        return this.#im;
    }
}

/**
* Class defining a complex number array implementing the accessor protocol.
*/
class Complex128Array {
    // Define private instance fields:
    #length; // array length
    #data;   // underlying data buffer

    /**
    * Returns a new complex number array instance.
    *
    * @param {number} len - array length
    * @returns {Complex128Array} complex array instance
    */
    constructor( len ) {
        this.#length = len;
        this.#data = new Float64Array( len*2 ); // accommodate interleaved components
    }

    /**
    * Returns the array length.
    *
    * @returns {number} array length
    */
    get length() {
        return this.#length;
    }

    /**
    * Returns an array element.
    *
    * @param {integer} index - element index
    * @returns {(Complex|void)} element value
    */
    get( index ) {
        if ( index < 0 || index >= this.#length ) {
            return;
        }
        const ptr = index * 2; // account for interleaved components
        return new Complex( this.#data[ ptr ], this.#data[ ptr+1 ] );
    }

    /**
    * Sets an array element.
    *
    * @param {Complex} value - value to set
    * @param {integer} index - element index
    * @returns {void}
    */
    set( value, index ) {
        if ( index < 0 || index >= this.#length ) {
            return;
        }
        const ptr = index * 2; // account for interleaved components
        this.#data[ ptr ] = value.re;
        this.#data[ ptr+1 ] = value.im;
    }
}

// Create a new complex number array:
const x = new Complex128Array( 10 );
// returns <Complex128Array>

// Retrieve the second element:
const z1 = x.get( 1 );
// returns <Complex>

const re1 = z1.re;
// returns 0.0

const im1 = z1.im;
// returns 0.0

// Set the second element:
x.set( new Complex( 3.0, 4.0 ), 1 );

// Retrieve the second element:
const z2 = x.get( 1 );
// returns <Complex>

const re2 = z2.re;
// returns 3.0

const im2 = z2.im;
// returns 4.0

If you are interested in a concrete implementation of complex number arrays, see the Complex128Array and Complex64Array packages provided by stdlib. We'll have more to say about these packages in future blog posts.

Sparse arrays

Applications of sparse arrays commonly arise in network theory, numerical analysis, natural language processing, and other areas of science and engineering. When data is "sparse" (i.e., most elements are non-zero), sparse array storage can be particularly advantageous in reducing required memory storage and in accelerating the computation of operations involving only non-zero elements.

In the following code sample, I define a minimal accessor protocol-compliant sparse array class using the dictionary of keys (DOK) format and supporting arbitrary fill values. Support for arbitrary fill values is useful as it extends the concept of sparsity to any array having a majority of elements equal to the same value. For such arrays, we can compress an array to a format which stores a single fill value and only those elements which are not equal to the repeated value. This approach is implemented below.

/**
* Class defining a sparse array implementing the accessor protocol.
*/
class SparseArray {
    // Define private instance fields:
    #length; // array length
    #data;   // dictionary containing array elements
    #fill;   // fill value

    /**
    * Returns a new sparse array instance.
    *
    * @param {number} len - array length
    * @param {*} fill - fill value
    * @returns {SparseArray} sparse array instance
    */
    constructor( len, fill ) {
        this.#length = len;
        this.#data = {};
        this.#fill = fill;
    }

    /**
    * Returns the array length.
    *
    * @returns {number} array length
    */
    get length() {
        return this.#length;
    }

    /**
    * Returns an array element.
    *
    * @param {number} index - element index
    * @returns {*} element value
    */
    get( index ) {
        if ( index < 0 || index >= this.#length ) {
            return;
        }
        const v = this.#data[ index ];
        if ( v === void 0 ) {
            return this.#fill;
        }
        return v;
    }

    /**
    * Sets an array element.
    *
    * @param {*} value - value to set
    * @param {number} index - element index
    * @returns {void}
    */
    set( value, index ) {
        if ( index < 0 || index >= this.#length ) {
            return;
        }
        this.#data[ index ] = value;
    }
}

// Create a new sparse array:
const x = new SparseArray( 10, 0.0 );

// Retrieve the second element:
const v1 = x.get( 1 );
// returns 0.0

// Set the second element:
x.set( 4.0, 1 );

// Retrieve the second element:
const v2 = x.get( 1 );
// returns 4.0

Lazy arrays

While less broadly applicable, situations may arise in which you want an array-like object supporting lazy materialization and random access. For example, suppose each element is the result of an expensive computation, and you want to defer the computation of each element until first accessed.

In the following code sample, I define a class supporting lazy materialization of randomly generated element values. When an element is accessed, a class instance eagerly computes all un-computed element values up to and including the accessed element. Once an element value is computed, the value is memoized and can only be overridden by explicitly setting the element.

/**
* Class defining an array-like object supporting lazy materialization of random values.
*/
class LazyRandomArray {
    // Define private instance fields:
    #data;   // underlying data buffer

    /**
    * Returns a new lazy random array.
    *
    * @returns {LazyRandomArray} new instance
    */
    constructor() {
        this.#data = [];
    }

    /**
    * Materializes array elements.
    *
    * @private
    * @param {number} len - array length
    */
    #materialize( len ) {
        for ( let i = this.#data.length; i < len; i++ ) {
            this.#data.push( Math.random() );
        }
    }

    /**
    * Returns the array length.
    *
    * @returns {number} array length
    */
    get length() {
        return this.#data.length;
    }

    /**
    * Returns an array element.
    *
    * @param {number} index - element index
    * @returns {*} element value
    */
    get( index ) {
        if ( index < 0 ) {
            return;
        }
        if ( index >= this.#data.length ) {
            this.#materialize( index+1 );
        }
        return this.#data[ index ];
    }

    /**
    * Sets an array element.
    *
    * @param {*} value - value to set
    * @param {number} index - element index
    * @returns {void}
    */
    set( value, index ) {
        if ( index < 0 ) {
            return;
        }
        if ( index >= this.#data.length ) {
            // Materialize `index+1` in order to ensure "fast" elements:
            this.#materialize( index+1 );
        }
        this.#data[ index ] = value;
    }
}

// Create a new lazy array:
const x = new LazyRandomArray();

// Retrieve the tenth element:
const v1 = x.get( 9 );
// returns <number>

// Set the tenth element:
x.set( 4.0, 9 );

// Retrieve the tenth element:
const v2 = x.get( 9 );
// returns 4.0

// Return the number of elements in the array:
const len = x.length;
// returns 10

stdlib

While array-like objects implementing the accessor protocol are useful in their own right, they become all the more powerful when combined with functional APIs which are accessor protocol-aware. Fortunately, stdlib treats accessor protocol-compliant objects as first-class citizens, providing support for them throughout its codebase.

For example, the following code sample uses @stdlib/array-put to replace the elements of an accessor protocol-compliant strided array at specified indices.

const put = require( '@stdlib/array-put' );

/**
* Class defining a strided array.
*/
class StridedArray {
    // Define private instance fields:
    #length; // array length
    #data;   // underlying data buffer
    #stride; // step size (i.e., the index increment between successive values)
    #offset; // index of the first indexed value in the data buffer

    /**
    * Returns a new StridedArray instance.
    *
    * @param {integer} N - number of indexed elements
    * @param {ArrayLikeObject} data - underlying data buffer
    * @param {number} stride - step size
    * @param {number} offset - index of the first indexed value in the data buffer
    * @returns {StridedArray} strided array instance
    */
    constructor( N, data, stride, offset ) {
        this.#length = N;
        this.#data = data;
        this.#stride = stride;
        this.#offset = offset;
    }

    /**
    * Returns the array length.
    *
    * @returns {number} array length
    */
    get length() {
        return this.#length;
    }

    /**
    * Returns the element located at a specified index.
    *
    * @param {number} index - element index
    * @returns {(void|*)} element value
    */
    get( index ) {
        return this.#data[ this.#offset + index*this.#stride ];
    }

    /**
    * Sets the value for an element located at a specified index.
    *
    * @param {*} value - value to set
    * @param {number} index - element index
    */
    set( value, index ) {
        this.#data[ this.#offset + index*this.#stride ] = value;
    }
}

// Define a data buffer:
const buf = new Float64Array( [ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 ] );

// Create a strided view over the data buffer:
const x1 = new StridedArray( 4, buf, 2, 1 );

// Retrieve the second element:
const v1 = x1.get( 1 );
// returns 4.0

// Retrieve the fourth element:
const v2 = x1.get( 3 );
// returns 8.0

// Replace the second and fourth elements with new values:
put( x, [ 1, 3 ], [ -v1, -v2 ] );

// Retrieve the second element:
const v3 = x1.get( 1 );
// returns -4.0

// Retrieve the fourth element:
const v4 = x1.get( 3 );
// returns -8.0

In addition to supporting accessor protocol-compliant array-like objects in utilities, linear algebra operations, and other vectorized APIs, stdlib has leveraged the accessor protocol to implement typed arrays supporting data types beyond real-valued numbers. To see this in action, see stdlib's Complex128Array, Complex64Array, and BooleanArray typed array constructors.

In short, the accessor protocol is a powerful abstraction which is not only performant, but can accommodate new use cases with minimal effort.

Conclusion

In this post, we dove deep into techniques for array-like object iteration. Along the way, we discussed the limitations of current approaches and identified opportunities for a lightweight means for element retrieval that can flexibly accommodate a variety of use cases, including strided arrays, arrays supporting deferred computation, shared memory views, and sparse arrays. We then learned about the accessor protocol which provides a straightforward solution for accessing elements in a manner consistent with built-in bracket notation and having minimal performance overhead. With the power and promise of the accessor protocol firmly established, we wrapped up by showcasing a few demos of the accessor protocol in action.

In short, we covered a lot of ground, but I hope you learned a thing or two along the way. In future posts, we'll explore more applications of the accessor protocol, including in the implementation of complex number and boolean typed arrays. We hope that you'll continue to follow along as we share our insights and that you'll join us in our mission to realize a future where JavaScript and the web are preferred environments for numerical and scientific computation. 🚀

If you'd like to view the code covered in this post on GitHub, please visit the source code repository.

How to call Fortran routines from JavaScript with Node.js

Athan — Wed, 24 Jul 2024 02:25:09 +0000

Fortran is a commonly used language for numerical and scientific computation, underpinning many of the higher-level numerical libraries and programming languages in use today. Since Fortran's original development in 1957, researchers and software developers have used Fortran as a primary language for high-performance computation and authored thousands of high-performance programs and libraries for astronomy, climate modeling, computational chemistry, fluid dynamics, simulation, weather prediction, and more.

Rather than attempt to re-implement the entirety of the Fortran ecosystem, programming languages, such as R, MATLAB, and Julia, and numerical libraries, such as NumPy and SciPy, have opted to provide language-specific wrappers around Fortran functionality. Despite significant interest in numerical computing on the web, no one has developed comprehensive JavaScript bindings for Fortran libraries. That is, until now.

In this post, we'll begin laying the groundwork for authoring high-performance Fortran bindings and explore how to call Fortran routines from JavaScript using Node.js. We'll start with a brief introduction to Fortran, followed by writing and compiling a simple Fortran program. We'll then discuss how to use Node-API to link a compiled Fortran routine to the Node.js runtime. And we'll conclude by demonstrating how to use stdlib to simplify the authoring of Node.js bindings.

By the end of this post, you'll have a good understanding of how to call Fortran routines from JavaScript using Node.js.

Prerequisites

Throughout this post, we'll be writing sample programs and performing various steps to compile and run Fortran programs. We'll assume that you have some familiarity with using the terminal, executing commands, and running JavaScript programs. For the most part, terminal commands will assume a Linux-based operating system. Some modifications may be required to successfully run commands and perform compilation steps on Windows.

If you're hoping to follow along, you'll need the following prerequisites:

You'll want to make sure you've installed the latest stable Node.js version. To check whether Node.js is already installed
```
$ node --version
```
where $ is the terminal prompt and node --version is the entered command.
We'll be using npm for installing Node.js dependencies, but you should be able to adapt any installation commands to your preferred JavaScript package manager (e.g., Yarn, pnpm, etc).
In order to generate build files appropriate for your operating system (OS), we'll be using node-gyp, which, in turn, has varying prerequisites depending on your OS, including the availability of Python. For more details, see the node-gyp installation instructions.
In order to compile Fortran programs, you'll need a Fortran compiler. In this post, we'll be using GNU Fortran (GFortran) to compile Fortran code. GFortran is an implementation of the Fortran programming language in the widely used GNU Compiler Collection (GCC), an open-source project maintained under the umbrella of the GNU Project. To check whether GFortran is already installed
```
$ gfortran --version
```
And finally, we'll be using GCC to compile and link C source code. To check whether GCC is already installed
```
$ gcc --version
```

If you don't have one or more of the above installed, you'll want to go ahead and install those now.

Introduction to Fortran

Fortran is a compiled, imperative programming language well-suited to numerical and scientific computation. Known for its high performance, versatility, and ease of use, Fortran is natively parallel and has built-in support for array handling. This makes Fortran a popular choice for scientific computing.

Many fundamental libraries for numerical computation, such as BLAS (basic linear algebra subprograms), LAPACK (linear algebra package), SLATEC, and MINPACK, among many others, are written in Fortran. These libraries serve as the foundation of popular open-source numerical computation libraries, such as NumPy and SciPy, and numerical programming languages, such as R, MATLAB, and Julia.

Given Fortran's widespread usage and decades of development, one could argue that most modern numerical programming languages and libraries are simply fancy wrappers around Fortran routines. Therefore, enabling JavaScript to call Fortran routines not only leverages these high-performance libraries but also positions JavaScript as a viable language for machine learning and other computation-intensive tasks.

Now, let's get started by compiling our first Fortran program!

Compiling our first Fortran program

Recognizing that some readers of this post may not be familiar with Fortran, let's kick things off by writing a "Hello world" program in Fortran for adding two numbers and printing the result. To begin, open up a text editor and create the file add.f90 containing the following code which contains a function definition for adding two integers and a main program which calls that function and prints the result.

! file: add.f90

!>
! Adds two integer values.
!
! @param {integer} x - first input value
! @param {integer} y - second input value
!<
integer function add( x, y )
    ! Define the input parameters:
    integer, intent(in) :: x, y
    ! ..
    ! Compute the sum:
    add = x + y
end function add

!>
! Main execution sequence.
!<
program main
    ! Local variables:
    character(len=999) :: str, tmp
    ! ..
    ! Intrinsic functions:
    intrinsic adjustl, trim
    ! ..
    ! Define a variable for storing the sum:
    integer :: res
    ! ..
    ! Compute the sum:
    res = add( 12, 15 )
    ! ..
    ! Print the results:
    write (str, '(I15)') res
    tmp = adjustl( str )
    print '(A, A)', 'The sum of 12 and 15 is ', trim( tmp )
end program

There are a few things to note in the above program. The first is that, in general, Fortran routines pass arguments by reference. A common practice is to define and pass output variables for storing results—something that we'll revisit later in this post.

Second, a best practice is to specify the intent(xx) of a variable. In the code above, intent(in) indicates that an argument must not be redefined or become undefined during the execution of a subroutine. Similarly, intent(out) indicates that an argument must be defined before the argument is referenced within a subroutine.

Third, in order to print formatted results, we need to perform various string manipulation steps, including writing to character buffers (write), adjusting alignment (adjustl), and trimming results (trim).

For the purposes of getting something working, our program defines a single variable res, which receives the result of passing two number literals to an add function. To test whether the code works, we first need to see if it compiles, and, to do this, we'll use the GNU Fortran (GFortran) compiler, which is part of the GNU Compiler Collection (GCC). While other Fortran compilers exist, such as the Intel Fortran Compiler, LLVM Flang, and LFortran, GFortran is one of the most widely used Fortran compilers, and what we cover in this post should readily translate elsewhere.

In a terminal, navigate to the directory containing add.f90, and execute the following command

$ gfortran add.f90 -o add.out && ./add.out

where add.f90 is the file path of the file to be compiled and add.out is the file path to use for storing a generated executable. If all went according to plan, you should see the following text as output

The sum of 12 and 15 is 27

Defining another Fortran subroutine

In add.f90, we defined a self-contained Fortran program which adds two numbers and prints the result. But what if we want to call Fortran functions and subroutines from another Fortran file or from outside of Fortran, such as from JavaScript running in Node.js?

To see how this is done, let's begin by creating another Fortran file mul.f90, this time containing a subroutine for multiplying two integers and returning an integer result.

! file: mul.f90

!>
! Multiplies two integer values.
!
! @param {integer} x - first input value
! @param {integer} y - second input value
! @param {integer} res - output argument for storing the result
!<
subroutine mul( x, y, res )
    integer, intent(in) :: x, y
    integer, intent(out) :: res
    res = x * y
end subroutine mul

Similar to add, mul takes two input parameters x and y, but this time mul is a subroutine which takes an output parameter res for the storing the result.

If we try compiling mul.f90 as we did with add.f90,

$ gfortran mul.f90 -o mul.out

we'll encounter an error message similar to the following

Undefined symbols for architecture arm64:
  "_main", referenced from:
      <initial-undefines>
ld: symbol(s) not found for architecture arm64
collect2: error: ld returned 1 exit status

In order to successfully generate a standalone executable, Fortran code must have a main program providing an entry point for execution. Without this entry point, a Fortran compiler does not where to begin executing code or where to look to identify the procedures and functions necessary to run a program.

For mul.f90, we're not wanting Fortran to drive execution, and, instead, we're interested in defining an entry point outside of Fortran which will enable a JavaScript runtime to drive execution. This means that we need to figure out a way to establish a bridge between a JavaScript runtime exposing native APIs and Fortran code containing APIs which we want to use. In order to establish such a bridge, we need to disentangle two compiler phases: compilation and linking.

Linking

At a high level, compilation is the process of translating one programming language into another programming language. Often this means taking expressions written in a higher-level language, such as Fortran, and translating them to a lower-level language, such as machine code, in order to create an executable program that a machine can natively understand. The output of compilation is one or more object files, which typically have .o or .obj filename extensions.

Linking is the process of taking one or more object files and combining them into a single executable file. During linking, a "linker" performs several tasks:

symbol resolution: resolving references to functions and variables across different object files.
address binding: assigning final memory addresses to a program's functions and variables.
library inclusion: including code from static or dynamic libraries as required.
executable creation: producing the final executable file that can be run on a target system.

When we ran the GFortran command above

$ gfortran mul.f90 -o mul.out

the compiler attempted to perform both compilation and linking. However, if we're trying to combine compiled Fortran code with a separate library (or a runtime such as Node.js), we need to split compilation and linking into separate steps.

Accordingly, in order to just generate the object file, we can amend the previous command as follows

$ gfortran -c mul.f90

where the -c flag instructs the compiler to compile, but not to link. After running this command from the same directory as mul.f90, you should see a mul.o (or mul.obj) file containing the compiled source code.

Linking Fortran files

To demonstrate linking as a separate phase, create a mul_script.f90 file containing the following code containing a main program which calls the mul function and prints the result.

! file: mul_script.f90

!>
! Main execution sequence.
!<
program main
    ! Local variables:
    character(len=999) :: str, tmp
    ! ..
    ! Intrinsic functions:
    intrinsic adjustl, trim
    ! ..
    ! Define a variable for storing the product:
    integer :: res
    ! ..
    ! Call the `mul` subroutine to compute the product:
    call mul( 4, 5, res )
    ! ..
    ! Print the results:
    write (str, '(I15)') res
    tmp = adjustl( str )
    print '(A, A)', 'The product of 4 and 5 is ', trim( tmp )
end program

We can then perform the same compilation step as we did for mul.f90.

$ gfortran -c mul_script.f90

At this point, we should have two object files: mul.o and mul_script.o (or mul.obj and mul_script.obj, respectively). To link them into a single executable, we can run the following command in which we define the path of the output executable and pass in the paths of the object files we wish to link.

$ gfortran -o mul_script.out mul.o mul_script.o

Once linked, we can test that everything works by running the generated executable.

$ ./mul_script.out

If all went according to plan, you should see the following text as output

The product of 4 and 5 is 20

At this point, we've successfully compiled and linked together separate Fortran source files, and we can now turn our attention to linking compiled Fortran to non-Fortran code.

Linking Fortran and C

A common scenario in numerical computing is exposing numerical computing libraries written in Fortran as C functions. C also happens to be the programming language used by Node.js to expose APIs for building native add-ons (i.e., extensions to the Node.js runtime). Accordingly, if we can figure out how to link Fortran to C, we'll be well on our way to creating a Node.js native add-on capable of calling Fortran routines.

Writing Fortran wrappers

While the mul function defined above can be used in conjunction with other Fortran files, we cannot simply call mul from C as we do in Fortran because Fortran expects arguments to be passed by reference rather than by value. It's also worth mentioning that, because Fortran functions can only return scalar values and not, e.g., pointers to arrays, general best practice is to expose Fortran functions as subroutines, which are the equivalent of C functions returning void and which allow passing pointers for storing output return values.

While mul is already a subroutine, if we wanted to expose add to C, we'd first need to wrap add as a subroutine in a manner similar to the following code snippet containing the subroutine wrapper addsub which forwards input arguments to add and assigns the result to an output argument res.

!>
! Wraps `add` as a subroutine.
!
! @param {integer} x - first input value
! @param {integer} y - second input value
! @param {integer} res - output argument for storing the result
!<
subroutine addsub( x, y, res )
    implicit none
    ! ..
    ! External functions:
    interface
        integer function add( x, y )
            integer :: x, y
        end function add
    end interface
    ! ..
    integer, intent(in) :: x, y
    integer, intent(out) :: res
    ! ..
    res = add( x, y )
    return
end subroutine addsub

Defining function prototypes in C

With those preliminaries out of the way, to help the C compiler reason about functions defined elsewhere (e.g., in a Fortran library or in other source files), we need to define function prototypes for any functions we plan to use before we use them. For our use case of calling a single Fortran routine, we can create a mul_fortran.h header file containing a single function declaration for the mul subroutine.

// file: mul_fortran.h

#ifndef MUL_FORTRAN_H
#define MUL_FORTRAN_H

#ifdef __cplusplus
extern "C" {
#endif

void mul( const int *x, const int *y, int *res );

#ifdef __cplusplus
}
#endif

#endif

One thing to note is that, in the above header file, we prevent name mangling by using extern "C". This is common practice in order to facilitate interoperation of C and C++, and preventing name mangling helps avoid compiler errors if we decide to use mul in C++ in the future.

Calling Fortran routines from C

Next, similar to how we created a Fortran program for calling a Fortran function defined in a separate file, we can create a main.c file containing a main function which calls mul and prints the result.

// file: main.c

#include "mul_fortran.h"
#include <stdio.h>

int main( void ) {
    int x = 4;
    int y = 5;
    int res;

    // Compute the product, passing arguments by reference:
    mul( &x, &y, &res );

    printf( "The product of %d and %d is %d\n", x, y, res );
    return 0;
}

Compiling C and Fortran

To compile our C program, we can run the following command

$ gcc -I mul_fortran.h -c main.c

where -I mul_fortran.h instructs the compiler to use the function declarations defined in the header file we created above.

Before linking main.o and mul.o, we first need to recompile mul.f90, making sure to instruct GFortran to not modify function names by appending underscores during compilation. This ensures that the name used in our C code matches the exported symbol from compiled Fortran. One should be careful, however, as non-mangled names may conflict with existing symbols defined in C.

To prevent GFortran from appending underscores to symbol names, we set the -fno-underscoring compiler option when calling GFortran.

$ gfortran -fno-underscoring -c mul.f90

Now that we've compiled our source files, it's time to generate an executable!

$ gcc -o main.out main.o mul.o

Depending on your operating system, if the previous command errors, you may need to modify the previous command to

$ gcc -o main.out main.o mul.o -lgfortran

where -lgfortran instructs GCC to link to the standard Fortran libraries. And finally, to test that everything works, we run the executable by entering the following command

$ ./main.out

If successful, you should see the following text as output

The product of 4 and 5 is 20

Phew! If you're new to Fortran and C, congratulations on making it this far!

Now that we've successfully managed to link Fortran and C code, we can turn our attention to using Node.js native add-ons to call Fortran routines from JavaScript.

Node-API

Node-API is an API for building Node.js native add-ons (i.e., extensions to the Node.js JavaScript runtime). There's a long history of add-on evolution and development in Node.js, of which I'll spare you the details. The real benefit of Node-API is in providing a stable Application Binary Interface (ABI), which insulates add-ons from changes in the underlying JavaScript engine (namely, V8) and which allows modules compiled for one version of Node.js to run on later versions of Node.js without recompilation. In short, Node-API provides the glue code, in the form of C APIs, necessary for us to extend Node.js capabilities with C/C++ code written and compiled independently of Node.js itself.

In order to access Node-API APIs, we need to do two things:

Include the <node_api.h> header in our C files.
Compile C source files using Node-API APIs with node-gyp, a build system based on Google's GYP, a meta-build system for generating other build systems.

So without further ado...

Creating an add-on file

Let's start by creating an addon.c file which will serve as an entry point for our native add-on. In this file, we'll define two functions—addon and Init—and register a Node-API module which exports a function in a manner similar to how we'd export a function if writing a module in vanilla JavaScript.

// file: addon.c

#include <node_api.h>
#include <assert.h>

/**
* Receives JavaScript callback invocation data.
*
* @param env    environment under which the function is invoked
* @param info   callback data
* @return       Node-API value
*/
static napi_value addon( napi_env env, napi_callback_info info ) {

    // NOTE: we'll add code here later in this post

    return NULL;
}

/**
* Defines the Node.js module "exports" object for the native add-on.
*
* @param env      environment under which the function is invoked
* @param exports  exports object
* @return         Node-API value
*/
static napi_value Init( napi_env env, napi_value exports ) {
    napi_value fcn;

    // Export the add-on function as a "default" export:
    napi_status status = napi_create_function( env, "exports", NAPI_AUTO_LENGTH, addon, NULL, &fcn );

    // Verify that we successfully wrapped the `addon` function as a JavaScript function object:
    assert( status == napi_ok );

    // Return the JavaScript function object to allow registering with the JavaScript runtime:
    return fcn;
}

/**
* Register a Node-API module which exports a function.
*/
NAPI_MODULE( NODE_GYP_MODULE_NAME, Init )

The addon.c file is comprised of three parts:

addon: this function receives JavaScript invocation data. If we assume foo() is a JavaScript function exposed by a native add-on, env is the environment in which the JavaScript code runs and info is an opaque object which can be used to retrieve function arguments and other contextual data when foo is invoked.
Init: similar to how module.exports defines the APIs a Node.js module exposes to other Node.js modules, this function defines the "exports" object and initializes exported values. In this context, initialization typically means wrapping C APIs as JavaScript objects so that a JavaScript engine can pass data back and forth between JavaScript and native code.
NAPI_MODULE: this is a macro exposed by Node-API for registering a Node-API module with the Node.js JavaScript runtime.

At this point, we're starting to accumulate a number of moving parts: Fortran source files, GFortran, C source files, GCC, Node-API, and a heretofore mentioned, but not explained, node-gyp.

As may be observed in the diagram above, a key component which we have yet to cover, but which is necessary to allow building a Node.js native add-on in a manner that is portable across platforms, is the binding.gyp file. It's this file and node-gyp that we'll dive into next.

node-gyp

node-gyp is a build system based on Google's GYP, which, in turn, is a meta-build system for generating other build systems. The key idea behind GYP is the generation of build files, such as Makefiles, Ninja build files, Visual Studio projects, and XCode projects, which are tailored to the platform on which a project is being compiled. Once GYP scaffolds a project in a manner tailored to the host platform, GYP can then perform build steps which replicate as closely as possible the way that one would have set up a native build of the project were one writing the project build system from scratch. node-gyp subsequently extends GYP by providing the configuration and tooling specific to developing Node.js native add-ons.

Configuring how to build an add-on

In order to describe the configuration necessary to build a Node.js native add-on, one needs to provide a binding.gyp file. This file is written in a JSON-like format and is placed at the root of a JavaScript package alongside a package's package.json file. GYP configuration files can be awkward to write, and, unfortunately, GYP has long been abandoned by the Google team responsible for its creation. Adding insult to injury, good documentation for authoring GYP files can be hard to come by, as the GYP documentation is incomplete and finding real-world examples doing exactly what you are wanting to do can be a time-consuming task, especially when authoring binding.gyp files requiring specialized configuration (e.g., as might be needed when compiling CUDA, OpenCL, or Fortran).

Nevertheless, persist we shall! Fortunately, writing a minimal binding.gyp file capable of supporting Fortran compilation is within reach. Start by creating a binding.gyp file specifying various configuration parameters, including build targets, source files, compiler flags, and rules for how to process files having a specific file type.

# file: binding.gyp

# A `.gyp` file for building a Node.js native add-on.
#
# [1]: https://gyp.gsrc.io/docs/InputFormatReference.md
# [2]: https://gyp.gsrc.io/docs/UserDocumentation.md
{
  # Define variables to be used throughout the configuration for all targets:
  'variables': {
    # Set variables based on the host OS:
    'conditions': [
      [
        'OS=="win"',
        {
          # Define the object file suffix on Windows:
          'obj': 'obj',
        },
        {
          # Define the object file suffix for other operating systems (e.g., Linux and MacOS):
          'obj': 'o',
        }
      ],
    ],
  },

  # Define compilation targets:
  'targets': [
    # Define a target to generate an add-on:
    {
      # The target name should match the add-on export name (see addon.c above):
      'target_name': 'addon',

      # List of source files:
      'sources': [
        # Relative paths should be relative to this configuration file...
        './addon.c',
        './mul.f90',
      ],

      # List directories which contain relevant headers to include during compilation:
      'include_dirs': [
        # Relative paths should be relative to this configuration file...
        './',
      ],

      # Define settings which should be applied when a target's object files are used as linker input:
      'link_settings': {
        # Define linker flags for libraries against which to link (e.g., '-lm', '-lblas', etc):
        'libraries': [],

        # Define directories in which to find libraries to link to (e.g., '/usr/lib'):
        'library_dirs': []
      },

      # Define custom build actions for particular source files:
      'rules': [
        {
          # Define a rule name:
          'rule_name': 'compile_fortran',

          # Define the filename extension for which this rule should apply:
          'extension': 'f90',

          # Set a flag specifying whether to process generated output as sources for subsequent steps:
          'process_outputs_as_sources': 1,

          # Define the pathnames to be used as inputs when performing processing:
          'inputs': [
            # Full path of the current input:
            '<(RULE_INPUT_PATH)',
          ],

          # Define the outputs produced during processing:
          'outputs': [
            # Store an output object file in a directory for placing intermediate results (only accessible within a single target):
            '<(INTERMEDIATE_DIR)/<(RULE_INPUT_ROOT).<(obj)',
          ],

          # Define the command-line invocation:
          'action': [
            'gfortran',
            '-fno-underscoring',
            '-c',
            '<@(_inputs)',
            '-o',
            '<@(_outputs)',
          ],
        },
      ],
    },
  ],
}

A few comments:

GYP configuration files support variables, conditionals, and expressions. In the configuration file above, <(RULE_INPUT_PATH), <(INTERMEDIATE_DIR), and <(RULE_INPUT_ROOT) are predefined variables provided by the GYP generator module. Variables such as <@(_inputs) and <@(_outputs) represent variable expansions and correspond to variables which should be expanded in list contexts.
While GYP attempts to automate and abstract away the generation of build files tailored to the operating system on which to compile, this doesn't absolve us from needing to consider platform variability. For example, the configuration file above includes a conditional for resolving an appropriate object file filename extension based on the target operating system.
Configuration files can quickly become complex depending on operating system variability, including the availability of specialized compilers, such as GFortran, and the need for bespoke rules for varying input file types.

Building an add-on

Now that we have a GYP configuration file, it's time to install node-gyp. In your terminal, run

$ npm install --no-save node-gyp

The node-gyp executable will subsequently be available in the ./node_modules/.bin directory. To generate the appropriate project build files for the current platform, run the following command

$ ./node_modules/.bin/node-gyp configure

This will generate a ./build directory containing platform-specific build files. To build the native add-on, we can run

$ ./node_modules/.bin/node-gyp build

which will generate an addon.node file in a ./build/Release sub-folder. To remove generated files, run

$ ./node_modules/.bin/node-gyp clean

As we continue to iterate on our addon.c file, we'll want to perform the clean-configure-build sequence each time we make changes. Accordingly, we can consolidate the above steps into a single command

$ ./node_modules/.bin/node-gyp clean && \
  ./node_modules/.bin/node-gyp configure && \
  ./node_modules/.bin/node-gyp build

Calling a Fortran routine from JavaScript

At this point, we've got almost all of the core building blocks for calling a Fortran routine from JavaScript. We're only missing two things:

Logic in addon.c which calls the Fortran routine.
A JavaScript file which invokes the function exposed by our native add-on.

Updating the add-on file

To start, let's revisit our addon.c file. In this file, we need to make four changes:

Retrieve provided arguments.
Convert from JavaScript objects to native C types.
Add logic to call our Fortran routine mul.
Return a result as a JavaScript object.

Luckily, we already have experience with (3) when we wrote main.c and linked against our compiled Fortran routine. As in main.c, we want to include the mul_fortran.h header, which we can do by making the following change in addon.c

// file: addon.c

+ #include "mul_fortran.h"
#include <node_api.h>
#include <assert.h>

Next, we'll want to modify the addon function in addon.c to include logic for calling the mul Fortran routine. In the snippet below, we copy the invocation logic used in main.c into the implementation of the addon function.

/**
* Receives JavaScript callback invocation data.
*
* @param env    environment under which the function is invoked
* @param info   callback data
* @return       Node-API value
*/
static napi_value addon( napi_env env, napi_callback_info info ) {

    // ...

    // Call the Fortran routine:
    int res;
    mul( &x, &y, &res );

    // ...

    return NULL;
}

Now on to argument munging. Fortunately, Node-API provides several APIs for converting from JavaScript objects to native C data types. In particular, we're interested in converting JavaScript numbers to C integers, which is demonstrated in the following code snippet which defines the number of expected input arguments, retrieves those arguments from provided callback info using napi_get_cb_info, and converts JavaScript objects to native C data types using napi_get_value_int32.

/**
* Receives JavaScript callback invocation data.
*
* @param env    environment under which the function is invoked
* @param info   callback data
* @return       Node-API value
*/
static napi_value addon( napi_env env, napi_callback_info info ) {
    napi_status status;

    // Define the expected number of input arguments:
    size_t argc = 2;

    // Retrieve the input arguments from the callback info:
    napi_value argv[ 2 ];
    status = napi_get_cb_info( env, info, &argc, argv, NULL, NULL );
    assert( status == napi_ok );

    // Convert each argument to a signed 32-bit integer:
    int x;
    status = napi_get_value_int32( env, argv[ 0 ], &x );
    assert( status == napi_ok );

    int y;
    status = napi_get_value_int32( env, argv[ 1 ], &y );
    assert( status == napi_ok );

    // Call the Fortran routine:
    int res;
    mul( &x, &y, &res );

    // ...

    return NULL;
}

And finally, we need to convert the integer result to a JavaScript object for use within JavaScript, which is demonstrated in the following code snippet which adds logic for converting a C signed 32-bit integer to an opaque object representing a JavaScript number using napi_create_int32.

/**
* Receives JavaScript callback invocation data.
*
* @param env    environment under which the function is invoked
* @param info   callback data
* @return       Node-API value
*/
static napi_value addon( napi_env env, napi_callback_info info ) {
    napi_status status;

    // Define the expected number of input arguments:
    size_t argc = 2;

    // Retrieve the input arguments from the callback info:
    napi_value argv[ 2 ];
    status = napi_get_cb_info( env, info, &argc, argv, NULL, NULL );
    assert( status == napi_ok );

    // Convert each argument to a signed 32-bit integer:
    int x;
    status = napi_get_value_int32( env, argv[ 0 ], &x );
    assert( status == napi_ok );

    int y;
    status = napi_get_value_int32( env, argv[ 1 ], &y );
    assert( status == napi_ok );

    // Call the Fortran routine:
    int res;
    mul( &x, &y, &res );

    // Convert the result to a JavaScript object:
    napi_value out;
    status = napi_create_int32( env, res, &out );
    assert( status == napi_ok );

    return out;
}

Putting it all together, we have the following addon.c file which defines the entirety of our native add-on bindings.


// file: addon.c

#include "mul_fortran.h"
#include <node_api.h>
#include <assert.h>

/**
* Receives JavaScript callback invocation data.
*
* @param env    environment under which the function is invoked
* @param info   callback data
* @return       Node-API value
*/
static napi_value addon( napi_env env, napi_callback_info info ) {
    napi_status status;

    // Define the expected number of input arguments:
    size_t argc = 2;

    // Retrieve the input arguments from the callback info:
    napi_value argv[ 2 ];
    status = napi_get_cb_info( env, info, &argc, argv, NULL, NULL );
    assert( status == napi_ok );

    // Convert each argument to a signed 32-bit integer:
    int x;
    status = napi_get_value_int32( env, argv[ 0 ], &x );
    assert( status == napi_ok );

    int y;
    status = napi_get_value_int32( env, argv[ 1 ], &y );
    assert( status == napi_ok );

    // Call the Fortran routine:
    int res;
    mul( &x, &y, &res );

    // Convert the result to a JavaScript object:
    napi_value out;
    status = napi_create_int32( env, res, &out );
    assert( status == napi_ok );

    return out;
}

/**
* Defines the Node.js module "exports" object for the native add-on.
*
* @param env      environment under which the function is invoked
* @param exports  exports object
* @return         Node-API value
*/
static napi_value Init( napi_env env, napi_value exports ) {
    napi_value fcn;

    // Export the add-on function as a "default" export:
    napi_status status = napi_create_function( env, "exports", NAPI_AUTO_LENGTH, addon, NULL, &fcn );

    // Verify that we successfully wrapped the `addon` function as a JavaScript function object:
    assert( status == napi_ok );

    // Return the JavaScript function object to allow registering with the JavaScript runtime:
    return fcn;
}

/**
* Register a Node-API module which exports a function.
*/
NAPI_MODULE( NODE_GYP_MODULE_NAME, Init )

To confirm that our Node.js add-on still compiles, we can re-run our build sequence defined above.

$ ./node_modules/.bin/node-gyp clean && \
  ./node_modules/.bin/node-gyp configure && \
  ./node_modules/.bin/node-gyp build

Creating a JavaScript file importing the native add-on

We're here! The moment that we've been waiting for! Time to create a JavaScript file which loads our Node.js native add-on and calls its public API. 🥁

Thankfully, loading a native add-on is just like loading any other JavaScript module. To see this in action, let's create a mul.js file which imports the native add-on module, calls the function exposed by the add-on, and prints the result.

// file: mul.js

// Import the native add-on module:
const addon = require( './build/Release/addon.node' );

// Compute the product of two integers:
const res = addon( 5, 10 );
console.log( 'The product of %d and %d is %d', 5, 10, res );

To test whether everything works as expected, we can run the script by passing the script's file path to the Node.js executable.

$ node ./mul.js

If all went according to plan, you should see the following text as output

The product of 5 and 10 is 50

That's it! We did it. 😅

Barring any platform quirks or dreaded compiler errors, we successfully called a Fortran routine from JavaScript. 🙌

Simplifying add-on authoring with stdlib

Depending on API complexity, authoring Node.js native add-ons can be verbose and error prone. This verbosity largely stems from the need for argument validation logic and status checks. For example, when handling typed arrays, one needs to perform multiple steps, such as verifying that an input argument is a typed array, verifying that an input argument is a typed array of the correct type, resolving the length of a typed array, converting a JavaScript object representing a typed array to a C pointer pointing to the start of the underlying typed array memory, and, for applications involving strided arrays, ensuring that typed array properties are consistent with other input arguments, such as strides and offsets.

While some validation logic can be performed in JavaScript or omitted entirely, a general best practice is to include such logic in order to ensure data integrity when calling APIs outside of Node-API APIs and to avoid hard-to-track down bugs leading to segmentation faults and buffer overflows. And furthermore, best practice requires that, after each invocation of a Node-API function, one must check napi_status return values to ensure that the JavaScript engine was able to successfully perform the requested operation. As a consequence, lines of code add up, and you find yourself writing the same logic over and over.

Macros for module registration and data type conversion

To simplify add-on authoring, stdlib provides several utilities, both functional APIs and macros, which abstract away common boilerplate. For example, we can refactor the addon.c file defined above to use stdlib's napi macros for retrieving input arguments, handling conversion to and from native C data types, and initializing and registering an exported function with Node.js.

// file: addon2.c

#include "mul_fortran.h"
#include "stdlib/napi/create_int32.h"
#include "stdlib/napi/argv_int32.h"
#include "stdlib/napi/argv.h"
#include "stdlib/napi/export.h"
#include <node_api.h>

static napi_value addon( napi_env env, napi_callback_info info ) {
    STDLIB_NAPI_ARGV( env, info, argv, argc, 2 ); // retrieve function arguments
    STDLIB_NAPI_ARGV_INT32( env, x, argv, 0 );    // convert to C data type
    STDLIB_NAPI_ARGV_INT32( env, y, argv, 1 );    // convert to C data type
    int res;
    mul( &x, &y, &res );
    STDLIB_NAPI_CREATE_INT32( env, res, out );    // convert to JavaScript object
    return out;
}

STDLIB_NAPI_MODULE_EXPORT_FCN( addon )

Specialized macros for common function signatures

The use case explored in this post—namely, calling a C/Fortran function which operates on and returns scalar values—is something that we do quite often in stdlib, especially for testing native C APIs and sharing test logic across JavaScript and C implementations. Accordingly, stdlib provides several more macro abstractions which abstract away all argument retrieval, argument validation, and module registration logic for certain input/output data type combinations.

// file: addon3.c

#include "mul_fortran.h"
#include "stdlib/math/base/napi/binary.h"

static int multiply( x, y ) {
    int res;
    mul( &x, &y, &res );
    return res;
}

STDLIB_MATH_BASE_NAPI_MODULE_II_I( multiply )

Two comments regarding the code above:

STDLIB_MATH_BASE_NAPI_MODULE_II_I is a macro for registering a Node-API module for an exported function accepting two signed 32-bit integer input arguments and returning a signed 32-bit integer output value. This signature is encoded in the macro name as II_I.
We need to wrap the Fortran routine in a C function, as the module registration macro assumes that a registered II_I function expects arguments to be passed by value, not by reference, and returns a scalar value.

Learning from real-world examples in stdlib

For more details on how we author Node-API native add-ons and leverage macros and various utilities for simplifying the add-on creation process, the best place to start is by browsing stdlib source code. For the examples explored in this post, we've brushed aside some of the complexity in ensuring cross-platform configuration portability (looking at you Windows!) and in specifying compiler options for optimizing compiled code. For those interested in learning more, you'll find many more examples throughout the codebase, and, if you have questions, don't be afraid to stop by and say hi! 👋

Conclusion

In this post, we explored several aspects when authoring Node.js native add-ons, with a particular eye toward being able to call Fortran routines from JavaScript. This effort involved compilation and linking, writing C interfaces, module registration, and build configuration. Along the way, we relied on a variety of tools for generating build artifacts, including Fortran and C compilers, Node-API, and node-gyp. We touched on best practices and potential pitfalls, and we observed how stdlib can make authoring Node.js native add-ons much easier.

All in all, it was a lot, with several moving parts and complex toolchains. But our exploration was well worth the effort. By leveraging Fortran's high-performance capabilities within Node.js, you can significantly enhance and accelerate your numerical and scientific computing tasks. With Node.js native add-ons, you can bridge the gap between modern web technologies and established scientific computing practices, providing a powerful toolset for you and others and opening the door to new and more powerful Node.js applications.

In future posts, we'll explore more complex use cases, including the ability to leverage hardware-optimized routines for linear algebra and machine learning. There's still a lot to learn and more ground to cover. We hope that you'll continue to follow along as we share our insights and that you'll join us in our mission to realize a future where JavaScript and the web are preferred environments for numerical and scientific computation. 🚀

If you'd like to view the code covered in this post on GitHub, please visit the source code repository.