Forem: nohehf

Introducing Goctopus: open-source, state-of-the-art GraphQL endpoint discovery & fingerprinting tool.

nohehf — Thu, 10 Aug 2023 16:18:17 +0000

Read this article more comfortably on Escape Blog_

In the fast-evolving domain of APIs, GraphQL has emerged as a powerful, data-oriented language. As its adoption soars, so does the need for robust tools to discover and fingerprint these APIs. Enter Goctopus, a Golang-based solution we developed at Escape to provide comprehensive, fast, and interoperable endpoint discovery and fingerprinting for GraphQL APIs. Engineered to overcome the limitations of existing tools, Goctopus extends its arms into realms of subdomain enumeration, route brute-force, authentication detection, schema introspection, and more, all while maintaining optimum speed and resource management. In this article, we dive into why we created Goctopus, what it does, and how it's pushing the boundaries of API security.

TL;DR: We built an open-source GraphQL endpoint discovery & fingerprinting tool: check out Goctopus on Github.

Unmasking APIs: The Importance of API Discovery & Fingerprinting

In the expansive realm of APIs, two practices stand out in their ability to shed light on the intricacies of your online presence: API Discovery and Fingerprinting.

API Discovery is the process of finding all the APIs an organization has in its ecosystem. The aim here is to unveil the totality of the digital surface that can be interacted with. In the era of microservices and distributed systems, discovering all your API endpoints is crucial for both maintaining service interoperability and ensuring security.

API Fingerprinting, on the other hand, takes this a step further. Once the endpoints are discovered, they're scanned to find specific information about them: technologies used, configuration, authentication details...

Endpoint discovery & fingerprinting are crucial tools in offensive and defensive security to:

Detecting Attack Surface: By understanding what endpoints exist and how they operate, you get an accurate view of your potential attack surface. This gives you the upper hand in securing your system, as you can preemptively locate and address vulnerabilities.
Internet-wide Research: If you're a security researcher or an organization keen on understanding the landscape of what's exposed on the internet, fingerprinting can give you a macro view. It helps to gather data about technologies, configurations, and security practices being used across various domains.
Classifying endpoints: Being able to determine which technologies and configurations are used on APIs is usually the first step in any security process (offensive or defensive).

For REST APIs there are a myriad of tools to perform those tasks. However, for GraphQL this is not the same story, as you will see in the next paragraph.

Identifying the Void: Why We Needed Goctopus

Regarding GraphQL APIs, we built Grafinder a while ago, and it has been serving us well for a few years. However, when we decided to implement our API Catalog, we needed a more robust, faster, all-in-one, and more interoperable solution.

Grafinder had some shortcomings that were hard to ignore. It was slow, hard to use programmatically, often missed inputs, lacked options, and its Python basis wasn't our preferred choice for this specific task. Moreover, we found ourselves needing a tool that could offer more comprehensive and efficient discovery and fingerprinting capabilities:

Broad and efficient discovery capabilities: (Subdomain enumeration, brute-force, script analysis).
Precise, fast and complete fingerprinting of GraphQL APIs: (Technology, authentication, schema, field suggestion attack).
Easy to use and enjoyable CLI.
Complete Package for programmatic usage.
Service interoperability: communicate with external services via webhooks.
Portability: run easily anywhere at any scale.
Versatile inputs: take a variety of inputs, domains, URLs, and IP addresses.
Speed: scan thousands of domains in minutes.
Scale: run on larger machines or scale horizontally.
Network resources: send a minimum amount of necessary requests and maximize throughput.

So, with those constraints in mind, we realized we needed a solution that could go beyond what was already there. A more holistic tool. A faster tool. A...well, better tool. We needed Goctopus.

Unleashing the Beast: Meet Goctopus

‌
Born out of the needs we've just discussed, Goctopus is our answer to the comprehensive, fast, and interoperable tool we need. Created with Golang for its superior speed and efficiency in networking, this octopus has its arms reaching everywhere. Let's delve into its various facets:

Discovery:

Subdomain Enumeration: Goctopus uses DNS records APIs via subfinder to enumerate subdomains.
Bruteforce: We built a custom list of popular GraphQL routes so that Goctopus has the best odds of finding endpoints on every domain it tests.
Script Analysis (Work-In-Progress): With this in the pipeline, Goctopus will soon be able to sniff out hidden GraphQL endpoints from Javascript & HTML files across the web.

Fingerprinting:

Authentication: It quickly identifies which endpoints require authentication and which are publicly exposed.
GraphQL Schema/Introspection: Goctopus checks if the schema is available or not.
Field Suggestions for Bruteforcing: If the schema is unavailable, this clever cephalopod can even figure out if the schema can be brute-forced using field suggestions attacks.
Engine Fingerprinting (Work-In-Progress): Once this feature is added, Goctopus will be capable of understanding the tech stack behind an endpoint.

Dx & usage:

CLI: Goctopus is easily usable as a CLI for manual operations.
Go package: It can be used as a Golang package programmatically to integrate into other tools.
Webhook: With this feature, Goctopus can forward its results via a webhook so that other services can retrieve them.
Variety of inputs: The tool accepts a wide variety of inputs: domains, URLs, and IP addresses.

Performance:

Speed: Designed for speed, Goctopus can scan thousands of domains in minutes.
Scale: This tool has no fear of the deep; it's built to scale on larger machines or spread horizontally across a cluster.
Network resources: Goctopus is mindful of its requests, only sending what's necessary.
Rate limiting: It also ensures not sending too many requests to a single endpoint concurrently to avoid rate limiting.

In a nutshell, Goctopus is what we envisioned when we realized the limitations of existing tools. It's comprehensive, fast, flexible, and extensible.

Quick Hands-On: Meet Goctopus

Interested in trying out Goctopus? Here's how to get started.

Installation: You have two options here. If Go is installed on your machine, just run go install -v github.com/Escape-Technologies/goctopus/cmd/goctopus@latest in your terminal. Prefer Docker? No problem, use docker run --rm -it escapetech/goctopus:latest <options> to run the latest version of Goctopus in a docker container.

Usage Example: Once installed, take Goctopus for a spin with this simple command goctopus -a example.com. This will start the discovery process on the specified domain with all features enabled (-a\ flag stands for "all").

For instance, running Goctopus on rickandmortyapi.com will result in the following:

‌

goctopus -a rickandmortyapi.com
                  _
  __ _  ___   ___| |_ ___  _ __  _   _ ___
 / _` |/ _ \ / __| __/ _ \| '_ \| | | / __|
| (_| | (_) | (__| || (_) | |_) | |_| \__ \
 \__, |\___/ \___|\__\___/| .__/ \__,_|___/ v0.0.14
 |___/                    |_|
[INF] Enumerating subdomains for 'rickandmortyapi.com'
[INF] Found 5 subdomains for 'rickandmortyapi.com' in 15 seconds 276 milliseconds
INFO[0016] Done fingerprinting rickandmortyapi.com
INFO[0016] Found: {"authenticated":false,"domain":"rickandmortyapi.com","schema_status":"OPEN","source":"rickandmortyapi.com","url":"https://rickandmortyapi.com/graphql"}
INFO[0016] Done. Found 1 graphql endpoints

‌

Visit our GitHub repo for detailed documentation and the latest release notes.

The Future of Goctopus: Contributing and Roadmap

We're actively looking for ways to enhance Goctopus, and we welcome contributions. Feel free to raise issues or PRs on our GitHub repo. We're excited to see where the community can take this project.

Looking ahead, we've got big plans. Improved subdomain enumeration, more intelligent endpoint discovery, engine fingerprinting - the roadmap for Goctopus is packed with exciting updates.

Goctopus at Escape: Providing Top-Notch Security for Our Users

Here at Escape, we're using Goctopus to provide enhanced security for our users.

API Catalog: We're utilizing Goctopus to provide users with detailed information about their attack surface. By continuously scanning for new APIs and updating our records, we can ensure an up-to-date vision and alert our users whenever they expose an unprotected API.

Suggestions: Goctopus is also integrated into our application creation process to automatically suggest our user's endpoints to secure to make our platform as smooth to use as possible.

‌

Kraken: We built an internal tool around Goctopus called Kraken to scale Goctopus to another level. It consists of an API, a database, and dozens of Goctopus instances managed by a Kubernetes cluster that allowed us to scan Millions of domains to gather data on +150k GraphQL endpoints. Stay tuned for an upcoming article on the topic!

‌

We invite you to join us in exploring what Goctopus can do. Try it out, contribute, and together, let's push the boundaries of API security! And don't forget to star the Goctopus repo on GitHub.

And if you want maximum security for your GraphQL APIs try out Escape security platform for free!

Handling multiple tokens, with a modern solidity architecture via Diamonds & ERC1155.

nohehf — Wed, 04 May 2022 18:58:48 +0000

Abstract:

The diamond pattern is a pretty recent standard that offers a solution to a lot of solidity issues:

Single address for multiple contracts
You can update your functions after contract deployment
No 24kb size limit
... but it also has some downsides or let's say compromises. One of those compromises is that you can't have multiple functions with the same footprint inside one diamond. This can make things complicated when you want to have several tokens under the same diamond (for example an ERC721 NFT contract with an ERC20 token contract). To know more about the Diamonds standard check: https://github.com/mudgen/awesome-diamonds I came across this pattern because I needed some kind of proxy architecture to scale up Parcels contracts (Parcels is a game & company I'm co-founding: https://twitter.com/parcelsgame). ## The issue Let me demonstrate what I mentioned above. This is not possible:

//FacetA.sol
contract FacetA {
    function my_function(uint256 number) external {
        //... Some logic
    }
}

//FacetB.sol
contract FacetB {
    function my_function(uint256 number) external {
        //... Some other logic, with the same footprint
    }
}

Because all facets functions will be exposed under one single contract address (the diamond address), it couldn't determine which one to use on a diamond.my_function(number) call.

For this example you could simply change footprints, like calling the functions respectively functionA and functionB, but you often need a presice footprint in order to implement standards & interfaces. This can for example happend with erc20 and erc721 that both have the same balanceOf(address _owner) function.

However, this is possible:

//FacetA.sol
contract FacetA {
    function function(uint256 number) external {
        //...
    }
}

//FacetB.sol
contract FacetB {
    function function(uint256 number, string str) external {
        //...
    }
}

Because both functions now have a different footprint, because of the arguments, even under a common name. For those familiar with object-oriented programming, this is pretty similar to how you can declare multiple constructors in most languages.

This means that you cannot have multiple facets implementing the same interface.
But this goes even further as different standards can still clash partly, like ERC20 and ERC721, meaning you cannot implement both at the same time inside a common diamond.

The ERC1155 standard

Fortunately, it happens to exist a standard that is designed to handle multiple token on one contract: ERC1155.
Basically ERC1155 is storing balances behind two parameters: id and address. It is this id that allows us to have multiple tokens at the same time on this contract.
These tokens can be fungible (if you allow minting more than one), or non-fungible (if you only allow one mint per id).
In order to distiguish the different token types you can split the ids in ranges:
Let's say we want an app where you have one tokens: $GLD (that would usually implement an ERC20 fungible token), and 1000 unique miners (that would usually implement an ERC721 non-fungible token). To implement these two tokens under our ERC1155 contract, we need to split the ids: One for the $GLD token and 1000 for the miners NFTs. What we can simply do here is assigning the first id (0) to $GLD, and set ids 1 - 1001 to the miners.
More generally what you would do is define à "base constant" for each token type, so here you would have something like:

uint256 constant GLD_ID = 0;
uint256 constant MINER_BASE_ID = 1;

So that you can access to the miner n°x with balance(MINER_BASE_ID + x).

If you need multiple wide ranges (like two types of nfts) you can set the base id by shifting bits:

uint256 constant GLD_ID = 0;
uint256 constant MINER_BASE_ID = 1;
uint256 constant OTHERNFT_BASE_ID = 1 <<< 128;

For more information on the ERC1155 standard please refer to: https://eips.ethereum.org/EIPS/eip-1155

Architecture

To implement the ERC1155 I advice solidstate-solidity LINK that is compatible with the diamond standard, or adapting OpenZepplin's ones rather than re-writing your own from scratch which could add vulnerabilities.

So, we now have a clear way of creating multiple tokens with our ERC1155 contract, but having all the functions in one single contract for every type of token would lead to messy code, and worse, could even be too much.
To address that we will separate each token logic in it's own Facet contract (see diamond pattern to understand facets).
But we will probably want to call funtions from one contract to another (for example you could want to access the $GLD balance to upgrade a miner). While this is possible by leaving the logic in separated facets, the best option is to move all the logic in Libraries, which simplifies calls. This also allows to only leave external getters / setters functions in the Facets, making the architecture even cleaner, as external exposed logic will be separated form internal logic (respectivly Facets & Libraries).

A simple token ($GLD) + NFT (Miner) architecture would be something like this:

Facets only expose external functions: ERC1155Facet for standard ERC1155 functions, TokenFacet & NFTFacet for custom funtions (like getGLDBalance(address of)).
Libraries LibToken & LibNFT handle all the logic (that will be used as internal and external) so that it can be reused across facets & other libraries.
LibStorage contains all the data that has to be stored which is not handled by ERC1155, see AppStorage / DiamondStorage pattern.
LibERC1155Internal is a copy of the functions defined in solidstate's ERC1155Internal.sol contract so that we can call internal functions across facets & libraries (like _mint, etc...). You just have to add events from the IERC1155Internal.sol interface and remove virtual keywords from functions (as library functions cannot / would not be overwritten). See https://gist.github.com/nohehf/3a1116e47d932bb9477bbc5332e61a9a .

This architecture makes sharing the logic across different parts of the application seamless while keeping concerns in separate codebases.

Snippets examples:

So for our $GLD / Miner example we would have the folowing structure (based on the hardhat-diamond-3 starter):

contracts
    ├── Diamond.sol
    ├── facets
    │   ├── DiamondCutFacet.sol
    │   ├── DiamondLoupeFacet.sol
    │   ├── ERC1155Facet.sol 🔺
    │   ├── GLDFacet.sol 🔺
    │   ├── MinerFacet.sol 🔺
    │   └── OwnershipFacet.sol
    ├── interfaces
    │   ├── IDiamondCut.sol
    │   ├── IDiamondLoupe.sol
    │   ├── IERC165.sol
    │   └── IERC173.sol
    ├── libraries
    │   ├── LibDiamond.sol
    │   ├── LibERC1155Internal.sol 🔺
    │   ├── LibGLD.sol 🔺
    │   ├── LibMiner.sol 🔺
    │   └── LibStorage.sol 🔺
    └── upgradeInitializers
        └── DiamondInit.sol

Only files marked with 🔺 are custom, the other ones are provided by the starter repo

ERC1155Facet.sol:

// SPDX-License-Identifier: UNLICENCED
pragma solidity ^0.8.9;

import {ERC1155} from "@solidstate/contracts/token/ERC1155/ERC1155.sol";

contract ERC1155Facet is ERC1155 {}

GLDFacet.sol:

// SPDX-License-Identifier: UNLICENCED
pragma solidity ^0.8.9;
import "../libraries/LibGLD.sol";
import {LibStorage, AppStorage, ArtefactType} from "../libraries/LibStorage.sol";
import {Modifiers} from "../libraries/LibStorage.sol";

contract ArtefactFacet is Modifiers {
    // ----- GETTERS -----
    function getMyGLDBalance(address addr) external view returns (uint256) {
        return LibGLD._getBalance(msg.sender);
    }

    // ...

    // ----- SETTERS -----
    // ...

}

LibERC1155Internal.sol (see https://gist.github.com/nohehf/3a1116e47d932bb9477bbc5332e61a9a)

LibGLD.sol:

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.9;

import {LibStorage, AppStorage, ArtefactType} from "./LibStorage.sol";
import "@solidstate/contracts/token/ERC1155/base/ERC1155BaseStorage.sol";
// import {ERC1155Facet} from "../facets/ERC1155Facet.sol";
import "../facets/ERC1155Facet.sol";
import "./LibERC1155Internal.sol";

// Handles all the $GLD token logic
library LibGLD {
    //CONSTANTS
    uint256 constant GLD_ID = 0;

    //STORAGE GETTERS:
    // common storage
    function s() internal pure returns (AppStorage storage) {
        return LibStorage.diamondStorage();
    }

    //erc1155 storage (NOTE: you should prefer calling LibERC1155, but it can be usefull)
    function s1155() internal pure returns (ERC1155BaseStorage.Layout storage) {
        return ERC1155BaseStorage.layout();
    }


    //GLD LOGIC
    function _getBalance(address addr) internal view returns (uint256) {
        return LibERC1155Internal._balanceOf(addr, GLD_ID);
    }

    function _mint(address to, uint256 amount) internal {
        LibERC1155Internal._mint(to, GLD_ID, amount, "");
    }

    // ...
}

Note that I only made examples for The miner would be pretty similar to GLD

Conclusion

Besides fixing the footprint colision problem between multiple tokens facets under the same diamond, this solitions provides a scalable, easy to use and test solution for multi-token Dapps. Calling the internal functions without having to rely on the deployed address of a contract is very handfull (and cheaper). Once established this pattern allowed me to run things smoothly and easly implement new features to each of my tokens, while keeping clean code & directories.

Possible improvements:

-> solidstate-solidity could move all the logic to Libs so we won't have to copy-paste everything.
-> Diamonds standard should remove the ERC165 implementation on the DiamondLoupeFacet, which we currently have to remove to add an ERC1155 facet (which is IMO a bit problematic).
-> Starter repo for ERC1155 & Diamonds.

I'm also discussing with solidstate & diamonds creators to improve their docs, or even
make a real framework with solidstate-solidity & diamonds along with step-by-step tutorial and docs, supporting natively this kinds of architectures.

Thanks for reading, and please ask me on twitter if you want more details / want to contribute: www.twitter.com/nohehf .