How APIs/Plugins Are Run

For most sites, plugins take up the majority of the build time. So what’s really happening when APIs are called?

Note: this section only explains how gatsby-node plugins are run. Not browser or ssr plugins

Early in the build

Early in the bootstrap phase, we load all the configured plugins (and internal plugins) for the site. These are saved into redux under the flattenedPlugins namespace. Each plugin in redux contains the following fields:

• resolve: absolute path to the plugin’s directory
• id: String concatenation of ‘Plugin ’ and the name of the plugin. E.g Plugin query-runner
• name: The name of the plugin. E.g query-runner
• version: The version as per the package.json. Or if it is a site plugin, one is generated from the file’s hash
• pluginOptions: Plugin options as specified in gatsby-config.js
• nodeAPIs: A list of node APIs that this plugin implements. E.g [ 'sourceNodes', ...]
• browserAPIs: List of browser APIs that this plugin implements
• ssrAPIs: List of SSR APIs that this plugin implements

In addition, we also create a lookup from api to the plugins that implement it and save this to redux as api-to-plugins. This is implemented in load-plugins/validate.js

apiRunInstance

Some API calls can take a while to finish. So every time an API is run, we create an object called apiRunInstance to track it. It contains the following notable fields:

• id: Unique identifier generated based on type of API
• api: The API we’re running. E.g onCreateNode
• args: Any arguments passed to api-runner-node. E.g a node object
• pluginSource: optional name of the plugin that initiated the original call
• resolve: promise resolve callback to be called when the API has finished running
• startTime: time that the API run was started
• span: opentracing span for tracing builds
• traceId: optional args.traceId provided if API will result in further API calls (see below)

We immediately place this object into an apisRunningById Map, where we track its execution.

Running each plugin

Next, we filter all flattenedPlugins down to those that implement the API we’re trying to run. For each plugin, we require its gatsby-node.js and call its exported API function. E.g if API was sourceNodes, it would result in a call to gatsbyNode['sourceNodes'](...apiCallargs).

Injected arguments

API implementations are passed a variety of useful actions and other interesting functions/objects. These arguments are created each time a plugin is run for an API, which allows us to rebind actions with default information.

All actions take 3 arguments:

1. The core information required by the action. E.g for createNode, we must pass a node
2. The plugin that is calling this action. E.g createNode uses this to assign the owner of the new node
3. An object with misc action options:
   • traceId: See below
   • parentSpan: opentracing span (see tracing docs)

Passing the plugin and action options on every single action call would be extremely painful for plugin/site authors. Since we know the plugin, traceId and parentSpan when we’re running our API, we can rebind injected actions so these arguments are already provided. This is done in the doubleBind step.

Waiting for all plugins to run

Each plugin is run inside a map-series promise, which allows them to be executed concurrently. Once all plugins have finished running, we remove them from apisRunningById and fire a API_RUNNING_QUEUE_EMPTY event. This in turn, results in any dirty pages being recreated, as well as their queries. Finally, the results are returned.

Using traceID to await downstream API calls

The majority of API calls result in one or more implementing plugins being called. We then wait for them all to complete, and return. But some plugins (e.g sourceNodes) result in calls to actions that themselves call APIs. We need some way of tracing whether an API call originated from another API call, so that we can wait on all child calls to complete. The mechanism for this is the traceId.

1. The traceID is passed as an argument to the original API runner. E.g

   apiRunner(`sourceNodes`, {
     traceId: `initial-sourceNodes`,
     waitForCascadingActions: true,
     parentSpan: parentSpan,
   })

2. We keep track of the number of API calls with this traceId in the apisRunningByTraceId Map. On this first invocation, it will be set to 1.

3. Using the action rebinding mentioned above, the traceId is passed through to all action calls via the actionOptions object.

4. After reducing the Action, a global event is emitted which includes the action information

5. For the CREATE_NODE and CREATE_PAGE events, we need to call the onCreateNode and onCreatePage APIs respectively. The plugin-runner takes care of this. It also passes on the traceId from the Action back into the API call.

6. We’re back in api-runner-node.js and can tie this new API call back to its original. So we increment the value of apisRunningByTraceId for this traceId.

7. Now, whenever an API finishes running (when all its implementing plugins have finished), we decrement apisRunningByTraceId[traceId]. If the original API call included the waitForCascadingActions option, then we wait until apisRunningByTraceId[traceId] == 0 before resolving.