![](\"/img/2022-05-03-add-webhook.png\")
Entity Providers are a more scalable and robust alternative to Entity Processors. The Backstage team introduced Entity Providers to solve problems that big deployments of Backstage were experiencing with the ingestion pipeline. If you take a look at their documentation on The Life of an Entity, it illustrates how your new ingestion pipelines should be structured.
\nBefore the Entity Providers came into the picture, the preexisting processors essentially took on the roles of both the entity providers and the processors described in the documentation. I bring this up as it may provide clarity to anyone confused by some of the overlapping functionality of the current processors and entity providers in the Backstage plugins; some of those processors have not yet been updated to adapt to the new proposed ingestion pipeline.
\nGetting back to why Entity Providers were introduced, there are some common issues with ingestion processors:
\nEntity providers eliminate these problems by giving developers complete control over the execution of ingestion:
\nSince entity providers do not have an implicit queue, you’ll need to specify what drives each entity provider. You could use a task scheduler and specify the frequency of entity provider executions. Ideally, your entity provider would respond to changes in the original data source and remain idle all other times. Developers can accomplish this with webhooks and streaming.
\nIn this tutorial, we'll walk you through the steps of adding GitHubOrgEntityProvider to your catalog and then show you how you can configure Github Webhooks to trigger mutations to your Backstage database.
Assuming you already have your own instance of Backstage, let's install the necessary packages for adding the Github Org Entity Provider:
\nyarn workspace backend add @backstage/plugin-catalog-backend-module-github @backstage/integrationThen you can import the provider to your catalog builder in packages/backend/src/plugins/catalog.ts. Be sure to replace the orgUrl with your own:
- import { CatalogBuilder } from '@backstage/plugin-catalog-backend';\n+ import { CatalogBuilder, EntityProvider } from '@backstage/plugin-catalog-backend';\n...\n+ import { ScmIntegrations, DefaultGithubCredentialsProvider } from '@backstage/integration';\n+ import { GitHubOrgEntityProvider } from '@backstage/plugin-catalog-backend-module-github';\n\nexport default async function createPlugin(\n env: PluginEnvironment,\n): Promise<Router> {\n const builder = await CatalogBuilder.create(env);\n builder.addProcessor(new ScaffolderEntitiesProcessor());\n\n+ const integrations = ScmIntegrations.fromConfig(env.config);\n+\n+ const githubCredentialsProvider = DefaultGithubCredentialsProvider.fromIntegrations(integrations);\n+\n+ const gitProvider = GitHubOrgEntityProvider.fromConfig(env.config, {\n+ id: \"github-org-entity-provider\",\n+ orgUrl: \"https://github.com/my-organization\", // 🚨 REPLACE\n+ logger: env.logger,\n+ githubCredentialsProvider\n+ });\n+\n+ builder.addEntityProvider(gitProvider as EntityProvider);\n\n const { processingEngine, router } = await builder.build();\n await processingEngine.start();\n\n return router;\n}If you try to run your Backstage app, you should notice nothing has changed in your catalog. As mentioned earlier, developers must provide the mechanism that will drive the entity provider. For the sake of this example, let's add the GitHubOrgEntityProvider's read() at the end of our catalog builder:
await processingEngine.start();\n+ await gitProvider.read();\n\n return router;\n}When you restart your Backstage, you should see the members and teams of your organization. If you do not, you should check the permissions of your Github authentication. Whether you're using a personal access token or a Github App, you need to make sure it has permissions for read:user and read:org.
Calling read() towards the end of the catalog building process will only update your database once during deployment so now we're going to configure a webhook to trigger the updates.
Let's start by creating your webhook on Github. From your Github organization's settings page, click Webhooks in the side bar and then Add webhook.
You can get a Payload URL from smee.io - this is a service that proxies payloads from your webhook for local development.
Set the content type to application/json and let's specify the events we want by selecting Let me select individual events. The GitHubOrgEntityProvider adds users and teams to your catalog so the webhook events we want to receive from Github are Orgnaization and Teams.
\n\nFor your actual deployment, you'll want to modify the Payload URL from the smee URL to the URL of your live Backstage app.
\n
Once the webhook is added, you should follow the instructions displayed on smee.io to install their client and use http://localhost:7007/api/catalog/github/webhook as the target URL.
Next, update your catalog builder so that it runs read() when a webhook event is posted to /github/webhook:
await processingEngine.start();\n- await gitProvider.read();\n+ router.post(\"/github/webhook\", async (req, _res) => {\n+ const event = req.headers[\"x-github-event\"];\n+ if (event == \"membership\" || event == \"organization) {\n+ await gitProvider.read();\n+ }\n+ })\n\n return router;\n}If you run the smee client and your Backstage app, you'll see that the entity provider will update the database only when a webhook event is posted by Github.
\nIf you have concerns of the possibily of a webhook being missed, you might want to consider using the Backstage task scheduler to run read() once a day for that extra assurance.
In this tutorial we quickly went over the steps of adding an entity provider to your catalog and using smee to proxy webhook events to your local environment, but this is just the beginning!
\nIf you look at the implementation of GitHubOrgEntityProvider, the read() function queries data directly from Github and runs a full mutation. In its current state, depending on the size of your organization, your read() function might end up triggering way too frequently - resulting in too many Github requests and performing a full mutation of your database each time.
When you create your own custom entity provider, you will want to create a function that applies a delta mutation just from the data received from the webhook events. You can read more about the mutation types here.