GitHub (Edge)

Configure GitHub as a Source in your Pipeline

Perform the following steps to configure your GitHub Source:

1. Click Pipelines in the Navigation Bar.

2. Click + Create Pipeline in the Pipelines List View.

3. On the Select Source Type page, select GitHub.

4. On the Select Destination Type page, select the type of Destination you want to use.

5. On the Select Pipeline Type page, click Edge, and then click Continue.

   This page appears only if the selected Destination type is supported in Edge and your Team has an existing GitHub Pipeline with the same Destination type. Otherwise, you can proceed to create an Edge Pipeline.

6. In the Configure Source screen, specify the following:

   

   • Source Name: A unique name for your Source, not exceeding 255 characters. For example, GitHub Source.

   • In the Connect to your GitHub account section, select the Authentication Method that you want to use for authenticating Hevo’s connection to your GitHub account. Default value: GitHub OAuth.

     • GitHub OAuth: Connect to your GitHub account using an authorization flow. Click Authenticate With GitHub, and then perform the following steps:

       1. If you are not already logged in to your GitHub account, log in when prompted. Otherwise, proceed to step 2.

       2. Review the permissions requested by Hevo. To replicate the data of an organization, click Grant next to the organization in the Organization access section. If you are not an owner of the organization, click Request to ask an organization owner to approve the access.

       3. Click Authorize to authorize Hevo to access your GitHub data.

       You are redirected to the Configure Source screen, where the username of the authorized GitHub account is displayed. To use a different GitHub account, click Reauthorize.

     • Personal Access Tokens: Connect to your GitHub account using one or more personal access tokens. Use this method if you want to replicate data from a self-hosted GitHub Enterprise Server installation.

       

       • Personal Access Tokens: The personal access token that you created in your GitHub account. To add a token, click + Add Token, paste the token in the Personal Access Token field, and then click Save. You can add up to 10 tokens, preferably from different GitHub accounts and of the same type. Hevo distributes its requests across the tokens to avoid rate limiting. Each classic token must have the repo and read:org scopes, and the read:project scope to replicate the Projects objects.

       • GitHub Enterprise Server: If enabled, Hevo connects to your self-hosted GitHub Enterprise Server installation instead of GitHub.com. Default value: Disabled. If you enable this option, specify the following:

         • Host: The hostname and API path of your GitHub Enterprise Server, without http:// or https://. For example, github.mycompany.com/api/v3.

     • GitHub App: Connect to your GitHub account using a GitHub App installed on your account or organization. GitHub Apps have higher rate limits and use fine-grained permissions.

       

       • App Client ID: The Client ID of the GitHub App that you obtained from your GitHub account.

       • Private Key (.pem): The private key that you generated for the GitHub App. Click the attachment icon, and then select the .pem file that GitHub downloaded when you generated the key.

   • In the Sync Settings section, specify the following. If you select the GitHub OAuth authentication method, this section is displayed after you authenticate your GitHub account.

     

     • Sync Mode: Select one of the following options:

       • Sync All Repositories: Syncs data from all the repositories that Hevo can access, including the repositories that you create later. For the GitHub App method, these are the repositories that the GitHub App installation can access. This option is selected by default.

       • Sync Specific Repositories: Syncs data only from the repositories that you select. If you select this option, in the Select Repositories section, select the repositories whose data you want to sync.

         

     • Default Branch Only: If enabled, Hevo syncs commits only from the default branch of each repository, for example, main or master. This option also applies to the Branch Commit Relation object. Disable this option to sync commits from all branches. Default value: Enabled.

     • Webhook Secret (Optional): A secret that Hevo uses to verify the signature of the webhook events that GitHub sends. If you specify a secret, specify the same value while creating the webhook in your GitHub repository. Hevo then rejects the events that have a missing or invalid signature. If you leave this field blank, Hevo accepts all webhook events without verifying their signature.

7. Click Test & Continue to test the connection to your GitHub Source.

   When you click this button, Hevo makes an API call to list the repositories that the authenticated account can access. For the GitHub App method, Hevo lists the repositories accessible to the App installation instead, and verifies that the App is installed on exactly one account or organization. For the Personal Access Tokens method, Hevo checks each token separately. If these checks pass, the connection test is marked as successful. You can then proceed to set up your Destination.

   Note: The connection test fails if:

   • A personal access token is invalid or has expired, does not have the repo and read:org scopes, or is not authorized for your organization’s SAML SSO. Hevo lists the issue for each token.

   • The Host that you specified for your GitHub Enterprise Server is incorrect.

   • The GitHub App is not installed on any account or organization, or is installed on more than one.

Data Replication

Hevo replicates data for all the objects selected on the Configure Objects page during Pipeline creation. By default, all supported objects and their available fields are selected. However, you can modify this selection while creating or editing the Pipeline.

Selecting a parent object automatically includes all its associated child objects for replication. However, you can select or deselect the child objects individually.

Hevo ingests the following types of data from your Source objects:

• Historical Data: The first run of the Pipeline ingests all available historical data for the selected objects and loads it into the Destination.

• Incremental Data: Once the historical load is complete, new and updated records for objects are ingested as per the sync frequency.

For the following objects and their child objects, Hevo ingests only the incremental data in subsequent Pipeline runs:

• Commit: Hevo ingests the commits created since the last Pipeline run, based on the committer date. If the Default Branch Only option is disabled, Hevo ingests the new commits of each branch in the same way.

• Issue: Hevo ingests the issues and issue comments updated since the last Pipeline run, based on the updated_at field.

• Pull Request: Hevo ingests the pull requests and review comments updated since the last Pipeline run, based on the updated_at field.

• Workflow Run: Hevo ingests the workflow runs created since the last Pipeline run, based on the created_at field.

For all other objects, Hevo ingests the entire data during each Pipeline run.

GitHub enforces rate limits on the number of requests that can be made to its API in a given period of time. Requests authenticated with a personal access token or OAuth are limited to 5,000 requests per hour per user. Requests made by a GitHub App installation start at 5,000 requests per hour and scale with the number of repositories and users, up to 12,500 requests per hour, or 15,000 requests per hour for GitHub Apps owned by a GitHub Enterprise Cloud organization. If these limits are exceeded, a rate limit exception occurs. To understand how Hevo handles such scenarios, read Handling Rate Limit Exceptions.

Note: You can create a Pipeline with this Source only using the Merge load mode. The Append mode is not supported for this Source.

Schema and Primary Keys

Hevo uses the following schema to upload the records to the Destination. For a detailed view of the objects, fields, and relationships, click the ERD.

Click to view ERD

Data Model

The following is the list of tables (objects) that are created at the Destination when you run the Pipeline: