Bulk Migrating GitHub Issues Across Repositories with Zenhub and Python

Managing GitHub issues efficiently is crucial for mature products, large teams, large codebases, efficient collaboration, project tracking, and streamlined workflow. The ArcGIS Native Maps SDK for Qt team faced a challenge where we had multiple Git repositories, one dedicated to issue tracking and another housing the codebase and we wanted to migrate to a single repository. In this blog post, I’ll share the process of how we achieved a bulk transfer of issues across multiple repositories into a single repository. I’ll share the approaches we considered to achieve this, along with the solution we ultimately chose using Zenhub and a custom python script that was best for our organizational needs. 

The Challenge 

Working with issues across multiple repositories can be inefficient for Developers, Product Owners, and Product Managers, due to context switching required to find relevant information. Our team found it was causing difficulties in areas such as tracking progress, linking issues, prioritizing tasks, and impacted on overall collaboration and communication, and we therefore wanted to improve our processes. 

To address these challenges, we needed to consolidate 1000+ issues and the code base into a single repository, creating a centralized hub for both issue management and development. By moving all issues into one repository, we decided to establish a unified source of truth, ensuring that all project discussions, bug reports, pull requests, and feature requests were directly tied to the codebase.  

To facilitate this transition and enhance our workflow further, we decided to leverage Zenhub, a tool which integrates seamlessly with GitHub and streamlines our project management processes. Several engineering teams within Esri and in the developer community rely on this tool for efficient project management. 

Introduction to Zenhub

Zenhub is a project management tool that natively integrates with GitHub’s user interface, enabling teams to track issues, manage tasks, and streamline workflows using Kanban boards, roadmaps, and burndown charts. Its Pipeline feature provides a visual representation of the workflow with customizable stages like “To Do”, “In Progress”, “Review”, “Done” etc. allowing teams to organize issues and pull requests based on status. Each pipeline can be tailored to specific development stages, helping teams track progress and prioritize work.

Comparison between Zenhub and GitHub Projects

In addition to Pipelines, Zenhub offers advanced capabilities that GitHub Projects lacks, such as sprints for Agile planning, estimations for effort tracking, epics for grouping related tasks, workflows for process automation and team alignment and releases for version planning. Its automation features reduce manual updates, while built-in velocity tracking, reporting, and dependency management provide data-driven insights for improved productivity. These features make Zenhub an ideal solution for our development teams at Esri, enabling efficient project management without leaving GitHub. 

When utilizing GitHub’s user interface, the process of moving issues from the source to the destination repository is limited to a single transfer at a time, which proves impractical for transferring a large number of issues efficiently. Additionally, following the transfer, issues accumulate in the initial pipeline of the destination repository without preserving the original issue pipelines. In contrast, by leveraging Zenhub’s Developer API, it becomes possible to conduct bulk transfers in a single operation while ensuring the continuity of the pipelines associated with the transferred issues.

Migration Use Case

To demonstrate how we migrated all open issues from Source to Destination, I’ll walk you through a straightforward example to demonstrate the key points, challenges and successes of the approach taken.  

To begin the migration process, ensure that any issue metadata such as labels, milestones, sprints, releases, author, assignees etc. are captured and accurate. Ensuring this up front will ensure you don’t have to go back in issue by issue to repopulate the metadata. In the following sections, I’ll share how we could achieve the issue migration in 3 ways, elucidating the advantages and drawbacks of each. While we ultimately opted for the custom Python solution using Zenhub, I have outlined the merits and drawbacks of alternative approaches, recognizing their relevance for differing objectives.

1st: Using GitHub Web Browser/UI 

This is a user-friendly interface that simplifies the process of transferring issues between repositories. Users can access the issue they want to transfer, click on the “Transfer issue” option, and select the target repository from a dropdown menu. The interface allows for the easy retention of essential metadata, such as labels and assignees, ensuring that the context of the issue remains intact. 

The benefits of this approach are that it works well when there are a small number of issues to migrate, and it has a simple interface that is easy to use. However, it does require manual process, and is impractival for when there are a large number of issues to transfer. For our team there is also the complication that this process doesn’t maintain the Zenhub pipeline assignment. Instead, when issues are transferred in this manner, they accumulate in the initial pipeline of the destination repository, resulting in missing pipeline data and requiring manual updating in Zenhub.

2nd: Using GitHub CLI – Command Line Interface 

Using GitHub CLI for bulk GitHub issue transfer offers distinct advantages over utilizing the GitHub UI outlined above. It provides a more streamlined and efficient process for transferring a large number of issues in one go between repositories. The following example workflow and script demonstrates how to transfer issues from one repository to another using the GitHub CLI. 

1. Install GitHub CLI: Begin by installing the GitHub CLI on your system if you haven’t already. You can download it from the official GitHub CLI repository. 

2. Authentication: After installation, authenticate the GitHub CLI with your GitHub account by running the gh auth login command and following the prompts. 

3. Clone Repositories: Clone both the source and destination repositories to your local machine using the git clone command. 

4. Transfer Issues: Use the gh issue list command to list all the issues in the source repository. Identify the issues you want to transfer and note down their issue numbers. 

5. Create a Script: Write a script that utilizes the GitHub CLI to transfer the selected issues from the source repository to the destination repository. You can use commands like gh issue transfer to achieve this. 

6. Use Filter: Use various filters to transfer certain set of issues matching the filter criteria. I will explain the filters in detail later. 

7. Execute the Script: Run the script in your terminal to initiate the bulk transfer of the selected issues. The GitHub CLI will handle the transfer process efficiently. 

Note: Ensure you have read and write access in the destination repository.  

Filters are essentially flags, that can be utilized to filter the issues you want to transfer from source to destination repositories matching the filter criteria. For example, commonly used flags include `–state` (filter issues based on if they are open or closed), `–assignee` (filter issues based on the GitHub user assigned to the issue) and `–label` (filter issues based on a specific label name). For a full list of filters available see the GitHub CLI doc. 

Example: Transfer all open issues with “Testlabel1” label 

% gh issue list --label "Testlabel1" --json number |\
 jq -r '.[] | .number' |\
 xargs -I% gh issue transfer % https.destinationrepo.git

Let’s break down what’s happening with the above script.  

gh issue list --label "Testlabel1" --json number: 

• % gh issue list : This part of the command initiates the GitHub CLI to list issues. It prompts the GitHub CLI to fetch and display a list of issues based on specified criteria.  
• --label “Testlabel1”: This is a flag used to filter the issues based on specific label, in this case “Testlabel1”. By including this flag, the command instructs the GitHub CLI to only list issues that are tagged with the label “Testlabel1”. 
• --json number: This flag is used to format the output of the command in JSON format, which is a structured data format that is easily readable by machines. The number argument specifies that only the issue numbers should be included in the JSON output. 

| jq -r '.[] | .number': 

• jq: The output from the previous command is piped (|) into jq, a command-line JSON processor. 
• -r: The -r option outputs raw strings (instead of JSON formatted strings). 
• '.[] | .number': This part iterates through each issue in the array and extracts just the issue numbers. 

| xargs -I% gh issue transfer % https://destinationrepo.git: 

• xargs: This is a command in Unix and Unix-like operating systems used to build and execute commands from standard input. It reads items from the standard input, delimited by blanks or newlines, and executes the specified command (in this case, gh issue transfer) with those items as arguments. 
• -I%: This option specifies that % will be replaced with each issue number. 
• gh issue transfer % https://destinationrepo.git: This is the GitHub CLI command used to transfer the GitHub issue represented by % to the destination repository specified by https://destinationrepo.git. 

Whilst the GitHub CLI approach provides multiple issue transfer capabilities alongside filter support, ultimately, we decided we wanted to maintain our pipelines in Zenhub, due to its enhanced flexibility in managing pipelines effectively. However, this increased flexibility comes with the trade-off of requiring a more customized approach, such as developing a Python script that leverages Zenhub’s API for transferring issues between repositories. It also provides the necessary customization option and control required for efficient repository management. 

3^rd: Hybrid – Use GitHub CLI and Zenhub API

Having identified that the approaches didn’t maintain the Zenhub pipelines that were critical to our solution, we required the Zenhub API to transfer issues. For this we required a customized solution as there was no pre-existing Zenhub UI solution tailored to the specific requirements of maintaining the issue pipeline. To transfer multiple issues using this solution, the following workspace ids, repository ids, Zenhub authentication token are requirements: 

• source_workspace_id = ” nq-5bd12345678e7654321dee60″ (accessed from the repository URL) 
• source_repo_id = 23645 (inspect page source to look for repository_id) 
• destination_workspace_id = “lb-12th3eu445a789876543b01234” (accessed from the repository URL) 
• destination_repo_id = 09876 (inspect page source to look for repository_id) 
• destination_repo_url = “https://destinationrepo.git” 
• Zenhub Authentication Token = “zenhub_token” 

Here are some tips before commencing this approach. 

• Source repository metadata must match the destination repository metadata (e.g: labels, sprints, releases, milestone, etc) 
• Repository subscribers will be notified of each transfer of issues, it’s recommended for the team to temporarily unsubscribe from the repository while the transfer is in progress. 
• Source and destination issue ids will be different after the transfer process. But all existing links to issues and pull requests are preserved. 
• Transfer can happen between public to public, public to private but not private to public type repository.   

Below I have added a small code snippet from the code we used in our solution for the customized python script. You can find the full python script here.

# Generate pipeline map from destination workspace 

def get_pipelines(destination_workspace_id, destination_repo_id): 
 
 # Construct the Zenhub API URL 
 zenhub_api_url = f"https://zenhub.com/api/p2/workspaces/{destination_workspace_id}/repositories/{destination_repo_id}/board" 

 # Set the headers for the API request 
 headers = {"X-Authentication-Token": zenhub_token} 

 # Make a GET request to the Zenhub API 
 response_data = get_request(zenhub_api_url, headers) 

 # Iterate through the pipelines in the response data 
 for pipeline in response_data["pipelines"]: 
 pipeline_name = pipeline["name"] 
 pipeline_id = pipeline["id"] 

 # Map the pipeline name to its ID 
 pipeline_map[pipeline_name] = pipeline_id 
 
# Function to transfer an issue to a destination repository 
def transfer(issue_number, destination_repo_url): 

 # Construct the GitHub CLI command 
 gh_command = f"gh issue transfer {issue_number} {destination_repo_url}" 
 new_issue = 0 

 try: 
 # Run the command using subprocess 
 gh_command_list = shlex.split(gh_command) 
 new_issue_url = subprocess.run(gh_command_list, text=True, capture_output=True, check=True) 
 new_issue = str(new_issue_url.stdout) 
 
 # Check if the transfer was successful 
 if new_issue is None or len(new_issue) == 0: 
 return 0 

 # Extract the new issue number from the URL 
 new_issue = int((new_issue.split('/')[-1]).strip()) 

 # Add a delay before proceeding 
 time.sleep(2) 

except subprocess.CalledProcessError as e: 

 # Handle any errors that occur during the transfer 
 print(f"Failed to transfer issue #{issue_number}: {e}") 
 failed_transfer_map[issue_number] = e 
 logging.info(f"Failed to transfer issue #{issue_number} error is {e}") 

return new_issue 

 

After the transfer process is completed, all existing links to issues and pull requests are preserved. Issue dependencies, as well as the relationships between epics and child issues, are retained post-transfer. There were no broken links; instead, any attempt to access the old links automatically redirected to the new URL. 

This approach had the benefits of allowing us to transfer multiple issues all at once, being able to select an issue based on a filter and crucially for us, enabling us to maintain the exact Zenhub pipeline data. As there was no out the box solution available, we spent time configuring, testing and debugging the customized python script that was specific to the requirements of our repositories. 

A ready-to-use version of this script is available here, which you can adapt for your own migrations by updating lines 137 to 141 to fit your repository setup. 

When you are utilizing the Python script for bulk GitHub issue transfer, consider the following tips to ensure a smooth and efficient process: 

• Ensure Proper Authentication: Make sure you have the necessary authentication tokens or credentials to access both the Zenhub and GitHub APIs securely. 
• Handle Errors Gracefully: Implement error handling mechanisms to address any issues that may arise during the transfer process. This includes network errors, API rate limits, or invalid data. 
• Test in a Controlled/Dummy Environment: Before performing bulk transfers on live data, test the script in a controlled environment to validate its functionality and identify any potential pitfalls. 
• Monitor API Usage: Keep track of your API usage limits on both Zenhub and GitHub to avoid hitting rate limits, which could disrupt the transfer process. 
• Backup Data: Consider backing up your GitHub repository data before initiating bulk transfers to prevent accidental data loss. 
• Review Transfer Results: After the transfer process is complete, review the transferred issues in the destination repository to ensure accuracy and completeness. 

Conclusion 

After consolidating all issues into a unified repository linked to the codebase, the goal was to boost visibility, teamwork, and productivity while minimizing manual tasks. Our engineering team successfully accomplished all the intended objectives following the transfer. 

Transferring over 1000+ GitHub issues between repositories may appear daunting initially, but with the appropriate tools and strategies, it becomes significantly more manageable. Throughout this process, we explored various techniques, ranging from manual transfers via the GitHub UI to utilizing automation tools such as GitHub’s CLI and custom Python scripts. While each method has its advantages and disadvantages, the most successful solution for our team involved employing a custom script that utilized the GitHub CLI and Zenhub API for batch processing. This approach not only saved considerable time but also ensured consistency across all transferred issues while reducing the likelihood of human error. The use of this custom script enabled us to complete the transfer of all 1000+ issues in approximately 3 hours, underscoring the efficiency and effectiveness of automation in streamlining large-scale transfers.  

This consolidation provided a clearer, real-time overview of project progress, making it easier to identify blockers, distribute workload effectively, and align priorities across teams. It also enabled less manual overhead, more efficient resource allocation and decision-making, as stakeholders had full visibility into the development pipeline, sprint planning, and outstanding tasks – all in one place. 

Here is a comparison table summarizing the benefits and considerations of the three approaches discussed above for migrating GitHub issues. 

Ultimately, the key to efficient issue management lies in understanding the scale of your use case and the tools at your disposal. Whether you’re transferring a handful of issues or thousands, selecting the correct approach will help you minimize manual issue transfer work, maintain a single source of truth, enhance team collaboration, and keep your project’s progress on track. By streamlining issue management, teams can focus more on what truly matters – writing great code and delivering value to their users. 

We’re always looking for talented engineers to join our team – check out the Esri career pages to learn more. 

About the author

Parijat Das

Serving as a Product Engineer and Scrum Master at ArcGIS Maps SDK for Native Apps, guiding agile teams and driving product quality through cross-functional collaboration, process optimization, and robust testing readiness.