GitHub Integration Guide

Learn how to set up GitHub integration with Kaizen Agent to automatically create pull requests with fixes and improvements.

Overview

Kaizen Agent can automatically create pull requests with fixes and improvements to your code. This is especially useful for:

Automated code improvements without manual intervention
Team collaboration through pull request reviews
Version control integration for tracking changes
CI/CD pipeline integration for automated testing and fixing

Prerequisites

Before setting up GitHub integration, you need:

GitHub account with access to the repository
Personal Access Token with appropriate permissions
Repository access (either as owner, collaborator, or with write permissions)

Step 1: Create GitHub Personal Access Token

Generate a New Token

Go to GitHub Settings > Developer settings > Personal access tokens
Click "Generate new token (classic)"
Give it a descriptive name (e.g., "Kaizen AutoFix")
Set an appropriate expiration date
Important: Select these scopes:
- ✅ repo (Full control of private repositories)
- ✅ workflow (if you want to trigger GitHub Actions)

Token Permissions Explained

Permission	Purpose	Required
`repo`	Full control of private repositories	✅ Yes
`workflow`	Trigger GitHub Actions workflows	Optional
`write:packages`	Publish packages to GitHub Package Registry	Optional

Security Note: The repo scope provides full access to your repositories. For production use, consider creating a dedicated GitHub App with more granular permissions.

Step 2: Set Up Environment Variables

Option 1: Using .env File (Recommended)

Create a .env file in your project root:

# Create .env file
cat > .env << EOF
GOOGLE_API_KEY=your_google_api_key_here
GITHUB_TOKEN=ghp_your_github_token_here
EOF

Option 2: Set Environment Variables Directly

# Set environment variables in your shell
export GOOGLE_API_KEY="your_google_api_key_here"
export GITHUB_TOKEN="ghp_your_github_token_here"

Option 3: Using Kaizen Setup Commands

# Create environment example file
kaizen setup create-env-example

# Check environment setup
kaizen setup check-env

Step 3: Test GitHub Access

Before running tests with PR creation, verify your GitHub setup:

# Test GitHub access
kaizen test-github-access --repo your-username/your-repo-name

This command will:

Verify your GitHub token is valid
Check repository access permissions
Test branch creation capabilities
Validate pull request creation permissions

Example Output

✅ GitHub access test successful!
✅ Token is valid and has required permissions
✅ Repository access confirmed: your-username/your-repo-name
✅ Can create branches and pull requests

Step 4: Run Tests with PR Creation

Once GitHub access is confirmed, you can run tests with automatic PR creation:

# Run tests with auto-fix and create PR
kaizen test-all --config kaizen.yaml --auto-fix --create-pr --repo your-username/your-repo-name

What Happens During PR Creation

Run tests with your configuration
Identify issues in your agent code or prompts
Generate fixes using AI
Create a new branch with the fixes
Commit changes with descriptive messages
Create a pull request with:
- Descriptive title and description
- List of changes made
- Before/after comparisons
- Test results summary

GitHub Commands Reference

Test GitHub Access

kaizen test-github-access --repo owner/repo-name

Options:

--repo: GitHub repository in format owner/repo-name
--token: Override GitHub token (optional)

Diagnose GitHub Access Issues

If you encounter problems with GitHub access:

kaizen diagnose-github-access --repo owner/repo-name

This command provides detailed diagnostics including:

Token validation
Repository permissions
Branch creation capabilities
Pull request permissions
Suggested fixes for common issues

Run Tests with PR Creation

kaizen test-all --config kaizen.yaml --auto-fix --create-pr --repo owner/repo-name

Required Options:

--config: Path to your YAML configuration file
--auto-fix: Enable automatic fixing
--create-pr: Create pull request with fixes
--repo: GitHub repository for PR creation

Optional Options:

--save-logs: Save detailed execution logs
--debug: Enable debug mode for more verbose output

Pull Request Details

PR Title Format

Kaizen Agent creates pull requests with descriptive titles:

🤖 Auto-fix: Improve EmailAgent prompts and error handling

Generated by Kaizen Agent test run on 2024-01-15

PR Description

Each pull request includes:

Summary of changes made
Test results showing improvements
Before/after comparisons of key metrics
Files modified with brief descriptions
Test configuration used for the run

Example PR Description

This Pull Request introduces fixes identified and implemented by the **Kaizen AutoFix Agent (v1.0.0)**, an automated code fixing agent using LLM-based analysis.

## Summary

This PR addresses critical issues identified in the email generation functionality, specifically concerning adherence to output format constraints. Previously, the system failed to produce only the email draft, often including additional explanatory text. The changes in this PR ensure that the output strictly contains only the improved email draft, resolving all previously failing test cases. As a result, all relevant tests now pass, significantly improving the reliability and compliance of the email generation feature.

| Test Case | Baseline | Attempt 1 | Attempt 2 | Final Status | Improvement |
|---|---|---|---|---|---|
| Basic Email Improvement | failed | passed | passed | passed | Yes |
| Too casual input | failed | passed | passed | passed | Yes |

## Detailed Results

### Baseline (Before Fixes)
**Status:** failed

**Test Case:** Basic Email Improvement
- **Input:** Can you help me improve this email: Dear John, I want to meet tomorrow to discuss the project. Thanks, Jane
- **Expected Output:** N/A
- **Actual Output:** Here's an improved version of the email draft, focusing on professionalism, clarity, and etiquette:

---

**Subject: Meeting Request: [Project Name] Discussion - [Your Name]**

Dear John,

I hope this email finds you well.

I would like to schedule a brief meeting tomorrow, [Insert Tomorrow's Date, e.g., Tuesday, October 26th], to discuss the [Project Name] project. I believe it would be beneficial to align on the current status and next steps.

Please let me know what time works best for you tomorrow. I am generally available [mention a broad availability window, e.g., "in the morning between 9:00 AM and 12:00 PM" or "after 1:00 PM"].

If tomorrow doesn't suit your schedule, please suggest an alternative time that works for you later this week.

Thank you for your time and consideration.

Best regards,

Jane

---

**Key Improvements and Why:**

1.  **Subject Line:**
    *   `Meeting Request:`: Clearly states the purpose of the email upfront.
    *   `[Project Name] Discussion`: Specifics the topic, so John knows what the meeting is about immediately.
    *   `- [Your Name]`: Helps John quickly identify who the email is from, especially if he receives many emails.

2.  **Professional Salutation:**
    *   `Dear John,`: Retained, as it's standard and appropriate.

3.  **Opening:**
    *   `I hope this email finds you well.`: A polite and common professional opening.

4.  **Clear Purpose and Intent:**
    *   `I would like to schedule a brief meeting tomorrow...`: More professional and less demanding than "I want to meet."
    *   `to discuss the [Project Name] project.`: Clearly states the topic.
    *   `I believe it would be beneficial to align on the current status and next steps.`: Adds a brief, professional reason for the meeting, showing the value of the discussion.

5.  **Flexibility and Specificity for Scheduling:**
    *   `tomorrow, [Insert Tomorrow's Date, e.g., Tuesday, October 26th]`: Adding the specific date avoids ambiguity.
    *   `Please let me know what time works best for you tomorrow.`: Puts the ball in John's court politely.
    *   `I am generally available [mention a broad availability window...]`: Offers helpful information without being overly rigid, making it easier for
- **Result:** FAIL
- **Evaluation:** The output successfully provided an improved email draft relevant to the test case. However, it failed to adhere to the strict instruction that the return should *only* contain the email draft, as it included an additional 'Key Improvements and Why' section.
- **Reasoning:** The core purpose of improving the email was met, and the quality of the improved email itself is high. However, the explicit constraint 'the return should only contain email draft, nothing else only one email draft' was violated by the inclusion of the detailed explanation section. This makes the output non-compliant with the specified format requirements.

**Test Case:** Too casual input
- **Input:** improve this email: hi there, are you free to meet tomorrow?
- **Expected Output:** N/A
- **Actual Output:** Here's an improved version of your email draft, focusing on professionalism, clarity, and effectiveness:

**Subject: Meeting Availability Tomorrow**

Hello [Recipient's Name],

I hope this email finds you well.

I'm writing to inquire if you might be available for a brief meeting tomorrow, [Date of Tomorrow, e.g., Wednesday, October 26th]. I'd like to discuss [brief topic, e.g., "the project update," "a quick question regarding X," or "a follow-up on our last conversation"].

Please let me know if you have any availability that works for you. If tomorrow doesn't suit, I'm happy to work around your schedule and find an alternative time later in the week.

Thank you for your time and consideration.

Best regards,

[Your Name]

---

**Why this is improved:**

1.  **Professional Tone and Language:**
    *   "Hello [Recipient's Name]," is a professional yet friendly salutation.
    *   "I hope this email finds you well" is a polite opening.
    *   "I'm writing to inquire if you might be available" is more formal and polite than "are you free."
    *   "Thank you for your time and consideration" and "Best regards" are standard professional closings.

2.  **Clear and Concise Communication:**
    *   **Subject Line:** Immediately tells the recipient the purpose of the email.
    *   **Specific Request:** Clearly states the desire for a "brief meeting tomorrow."
    *   **Context/Purpose:** The placeholder `[brief topic]` encourages you to add a specific reason for the meeting, making it more efficient for the recipient to decide. If the context is already very clear (e.g., you just finished a call and this is a pre-arranged follow-up), you can omit it, but generally, it's good practice.
    *   **Date Specificity:** Adding `[`
- **Result:** FAIL
- **Evaluation:** The test output successfully provided an improved email draft, which aligns with the implicit request to make a 'too casual input' more professional. However, it failed to adhere to the strict criterion that the output 'should only contain email draft, nothing else'. The additional 'Why this is improved' section, while helpful for understanding, violates this specific instruction.
- **Reasoning:** The core issue is the inclusion of explanatory text ('Why this is improved') after the email draft. The evaluation target explicitly states that 'the return should only contain email draft, nothing else only one email draft'. This additional content causes the overall failure, despite the email draft itself being appropriate and well-formed.

### Best Attempt (Attempt 1)
**Status:** passed

**Test Case:** Basic Email Improvement
- **Input:** Can you help me improve this email: Dear John, I want to meet tomorrow to discuss the project. Thanks, Jane
- **Expected Output:** N/A
- **Actual Output:** Dear John,

I would like to schedule a meeting tomorrow to discuss the project. Please let me know your availability.

Best regards,

Jane
- **Result:** PASS
- **Evaluation:** The actual output fully meets all specified criteria for the 'email quality' target. It is a clean, single email draft, and directly addresses the 'Basic Email Improvement' test case by providing an email.
- **Reasoning:** The output 'Dear John,...Jane' is a complete and singular email draft. It contains no additional text, boilerplate, or multiple drafts, thus satisfying the first criterion of 'only contain email draft, nothing else only one email draft'. As the Test Case is explicitly 'Basic Email Improvement', the output correctly provides an improved email draft, fulfilling the second criterion. The third criterion regarding 'I'm sorry, I can't help with that.' is not applicable as the input *was* about improving an email, and the output correctly did not return that phrase.

**Test Case:** Too casual input
- **Input:** improve this email: hi there, are you free to meet tomorrow?
- **Expected Output:** N/A
- **Actual Output:** Subject: Meeting Availability Tomorrow

Hello [Recipient Name],

Would you be available to meet tomorrow to discuss [brief topic]? Please let me know your availability or if another time works better.

Best regards,
[Your Name]
- **Result:** PASS
- **Evaluation:** The test case successfully generated a professional email draft from what is described as a 'too casual input'. The output adheres to all specified criteria for 'email quality'. It contains only one email draft, with no additional text. Given the input implies the need for a more formal output (improving the casualness), the generated email serves as the appropriate 'improved' draft.
- **Reasoning:** The output perfectly matches the expected format and content based on the provided target criteria. It is a single, clean email draft, directly addressing the implicit need to formalize or generate an email from a casual prompt.

Branch Naming Convention

Kaizen Agent creates branches with the following naming convention:

kaizen-auto-fix-YYYY-MM-DD-HHMMSS

Example: kaizen-auto-fix-2024-01-15-143022

Best Practices

1. Review PRs Before Merging

Always review the generated pull requests before merging:

Check the changes make sense
Verify test improvements are real
Ensure no unintended side effects

2. Use Descriptive Commit Messages

Kaizen Agent generates descriptive commit messages, but you can customize them:

# Custom commit message
kaizen test-all --config kaizen.yaml --auto-fix --create-pr --repo owner/repo-name --commit-message "Fix email agent prompt issues"

3. Set Up Branch Protection

Consider setting up branch protection rules:

Require pull request reviews
Require status checks to pass
Restrict direct pushes to main branch

4. Monitor PR Activity

Regularly check for new pull requests:

Set up notifications for new PRs
Review and merge promptly
Provide feedback on improvements

Troubleshooting

Common Issues

1. Token Permission Errors

❌ Error: Token does not have required permissions

Solution: Ensure your token has the repo scope enabled.

2. Repository Access Denied

❌ Error: Cannot access repository

Solution:

Verify the repository exists and is accessible
Check your token has access to the repository
Ensure you're using the correct repository format (owner/repo-name)

3. Branch Creation Failed

❌ Error: Cannot create branch

Solution:

Check if the branch name already exists
Verify you have write permissions to the repository
Ensure the base branch exists

4. Pull Request Creation Failed

❌ Error: Cannot create pull request

Solution:

Check if a PR already exists for the same changes
Verify the repository allows pull requests
Ensure the base branch is protected (if applicable)

Debug Mode

Enable debug mode for detailed troubleshooting:

kaizen test-all --config kaizen.yaml --auto-fix --create-pr --repo owner/repo-name --debug

Getting Help

If you continue to have issues:

Check the logs: Review detailed logs in test-logs/ directory
Run diagnostics: Use kaizen diagnose-github-access
Join Discord: Get help from the community at Discord
Check GitHub status: Ensure GitHub services are operational

Security Considerations

Token Security

Never commit tokens to version control
Use environment variables or secure secret management
Rotate tokens regularly (set appropriate expiration dates)
Use minimal permissions when possible

Repository Security

Review all changes before merging
Set up branch protection rules
Use code review for all automated changes
Monitor for unusual activity

Integration with CI/CD

You can integrate Kaizen Agent with your CI/CD pipeline:

# Example GitHub Actions workflow
name: Kaizen Agent Testing

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - name: Install Kaizen Agent
        run: pip install kaizen-agent
      - name: Run Kaizen tests
        run: kaizen test-all --config kaizen.yaml --auto-fix --create-pr --repo ${{ github.repository }}
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

For more help with GitHub integration, see our FAQ or join our Discord community.

Overview​

Prerequisites​

Step 1: Create GitHub Personal Access Token​

Generate a New Token​

Token Permissions Explained​

Step 2: Set Up Environment Variables​

Option 1: Using .env File (Recommended)​

Option 2: Set Environment Variables Directly​

Option 3: Using Kaizen Setup Commands​

Step 3: Test GitHub Access​

Example Output​

Step 4: Run Tests with PR Creation​

What Happens During PR Creation​

GitHub Commands Reference​

Test GitHub Access​

Diagnose GitHub Access Issues​

Run Tests with PR Creation​

Pull Request Details​

PR Title Format​

PR Description​

Example PR Description​

Branch Naming Convention​

Best Practices​

1. Review PRs Before Merging​

2. Use Descriptive Commit Messages​

3. Set Up Branch Protection​

4. Monitor PR Activity​

Troubleshooting​

Common Issues​

1. Token Permission Errors​

2. Repository Access Denied​

3. Branch Creation Failed​

4. Pull Request Creation Failed​

Debug Mode​

Getting Help​

Security Considerations​

Token Security​

Repository Security​

Integration with CI/CD​

Overview

Prerequisites

Step 1: Create GitHub Personal Access Token

Generate a New Token

Token Permissions Explained

Step 2: Set Up Environment Variables

Option 1: Using .env File (Recommended)

Option 2: Set Environment Variables Directly

Option 3: Using Kaizen Setup Commands

Step 3: Test GitHub Access

Example Output

Step 4: Run Tests with PR Creation

What Happens During PR Creation

GitHub Commands Reference

Test GitHub Access

Diagnose GitHub Access Issues

Run Tests with PR Creation

Pull Request Details

PR Title Format

PR Description

Example PR Description

Branch Naming Convention

Best Practices

1. Review PRs Before Merging

2. Use Descriptive Commit Messages

3. Set Up Branch Protection

4. Monitor PR Activity

Troubleshooting

Common Issues

1. Token Permission Errors

2. Repository Access Denied

3. Branch Creation Failed

4. Pull Request Creation Failed

Debug Mode

Getting Help

Security Considerations

Token Security

Repository Security

Integration with CI/CD