Add test results report azure integration

Author: NayeemJohnY

.github/workflows/test-execution.yml

@@ -94,6 +94,15 @@ jobs:
           path: test-results
           retention-days: 3
 
+      - name: Post Test Results to Azure DevOps
+        if: ${{ (success() || failure()) && inputs.publish_report }}
+        uses: NayeemJohnY/actions/post-test-results-to-azure-devops@main
+        with:
+          test-results-json: test-results/test-results-report.json
+          org-url: ${{ vars.AZURE_ORG_URL }}
+          project: ${{ vars.AZURE_PROJECT }}
+          azure-token: ${{ secrets.AZURE_TOKEN }}
+
       - name: Prepare GitHub Pages Content
         if: success() || failure()
         uses: NayeemJohnY/actions/prepare-github-pages@main

README.md

@@ -189,10 +189,35 @@ This marker system enables flexible test execution strategies, from quick smoke
 - **Parallel test issues**: Make sure test data is unique per worker or run tests serially.
 - **HTML report not generated**: Ensure `pytest-html` is installed and use the `--html` option.
 ---
-## CI/CD & Automation
-
-This project uses GitHub Actions for:
-- **Code Analysis**: Automated linting and static analysis on every push/PR.
-- **Test Execution**: Runs all tests and generates reports for every push/PR.
-
-See the badges at the top of this README for the latest status of these workflows.
+## CI/CD & GitHub Actions
+
+This project uses GitHub Actions for automated workflows:
+- **Code Analysis**: Runs linting and static analysis on every push and pull request.
+- **Test Execution**: Runs all tests and generates reports for every push and pull request, including parallel execution and publishing Allure/HTML reports.
+- **Badges**: See the top of this README for live status of these workflows.
+
+Workflow files are located in `.github/workflows/`.
+
+## Running the Test Execution Workflow (GitHub Actions)
+
+### How to Trigger
+- The workflow runs automatically on every push and pull request to the repository.
+- You can also trigger it manually from the GitHub Actions tab by selecting the `Test Execution & Publish Report` workflow and clicking 'Run workflow'.
+
+### What the Workflow Does
+- Installs Python and all dependencies from `requirements.txt`.
+- Runs all tests using pytest (including parallel execution with xdist).
+- Publishes test results as HTML and Allure reports (if configured).
+- Uploads the reports as workflow artifacts for download and review.
+- Updates status badges at the top of the README to reflect the latest run.
+
+### New: Post Results to Azure
+- After test execution, the workflow now includes a step to post the test results to Azure (e.g., Azure DevOps, Azure Storage, or a custom API endpoint).
+- This enables centralized reporting, dashboard integration, or further automation in your Azure environment.
+- The step uses secure credentials and API endpoints configured in your repository secrets.
+- You can customize the target Azure service and payload format as needed for your organization.
+
+### Viewing Results
+- Go to the 'Actions' tab in your GitHub repository.
+- Select the latest run of `Test Execution & Publish Report`.
+- Download the HTML/Allure report artifacts for detailed results.

conftest.py

@@ -38,18 +38,34 @@ def api_client():
     return APIClient(BASE_URI, BASE_PATH)
 
 
-def collect_test_results(test_name: str, test_params: str, report: TestReport):
+def collect_test_results(test_name: str, test_params: str, report: TestReport, call=None):
     """
     Collects and aggregates test results for each test case.
     Stores outcome, duration (in ms), and iteration details if test parameters are present.
     """
+    def get_outcome():
+        if report.outcome == "skipped":
+            return "Error"
+        return report.outcome.capitalize()
+    
+    def get_error_message():
+        if call and call.excinfo:
+            # Get simple exception message from call.excinfo
+            exception_type = call.excinfo.typename
+            exception_value = str(call.excinfo.value)
+            return f"{exception_type}: {exception_value}"
+        return ""
+        
     test_case_id: str = test_case_mappings[test_name]["testCaseId"]
     # Convert duration to milliseconds
     duration_ms = int(float(report.duration) * 1000)
+    error_message = get_error_message()
+    new_outcome = get_outcome()
+    
     result = test_results.setdefault(
         test_case_id,
         {
-            "outcome": report.outcome.capitalize(),
+            "outcome": new_outcome,
             "durationInMs": 0,
             "comment": f"Test Name: {test_name}",
             "iterationDetails": [],
@@ -58,17 +74,31 @@ def collect_test_results(test_name: str, test_params: str, report: TestReport):
 
     if test_params:
         iteration_id = len(result["iterationDetails"]) + 1
+        error_message = f"Iteration {iteration_id}: {error_message}" if error_message else error_message
         iteration_result = {
             "id": iteration_id,
-            "outcome": report.outcome.capitalize(),
+            "outcome": new_outcome,
             "durationInMs": duration_ms,
-            "comment": f"Test Parameters: {json.dumps(test_params)}",
+            "errorMessage": error_message,
+            "comment": f"DataDriven: Test Parameters: {json.dumps(test_params)}",
         }
         result["iterationDetails"].append(iteration_result)
 
     result["durationInMs"] += duration_ms
-    if result["outcome"] == "Failed" or report.outcome.capitalize() == "Failed":
-        result["outcome"] = "Failed"
+    
+    # Update outcome based on current and new results
+    current_outcome = result["outcome"]
+    # Determine final outcome: same = keep, different = Inconclusive
+    if current_outcome != new_outcome:
+        result["outcome"] = "Inconclusive"
+    
+    # Append error messages from all iterations
+    if error_message:
+        existing_error = result.get("errorMessage", "")
+        if existing_error:
+            result["errorMessage"] = f"{existing_error}\n{error_message}"
+        else:
+            result["errorMessage"] = error_message
 
 
 @pytest.hookimpl(hookwrapper=True)
@@ -79,10 +109,16 @@ def pytest_runtest_makereport(item: Function, call):
     """
     outcome: Result = yield
     report: TestReport = outcome.get_result()
+    test_name: str = item.originalname
+    test_params = item.callspec.params if hasattr(item, "callspec") else None
+    
+    # Collect results for call phase (actual test execution)
     if report.when == "call":
-        test_name: str = item.originalname
-        test_params = item.callspec.params if hasattr(item, "callspec") else None
-        collect_test_results(test_name, test_params, report)
+        collect_test_results(test_name, test_params, report, call)
+    
+    # Handle setup failures - mark as error
+    elif report.when == "setup" and report.outcome in ["failed", "skipped"]:
+        collect_test_results(test_name, test_params, report, call)
 
 
 def pytest_sessionfinish(session: Session, exitstatus):
@@ -115,9 +151,14 @@ def pytest_sessionfinish(session: Session, exitstatus):
             with open(temp_file_path, "r", encoding="utf-8") as temp_results_file:
                 worker_data = json.load(temp_results_file)
                 test_results.update(worker_data)
+        report = {
+            "testPlanName": test_plan_suite['testPlanName'],
+            "testSuiteName": test_plan_suite['testSuiteName'],
+            "testResults": test_results
+        }
 
         with open(TEST_RESULTS_PATH, "w", encoding="utf-8") as out:
-            json.dump(test_results, out, indent=4)
+            json.dump(report, out, indent=4)
             logger.info(
                 "Test results report generated successfully: %s", TEST_RESULTS_PATH
             )

github-pages/images/ado-run-summary.png

@@ -1,52 +1,64 @@
 <!DOCTYPE html>
 <html lang="en">
+
 <head>
     <meta charset='utf-8'>
     <meta http-equiv='X-UA-Compatible' content='IE=edge'>
     <title>Book API Testing Framework</title>
     <meta name='viewport' content='width=device-width, initial-scale=1'>
     <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
     <meta property="og:title" content="Book API Testing Framework">
-    <meta property="og:description" content="Comprehensive API Testing Framework for Book system with Python, pytest, and requests.">
+    <meta property="og:description"
+        content="Comprehensive API Testing Framework for Book system with Python, pytest, and requests.">
     <meta property="og:type" content="website">
     <meta name="twitter:card" content="summary_large_image">
     <link rel="icon" href="favicon.ico" type="image/x-icon">
     <link rel="stylesheet" href="style.css">
 </head>
+
 <body>
-    <a href="https://github.com/NayeemJohnY/book-api-python-requests-pytest-automation" class="github-corner" target="_blank" title="View on GitHub">
+    <a href="https://github.com/NayeemJohnY/book-api-python-requests-pytest-automation" class="github-corner"
+        target="_blank" title="View on GitHub">
         <img src="https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png" alt="GitHub Repository">
     </a>
-    
+
     <div class="main-container">
         <header>
             <h1><i class="fas fa-book"></i> Book API Testing Framework</h1>
-            <p class="subtitle">A robust, maintainable API test automation framework built with Python, pytest, and requests</p>
+            <p class="subtitle">A robust, maintainable API test automation framework built with Python, pytest, and
+                requests</p>
         </header>
-        
+
         <nav class="quick-links">
             <a href="./allure-report/index.html" class="nav-link">
                 <i class="fa-solid fa-chart-line"></i>
                 View Allure Report
             </a>
+            <a href="./test-results-report.json" class="nav-link">
+                <i class="fa-solid fa-file-code"></i>
+                Test Results JSON
+            </a>
         </nav>
-        
+
         <div class="content-grid">
             <div class="card">
                 <h2><i class="fas fa-star"></i> Key Features</h2>
                 <ul>
                     <li><strong>Custom APIClient</strong> with built-in retry logic and comprehensive logging</li>
-                    <li><strong>Reusable Validators</strong> for status codes, error messages, and response validation</li>
+                    <li><strong>Reusable Validators</strong> for status codes, error messages, and response validation
+                    </li>
                     <li><strong>DRY Architecture</strong> using pytest fixtures and base classes</li>
                     <li><strong>Parallel Execution</strong> with pytest-xdist and worker-safe test data</li>
                     <li><strong>Rich Reporting</strong> with HTML and Allure integration</li>
                     <li><strong>Comprehensive Logging</strong> for all requests, responses, and assertions</li>
                 </ul>
-                 <ul>
-                     <li><strong>CI/CD Workflows:</strong> Automated <code> Code Analysis</code> and <code>Test Execution</code> via GitHub Actions</li>
-                 </ul>
+                <ul>
+                    <li><strong>CI/CD Workflows:</strong> Automated <code> Code Analysis</code> and
+                        <code>Test Execution</code> via GitHub Actions
+                    </li>
+                </ul>
             </div>
-            
+
             <div class="card">
                 <h2><i class="fas fa-rocket"></i> Quick Start</h2>
                 <ol>
@@ -56,10 +68,12 @@ <h2><i class="fas fa-rocket"></i> Quick Start</h2>
                     <li>Run tests in parallel:<br><code>pytest -n auto --dist loadfile</code></li>
                     <li>Run smoke tests quickly:<br><code>pytest -m smoke -n auto --dist loadfile</code></li>
                     <li>Generate Allure report:<br><code>allure serve test-results/allure-results</code></li>
-                    <li>To generate a static Allure HTML report:<br><code>allure generate test-results/allure-results --clean -o test-results/allure-report</code></li>
+                    <li>To generate a static Allure HTML
+                        report:<br><code>allure generate test-results/allure-results --clean -o test-results/allure-report</code>
+                    </li>
                 </ol>
             </div>
-            
+
             <div class="card">
                 <h2><i class="fas fa-cogs"></i> Best Practices</h2>
                 <ul>
@@ -70,7 +84,7 @@ <h2><i class="fas fa-cogs"></i> Best Practices</h2>
                     <li><strong>Parallel Safety:</strong> Unique test data per worker prevents collisions</li>
                 </ul>
             </div>
-            
+
             <div class="card">
                 <h2><i class="fas fa-tools"></i> Tech Stack</h2>
                 <ul>
@@ -102,24 +116,90 @@ <h2><i class="fas fa-tags"></i> Pytest Markers</h2>
             <div class="card">
                 <h2><i class="fas fa-link"></i> Test Case Mapping & Result Collection</h2>
                 <ul>
-                    <li><strong>Test Case Mapping:</strong> Each test function is mapped to a unique test case ID using <code>test-plan-suite.json</code> for traceability and reporting.</li>
-                    <li><strong>Result Collection:</strong> Test results are collected for each test case, including outcome, duration (ms), and iteration details. Results are aggregated and written to <code>test-results/test-results-report.json</code> after each run, supporting both serial and parallel execution.</li>
+                    <li><strong>Test Case Mapping:</strong> Each test function is mapped to a unique test case ID using
+                        <code>test-plan-suite.json</code> for traceability and reporting.
+                    </li>
+                    <li><strong>Result Collection:</strong> Test results are collected for each test case, including
+                        outcome, duration (ms), and iteration details. Results are aggregated and written to
+                        <code>test-results/test-results-report.json</code> after each run, supporting both serial and
+                        parallel execution.
+                    </li>
                 </ul>
             </div>
+
+            <div class="card">
+                <h2><i class="fa-brands fa-github"></i> CI/CD & GitHub Actions</h2>
+                <ul>
+                    <li><strong>Code Analysis:</strong> Automated linting and static analysis on every push and pull
+                        request.</li>
+                    <li><strong>Test Execution:</strong> Runs all tests and generates reports for every push and pull
+                        request, including parallel execution and publishing Allure/HTML reports.</li>
+                </ul>
             </div>
         </div>
-        
-        <footer class="footer">
-            <div>
-                <a href="https://www.linkedin.com/in/nayeemjohny/" target="_blank" class="linkedin-link">
-                    <i class="fa-brands fa-linkedin"></i>Connect with me on LinkedIn
-                </a>
+        <div class="ado-images-container">
+            <h2>Azure DevOps Test Plan Integration</h2>
+            <p>Below are sample screenshots showing how test results are posted and visualized in Azure DevOps Test
+                Plans:</p>
+            <div class="ado-images-grid">
+                <figure>
+                    <img src="images/ado-progress-report.png" alt="ADO Progress Report" class="ado-img"
+                        onclick="enlargeImage(this)">
+                    <figcaption>Progress Report</figcaption>
+                </figure>
+                <figure>
+                    <img src="images/ado-run-summary.png" alt="ADO Run Summary" class="ado-img"
+                        onclick="enlargeImage(this)">
+                    <figcaption>Run Summary</figcaption>
+                </figure>
+                <figure>
+                    <img src="images/ado-test-result-summary.png" alt="ADO Test Result Summary" class="ado-img"
+                        onclick="enlargeImage(this)">
+                    <figcaption>Test Result Summary</figcaption>
+                </figure>
+                <figure>
+                    <img src="images/ado-test-results.png" alt="ADO Test Results" class="ado-img"
+                        onclick="enlargeImage(this)">
+                    <figcaption>Test Results in Azure DevOps</figcaption>
+                </figure>
             </div>
-            <div class="copilot-credit">
-                <i class="fa-solid fa-robot"></i>
-                Thanks to <span class="copilot-highlight">GitHub Copilot</span> for assisting with documentation
+            <div id="adoModal" class="ado-modal" onclick="closeModal(event)">
+                <div class="ado-modal-content">
+                    <div class="ado-modal-image-container">
+                        <img id="adoModalImg" src="" alt="Enlarged Azure DevOps Screenshot">
+                    </div>
+                    <button class="ado-modal-close" onclick="closeModal(event)">Close</button>
+                </div>
             </div>
-        </footer>
+        </div>
+        <script>
+            function enlargeImage(img) {
+                var modal = document.getElementById('adoModal');
+                var modalImg = document.getElementById('adoModalImg');
+                modalImg.src = img.src;
+                modal.classList.add('active');
+            }
+            function closeModal(event) {
+                var modal = document.getElementById('adoModal');
+                if (event.target.classList.contains('ado-modal') || event.target.classList.contains('ado-modal-close')) {
+                    modal.classList.remove('active');
+                }
+            }
+        </script>
+    </div>
+
+    <footer class="footer">
+        <div>
+            <a href="https://www.linkedin.com/in/nayeemjohny/" target="_blank" class="linkedin-link">
+                <i class="fa-brands fa-linkedin"></i>Connect with me on LinkedIn
+            </a>
+        </div>
+        <div class="copilot-credit">
+            <i class="fa-solid fa-robot"></i>
+            Thanks to <span class="copilot-highlight">GitHub Copilot</span> for assisting with documentation
+        </div>
+    </footer>
     </div>
 </body>
-</html>
+
+</html>