Skip to content

Conversation

@KlausDornsbach
Copy link
Contributor

@KlausDornsbach KlausDornsbach commented Aug 8, 2024

Description

API access keypairs are primarily used to support interactions between systems, without the need to create sessions (through user and password authentication). Currently, CloudStack's implementation of API keypairs does not allow you to specify permissions for each keypair, simply using the account's default permissions. Additionally, the number of keypairs is limited to one per user and they have no start and end dates.

An extension of the API keypairs functionality was implemented, adding several new features that increase flexibility and security. It is now possible to specify a subset of permissions (from the base account) for each keypair, as well as create more than one key per user. It is also possible to define start and end dates for the validity of a keypair. A key created without an expiration date will always be valid up until it is deleted. It should be noted that creating API keypairs without specifying permissions just creates an API keypair with all account's base permissions. Also, API keypairs older than this patch will always be viewed as keypairs with full account permissions.

The following endpoints were created:

A new listUserKeys API was added. Through this API the user will be able to specify a single keypairid to fetch its specific properties, or apikeyfilter to return a specific keypair based on an apikey. The user can inform an userid to fetch an user's api keypair list. If no keypairid, apikeyfilter or userid is provided, the API defaults to fetching information on the calling user. The listall property allows for fetching all keypairs in the structure that are visible based on the calling user/keypair permissions, if not specified, it defaults to false, fetching only the apikeys on the level of the calling user/keypair. Also, it is possible to inform showpermissions to list all permissions associated with each returned apikey.

Name Description Required Default
userid id of the owner of the keypairs no none
keypairid id of the keypair no none
listall list all accessable keypairs no false
apikeyfilter apikey of they keypair no none
showpermissions lists all associated apikey permissions no false

The API getUserKeys was modified preserving backwards compatibility. It now fetches the last keypair created for the informed user.

The api registerUserKeys was modified so the new API keypair parameters could be specified on creation:

Name Description Required Example Default
id user id Yes b8914774-771e-11ee-8e59-5254003754dc none
name name of the keypair No MyKey userId + " - API Keypair"
startdate date when key becomes valid No 2024-04-09 none
enddate date when key expires No 2024-04-09 never expires
description keypair description No read only permissions none
rules list of API access rules No rules[1].rule=list* rules[1].permission=allow all user API permissions based on Account Role

A new keypair deletion API was added (deleteUserKeys). It will accept only one required argument, the keypair id.

Name Description Required
keypairid id of the keypair yes

I also added a listUserKeyRules api, allowing the user to list the rules associated with an API keypair.

Name Description Required
keypairid id of the keypair yes

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

API Key Creation and Basic Testing

Single Key (via UI):

  1. I was able to create API keypairs through the UI through the button on the top right of the user view;
  2. Through Cloud Monkey, validated that the keypair had the same permissions as the base user by calling a series of APIs;

Multiple Keys (via Cloud Monkey):

  1. Created multiple keys;
  2. Validated the operation was successful on the DB;
  3. Tested creating a key that was not valid and would become valid in the future, with success;
  4. Tested creating a key that was valid and would become invalid in the future, with success;
  5. Tested trying to create keys that were already invalid and got errors;
  6. All permissions of the API key pairs were consistent with each key pair tested.

Permissions Validation

Tested the permissions of keyrules listing, keypair listing, keypair deletion and keypair cretion with the following user/account/domain setup:

  • domain /ROOT
    • account root admin

      • user root admin
      • user user1
    • domain subdomain

      • account domain admin
        • user domain admin
      • account userAccount
        • user user2
        • user user3

The following table describes the results obtained when the user on the first column attempted to operate on the keypairs of users on the first row (V: operation was possible, F: operation was not possible).

user rootAdm user1 domainAdm user2 user3
rootAdm V V V V V
user1 V V V V V
domainAdm F F V V V
user2 F F F V F
user3 F F F F V

Migration to api_keypair Table

  1. A key was created for a read-only user
  2. The database was updated from version 4.19 to 4.20
  3. The API key data was successfully migrated to the api_keypair table, with the corresponding columns in the user table deleted.
  4. Confirmed correct permission handling of the api key.

General validations

  • Could not create an API keypair with more permissions than the base keypair.
  • Deleting system keys was not possible.
  • After deleting a user or account, the API keypairs were invalidated.

@codecov
Copy link

codecov bot commented Aug 12, 2024

Codecov Report

Attention: Patch coverage is 16.95958% with 945 lines in your changes missing coverage. Please review.

Project coverage is 16.58%. Comparing base (650b5ec) to head (493852d).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...c/main/java/com/cloud/user/AccountManagerImpl.java 35.19% 173 Missing and 13 partials ⚠️
...he/cloudstack/api/response/ApiKeyPairResponse.java 0.00% 127 Missing ⚠️
...ck/api/command/admin/user/RegisterUserKeysCmd.java 0.00% 92 Missing ⚠️
...n/java/org/apache/cloudstack/acl/ApiKeyPairVO.java 26.40% 92 Missing ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java 0.00% 61 Missing ⚠️
...g/apache/cloudstack/acl/dao/ApiKeyPairDaoImpl.java 0.00% 59 Missing ⚠️
...g/apache/cloudstack/acl/ApiKeyPairManagerImpl.java 0.00% 38 Missing ⚠️
.../cloudstack/discovery/ApiDiscoveryServiceImpl.java 12.82% 32 Missing and 2 partials ⚠️
...dstack/api/command/admin/user/ListUserKeysCmd.java 0.00% 30 Missing ⚠️
...c/main/java/com/cloud/user/dao/AccountDaoImpl.java 0.00% 29 Missing ⚠️
... and 29 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9504      +/-   ##
============================================
+ Coverage     16.53%   16.58%   +0.04%     
- Complexity    13819    13887      +68     
============================================
  Files          5717     5729      +12     
  Lines        506626   508035    +1409     
  Branches      61481    61628     +147     
============================================
+ Hits          83782    84245     +463     
- Misses       413454   414356     +902     
- Partials       9390     9434      +44     
Flag Coverage Δ
uitests 3.96% <ø> (-0.01%) ⬇️
unittests 17.45% <16.95%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland
Copy link
Contributor

nice feature @KlausDornsbach , some suggestions,

  1. would it make sense to be able to delete a kay based on name?
  2. also is admin allowed to delete keys for other users? (would make sense from a maintainance point of view, would it?)

@KlausDornsbach
Copy link
Contributor Author

Hey @DaanHoogland, thanks for taking a look!

It would make sense to be able to delete a keypair by name, we would just need to block users from creating multiple API keypairs with the same name.

At the moment an admin is allowed to delete keypairs from users it has access, for example, a root admin user could delete any keypair in the platform, domain admin users can delete any keypair in the domain, normal users can only delete their own keys. These permissions are also true for visualization and creation APIs.

@DaanHoogland
Copy link
Contributor

It would make sense to be able to delete a keypair by name, we would just need to block users from creating multiple API keypairs with the same name.

Well, I think a unique constraint on UserId/KeyPairName makes sense also from a usability sense.

At the moment an admin is allowed to delete keypairs from users it has access, for example, a root admin user could delete any keypair in the platform, domain admin users can delete any keypair in the domain, normal users can only delete their own keys. These permissions are also true for visualization and creation APIs.

👍

@rajujith rajujith self-assigned this Aug 19, 2024
@rajujith
Copy link

@blueorangutan package

@github-actions
Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10714

@rajujith
Copy link

@blueorangutan package

@blueorangutan
Copy link

@rajujith a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10720

@rajujith
Copy link

@blueorangutan package

@blueorangutan
Copy link

@rajujith a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10736

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@nicoschmdt
Copy link
Collaborator

@blueorangutan package

@blueorangutan
Copy link

@nicoschmdt a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13529

@nicoschmdt
Copy link
Collaborator

@blueorangutan package

@blueorangutan
Copy link

@nicoschmdt a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13551

@nicoschmdt
Copy link
Collaborator

@blueorangutan package

@nicoschmdt
Copy link
Collaborator

@blueorangutan package

@blueorangutan
Copy link

@nicoschmdt a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13578

@nicoschmdt
Copy link
Collaborator

@DaanHoogland could we run the tests again?

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian Build Failed (tid-13460)

@nicoschmdt
Copy link
Collaborator

Hey @DaanHoogland could you send me the log file so that I can identify the error?

@blueorangutan
Copy link

[SF] Trillian test result (tid-13462)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 94922 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9504-t13462-kvm-ol8.zip
Smoke tests completed. 118 look OK, 23 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_2FA_setup Error 18.12 test_2fa.py
test_01_user_access Failure 5.58 test_account_access.py
test_02_domain_admin_access Failure 5.11 test_account_access.py
test_user_cannot_renew_other_keys Error 7.52 test_accounts.py
test_user_key_renew_same_account Error 8.20 test_accounts.py
test_console_endpoint_permissions Error 11.79 test_console_endpoint.py
test_list_vms_metrics_user Error 115.61 test_metrics_api.py
test_nic_secondaryip_add_remove Error 1519.38 test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup Error 0.00 test_nested_virtualization.py
ContextSuite context=TestNetworkACL>:setup Error 0.00 test_network_acl.py
ContextSuite context=TestIpv6Network>:setup Error 0.00 test_network_ipv6.py
test_02_network_permission_on_user_network Failure 3.97 test_network_permissions.py
test_03_network_operations_on_created_vm_of_otheruser Failure 94.54 test_network_permissions.py
test_03_network_operations_on_created_vm_of_otheruser Error 94.55 test_network_permissions.py
test_04_deploy_vm_for_other_user_and_test_vm_operations Failure 0.82 test_network_permissions.py
test_delete_account Error 1518.43 test_network.py
test_delete_network_while_vm_on_it Error 1.21 test_network.py
test_deploy_vm_l2network Error 1.20 test_network.py
test_l2network_restart Error 2.37 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 3.64 test_network.py
ContextSuite context=TestPublicIP>:setup Error 10.43 test_network.py
test_reboot_router Failure 0.10 test_network.py
test_releaseIP Error 5.20 test_network.py
test_releaseIP_using_IP Error 5.54 test_network.py
ContextSuite context=TestRouterRules>:setup Error 5.63 test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1520.84 test_network.py
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
ContextSuite context=TestAdapterTypeForNic>:setup Error 0.00 test_nic_adapter_type.py
ContextSuite context=TestNonStrictAffinityGroups>:setup Error 0.00 test_nonstrict_affinity_group.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup Error 0.00 test_portforwardingrules.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 1529.63 test_projects.py
test_user_userdata_crud Error 12.22 test_register_userdata.py
test_01_sys_vm_start Failure 0.11 test_secondary_storage.py
test_03_scale_down_verify Error 5.49 test_vm_autoscaling.py
test_04_stop_remove_vm_in_vmgroup Failure 13.45 test_vm_autoscaling.py
test_01_register_vnf_template Error 0.66 test_vnf_templates.py
test_02_list_vnf_template Error 0.64 test_vnf_templates.py
test_03_edit_vnf_template Error 0.64 test_vnf_templates.py
test_04_deploy_vnf_appliance Error 0.64 test_vnf_templates.py
test_05_delete_vnf_template Error 0.66 test_vnf_templates.py
test_01_webhook_deliveries Error 12.19 test_webhook_delivery.py
test_04_create_webhook_domainadmin_local Error 12.73 test_webhook_lifecycle.py
test_05_create_webhook_domainadmin_subdomain Error 13.48 test_webhook_lifecycle.py
test_06_create_webhook_domainadmin_global_negative Failure 12.76 test_webhook_lifecycle.py
test_07_create_webhook_user_local Error 11.86 test_webhook_lifecycle.py
test_08_create_webhook_user_domain Error 12.07 test_webhook_lifecycle.py
test_09_create_webhook_user_gloabl Error 11.78 test_webhook_lifecycle.py
test_11_update_webhook Error 11.85 test_webhook_lifecycle.py
test_12_list_user_webhook_deliveries Error 11.64 test_webhook_lifecycle.py
test_13_webhook_execute_delivery Error 12.10 test_webhook_lifecycle.py

@DaanHoogland
Copy link
Contributor

@nicoschmdt , there is now clearly some API that work so the main issue is no longer that the keys can not be found. The zip should contain any logs you need.

[SF] Trillian test result (tid-13462) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 94922 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9504-t13462-kvm-ol8.zip Smoke tests completed. 118 look OK, 23 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_2FA_setup Error 18.12 test_2fa.py
test_01_user_access Failure 5.58 test_account_access.py
test_02_domain_admin_access Failure 5.11 test_account_access.py
test_user_cannot_renew_other_keys Error 7.52 test_accounts.py
test_user_key_renew_same_account Error 8.20 test_accounts.py
test_console_endpoint_permissions Error 11.79 test_console_endpoint.py
test_list_vms_metrics_user Error 115.61 test_metrics_api.py
test_nic_secondaryip_add_remove Error 1519.38 test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup Error 0.00 test_nested_virtualization.py
ContextSuite context=TestNetworkACL>:setup Error 0.00 test_network_acl.py
ContextSuite context=TestIpv6Network>:setup Error 0.00 test_network_ipv6.py
test_02_network_permission_on_user_network Failure 3.97 test_network_permissions.py
test_03_network_operations_on_created_vm_of_otheruser Failure 94.54 test_network_permissions.py
test_03_network_operations_on_created_vm_of_otheruser Error 94.55 test_network_permissions.py
test_04_deploy_vm_for_other_user_and_test_vm_operations Failure 0.82 test_network_permissions.py
test_delete_account Error 1518.43 test_network.py
test_delete_network_while_vm_on_it Error 1.21 test_network.py
test_deploy_vm_l2network Error 1.20 test_network.py
test_l2network_restart Error 2.37 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 3.64 test_network.py
ContextSuite context=TestPublicIP>:setup Error 10.43 test_network.py
test_reboot_router Failure 0.10 test_network.py
test_releaseIP Error 5.20 test_network.py
test_releaseIP_using_IP Error 5.54 test_network.py
ContextSuite context=TestRouterRules>:setup Error 5.63 test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1520.84 test_network.py
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
ContextSuite context=TestAdapterTypeForNic>:setup Error 0.00 test_nic_adapter_type.py
ContextSuite context=TestNonStrictAffinityGroups>:setup Error 0.00 test_nonstrict_affinity_group.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup Error 0.00 test_portforwardingrules.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 1529.63 test_projects.py
test_user_userdata_crud Error 12.22 test_register_userdata.py
test_01_sys_vm_start Failure 0.11 test_secondary_storage.py
test_03_scale_down_verify Error 5.49 test_vm_autoscaling.py
test_04_stop_remove_vm_in_vmgroup Failure 13.45 test_vm_autoscaling.py
test_01_register_vnf_template Error 0.66 test_vnf_templates.py
test_02_list_vnf_template Error 0.64 test_vnf_templates.py
test_03_edit_vnf_template Error 0.64 test_vnf_templates.py
test_04_deploy_vnf_appliance Error 0.64 test_vnf_templates.py
test_05_delete_vnf_template Error 0.66 test_vnf_templates.py
test_01_webhook_deliveries Error 12.19 test_webhook_delivery.py
test_04_create_webhook_domainadmin_local Error 12.73 test_webhook_lifecycle.py
test_05_create_webhook_domainadmin_subdomain Error 13.48 test_webhook_lifecycle.py
test_06_create_webhook_domainadmin_global_negative Failure 12.76 test_webhook_lifecycle.py
test_07_create_webhook_user_local Error 11.86 test_webhook_lifecycle.py
test_08_create_webhook_user_domain Error 12.07 test_webhook_lifecycle.py
test_09_create_webhook_user_gloabl Error 11.78 test_webhook_lifecycle.py
test_11_update_webhook Error 11.85 test_webhook_lifecycle.py
test_12_list_user_webhook_deliveries Error 11.64 test_webhook_lifecycle.py
test_13_webhook_execute_delivery Error 12.10 test_webhook_lifecycle.py

@nicoschmdt
Copy link
Collaborator

@DaanHoogland Thanks! I'll look into it as soon as possible

@github-actions
Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@shwstppr
Copy link
Contributor

Hi @KlausDornsbach do you plan to work on this to resolve conflicts? Do you think this can be included in 4.22?

@DaanHoogland DaanHoogland marked this pull request as draft October 10, 2025 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

9 participants