Fixing recording of query to external db sources like MySQL or InfluxDB #61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When running with concurrent streams, not all queries were recorded successfully to the database like MySQL due to the timeToExit signal channel being received first which would prematurely end the loop that records the query to the database even though that resultChan is not done receiving all the queries report yet. Fix was to close the resultChan and modify the loop to allow for full processing of the resultChan even if the timeToExit signal channel was received.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixing recording of query to external db sources like MySQL or InfluxDB.
When running with concurrent streams, not all queries were recorded successfully to the database like MySQL due to the timeToExit signal channel being received first which would prematurely end the loop that records the query to the database even though that resultChan is not done receiving all the queries report yet.
Fix was to close the resultChan and modify the loop to allow for full processing of the resultChan even if the timeToExit signal channel was received.
Example:
Ran pbench locally.
100 streams of 70 simple queries = 7000 queries expected to be reported
only 6984 queries were reported in the queries_ran column
https://grafana.ibm.prestodb.dev/d/cf1f9e9c-9c17-4f82-804c-4cdc99e14f37/run-comparison?orgId=1&var-base_run=mcao_pbench_main_stream_test_biday3_sf1_250909-104401&var-target_run=None&var-query_idx=0&from=now-30d&to=now&refresh=5s
NOTE:
For some reason, when running the pbench jenkins pipeline, it does accurately reports the number of queries ran.
https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-pbench-ibm-concurrency-testing/21/
https://grafana.ibm.prestodb.dev/d/cf1f9e9c-9c17-4f82-804c-4cdc99e14f37/run-comparison?orgId=1&var-base_run=mcao_biday3_sf1_prestissimo_100streams_run_simple_test_for_allen_pbench_no_random_dup_250908-211518&var-target_run=None&var-query_idx=0&from=now-30d&to=now&refresh=5s