rptest: increase message count in metrics test

The metrics test checks for a non-zero value for lag. Whether the metric
is non-zero by the time the check occurs depends on whether all messages
have been produced and if the shadow cluster has caught up. This may not
always be the case depending how the test is ran(i.e, in CDT things will
run much faster due to the greater resources).

Before the default max partition bytes was 5MiB in the direct consumer
the shadow cluster could at most consume 1MiB/s from the single
partition topic in the test. This meant that more likely than not the
shadow cluster would still be catching up by the time the metric is
checked. However, not that the default max partition bytes is 5MiB the
throughput limit is no longer the case and the shadow cluster is more
likely than not to be caught up by the time the metric is checked.

This commit increases the messages being produced to be much higher than
before to ensure that the shadow cluster is still catching up by the
time the lag metric is checked. This change also increases the test
runtime from 1min to 2min.

Author: ballard26

tests/rptest/tests/cluster_linking_e2e_test.py

@@ -3347,7 +3347,7 @@ def validate_metrics(
         self.start_producer_consumer(
             topic=topic_3.name,
             msg_size=128,
-            msg_cnt=100000,
+            msg_cnt=5000000,
             use_transactions=True,
             producer_properties={
                 "msgs_per_transaction": "10000",