HIVE-27126: queue level resource stats for YARN RM.

[architjainjain]

What changes were proposed in this pull request?

This PR adds real-time YARN queue resource metrics display alongside Tez job progress during query execution. Users can now see memory, vCores, capacity utilization, running/pending applications, and allocated/pending containers for the queue being used by their queries.

Key Components:

• YarnQueueMetricsCollector: Collects queue metrics from YARN ResourceManager
• QueueMetricsCache: JVM-wide shared cache with expiry-based refresh (fetches only when cached data expires)
• QueueMetricsRefreshPool: Scheduled executor pool for non-blocking metric collection
• TezProgressMonitor: Enhanced to display queue metrics alongside task progress

What is NOT Changed:

• Tez query execution logic remains unchanged
• No modifications to YARN ResourceManager communication protocols
• DAG submission, task scheduling, and container allocation logic untouched
• Existing progress monitoring and logging infrastructure preserved

Technical Highlights:

• Queue-level RM API calls: Single ResourceManager call per queue (not per session), only when cache expires
• Shared caching: Multiple sessions using the same queue share cached metrics, reducing RM load
• Dynamic refresh scheduling: Cache automatically adjusts to the lowest refresh interval among all active sessions for a queue
• Jitter: Random 0-10% delay added to refresh intervals to prevent thundering herd when multiple queues expire simultaneously
• Circuit breaker: Automatic fallback to no-op collector after 3 consecutive RM failures, preventing cascading failures

Configuration:

• hive.tez.queue.metrics.refresh.interval: Per-session refresh interval (default: 0s)
• hive.server2.tez.queue.metrics.refresh.threads: Pool size for metric collection (default: 4)

Important Note: An HTTP exporter and Prometheus/Grafana integration were implemented as a local testing setup to validate the feature during development. These components are NOT part of this PR, NOT included in production code, and were only used in a local Docker environment for validation purposes.

Why are the changes needed?

Currently, when a Hive query is slow or stalled, users cannot determine if the issue is due to insufficient queue resources or other factors. They see Tez task progress (e.g., "Map 1: 3/10") but have no visibility into whether the queue has available memory/vCores to execute tasks in parallel.

This enhancement provides real-time queue resource information, enabling users to:

• Identify resource bottlenecks immediately (e.g., queue at 95% memory usage)
• Distinguish between slow queries due to lack of resources vs. data processing overhead
• Make informed decisions about queue selection or query timing
• Understand if pending containers are waiting due to queue capacity limits

Backward Compatibility:

• Fully backward compatible - feature is opt-in via configuration (disabled by default in this implementation, can be enabled by setting refresh interval)
• When disabled (interval=0s), behavior is identical to previous Hive versions
• No changes to existing APIs, query execution, or Tez integration
• No impact on existing queries, scripts, or workflows

Performance Impact:

• Minimal overhead: Shared cache with single RM call per queue (not per session)
• Non-blocking: Metrics collected in background thread pool
• Graceful degradation: Circuit breaker prevents cascading failures
• Zero impact when disabled

Does this PR introduce any user-facing change?

Yes. When hive.tez.queue.metrics.refresh.interval is set to a positive value (default: 0s), users will see queue-level metrics displayed with Tez job progress:

In-place mode (hive.server2.in.place.progress=true):

Map 1: 3/10   Reducer 2: 0/5
QUEUE: default
MEMORY: 1.5/5.4 GB (27.78% used) | VCORES: 3/7 (42.86% used)
CAPACITY: 46.30% (used), 60.00% (allocated)
APPS: 1 running, 0 pending | CONTAINERS: 4 allocated, 0 pending

Log file mode (hive.server2.in.place.progress=false):

INFO  : Map 1: 3/10   Reducer 2: 0/5    QUEUE: default | MEMORY: 1.5/5.4 GB (27.78% used) | VCORES: 3/7 (42.86% used) | CAPACITY: 46.30% (used), 60.00% (allocated) | APPS: 1 running, 0 pending | CONTAINERS: 4 allocated, 0 pending

When disabled (set hive.tez.queue.metrics.refresh.interval=0s):
No queue metrics are displayed, behavior remains identical to previous versions.

How was this patch tested?

The patch was tested in a multi-node YARN cluster environment with concurrent queries running on different queues.

Testing included:

1. Functional testing: Queue metrics displayed correctly with various refresh intervals (1s, 5s, 10s)
2. Feature toggle testing: Metrics disabled when refresh interval set to 0s
3. Multi-session testing: Multiple concurrent sessions with different queues and refresh intervals
4. Dynamic scheduling: Verified cache adjusts to lowest interval when sessions with different settings share a queue
5. Cross-mode testing: Verified consistent output in both in-place and log file modes
6. Performance testing: Validated minimal overhead with shared cache and background refresh pool
7. Efficiency validation: Confirmed single RM API call per queue (not per session) only on cache expiry
8. Jitter behavior: Verified 0-10% randomization prevents simultaneous refresh across queues
9. Circuit breaker: Validated automatic failover to no-op collector after consecutive RM failures

Test Environment:

• YARN cluster with multiple queues (default, analytics, batch)
• HiveServer2 with Tez execution engine
• Concurrent query workloads

Validation artifacts (local testing setup only - NOT in production):

• Prometheus exporter and Grafana dashboards were implemented in a local Docker environment to validate metric accuracy, cache behavior, and refresh scheduling
• Multi-session test scripts for concurrent scenario validation
• These testing tools are separate from the production code and not included in this PR

Screenshots:

1. Terminal output with queue metrics enabled (hive.tez.queue.metrics.refresh.interval=10s):

2. Terminal output with queue metrics disabled (hive.tez.queue.metrics.refresh.interval=0s):

3. Grafana dashboard showing backend metrics validation (local testing setup only):

Note: The Grafana and Prometheus infrastructure shown above was created exclusively for local testing and validation purposes. This monitoring stack is NOT part of the production code, NOT included in the deployment, and was only used in a local Docker environment to validate the feature's correctness.

Configuration tested:

<!-- Enable queue metrics with 10-second refresh -->
<property>
  <name>hive.tez.queue.metrics.refresh.interval</name>
  <value>10s</value>
</property>

<!-- Disable queue metrics -->
<property>
  <name>hive.tez.queue.metrics.refresh.interval</name>
  <value>0s</value>
</property>

[abstractdog]

thanks @architjainjain so far, dropped some comments, but I wasn't able to fully read it through, I'll get back after, in the meantime I can do some testing too hopefully

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds optional, real-time YARN queue resource metrics to the Tez progress display during Hive query execution, propagating the data through HS2/Thrift and Beeline in-place updates.

Changes:

• Introduces YARN queue metrics collection (collector + per-queue state + shared cache + scheduled refresh pool with jitter/circuit breaker).
• Integrates queue metrics into Tez progress rendering (in-place + log-to-file) and publishes it over Thrift (TProgressUpdateResp).
• Adds unit tests covering cache/state/pool behavior and Tez monitor formatting/initialization.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
service/src/java/org/apache/hive/service/server/HiveServer2.java	Initializes the shared refresh pool during HS2 Tez session pool startup.
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java	Populates new Thrift queueMetrics field and applies Thrift “isset” workaround for startTime.
service/src/java/org/apache/hive/service/cli/JobProgressUpdate.java	Adds queue metrics string to progress update model.
service-rpc/if/TCLIService.thrift	Adds optional queueMetrics to TProgressUpdateResp.
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java	Implements queueMetrics() default for the progress monitor facade.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/QueueMetricsCollector.java	New interface for queue metrics collectors.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/NoOpQueueMetricsCollector.java	Null-object collector when feature is disabled.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/QueueMetricsSnapshot.java	Immutable snapshot of queue resource stats fetched from RM.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/QueueMetricsState.java	Per-queue state: interval registration, scheduling, refresh lock, circuit breaker.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/QueueMetricsCache.java	JVM-wide Guava cache mapping queue name → state with expiry.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/QueueMetricsRefreshPool.java	JVM-wide scheduled executor singleton to run refresh tasks (with jitter).
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/YarnQueueMetricsCollector.java	Active collector that coordinates with shared cache/state and refresh pool.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java	Formats and returns multi-line queue metrics for progress rendering.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java	Creates/shuts down the metrics collector based on config and wires it into progress monitor.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java	Appends queue metrics into log-to-file progress report (single-line rendering).
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java	Adds per-session YarnClient lifecycle to support collector RM queries.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java	Exposes YarnClient via pooled session wrapper.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java	Extends TezSession interface with getYarnClient().
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java	Adds new configuration keys for refresh interval and refresh thread pool size.
common/src/java/org/apache/hadoop/hive/common/log/ProgressMonitor.java	Adds queueMetrics() to the ProgressMonitor contract.
common/src/java/org/apache/hadoop/hive/common/log/InPlaceUpdate.java	Renders queue metrics block (multi-line) under the standard progress output.
beeline/src/java/org/apache/hive/beeline/logs/BeelineInPlaceUpdateStream.java	Reads queueMetrics from Thrift progress updates for in-place rendering.
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/monitoring/yarnqueue/TestQueueMetricsState.java	Unit tests for per-queue state logic (intervals, refresh lock, circuit breaker).
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/monitoring/TestTezProgressMonitorQueueMetrics.java	Unit tests for queue metrics formatting and edge cases in TezProgressMonitor.
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/monitoring/TestTezJobMonitorQueueMetrics.java	Unit tests for TezJobMonitor collector initialization decisions.
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestYarnQueueMetricsCollector.java	Unit tests for collector/cache/pool integration behavior and circuit breaker.
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestQueueMetricsRefreshPool.java	Unit tests for refresh pool singleton, scheduling, and jitter determinism/range.
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestQueueMetricsCache.java	Unit tests for cache placeholder/put semantics and concurrency behavior.
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestNoOpQueueMetricsCollector.java	Unit tests for NoOp collector behavior and singleton semantics.

Files not reviewed (1)

• service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProgressUpdateResp.java: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[abstractdog]

thanks @architjainjain for the patch so far, left comments

[abstractdog]

thanks @architjainjain for the patch so far, left comments

[architjainjain]

thanks @architjainjain for the patch so far, left comments

I will look into this and provide fix for the review comments added.

[architjainjain]

@abstractdog All commnets are addressed and updated/added/removed test case as per the changes.
Thanks for the review.

[abstractdog]

thanks a lot @architjainjain for taking care of my comments, only a few of them left (I resolved the rest)
please take a look at the latest SonarQube warning if any of them can be fixed in a convenient way (it's not the case always :) )

[architjainjain]

@abstractdog all comments addressed and sonar issues solved. Please review and approve

[abstractdog]

@architjainjain , minor nits, sorry for those :) rest LGTM

[abstractdog]

nit: "particularly focusing on separator line positioning when queue metrics are displayed." remove this as it's going to get outdated silently; TestInPlaceUpdate makes the reader think it's generic and tests InPlaceUpdate, even if it's only testing the queue metrics scenario at the moment

[architjainjain]

done

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud