Health Manager

Whether the pipeline behind your reports is keeping up. Health Manager shows queued data changes, Kafka consumer lag, ingestion connector state, and the consumer events worth investigating.

What Health Manager watches

Health Manager is the operator's view of the machinery between an SDK request and a report. Everything on this page is about the server itself, not about one project, so the numbers do not change when you switch the project in the top bar.

It answers three questions. Is data arriving and being written. Is anything falling behind, and by how much. Did a requested change to stored data actually reach every place that data lives.

The page is admin only and refreshes itself every 30 seconds while the browser tab is in front. When the tab is in the background the polling stops, so a laptop left open overnight does not keep querying the server.

The pipeline behind a report

An event travels through four stages, and Health Manager gives each one a tab.

Ingestion. Kafka Connect writes what the SDKs sent into the datastore. The Ingestion Status tab shows the sink connectors, their tasks, and how far behind they are.

Aggregation. Consumer groups read the raw stream and build the aggregates that reports read. The Aggregator Status tab shows their lag, their batches, and how often they have been reshuffled.

Consumer events. When a consumer notices something it cannot silently absorb, a cluster id that changed or an offset that jumped backward, it records an event. The Kafka Events tab is the log of those.

Mutation. A delete or update of already stored data cannot happen in one write, because the same records live in Mongo and in ClickHouse. Requests are queued and worked through by a background job. The Mutation Status tab is that queue.

Lag in an early stage shows up as staleness in every later one, so the tabs are worth reading in pipeline order when something looks wrong.

Use Cases

Find out why the numbers look behind

Someone reports that today's numbers stopped moving, or that a report is behind by an hour.

1. Open Ingestion status. If Connectors or Tasks show anything other than a full count, the pipeline has stopped at its last stage and nothing further along will catch up until the connector is running again.
2. Read Sink lag on the same tab. A large and growing figure means data is arriving faster than it is being written.
3. Open Aggregator status and read Total lag. This is the earlier stage. Lag here explains stale aggregates even when the sink looks healthy.
4. Open the Consumer groups table and find which group carries the lag. One group behind means one report family is stale. Every group behind points at the cluster rather than at a consumer.
5. Check Rebalances and Errors for that group. Both climbing at once usually means a consumer is crash looping rather than merely running slowly.

If every stage is healthy and the numbers are still behind, the delay is downstream of this page, in caching or in the report's own date range.

Confirm a deletion reached every datastore

A deletion or a bulk property update was requested, and someone needs to know whether it finished. Countly queues these instead of applying them inline, so "the request was accepted" and "the data is gone" are different moments.

1. Open Mutation status and read the Status summary bar. Queued and running counts tell you the work is still in progress.
2. Open the Mutation records table and sort by Created to find the request. The Type, Collection, and Query columns identify it.
3. Follow its Status. A record reaches completed only after both the Mongo write and, when ClickHouse is enabled, the matching ClickHouse mutation have been validated.
4. If the status is failed, read the Error column and the Retries count. A record at its retry ceiling will not be retried automatically and needs the request to be reissued.

A red banner at the top of the tab appears whenever any record has exhausted its retries, so a failure is visible without opening the table.

Check the pipeline after a deploy

After a server restart, a version upgrade, or a change to the Kafka cluster, the pipeline recovers on its own. This is how to confirm it did.

1. Open Kafka events and set the type filter to All types. An upgrade can produce STATE_MIGRATED entries, and those are expected once.
2. Look for CLUSTER_MISMATCH or OFFSET_BACKWARD. Either one means the consumers found stored state that no longer matches the cluster they are reading, which is worth understanding before it repeats.
3. Open Aggregator status and check that Rebalances has settled. A short burst during a restart is normal, a number that keeps climbing is not.
4. Open Ingestion status and confirm Connectors and Tasks are both at their full count, then watch Sink lag drain over a few refresh cycles.

Lag right after a restart is expected. Lag that is still growing several minutes later is the signal.

Track down repeated consumer rebalances

Rebalances pause processing while partitions are reassigned. A few are routine. A group that rebalances continuously never gets far enough through a batch to commit, and its lag grows even though nothing has errored.

1. Open Aggregator status and read the Rebalances card. The tab flags it once the count is above a handful.
2. Open the Consumer groups table and compare Rebalances and Last rebalance across groups to find the one that is churning.
3. Read Duplicates skipped for that group in Partition stats. Duplicates rise with rebalances, because a reassigned partition replays a little of what the previous consumer had already handled. Deduplication catching them is the system working, not a fault.

The usual causes are a consumer that is being restarted repeatedly, or one whose processing exceeds the session timeout so the cluster considers it dead.

Recover an event the sink could not store

An event never arrived in Drill and the project's numbers are short. Nothing failed loudly, so the pipeline pages all read healthy.

Open Health Manager, then Dead-letter queue. Filter Status to pending and read the reason column.

Expect a short list. The healthy records that were set aside alongside a bad one are re-inserted automatically, so what remains is the records ClickHouse refuses on their own. A single bad value shows up once, not as the hundreds of records that happened to share its batch.

Expand a record to see the payload and the error. Read the one from the record's own insert rather than the connector's: the connector reports what it saw when it tried the whole batch, which can name a value from a different record. A number past the range ClickHouse stores, a text value where a number belongs, and a timestamp outside the supported range are the three that come up most.

Fix the value in the editor and replay. The record goes back through the same validation live ingestion uses, and is then written straight to ClickHouse, so you find out immediately whether the edit worked. A payload ClickHouse still refuses leaves the record pending for you to try again rather than disappearing.

Discard the records that should not be recovered, such as load-test traffic or a payload the SDK has since stopped sending, and leave a note saying why. Discarded records stay listed until they expire, so the note is what tells the next reader the queue was triaged rather than ignored.

Fix the cause at the source as well. The SDK that sent the value will keep sending it until the app changes, and every new one costs its whole batch a round of re-insertion before it lands here again.

FAQs

Why do the numbers not change when I switch project?

They are server wide. Health Manager reports on the pipeline that serves every project on this deployment, so there is nothing to scope to one of them.

Why is there no ClickHouse capacity card on Mutation status?

That card, and the awaiting validation status, only apply where ClickHouse is enabled. Without a second datastore there is nothing to reconcile, so the stage does not exist.

Why does Aggregator Status say there are no Kafka consumer stats yet?

None of the aggregator's consumers has reported yet. That is normal for a few moments after the aggregator starts. If it lasts, check that the aggregator is running and can reach Kafka.

Is lag after a restart a problem?

Not by itself. Consumers pick up from their last committed offset and work through the backlog, so lag rises and then drains. Lag that is still growing several minutes later is the signal, especially alongside a rising rebalance count.

Can I export what I am looking at?

Mostly. Every table on the Mutation, Aggregator and Ingestion tabs has an export in its card header, as CSV, XLSX or JSON, under a filename you choose. The Aggregator tab exports whichever of Consumer Groups or Partition Stats is showing. The Kafka events list has no export, and the Dead-letter queue has none on purpose, because it holds raw payloads from every project.

Related Resources