Incoming Data & Filtering

Every SDK request, exactly as it arrived, plus the rules that throw the unwanted ones away. Incoming Data & Filtering shows the raw requests the server received for a project, what each one carried, and the per-project rules that drop data at the door before anything is counted.

What the request log is

The request log is Countly's record of what an SDK actually put on the wire. Not the events after processing, not the aggregates on a dashboard: the raw request, its query string, its headers and the response the server sent back.

That makes it the one place where "the SDK says it sent it" and "the dashboard does not show it" can be settled. If the request is here, it arrived. If it is not, it never reached this server, and the problem is on the device or in the network rather than in Countly.

Two things follow from being a raw record. It is per project, because a request carries an app key and lands under exactly one of them. And it is short lived: only the newest entries are kept, so this is a tool for looking at what is happening now or what happened minutes ago, never a history you can query later.

It also has to be switched on. Logging every request costs storage and write throughput on a busy server, so a project logs requests only while its Logger setting says so, and the Automatic default stops logging as soon as traffic gets heavy. An empty tab usually means logging is off rather than that nothing arrived.

The second tab, Filtering Rules, is the other half of the same subject. The log tells you what arrived. Filtering rules decide what is allowed to count.

Filtering happens at the door

A filtering rule is not a report filter and it is not a delete. It runs while the request is being received, before anything is processed, aggregated or stored. Data a rule drops was never counted, so it does not appear in any chart, any drill query, any profile or any export, and there is nothing to clean up afterward.

That is the point, and it is also the risk. Filtering is the cheapest way to keep junk out of a project, and it is the only way that leaves no trace to undo. A rule that is too broad quietly loses real data for as long as it is enabled, and no amount of reprocessing brings it back.

There are three kinds of rule and each drops a different amount of the request.

All requests cancels the whole request. Nothing in it is recorded.

Sessions strips the session parts out of a request and lets the rest through. Session begin, duration and end are removed, along with any session events riding in the same request, but events, crashes and profile updates in that request still count.

Events removes matching events from the request and leaves everything else, including the session, intact.

Rules live on the project, so each project has its own set, and they take effect immediately: the next request through the door is judged by them.

The log and the rules meet in one place, but only halfway. A request canceled by an All requests rule is still written to the log, with Canceled giving the ID of the rule that stopped it, so that kind of rule can be confirmed by looking. Session and event rules do not cancel anything, and the entry is written after the rule has already taken its part out, so those entries show what survived rather than what arrived.

Use Cases

Check what the SDK is actually sending

An SDK has just been wired into an app and you need to see the requests it produces before trusting any dashboard built on them.

1. Open Incoming Data & Filtering from the Manage menu, and pick the project the SDK is reporting to.
2. Confirm logging is on. If the tab shows a banner saying request logging is off, follow it into the Logger settings and set the state to Always on for the duration of the test.
3. Trigger the thing you want to see on the device or in the browser: open the app, fire the event, submit the form.
4. Press Live if it is paused, or press it twice to force a fetch now. Entries appear within seconds of arriving.
5. Find your request by device ID. Search matches the device identifier, which is the one value you can predict from the client side.
6. Open the row and read the Request query tab. This is the query string as the server received it, so an event key with a typo, a missing app version or a segmentation value that arrived as a string rather than a number is visible here and nowhere else.
7. Read the Response tab. A success means the server accepted the request. Anything else, and the Problems tab names what the server could not use.

Set the Logger state back to Automatic when you are done. Always on keeps writing a request log for as long as it is set, which costs storage and write throughput on a project with real traffic.

Explain why an event is missing

An event that should be there is not in the dashboard, and you need to know at which point it went missing.

1. Open the Incoming Data tab for the project and set Type to Events.
2. Reproduce the event from the client, then look for its request. If nothing arrives at all, the request never reached this server and the search moves to the client: consent not granted, the SDK not initialized, a wrong app key, an offline queue that has not flushed yet, or a firewall.
3. If the request is there, open it and read the Request query tab. An event only counts if the key is exactly what the dashboard expects, so compare character for character. A key that arrives with a trailing space or different casing is a different event.
4. Check the Problems tab. A request that arrived but could not be processed says so here, for example an unparseable events payload or a plugin that handles this kind of data being switched off.
5. If the request looks correct and has no problems, go to the Filtering Rules tab. A rule is the one thing that can accept a request and still discard its contents. Read every enabled rule of type Events, remembering that a rule can match by exact key, by substring or by regular expression, so a rule aimed at something else can catch your key by accident.
6. Disable the suspect rule, reproduce the event once more, and see whether it lands.

One trap is worth knowing. When an event rule removes an event, the log entry is written afterward, so the entry shows the request without that event rather than showing it being dropped. An event missing from the Request query of a request you are certain contained it is a sign to check the rules rather than the client.

Keep test data out of your analytics

Something is polluting a project's numbers, a staging build, a load test, a QA device, an event the app fires far too often, and you want it to stop counting without asking anyone to change the app.

1. Open the Incoming Data tab first and find one example of the traffic you want gone. You need to know what distinguishes it: an event key, a segmentation value such as a sandbox currency, a device identifier, an address range, a hostname.
2. Switch to the Filtering Rules tab and press Create filtering rule.
3. Choose what to filter. All requests throws the whole request away and is right for a machine or a network that should not be reporting at all. Sessions removes the session parts and keeps the rest, which is what you want when the traffic is real but its session counts are not. Events drops matching events and leaves the session alone.
4. For an event rule, pick how the key is matched. A specific event key is the safest. Contains and regular expression are for families of keys, and both match more than they look like they do.
5. On the second step, decide between Block all and Add condition. Block all drops everything the first step described. A condition narrows it, so the rule only fires when a property or a segmentation value also matches.
6. Create it enabled, then reproduce the traffic and check that the numbers stop moving.

Start narrower than feels necessary. Filtering is not reversible: data a rule drops was never recorded, so a rule that turns out to be too broad has already cost you the days it was enabled. A rule on one exact event key that you widen later is cheap, and a regular expression that matched more than you meant is not.

If the traffic is a whole machine or network rather than a kind of data, an All requests rule with a condition on the address, the hostname or the device identifier is the cheapest shape: the server can judge those from the request alone, so it refuses the request before reading anything else.

Confirm a rule is dropping what you meant

A rule is enabled and you want evidence that it is matching, not just an absence in a chart that could have several causes.

1. Open the Filtering Rules tab and look at Last used. A timestamp is proof the rule matched. A dash is not proof of the opposite: the server only stamps this for All requests rules that it can judge from the request alone, and even then no more often than once every ten minutes.
2. For an All requests rule, go to the Incoming Data tab and open a request you expect it to catch. The entry is written even when the rule cancels the request, and its Canceled field reads Blocked by rule, followed by the ID of the rule that stopped it. That is the clearest confirmation available in the dashboard.
3. For a session or event rule, compare instead of looking for a marker. Note the Types column and the Request query of a request before you enable the rule, enable it, send the same request again, and read the new entry. The session or the event should be absent from the second one.
4. If you need certainty on a session or event rule, turn the API log up. Add block to the modules logged at info level in the server's log settings, then read the API log where your deployment collects it, or ask whoever runs your server to. The rule engine writes a line naming the rule each time it removes something.
5. Turn the log level back down when you have your answer. Info level on the filtering engine on a busy project fills a log file quickly.

Both directions of error are worth ruling out. A rule that never fires is usually a key that does not match exactly, a regular expression that is not matching what you think, or a condition on a property the request does not carry. A rule that fires too much shows up as real data disappearing, which is why it is worth checking a rule in the first days after you create it rather than only when someone complains.

FAQs

The Incoming Data tab is empty. Is nothing arriving?

Check whether request logging is on before anything else. The default setting stops logging once traffic passes its limit and records that by turning the project's setting off, so a project that logged yesterday can be silent today. If logging is on and the tab is still empty, then nothing is arriving for that project and the search moves to the client.

How far back does the log go?

To the thousandth most recent request for that project, whenever that was. On a quiet project that is hours. On a busy one it can be under a minute. There is no date range and no way to ask for more.

Why is the count sometimes above the stated maximum?

Because trimming is a job that runs every five minutes rather than a limit applied as each entry is written. Between runs the count can sit slightly above the ceiling.

Can I search for an event key or something in the payload?

No. Search matches the device identifier only. To find a request by anything else, filter by type to narrow the list and read the entries.

Why does Received differ from the SDK's own timestamp?

Because the SDK recorded the thing at one moment and uploaded it at another. Devices go offline, SDKs batch events, and uploads get throttled. A large gap is worth noticing, because it is the usual reason an event lands on a different day from the one it happened on.

What does a Problem on an entry mean?

That the server accepted the request but could not use part of it. A payload would not parse, a crash report was missing a field it needs, or the payload was for a feature whose plugin is switched off. All three leave the request in the log, and a payload that would not parse or whose plugin is off produces nothing downstream.

Does the log contain credentials?

The cookie header and the Countly token header are blanked before an entry is stored. Everything else the client sent is kept as sent, so treat the request log as sensitive: it holds device identifiers, addresses and whatever the app put in its payloads.

Is turning on Always on safe on a production project?

For a debugging session, yes. As a permanent setting on a busy project, no. Every request becomes a write, which costs storage and throughput for a record that is discarded within minutes.

What is the difference between a filtering rule and deleting data?

A filtering rule prevents data from ever being recorded. Deleting removes something already recorded. A rule cannot clean up the past, and deleting cannot stop the future.

I deleted a rule. Does the data it dropped come back?

No. Dropped data was never recorded, so there is nothing to restore. Removing or disabling a rule only changes what happens to requests from that moment on.

Why does my event rule not seem to fire?

Usually the key does not match as exactly as it looks. Compare the key in the Request query tab against the rule character for character, including case and trailing spaces. If the rule uses a regular expression, test the pattern against the key itself: the form refuses a pattern that is not valid, so a saved one compiles, but it can still match something other than what you meant.

Why is Last used empty on a rule I know is working?

Because that stamp is only written for All requests rules the server can judge from the request alone, and even then no more often than once every ten minutes. A session or event rule can work perfectly and never show a timestamp.

Can a rule keep data instead of dropping it?

No. Every rule is a drop. To keep only some data, describe what you do not want and drop that.

Why can I not use a greater than or a date comparison in a condition?

Because the filtering engine cannot evaluate them, and it drops anything it cannot evaluate rather than letting it pass. Saving a condition with one of those operators would turn a narrow rule into one that drops everything, so the form refuses to save it and names the operator.

Every request for this project has stopped. What did I do?

Look for a red banner on the Filtering Rules tab. An All requests rule set to block all, with no condition, cancels every request for the project. Disabling it resumes ingestion immediately.

Why can my project admin not open this page?

Each tab needs its own permission, and the Filtering Rules tab also needs its plugin enabled on the server. An account can be able to read the request log and still not see or change rules.

Related Resources