Background Tasks

The work the server does for itself, on a schedule, and the controls for it. Background Tasks lists every job the server runs in the background, shows when each one last ran and whether it succeeded, and lets an administrator reschedule it, run it now or switch it off.

What a job is

Most of what Countly shows you is not computed when you look at it. Session aggregates, retention matrices, cohort membership, push delivery, drill caches, symbolication: all of it is produced ahead of time by background work, on a schedule, by a part of the server that has no user interface of its own.

A job is one of those pieces of work. It has a name in two parts, the area it belongs to and what it does, like sessions:aggregate_sessions. It has a schedule written as a cron expression. It runs, it succeeds or it fails, and it records what happened.

Jobs are not something you create. They exist because a plugin's code declares them, so the list on this page is the set of jobs the running server actually has. Installing a plugin adds its jobs; removing one takes them away. What this page gives you is control over work that already exists: when it runs, whether it runs at all, and whether to run it right now.

That control is server wide and it is powerful, which is why it is restricted to global administrators. A job that is switched off stops producing whatever it produced, and nothing in the product will tell the people relying on that data why their numbers stopped moving.

The page sits inside Background Tasks, alongside three other tabs covering long running queries, exports and snapshots. Those are individual pieces of work somebody asked for. Jobs are the recurring work nobody asked for, because the server needs it.

Two kinds of run

A job runs in two different ways, and the difference explains almost everything about how its history is recorded.

Scheduled runs happen because the cron expression came due. The server keeps one record per job for this, and that record is overwritten each time: when it last started, when it last finished, whether it failed, and how many times it has run in total. There is no per run entry, so there is no list of past scheduled runs to read. The job holds its latest state and a counter, and that is all.

Runs you trigger are different. Pressing Run now queues a separate one off run, and that run gets a record of its own that survives afterward. This is why the detail page has a table of manual runs with their results, and why it has no equivalent table of successful scheduled runs.

Failures are the exception to both. Every failed run, scheduled or triggered, is written to a separate history with its error message. That is the one complete record on the page: you can see every failure a job has ever had, but only the most recent success.

Run now is also less immediate than it sounds. It asks the server to queue the job rather than executing it in the browser's request, so the run starts when a worker picks it up. On a busy server that is usually seconds, and the job's detail page shows Running once it starts.

One more thing follows from all this. Several server processes can run jobs, and they coordinate by taking a lock on the job while it runs, so runs of the same job do not trample each other. A process that dies mid run leaves a lock that expires rather than a job stuck forever, and a run that hangs is failed once it passes its own time limit.

Use Cases

Find out why something is not updating

A dashboard, a cohort count or a retention chart has stopped moving, and you suspect the work behind it rather than the data going in.

  1. Open Background Tasks from the Manage menu. The Jobs tab lists every job on the server.
  2. Search for the area. Job names start with the feature they belong to, so retention, cohorts or sessions narrows the list to the handful worth reading.
  3. Read the Status column first, or choose Disabled from the All jobs menu above the table to list only those. Disabled is the answer more often than anything else, and it means somebody switched the job off, possibly a long time ago.
  4. Read Last run and Last run status. A last run from three days ago on a job scheduled hourly means it has been failing or not being picked up, and a Failed status says which.
  5. Open the job by clicking its name. The detail page carries the error message from the last failure and the full list of every failure before it.
  6. Read the failure reasons rather than counting them. The same message repeating is one problem; messages that change are usually a symptom of something else, such as the database being unreachable at particular times.

If the job looks healthy and the numbers still have not moved, the problem is upstream of it. Check that the data is arriving at all before coming back here.

Run a job now instead of waiting

Something needs to be recomputed before its next scheduled run, usually because you have just fixed the reason it was wrong.

  1. Find the job in the list, or open its detail page.
  2. Confirm it is enabled. Run now on a disabled job is not the way to bring it back; enable it first, so it keeps running afterward.
  3. Choose Run now from the row menu, or from the menu on the detail page.
  4. Wait on the detail page, which refreshes itself every ten seconds. The run is queued rather than executed on the spot, so Last status changes to Running when a worker picks it up and to Completed or Failed when it ends.
  5. Read the result on the detail page. The run appears in the manual run table with its duration and, when the job returns one, the result it produced.

Two things to know before you press it. A triggered run does not replace the next scheduled one, so the job will still run on its own schedule afterward. And a job that is already running may hold the new run back until the current one finishes, because each job declares how many of its runs may overlap and most allow only one.

Think about cost on a busy server. Some of these jobs read the whole of a project's data, and running several by hand at once competes with the traffic coming in.

Change how often a job runs

A job's default schedule does not suit this server, usually because it is too frequent for the load or not frequent enough for the freshness people expect.

  1. Open the job's detail page and read Base configuration first. That is the schedule the job ships with, written out in words underneath, and it is what you are departing from.
  2. Choose Change schedule from the actions menu.
  3. Type a cron expression. The field reads it back to you in plain language as you type, so you can check the meaning without running anything, and it refuses to save an expression it cannot parse.
  4. Save. The new schedule takes effect without restarting anything, and the next run time updates to match.
  5. Check the detail page again. Your expression now appears as a Schedule override alongside the base schedule, which is how the page distinguishes what you set from what the job came with.

Aim below the job's own running time, not just at the interval you want. A job scheduled every five minutes that takes seven will not overlap, because it locks itself, but it will fall behind steadily and the schedule stops meaning anything.

An override is a departure from the product's default, so write down why you made it. There is nowhere in the product to record the reason, and a schedule nobody can explain tends to be reverted by the next person.

Turn a job off

A job is hurting the server, or producing something this deployment does not use, and you want it to stop.

  1. Find the job and confirm what it feeds. The name tells you the area; the feature that owns it is where the effect will show. Switching off a sessions job stops session aggregates, not just the job.
  2. Check whether rescheduling would do instead. Most of the time the problem is frequency rather than the work itself, and a longer interval keeps the data coming.
  3. Choose Disable from the row menu or the detail page menu.
  4. Confirm the Status column now reads Disabled.
  5. Tell whoever depends on it. Nothing in the product warns anyone that the numbers they read have stopped being produced.

Disabling stops future runs; it does not stop a run already in progress, and it does not remove anything the job produced. Re enabling it starts the schedule again from the next due time, and the gap while it was off is not backfilled unless the job itself catches up.

A job disabled for a temporary reason is worth a calendar reminder. The switch stays where you left it through restarts, and through upgrades unless the upgrade changes the job itself, which puts it back to its default. Even so, a job switched off during an incident two years ago is a common thing to find on an old server.

FAQs

What are these jobs and did I create them?

No. Jobs are declared by the code the server runs, so the list is whatever the installed plugins bring with them. This page controls them; it does not define them.

I pressed Run now and nothing happened.

The run is queued rather than started on the spot, so give it a few seconds and watch the job's detail page, which refreshes itself. The list does not show triggered runs, so it will not change. If the detail page still has not changed, check whether the job is already running, since a job that allows only one run at a time holds a second run back until the first finishes.

Why can I see every failure but not every success?

Because they are recorded differently. Each failure is written to its own history with its error message. Successes update the job's single record in place, so only the most recent one survives.

The manual run table is empty. Does that mean the job never runs?

No. That table holds only the runs somebody started with Run now. A job running perfectly on its schedule will have an empty table there.

A job says Disabled but its last run succeeded. Which is right?

Both. Status is the switch and describes the future; Last run status is the outcome of something that already happened. A job switched off yesterday will show Disabled and a successful run from before that.

Last run status is a dash. What does that mean?

That the job has never run, so there is no outcome to report. Next run tells you when it first will.

What time zone is the schedule in?

The server's. There is no per user time zone here, so an expression that says two in the morning means two in the morning where the server is.

Can I schedule a job to run once at a particular time?

Not from this dialog. It takes a repeating cron expression only.

I changed a schedule and after an upgrade it was back to the original.

That is expected behavior rather than a fault. When a job's implementation changes, the server resets that job's configuration to the new defaults, including whether it is enabled. Overridden schedules and disabled jobs are worth re checking after any upgrade.

How do I put a schedule back to the default?

By typing the default expression yourself. There is no reset, which is why the base schedule on the detail page is worth noting before you change it.

Is it safe to disable a job?

It is safe for the server and not necessarily safe for the people using the feature it feeds. Whatever the job produces stops being produced, with no warning anywhere in the product. Reschedule rather than disable when the problem is load.

A job keeps failing with a message about exceeding a time limit.

That is a run that never finished. Each job has a limit on how long one run may take, and a run that passes it is failed deliberately so that it stops holding its slot. The usual causes are more data than the job was written for, or something it depends on being unavailable.

Where do I see the actual work a job did?

Only where the job reports one. A triggered run shows whatever result the job returned, in the manual run table. Most jobs return nothing, and their real output is the data they wrote, which you check in the feature that owns them.

Why can my project admin not use these controls?

Because a job runs for the whole server rather than for one project, so changing one affects every project at once. The page and its controls are restricted to global administrators.

Related Resources

Was this page helpful?
Reach out to us for any other questions.
Helpful?

Looking For More Help?