Observing your App

Whether your app is running as expected or not, there are various ways to find out what is wrong.

Logs

To view logs, you can either use Cockpit or nctl. Various components of Deploio can emit logs:

• Build: these are logs produced by the build process. Useful if there's an issue with building your app.
• App: these logs contain what the app replicas are outputting to stdout/stderr during runtime.
• Worker Job: these logs contain what the worker job is outputting to stdout/stderr during runtime.
• Deploy Job: these logs contain what the deploy job is outputting to stdout/stderr during runtime.
• Scheduled Job: these logs contain what the scheduled job is outputting to stdout/stderr during runtime.

The logs are retained for 30 days.

Events

Besides the output of your app, the logs contain events about the replicas of your app, for example when a replica is restarted or a health probe fails. They help you find out why your app doesn't start or keeps restarting. Event lines are labeled with source="event".

$ nctl logs app go --type event
2026-11-09T10:12:03+01:00 {replica="go-59dcf7656f-9wr52", source="event"} failed to start container go: command "start.sh" not found
2026-11-09T10:12:05+01:00 {replica="go-59dcf7656f-9wr52", source="event"} back-off restarting failed container go

Some events tell you what to do, for example:

• failed to start container <name>: command "<command>" not found: the command of your app doesn't exist in the image.
• failed to start container <name>: command "<command>" is not executable: for scripts, make sure that the executable bit is set, for example with chmod +x.
• Container <name> was killed because it ran out of memory ...: the replica used more memory than its limit. Reduce the memory usage of your app or choose a bigger size.
• back-off restarting failed container <name>: the container exited and is restarted with an increasing delay. The log lines of the replica before this event usually show why it exited.
• <probe> probe failed for container <name>: a health probe of the replica failed. See Health Probe.

Build logs only contain the events which explain a failed or stuck build, for example when the build ran out of memory. Scheduled jobs only show when they were started and their failures.

The timestamps of events only have a precision of seconds. Events can therefore appear slightly out of order compared to each other and to the log lines of your app within the same second.

Log Line Format

Each line your app writes to stdout or stderr must end with a newline character (\n). Deploio treats output without a trailing newline as an incomplete line and holds it back until the line is completed. As a result, the output can show up in the logs much later, or not at all.

This mostly affects short-lived processes such as scheduled jobs or deploy jobs that write their last line without a newline and then exit.

Common causes are:

• printf "done" instead of printf "done\n" in shell scripts
• echo -n
• process.stdout.write("done") in Node.js
• print("done", end="") in Python
• $stdout.print "done" in Ruby

Most logging libraries add the newline for you.

Examples

Display all logs of an app. These examples all use an app named go:

$ nctl logs app go
2024-10-22T16:07:55+02:00 {build="go-build-1", replica="go-build-1-build-pod"} Build successful
2024-10-22T16:07:59+02:00 {replica="go-59dcf7656f-9wr52"} 2024/10/22 14:07:59 starting HTTP server on :5678

The labels in between the curly braces tells you which component each log line belongs to. The labels can be turned off with the option --no-labels.

Display only the main app logs.

nctl logs app go --type app

Display worker job logs of an app.

nctl logs app go --type worker_job

Display deploy job logs of an app.

nctl logs app go --type deploy_job

Display scheduled job logs of an app.

nctl logs app go --type scheduled_job

Display only the events of an app.

nctl logs app go --type event

Display all build logs of an app.

nctl logs app go --type build

Alternatively, you can also view the logs of a specific build using the logs build command.

nctl logs build go-build-1

You can add the -f flag to keep following the logs as they are emitted.

nctl logs app go -f

To get more lines, use the -l/--lines flag.

nctl logs app go-example -l 1000

To go further back in time, use the -s/--since flag. Durations can be specified using 1s (1 second), 1m (1 minute) and 1h (1 hour).

nctl logs app go-example -s 48h

Stats

To get an overview of the app replicas, their status and resource usage you can use the -o stats flag for the nctl get app command.

$ nctl get app rails -o stats
PROJECT        NAME    REPLICA                  STATUS   CPU   CPU%   MEMORY   MEMORY%   RESTARTS   LASTEXITCODE
nine-staging   rails   rails-9f6bdb676-crlrf    ready    1m    0.8%   121MiB   47.4%     0          <none>
nine-staging   rails   rails-9f6bdb676-2pfps    ready    1m    0.8%   123MiB   48.4%     0          <none>
nine-staging   rails   rails-9f6bdb676-sd9mm    ready    1m    0.8%   122MiB   48.0%     1          137 (Out of memory)

If you have worker jobs defined, they will also show up here. The help output of nctl get app -h describes all the columns:

REPLICA: The name of the app replica.
STATUS: Current status of the replica.
CPU: Current CPU usage in millicores (1000m is a full CPU core).
CPU%: Current CPU usage relative to the app size. This can be over 100% as Deploio allows bursting.
MEMORY: Current Memory usage in MiB.
MEMORY%: Current Memory relative to the app size. This can be over 100% as Deploio allows bursting.
RESTARTS: The amount of times the replica has been restarted.
LASTEXITCODE: The exit code the last time the replica restarted. This can give an indication on why the replica is restarting.

Metrics

Some basic application metrics like CPU and RAM usage can be found directly in Cockpit.

If you want more detailed metrics and want to visualize them with your own dashboards, install a Grafana instance to access them.

Dashboard

As soon as your organization has at least one Deploio app, we install a Deploio dashboard into your Grafana instance. It shows the CPU and memory usage, the replicas, the restarts and the HTTP requests of your apps.

Available Metrics

All metrics have an app label with the name of the app and a nine_project label with the name of its project. The metrics that are labeled by replica contain one series per app replica.

The CPU usage metrics, the HTTP request metrics and the metrics ending in _total are counters. Use them with rate() or increase().

App

Metric	Labels	Description
application_cpu_usage_seconds	replica	CPU time used by the replica, in seconds.
application_cpu_limit_cores	replica	CPU limit of the replica, in cores.
application_memory_usage_bytes	replica	Memory used by the replica, in bytes.
application_memory_limit_bytes	replica	Memory limit of the replica, in bytes.
application_restarts_total	replica	Number of times the replica has been restarted.
application_replicas_desired		Number of replicas the app should have.
application_replicas_ready		Number of replicas that are ready.
application_replicas_total		Number of replicas that currently exist, including the ones that are not ready yet.

HTTP

These metrics are measured at the load balancer in front of your app.

Metric	Description
application_http_requests_total	Number of HTTP requests to the app.
application_http_requests_2xx	Number of HTTP requests answered with a 2xx status code.
application_http_requests_3xx	Number of HTTP requests answered with a 3xx status code.
application_http_requests_4xx	Number of HTTP requests answered with a 4xx status code.
application_http_requests_5xx	Number of HTTP requests answered with a 5xx status code.
application_http_response_duration_avg	Average response time of the last 1024 requests, in seconds.
application_http_response_duration_max	Maximum response time since the load balancer was last restarted, in seconds.

Worker Jobs

All worker job metrics have a worker_job label with the name of the worker job.

Metric	Labels	Description
worker_job_cpu_usage_seconds	replica	CPU time used by the worker job replica, in seconds.
worker_job_cpu_limit_cores	replica	CPU limit of the worker job replica, in cores.
worker_job_memory_usage_bytes	replica	Memory used by the worker job replica, in bytes.
worker_job_memory_limit_bytes	replica	Memory limit of the worker job replica, in bytes.
worker_job_restarts_total	replica	Number of times the worker job replica has been restarted.

Builds

Metric	Labels	Description
application_build_last_success		1 if the latest finished build succeeded, 0 if it failed.
application_build_last_failed_stage	stage	Stage in which the latest finished build failed. Only exists if the build failed and the stage is known.
application_build_last_duration_seconds		Duration of the latest finished build, from its creation until it finished, in seconds. Includes the time queued.
application_build_last_completion_timestamp_seconds		Unix timestamp at which the latest finished build finished.
application_build_last_trigger	trigger	Triggers of the latest finished build, one series per trigger.
application_build_running		1 if a build is queued or running, 0 otherwise.
application_build_running_creation_timestamp_seconds		Unix timestamp at which the running build was created.
application_build_running_trigger	trigger	Triggers of the running build, one series per trigger.
application_build_finished_total	result, stage	Number of finished builds. result is success or failure, failed builds are also labeled by stage.

The stage label tells you in which part of the build it failed:

• source_fetch: fetching the source code from your Git repository.
• detection: detecting how to build your app, for example its language.
• build: building your app, including Dockerfile builds.
• platform: preparing the build, the build cache or storing the built image. These failures are most likely not caused by your app. Retry the build, and contact us if it keeps failing.
• unknown: the stage could not be determined. Only used in application_build_finished_total.

The trigger label tells you why a build was started. A build can have several triggers:

• source: a new commit in your Git repository.
• config: a change to the build configuration, for example a build environment variable. The first build of an app has this trigger as well.
• retry: the build was retried, see Retry a Build.
• buildpack_update: we updated the buildpacks.
• base_image_update: we updated the base image of your app.
• unknown: the trigger could not be determined.

Example Queries

CPU usage of each replica relative to its limit:

sum by (nine_project, app, replica) (rate(application_cpu_usage_seconds[5m]))
  / sum by (nine_project, app, replica) (application_cpu_limit_cores)

Share of HTTP requests that were answered with a 5xx status code:

sum by (nine_project, app) (rate(application_http_requests_5xx[5m]))
  / sum by (nine_project, app) (rate(application_http_requests_total[5m]))

Apps whose latest build failed:

application_build_last_success == 0

Failed builds of the last 24 hours, by stage:

sum by (stage) (increase(application_build_finished_total{result="failure"}[24h]))

Builds that are running for more than 30 minutes:

time() - application_build_running_creation_timestamp_seconds > 1800