Skip to main content

Troubleshooting and Advanced Topics

How Deploio Works​

Understanding the backend process helps you troubleshoot deployment issues. During a successful app creation, Deploio performs these steps:

  • Clones the Git repository at the specified revision
  • Detects the app's language
  • Builds and packages the app into an OCI container with language-appropriate tools
  • Uploads the built app image to the Deploio image registry
  • Creates a release that executes the built image
  • Exposes the application on an HTTP endpoint

The build and release steps are where most issues occur. If there's an error during cloning or building, nctl shows the build logs to help with troubleshooting. If the build succeeds but a release doesn't become ready, check the app logs for an indication of the problem. If your app fails silently on start, the app log might be empty.

Retry a Build​

Builds can fail for multiple reasons. Most of the time, a change to the application code or adding a missing environment/build variable fixes the issue. However, a temporary internal system issue might also cause the failure. In those cases, retry the build using nctl:

$ nctl update app go-example --retry-build

Retry a Release​

The retry release trigger lets you manually initiate a new release without code changes or rebuilding your application. This is especially useful when a release failed due to external factors, infrastructure issues, or temporary network failures. For example, if your release process failed due to a broken or incorrect database migration and you've resolved the issue at the database level, use the retry release feature to redeploy the existing build:

$ nctl update app go-example --retry-release

Inspect and Run the Build Output Locally​

Since the build output is an OCI/Docker image, you can pull it to your local machine for inspection. This helps when the build succeeds but the release fails to start the app. You can only pull images for debugging; the registry doesn't accept pushes.

$ nctl get build go-example-build-1 --pull-image
Pulling image of build go-example-build-1
739cb80086f0: Download complete
1bfd697887fa: Download complete
aea73d1e202c: Download complete
e4fd482e7de7: Download complete
 ✓ Pulled image deploio.296dc8b.registry.nineapis.ch/nine/go-example 💾

After downloading the image, you can interact with it using Docker or Podman (or anything that can interact with OCI images) to inspect and run the image locally.

$ docker run deploio.296dc8b.registry.nineapis.ch/nine/go-example

Execute a Shell or Command in a Deploio Application​

note

In some Deploio application images, especially those built using a Dockerfile from distroless or scratch base (for example, the sample Dockerfile app), the shell isn't included, preventing command execution. In such cases, use alternative debugging tools and techniques.

You can start a shell in your Deploio application after a successful release. Your application must run in a stable state for a shell to start. If your application constantly restarts because of an error, starting a shell isn't possible.

Start a shell using nctl. This starts a shell in the first available Deploio replica.

$ nctl get app
NAME     HOSTS                        UNVERIFIED_HOSTS
rails    rails.7f0b629.deploio.app    none

$ nctl exec app rails
cnb@rails-5fcc67cc-89fd2:/workspace$

You can also start a single command by separating it with --.

$ nctl exec app rails -- hostname
rails-5fcc67cc-89fd2

Every command you start counts towards your memory quota, which depends on your configured Deploio application size. If too much memory is used, Deploio terminates the entire application replica and you see an exit code of 137:

nctl: error: command terminated with exit code 137

Similarly, if your application exceeds its local ephemeral storage limit, Deploio immediately terminates it. The session is forcibly cut off and the following error message appears:

nctl: error: command terminated with exit code 137

Any changes to local files made in a shell session are lost when the Deploio replica restarts.

Let's Encrypt Certificates​

Deploio exposes applications via a randomly generated URL by default, letting you access your application during development without your own domain name. Deploio secures the default URL with a Let's Encrypt TLS certificate, generated automatically when you create an application. For Let's Encrypt certificates, Deploio uses the HTTP-01 challenge type. View the certificate status for the default URL using nctl:

nctl get app <APP NAME> -o yaml

kind: Application
apiVersion: apps.nine.ch/v1alpha1
...
status:
  atProvider:
    ...
    defaultHostsCertificateStatus: Issued

When you add custom host names to your Deploio application, Deploio issues a corresponding Let's Encrypt TLS certificate for all your custom host names. View the certificate status using nctl:

nctl get app <APP NAME> -o yaml

kind: Application
apiVersion: apps.nine.ch/v1alpha1
...
status:
  atProvider:
    ...
    customHostsCertificateStatus: Pending

Because Deploio uses the Let's Encrypt HTTP-01 challenge type, the certificate is only issued once all of your custom hostnames point to the Deploio infrastructure. Deploio uses an optimized DNS resolving path to quickly react to DNS changes, but certificate issuance might still take a few minutes. This is especially important when migrating an application hosted elsewhere to Deploio. Since the transition can take several minutes, consider migrating during non-business hours.

Let's Encrypt favors IPv6 DNS entries over IPv4. If you have DNS AAAA records for your custom hostnames, delete them when migrating to Deploio, as Deploio doesn't currently support IPv6.

Local Ephemeral Storage Limitations​

Ephemeral storage is a temporary storage mechanism that your app's container uses while running. Deploio limits local ephemeral storage to 2 GiB per application. These limits prevent any single application from consuming excessive resources, ensuring fair distribution across the system.

What Constitutes Ephemeral Storage?​

Containers use local ephemeral storage for scratch space, caching, and logs. Writing data through an exec session into a running app's filesystem also counts toward total ephemeral storage usage. Exceeding the storage limit can cause the app to restart, resulting in the loss of all ephemeral storage data.

Best Practices for Managing Ephemeral Storage​

To manage ephemeral storage effectively:

  • Minimize writes to the writable layer
  • Avoid excessive logging and unnecessary temporary file generation
  • Store critical data in persistent storage to prevent loss
note

The ephemeral storage limit does not include the size of the Deploio application image.

What Happens When You Exceed Ephemeral Storage?​

If an application exceeds the ephemeral storage limit, Deploio terminates the container and returns exit code 137.

Impact on Exec Sessions​

If an exec session is active inside a container that exceeds its ephemeral storage limit, Deploio immediately terminates the container. The session is forcibly cut off and the following error message appears:

nctl: error: command terminated with exit code 137

Writable Layer​

The writable paths and behavior in the app container depend on whether you built the application using Buildpacks (default) or a Dockerfile. Different approaches result in different restrictions and permissions for writable directories.

Applications Built with Buildpacks (Default)​

Buildpacks define a strict set of writable paths based on conventions designed to enhance security and reproducibility. In most Buildpack-built apps, writable paths are restricted to specific directories, commonly:

/workspace
/tmp

Buildpacks create immutable layers for application dependencies and base files. Modifications at runtime should only occur in ephemeral directories to maintain container integrity.

The writable directories in a Buildpack-built container are bound by the app's ephemeral storage limit, meaning excessive writing can exhaust available space.

Applications Built with a Dockerfile​

When you build an application using a Dockerfile, writable paths and behavior depend on your choices and can vary widely based on application needs. Similar to Buildpacks, writable paths in a Dockerfile-built application are also limited by the app's ephemeral storage restrictions.

Check Which User Runs the Container​

Before testing writability, determine which user runs the container by executing:

id

Example output:

uid=1000(cnb) gid=1000(cnb) groups=1000(cnb)

The user ID might differ between build time and runtime, depending on the Buildpack architecture. Some Buildpacks use different users during build and runtime, while others maintain the same user throughout. This can result in different runtime permissions. For more information, see Cloud Native Buildpack policy. Dockerfile-based builds give you full control over the user, and configurations can vary significantly.

Determine Writable Directories in Your App​

First, start a shell in your Deploio application. There is no direct command to list only writable directories, but exploration can help identify them. The following tips can help.

To check for writable permissions:

find / -writable -type d 2>/dev/null | sort

This command might not be available in all container images. If find is missing but ls is present, inspect directory permissions manually:

ls -ld /path/to/directory

If you're unsure about writable directories, test them manually by attempting to create a file:

touch /tmp/testfile && echo "/tmp is writable" || echo "/tmp is not writable"

If touch is unavailable, test writing to a file with echo:

echo "test" > /tmp/testfile && echo "/tmp is Writable" || echo "/tmp is NOT Writable"
rm -f /tmp/testfile

You can also try mkdir:

mkdir /tmp/testdir && echo "/tmp is Writable" || echo "/tmp is NOT Writable"
rmdir /tmp/testdir