Skip to main content

Troubleshooting and advanced topics

How it works behind the scenes​

To understand and be able to troubleshoot issues when deploying your app, it helps to know a bit about what is going on in the backend. This is what happens on a high level during a successful app creation:

  • Git repository is cloned at whatever revision was specified
  • The language of the app gets detected
  • The app is being built and package into a OCI container with language-appropriate tools
  • The built app image is uploaded to the Deploio image registry
  • A release is created which executes the previously built image
  • The application is exposed on an HTTP endpoint

The build and release step is where most things can go wrong. If there's an error or incompatibility within the cloning or build step, nctl will show the build logs to help with troubleshooting. If the build is successful but a release does not become ready, usually there should be an indication about the problem in the app logs. Of course if your app fails silently on start, the app log might be empty.

Retry build​

The build may fail due to multiple reasons. Most of the time a change on the application code or a missing environment/build variable will fix the issue, but it might happen that some temporary internal system issue failed the build. In those cases you can retry the build by using nctl:

$ nctl update app go-example --retry-build

Retry release​

The retry release trigger allows you to manually initiate a new release without requiring code changes or rebuilding your application. This is especially useful when a release has previously failed due to external factors, infrastructure issues, or temporary network failures. For example, if your release process failed due to a broken or incorrect database migration, and you've manually resolved this at the database level, you can directly use the retry release feature to redeploy the existing build:

$ nctl update app go-example --retry-release

Inspecting and running the build output locally​

As the build output is just an OCI/Docker image, it's possible to pull the image to your local machine to inspect it. This can be helpful if the build succeeds but the release fails to start the app. Note that it's just possible to pull images for debugging, pushing images to the registry is disabled.

$ nctl get build go-example-build-1 --pull-image
Pulling image of build go-example-build-1
739cb80086f0: Download complete
1bfd697887fa: Download complete
aea73d1e202c: Download complete
e4fd482e7de7: Download complete
 ✓ Pulled image deploio.296dc8b.registry.nineapis.ch/nine/go-example 💾

After downloading the image, you can interact with it using Docker or Podman (or anything that can interact with OCI images) to inspect and run the image locally.

$ docker run deploio.296dc8b.registry.nineapis.ch/nine/go-example

Executing a shell or command in a Deploio application​

You can start a shell in your Deploio application once it was released successfully. Please note that your application needs to already run in a stable way so that a shell can be started. If your application constantly restarts because of an error, starting a shell will not be possible.

You can start a shell with the help of nctl. This will start a shell in the first Deploio replica which can be found.

$ nctl get app
NAME     HOSTS                        UNVERIFIED_HOSTS
rails    rails.7f0b629.deploio.app    none

$ nctl exec app rails
cnb@rails-5fcc67cc-89fd2:/workspace$

You can also start a single command by separating it with --.

$ nctl exec app rails -- hostname
rails-5fcc67cc-89fd2

Please note, that every command you start will be counted towards your memory quota which depends on your configured Deploio application size. If too much memory is used the whole Deploio application replica will be terminated and you will most probably see an exit code of 137:

nctl: error: command terminated with exit code 137

Similarly, If your application exceeds its local Ephemeral Storage limit, it is immediately terminated. The session is forcibly cut off, and the following error message appears:

nctl: error: command terminated with exit code 137

Furthermore, any change to local files being done in a shell session will be lost, once the Deploio replica was restarted.

Lets Encrypt certificates​

Applications on Deploio are exposed via a random generated URL by default. That way you can access your application during development time without the need for an own domain name. The default URL is also secured by a Lets Encrypt TLS certificate which gets directly generated when you create a Deploio application. For Lets Encrypt certificates, the HTTP-01 challenge type is generally used. You can see the status of the default URLs Lets Encrypt certificate via nctl:

nctl get app <APP NAME> -o yaml

kind: Application
apiVersion: apps.nine.ch/v1alpha1
...
status:
  atProvider:
    ...
    defaultHostsCertificateStatus: Issued

Whenever you add custom host names to your Deploio application, a corresponding Lets Encrypt TLS certificate will be issued for all your custom host names. You can see the status of that certificate via nctl as well:

nctl get app <APP NAME> -o yaml

kind: Application
apiVersion: apps.nine.ch/v1alpha1
...
status:
  atProvider:
    ...
    customHostsCertificateStatus: Pending

As we are using the Lets Encrypt HTTP-01 challenge type, the certificate will only be successfully issued once all of your custom hostnames point to the Deploio infrastructure. We are using an optimized DNS resolving path to quickly react to DNS changes, but it might still take some minutes before the certificate can be issued. This is especially important if you migrate an application which is elsewhere hosted to Deploio. As the transition can take some minutes, please consider a migration during non-business hours.

Please also keep in mind that Lets Encrypt favors IPv6 DNS entries over IP4 ones. So if you have DNS AAAA records for your custom hostnames, make sure to delete them when migrating to Deploio (as Deploio does not support IPv6 currently).

Local Ephemeral Storage limitations for Deploio applications​

Ephemeral Storage is a temporary storage mechanism used by your app, or more precisely, by its container while it is running. Currently we limit local Ephemeral Storage to 2 GiB per Deploio application. These limits are in place to prevent any application from consuming excessive resources, ensuring a fair distribution across the system.

What constitutes Ephemeral Storage?​

Containers use local Ephemeral Storage for scratch space, caching, and for logs. Writing data through an exec session into a running app's filesystem also counts toward the total Ephemeral Storage usage. Exceeding the storage limit may cause the app to restart, resulting in the loss of all Ephemeral Storage data.

Best practices for managing Ephemeral Storage​

For effective Ephemeral Storage management, users should minimize writes to the writable layer. Excessive logging and unnecessary temporary file generation should be avoided. Critical data should be stored in persistent storage to prevent loss.

note

Ephemeral Storage limit does not include the size of the Deploio application image.

What happens when Ephemeral Storage is exceeded?​

If an application surpasses the Ephemeral Storage limit, its container is terminated, and the exit code 137 is returned upon termination.

Impact on exec sessions​

If an exec session is active inside a container that exceeds its Ephemeral Storage limit, the container is immediately terminated. The session is forcibly cut off, and the following error message appears:

nctl: error: command terminated with exit code 137

The writable layer of the Deploio application​

The writable paths and behavior in the app container depend on whether the application was built using Buildpacks (default) or a Dockerfile. Different approaches result in different restrictions and permissions for writable directories.

Applications built using Buildpacks (default)​

Buildpacks define a strict set of writable paths based on conventions designed to enhance security and reproducibility. In most Buildpack-built apps, writable paths are restricted to specific directories, commonly:

/workspace
/tmp

Why this restriction? Buildpacks are designed to create immutable layers for application dependencies and base files. Any modifications at runtime should only occur in ephemeral directories to maintain container integrity.

The writable directories in a Buildpack-built container are bound by the app's Ephemeral Storage limit, meaning excessive writing can exhaust available space.

Applications built using a Dockerfile​

When an application is built using a Dockerfile, writable paths and behavior depend on the developer's choices, and configurations can vary widely based on application needs. Similar to Buildpacks, writable paths in a Dockerfile-built application are also limited by the app's Ephemeral Storage restrictions.

Checking Which User is Running the Container​

Before testing writability, it is helpful to determine which user is running the container. This can be done executing:

id

Example output:

uid=1000(cnb) gid=1000(cnb) groups=1000(cnb)

The User ID may differ between build time and runtime, depending on the Buildpack architecture. Some Buildpacks may use different users during build and runtime, while others may maintain the same user throughout. This may result in different runtime permissions. For more info see Cloud Native Buildpack policy. Dockerfile-based builds allow full control over the user and configurations can vary significantly.

Determining writable directories in your app​

First, you need to start a shell in your Deploio application. There is no direct command to list only writable directories, but some exploration can help identify them. Below are some tips that may be helpful.

A useful command to check for writable permissions is:

find / -writable -type d 2>/dev/null | sort

This command may not be available in all container images. If find is missing but ls is present, directory permissions can be inspected manually using:

ls -ld /path/to/directory

If unsure about writable directories, you can also test them manually by attempting to create a file:

touch /tmp/testfile && echo "/tmp is writable" || echo "/tmp is not writable"

If touch is unavailable, writing to a file can be tested with echo:

echo "test" > /tmp/testfile && echo "/tmp is Writable" || echo "/tmp is NOT Writable"
rm -f /tmp/testfile

You can also try mkdir:

mkdir /tmp/testdir && echo "/tmp is Writable" || echo "/tmp is NOT Writable"
rmdir /tmp/testdir
note

In some Deploio application images, especially those built using a Dockerfile from distroless or scratch base (for example our sample Dockerfile app), the shell may not be included, preventing any command from being executed. In such cases, you should use alternative debugging tools and techniques.