# Nine Internet Solutions AG - Documentation > Official documentation for Nine's cloud products and services This file contains all documentation content in a single document following the llmstxt.org standard. ## Cloudflare FAQs The activation of Cloudflare may have some implications for your websites/applications. We have therefore collected the most important information for you below. If your question is not included, our customer support will be happy to help you at any time. ## Upload Limit Cloudflare imposes an upload limit (HTTP Post request size) depending on your account type: - 100MB Free and Pro - 200MB Business - 500MB Enterprise (an increase of this limit can be requested via the CF customer support) ## Sub-subdomains Cloudflare automatically creates a free wildcard TLS certificate for the main and subdomains. If you want to use a sub-subdomain such as "sub.site.example.org", you need an additional certificate. For this purpose Cloudflare offers the service "Advanced Certificate Manager", which allows for $10/month, to issue additional certificates. Please note that these certificates should be active before the site is routed through Cloudflare. Otherwise, your website will not work. You can read more about certificates in the following Cloudflare articles: https://blog.cloudflare.com/advanced-certificate-manager/ https://developers.cloudflare.com/ssl/edge-certificates/advanced-certificate-manager ## Timeouts Cloudflare's default connection timeout is 100 seconds. Enterprise customers can increase the "Error 524: timeout" up to 6000 seconds. This can be done via the Cloudflare API or [contact our support](/docs/general/contact) to request the timeout increase. https://api.cloudflare.com/#zone-settings-change-proxy-read-timeout-setting ## Content Security Policy Content Security Policy (CSP) is a policy implemented via HTTP header that is used to prevent certain types of attacks on websites where the sites are modified to include and run malicious foreign resources. If you are using CSP headers, you may need to adapt them to use some Cloudflare features. Consult the following linked documentation from Cloudflare for more information: https://developers.cloudflare.com/fundamentals/get-started/reference/content-security-policies/. ## CNAME setups A CNAME setup can be used in cases where you want to use your own DNS server instead of Cloudflare's DNS. However, this function is only available for customers with a Business or Enterprise license. :::warning For a CNAME setup, your DNS server must provide CNAME flattening for the root entry. ::: A detailed explanation of how this works and can be implemented can be found here: https://support.cloudflare.com/hc/en-us/articles/360020348832-Understanding-a-CNAME-Setup ## Mutual TLS Since Cloudflare terminates the initial TLS connections, existing Mutual TLS (mTLS) won't work after putting Cloudflare in front of your application. In this case, the service "Cloudflare Access" is what you're looking for. For more information and a detailed explanation, please look at the following resources: https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/ https://blog.cloudflare.com/using-your-devices-as-the-key-to-your-apps/ ## Multiple Domains per Account You can have multiple domains per account. However, every domain needs a separate license. If you use a second domain mainly for forwarding to your main domain, a free license is sufficient in most cases. But, since we can only manage Business and Enterprise accounts, the customer would have to create and manage a separate account for this on his own. ## API Tokens Maybe you want to automate CF actions or Worker deployments. To do this, you can access the Cloudflare API with a Security Token. Detailed instruction for this can be found in this documentation: https://developers.cloudflare.com/api/tokens/create --- ## Price & Product Overview Cloudflare ## Official Cloudflare Partner We offer multiple options to use Cloudflare in combination with Nine services, ranging from fully managed solutions operated by Nine to support for customer-owned Cloudflare licenses. ## Cloudflare Enterprise and Fully Managed Cloudflare Options The **Cloudflare Enterprise License** offers you the highest level of performance, security, and flexibility, tailored to your specific business needs. This option gives you access to the full Cloudflare Enterprise feature set, including advanced DDoS protection, Enterprise WAF with extended rules and features, and global traffic acceleration. You retain full access to the Cloudflare dashboard and benefit from direct support from Cloudflare, as well as consulting and configuration services provided by Nine. With the **fully managed Cloudflare** options, you benefit from having all operational responsibilities handled by Nine. The Nine team takes care of all configuration, monitoring, and change requests on your behalf. You do not receive access to the Cloudflare dashboard, allowing Nine to manage all technical aspects. These fully managed options are available only if you already use Nine services and need infrastructure protected through Cloudflare. ProductPrice, ProductSetupPrice, FormatPrice, } from "@site/src/components/ProductPrice" [Price Calculator](https://nine.ch/en/price-calculator/?category=Cloudflare) [Cloudflare Product Page](https://nine.ch/products/cloudflare/) | | Enterprise License | Fully Managed Cloudflare Enterprise Light License | Fully managed Cloudflare Business License | | :------------------------------------------- | :-------------------------------: | :----------------------------------------------------------------------------: | :---------------------------------------------------------------------: | | **Monthly fee** | On request | | | | **Setup fee** | | | | | **License availability** | Unrestricted | Restricted \* | Restricted \* | | **Contract term** | 12 months | 1 month | 1 month | | **Cancellation period** | Cloudflare terms | 1 month | 1 month | | **License managed by** | Nine | Nine | Nine | | **Primary domains** | Individual contract | 1 | 1 | | **Access to Cloudflare dashboard** | ✅ | ❌ | ❌ | | **Access to Cloudflare API** | ✅ | ❌ | ❌ | | **Additional services** | Optional | ❌ | ❌ | | **China Network Access** | Optional | ❌ | ❌ | | **DDoS protection** | ✅ | ✅ | ✅ | | **Enterprise DDos Mitigation** | ✅ | ✅ | ❌ | | **DDoS Priority** | ✅ | ✅ | ❌ | | **Web Application Firewall (WAF)** | Advanced Ruleset/Features | Advanced Ruleset/Features | Standard Ruleset | | **Custom WAF rules** | 1000 | 1000 | 100 | | **DNS Records per Zone** | According to individual agreement | 3500 | 3500 | | **DNS Analytics** | 30 days | 30 days | 7 days | | **Max timeout HTTP requests** | \> 100 s, customizable | \> 100 s, customizable | 100 s | | **Max file size Cache** | 5 GB | 5 GB | 512 MB | | **Upload Limit (POST/PUT/PATCH)** | 500+ MB | 500+ MB | 200 MB | | **Support by Nine** | 24/7 | 24/7 | 24/7 | | **Hourly Rate Support/Consulting by Nine** | 2h/month included, then CHF 250/h | 2h/month included, then CHF 250/h | CHF 250/h | | **Change Requests handled by Nine** | Included | Included | Included | | **Direct Support by Cloudflare** | ✅ | ❌ | ❌ | | **Uptime SLA** | 100 % | ❌ | ❌ | | **Uptime Service Credits** | ✅ | ❌ | ❌ | | **ISO 27001 compliance** | ✅ | ✅ | ✅ | | **SOC 2 Type II compliance** | ✅ | ✅ | ❌ | | **PCI DSS 4.0 compliance (online payments)** | ✅ | ✅ | ❌ | \* Only available for customers with existing Nine products that are to be protected via Cloudflare. ## Support Offering for Enterprise License Customers In addition to 24x7x365 tickets, chat, phone, community forums, and 24x7x365 emergency phone support by Cloudflare, an Enterprise license offers the following additional options to get familiar with Cloudflare features: - **Workshops**: Led by Cloudflare engineers and product managers, covering best practices, how-tos, and technical insights across application, security, networking, and developer topics. - **Office hours**: Open (several customers together) Q&A sessions with technical specialists, offering quick guidance and configuration help. - **Private appointments**: 1:1 sessions with subject matter experts for deep dives into specific use cases. Available weekly on a first-come, first-served basis. Access these options through your [Cloudflare dashboard](https://www.cloudflare.com/ecp/successevents/). ## Support Access for Customer Cloudflare Licenses This option is intended for customers who operate their own Cloudflare license and wish to retain full control via the Cloudflare dashboard. By subscribing to this option, you grant Nine ongoing access to your license, allowing the team to provide support when needed. Active management and monitoring are not included. You can request support during business hours, and it will be billed separately at standard hourly rates. This option is available only if you already use Nine services and need infrastructure protected through Cloudflare. | | Support Access for Customer Cloudflare Licenses | | :----------------------------------------- | :---------------------------------------------------------------------: | | **Monthly fee** | | | **License availability** | Restricted \* | | **Contract term** | Min. 6 months | | **Cancellation period** | 1 month | | **License managed by** | Customer | | **Access to Cloudflare dashboard** | Full access | | **Access to Cloudflare API** | Full access | | **Hourly Rate Support/Consulting by Nine** | CHF 250/h | | **Change Requests handled by Nine** | CHF 250/h | \* Only available for customers with existing Nine products that are to be protected via Cloudflare. --- ## Cockpit Account Suspension ## Situation You open the Cockpit or try to create a self-service resource and you get a message, that your customer account is suspended. ## Explanation If you see this message, your account has been suspended due to a violation of our General Terms and Conditions (GTC) or our Acceptable Use Policy. If you believe this is was an error, please contact our support team by phone at +41 44 637 40 40 or by email at . ## Restrictions with the suspension If your customer account is suspended, you can no longer create or edit self-service resources. --- ## Contact Management and Cockpit Access From February 2023, our customers will be able to manage their contacts themselves via . People will now have access to Cockpit with an individual user account. How to manage these contacts and what to keep in mind, we show you here. ## Terms - **User profile**: A user profile is linked to an individual login. A user profile can be granted access to one or several customer accounts. A user profile is identified by their email address. - **Customer account**: A customer account is an organisational unit that contains resources (VMs, servers, racks, etc.). Each customer account is identified by a customer number and an identifier. Visit the article [Difference between your User Profile and a Customer Account](./difference-between-your-user-profile-and-a-customer-account) for a detailed comparison of the two types. ## Roles Persons (and thus user accounts) are linked to a customer account via different roles. - **Cockpit Access**: The person is granted access to Cockpit and is allowed to interact with the resources of the customer account in it. For administrative functions (e.g. viewing invoices, managing contacts and addresses), the _Customer Admin_ role is also required. - **Technical Contact**: Technical contacts are allowed to create tickets and receive technical information from us about a customer account's resources. If you also have the _Cockpit Access_ role, these people can interact with the resources of a customer account in . - **Customer Admin**: Customer admins are main contacts of the customer account. They have the most rights and can do everything in the cockpit (provided they also have the _Cockpit Access_ role). Admins also receive all important information from us via e-mail. ## Add new contacts As a _Customer Admin_ (with Cockpit access) you have the possibility to manage contacts in the yourself: https://cockpit.nine.ch/en/customer/contacts. To give a new person access to your customer account, create an invitation [in Cockpit](https://cockpit.nine.ch/en/customer/contacts). The person will receive a link by e-mail. To accept the invitation, the person must log in with their existing or a new user account. **Caution**: Make sure that the e-mail address in the invitation matches the e-mail address of the user account! If the person cannot or does not want to create a personal user account, you can still let us do the contact management for you. Just drop us an e-mail from an address we know to or call . ## Manage roles You can manage the roles of existing contacts in the [Cockpit](https://cockpit.nine.ch/en/customer/contacts). A role can be removed by a specified date (in the future) or immediately. The specified date is inclusive. This means that the person will still have the role and thus the corresponding access until the end of the given day. ## Logins for existing contacts If a person is already in the list of contacts that have access to their customer account, this person can [create a new user account](https://cockpit.nine.ch/signup) (if not done already) and will have access automatically. ## Delete user account User accounts are personal and are managed by the individuals themselves. As a customer admin you cannot delete user accounts, but you can revoke cockpit access and other roles from individuals (see above). --- ## Creating New Customer Account Creating a customer account, whether for personal use or an organization, is a simple and straightforward process. Both types of accounts allow for managing multiple users and assigning specific roles and permissions. ## Personal vs. Organizational Accounts - **Personal Customer Account**: Ideal for individuals managing their own resources. - **Organizational Customer Account**: Designed for businesses or teams that require shared access and role-based permissions. --- ## How to Create a Customer Account To create a new customer account: 1. Visit and open the **Account Selection** in the side navigation. Then and choose `All`. ![Switch Account Screenshot](/img/switch_account.png) 2. Scroll to the bottom of the page and click the **Sign Up Customer Account** button. ![Sign Up Button Screenshot](/img/sign_up.png) 3. Choose whether the account is for an individual or a company. Fill in the required details, such as name, email address, and address. ![Sign Up Form Screenshot](/img/signup_form.png) 4. After submitting the form, your account will undergo verification. Once approved, you'll gain full access to all account features. ![Verification Message Screenshot](/img/verifier_message.png) --- ## Contact Management and Access Control Once your account is active, you can manage the access rights under `Account -> Contacts` in . This page allows you to: - Define individuals or teams who can access your account. - Assign roles and permissions. - Ensure appropriate access control for managing resources and services. For more information, refer to the article on [Contact Management and Cockpit Access](./contact-management-and-cockpit-access). --- ## Next Steps: Creating Self-Service Resources After your account is set up, you're ready to begin creating self-service resources. Here are some options to get started: - **Deploio**: A fully managed platform that transforms Git repositories into applications. [Learn more about Deploio](../deplo-io). - **Object Storage**: S3-compatible storage for your data needs. [Learn more about Object Storage](/docs/object-storage). - **On-Demand-Services**: Create and manage on-demand services easily. [Learn more about On-Demand services](/docs/on-demand-services/). - **CloudVM**: Create and manage cloud server instances. [Learn more about CloudVM](/docs/root-server). - **Managed Kubernetes**: Run containers on a fully managed software stack based on Kubernetes. [Learn more about Managed Kubernetes](/docs/category/managed-kubernetes). - **DNS**: Create and manage DNS records for your domains. [Learn more about DNS](/docs/category/dns). --- ## Difference between your User Profile and a Customer Account In our cockpit, we provide two types of accounts. This page gives an overview of the two types and how they are different from each other. ## Overview of the Account Types - **User profile**: A user profile is linked to an individual login. A user profile can be granted access to one or several customer accounts. A user profile is identified by their email address. - **Customer account**: A customer account is an organisational unit that contains resources (VMs, servers, racks, etc.). Each customer account is identified by a customer number and an identifier. ## Differences between the Account Types | User profile | Customer account | | ----------------------------------------- | ---------------------------------------------------------------------- | | Connected to a login to access Cockpit. | Requires a user with the role "Cockpit Access" to access. | | Represents you as an individual user. | Represents a legal entity, e.g. a company or a single person. | | Contains no resources. | Contains all your resources, such as VMs or Applications. | | Creation is for free. | Creation is for free, but all contained resources will generate costs. | | Identification is done via email address. | Identification is done via customer number and a customer identifier. | ## Manage User Profile In order to manage your user profile, do the following in : 1. Click on your user profile icon in the side navigation on the left 2. Choose `Manage User Profile` ![Manage User Profile](/img/manage_user_profile.png) For help on how to change the User Password, please visit [How to change your Cockpit credentials](how-to-change-your-cockpit-credentials). ## Manage Customer Account In order to manage your customer account, do the following in : 1. Click on the cog icon next to the customer account selection dropdown list ![Manage Customer Account](/img/manage_customer_account.png) Note: If you want to manage a different customer account than the currently selected one, make sure to first select the customer in the customer selection dropdown on the side navigation on the left. ![Customer Selection Dropdown](/img/customer_selection_dropdown.png) --- ## How to change your address This page describes how to change your address. There's different types of addresses, which are all covered in this article. ## Change Customer Account Addresses Your customer account has two different types of addresses: a general address, and optionally an invoice address. For more information about the difference between a customer account and a user profile, please visit [Difference between your User Profile and a Customer Account](difference-between-your-user-profile-and-a-customer-account) ### Change General Account Address The general address of a customer account is used for the main correspondence regarding the customer account. To change the general address, please go through following steps: 1. Log into with administrator rights to the customer account - Click the `Settings` icon in the side navigation navigation - Click the tab `Details` - Click the edit button next to the General Address 2. Change the address and confirm by clicking the button `Save` ![Change General Address](/img/change_general_address.png) ### Change Invoice Address The invoice address of a customer number is used for invoices. To add an invoice address, please go through following steps: 1. Log into with administrator rights to the customer account - Click the `Settings` icon in the side navigation navigation - Click the tab `Details` - Click the button `Add invoice address` ![Add Invoice Address](/img/add_invoice_address.png) ## Change User Profile Address The user profile address will be used as a template for new customer accounts. In order to change your user profile address, please go through the following steps: 1. Log into - Click the `User Profile` icon in the side navigation navigation - Click on `Manage User Profile` - Click the edit button under `User profile address` 2. Change the address and confirm by clicking the button `Save` ![Change User Profile Address](/img/change_user_profile_address.png) Please note: As the primary email address is used to identify the user, it cannot be changed. How the first name and last name can be changed you can find in the next section. For more information about the difference between a customer account and a user profile, please visit [Difference between your User Profile and a Customer Account](difference-between-your-user-profile-and-a-customer-account) ### Change first name and last name In order to change your user profile's first and last name, please do the following steps: - Click the `User Profile` icon in the side navigation navigation - Click on `Manage User Profile` - Click the edit button under `User identification` ![Change User Identification](/img/change_user_identification.png) --- ## How to change your Cockpit credentials This page describes, how you can manage your user profile credentials. For more information about your user profile, please visit [Difference between your User Profile and a Customer Account](difference-between-your-user-profile-and-a-customer-account). ## Change Password In order to change your password, do the following in : 1. Click on your user profile icon in the side navigation on the left 2. Choose `Manage Login Credentials` ![Manage Login Credentials](/img/manage_login_credentials.png) A new page should open up. Now do the following: 1. Choose `Signing in` under `Account Security` ![Signing in](/img/signing_in_keycloak.png) 2. Choose `Update` under `My Password` ![Update password](/img/update_password.png) ## Enable Two-Factor Authentication In order to enable Two-Factor Authentication, do the following in our : 1. Click on your User Profile Picture in the side navigation on the left 2. Choose `Manage Login Credentials` ![Manage Login Credentials](/img/manage_login_credentials.png) A new page should open up. Now do the following: 1. Choose `Signing in` under `Account Security` ![Signing in](/img/signing_in_keycloak.png) 2. Now you can manage the settings under `Two-factor authentication` ![Two-Factor Authentication](/img/two_factor_authentication.png) --- ## Datacenter access Your own access enables you to enter the datacenter and access your rack or housing at any time (24/7) using access control and a key. ## Costs You can find the current prices at [Price & Product Overview Colocation & Server Housing](./) ## Cert+ NTT Global Datacenters (formerly e-Shelter) Access to the NTT Global Datacenter is controlled via a personal badge. This is issued by the NTT counter on site for the duration of your stay in the datacenter in exchange for a valid ID card. ## NTS / Colozüri Access control in the Colozüri datacenter is carried out using fingerprint scanners.\ The fingerprints are recorded in our office. As soon as you order access, we will invite you to an appointment. Our address:\ https://goo.gl/maps/KmdwdgJXhwm Nine Internet Solutions AG\ Badenerstrasse 47\ 8004 Zurich ### Security Procedures at NTS / Colozüri The NTS / Colozüri datacenter uses personnel airlocks (mantraps) and goods airlocks to ensure secure access. Please review the relevant documentation for the floor you are accessing: - **Ground Floor (CZ 0.1):** [Personnel and Goods Airlock Procedures (PDF)](/uploads/Informationen-zur-Personenvereinzelung-_-Warenschleuse-coloZH-EG-EN.pdf) - **4th Floor (CZ 4.1 & 4.2):** [Personnel and Goods Airlock Procedures (PDF)](/uploads/Informationen-zur-Personenvereinzelung-_-Warenschleuse-coloZH-EN.pdf) ### Site Rules NTS / Colozüri Please familiarize yourself with the site rules before visiting the datacenter: - [Site Rules NTS / Colozüri (PDF)](/uploads/Site-Rules-coloZH-EN.pdf) ## Keysafe NTS / Colozüri The Colozüri datacenter has a key safe that allows rack keys to be deposited. These can be removed from the safe using a PIN code. ## Keysafe NTT / E-Shelter The NTT datacenter has a key safe that allows rack keys to be deposited. These can be removed from the safe using a PIN code. ## Request datacenter access Of course, it is also possible to request datacenter access by email at . We need the following information: - First and last name - Copy of ID card in color, front and back - Mobile phone number - Email address - Only for NTT access: Portrait photo in color, collarbone upwards For NTS Colozüri access, we will contact you after receiving the above information to make an appointment for fingerprint scanning. ## Temporary guest access In the event that you would like to allow additional persons access to the datacenter on a temporary basis, e.g. for technicians requested by the hardware manufacturer, the procedure would be as follows: ### NTT / E-Shelter Please [contact our support](/docs/general/contact) 24 hours before the planned visit with the following details for the visitors to be registered: - First and last name - Company name - Date and duration of the visit (maximum 1 week) If it is not possible to meet the 24-hour lead time, please let us know so that we and the datacenter can prioritize the registration. ### NTS / Colozüri No registration is required for the Colozüri, provided that the visitor is accompanied by a person with regular datacenter access. Using their own fingerprint access, this person can let visitors in and out through the personnel airlock. --- ## How can I restart the server (Remote Hands)? As we do not always have staff on-site at the data center, we ourselves have to organise an employee to carry out even a simple restart of a server. If you do not take up the remote reboot option or if the server does not start up despite a remote reboot, we perform a restart or to analyse the problem on-site for a fee. --- ## Price & Product Overview Colocation & Server Housing ProductPrice, ProductSetupPrice, FormatPrice, } from "@site/src/components/ProductPrice" [Colocation & Server Housing Product Page](https://nine.ch/products/colocation/) **Pricecalculator:** - [Server Housing](https://calculator.nine.ch/?category=Colocation&segment=3%29+Server+Housing) - [Colocation](https://calculator.nine.ch/?category=Colocation&segment=1%29+Colocation) ## Colocation and Server Housing {/* prettier-ignore */} Server Housing 1U Quarter Rack Half Rack Full Rack Premium Half Rack Premium Full Rack **Monthly fees** **Setup fees** **Location** NTS / Colozüri (Zürich City, Altstetten) NTT / E-Shelter (Rümlang, Canton Zürich) Cert+ ISO 27001 certified datacenter ❌ ✅ **Number of height units (44.45 mm or 1.75 inches)** 1U 11U(1U each for patch panel and power supply) 22U(1U each for patch panel and power supply) 47U(1U for patch panel) 23U(1U for patch panel) 52U(1U for patch panel) **Power included** 160W 500W 1000W 2000W 1000W 2000W **Copper connection** ❌ ✅ **Fiber connection** ❌ Optional ✅ **Connection / Bandwidth** 1 Gbps flat (Fair Usage) **Network availability** 99.9% **IPv4** Optional (order separately) **IPv6** Up to /48 prefix **24/7 datacenter and rack access** Optional ✅ **Further included services** Access control Video surveillance Fire protection (gas) Emergency power Redundant climate control 24/7 stand-by duty (for costs see [Support Services](#support-services)) ## IPv4 Subnets IPv4 subnets are available as an add-on for all Colocation and Server Housing products. A /48 IPv6 subnet is included with every product. | Subnet | Usable IPv4 addresses | Price per month | One time fee | | :------------- | :-------------------: | :-----------------------------------: | :----------------------------------------: | | **/30 subnet** | 2 | | | | **/29 subnet** | 3 | | | | **/28 subnet** | 11 | | | | **/27 subnet** | 27 | | | | **/26 subnet** | 59 | | | | **/25 subnet** | 123 | | | | **/24 subnet** | 251 | | | For Server Housing, a single IPv4 address is also available (see [Options for Server Housing](#options-for-server-housing)). Larger IPv4 subnets, BGP announcements, and BGP IP-Transit are available on request. ## Options for Rack Colocation | Option | Price per month | One time fee | | :------------------------------------------------------------------------ | :-----------------------------------------------------: | :----------------------------------------------------------: | | **Dualpower with additional powersupply bar** | | | | **Additional 1kW power supply (NTS)** | | - | | **Additional 1kW power supply (NTT)** | | - | | **Network redundancy** | | | | **Port for out-of-band management** | | | | **Increased bandwidth** | On request | - | | **IPv4 subnets** | See [IPv4 Subnets](#ipv4-subnets) | - | | **BGP announcement** | On request | - | | **BGP IP-Transit** | On request | - | | **Passive (cat. 6 copper) connection between two racks in the same room** | - | | | **Climate-responsible operation (Compensation of CO2 emissions)** | | - | | **24/7 datacenter and rack access** | Included | - | | **Additional 24/7 datacenter access** | Included | - | | **Additional Key for Rack** | Included | deposit per key | | **Keysafe in Colozüri** | | | | **Keysafe in NTT** | | | ## Options for Server Housing | Option | Price per month | One time fee | | :----------------------------------------------------------------- | :--------------------------------------------------------------------: | :---------------------------------------------------------------------------------: | | **Additional height unit (for servers that require more than 1U)** | | - | | **Additional 115 W power supply** | | - | | **High availability (dual power and out-of-band port)** | | | | **Per IPv4 address** | | - | | **IPv4 subnets** | See [IPv4 Subnets](#ipv4-subnets) | - | | **Additional switchport** | | | | **Climate-responsible operation (Compensation of CO2 emissions)** | | - | | **24/7 datacenter and rack access** | | Setup: Deposit for key: | | **Additional 24/7 datacenter access** | | | | **Keysafe in Colozüri** | | | ## Support services | Service | Price | | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------: | | **Remote Console IP-KVM for 6h** | | | **Remote Hands, Office Hours****_(Mo-Fr, 09:30 - 18:00, except on public holidays)_**Hourly rate (A minimum of 15 minutes will be charged) | /h | | **Remote Hands, Pikett****_(outside office hours & on public holidays)_**Hourly rate (A minimum of 1 hour will be charged)Assignment fee outside business hoursOn-Site Service Fee (If on-site travel is necessary) | /h | --- ## Remote Console IP-KVM For Root-Server, Housing and Colocation customers we offer an IP-KVM device that will be connected to your server. ## Usage This guidance is limited to the most commonly used IP-KVM model at Nine, the Raritan DKX2-101-V2 device. The officially supported browsers are Firefox and Internet Explorer. Other browsers are not supported and might not work. Some browser plugins and add-ons can also affect the functionality. How to use the IP-KVM: - You will receive an e-mail containing the credentials and the URL of the device - Open the URL in your browser and login with the provided credentials - Accept all Java messages and pop-ups - Click on "Your Server" and then on "Connect" - A pop-up window will be opened where you can control your server Please note, that the access is limited to one user at a time. ## Embed an ISO file In the server console window please open Virtual Media -> Connect CD-ROM / ISO and choose the file you would like to attach. The ISO file is being attached to the server as a removable device. You can use it to boot the server and install your operating system. Please note that the BIOS access is password restricted for root servers. If you need to boot your server from the IP-KVM device, please contact . ## Troubleshooting **An error message appears instead of "Connect" when clicking on "Your Server"** Make sure that you run the most recent version of Java and accepted all pop-ups. Sometimes, a reload of the page or deactivating browser add-ons could help as well. **The cursor on the server is not matching the cursor on my local computer** Click in the menu bar on "Mouse" and synchronize it. If that does not help, configure the mouse settings according to this manual: http://support.raritan.com/dominion-kx-ii-101/version-3.6.0/QSG-KX2101V2-v3.6.0-0E-E.pdf On the second half of the first page you will find the recommended mouse settings for different operating systems. After applying the settings, the cursor should be synchronized again. You can also choose another mouse modus (default is "Intelligent"). **Java doesn't work on my Linux client** When using the IP-KVM there may be problems with the Java console output on Linux. This short manual instructs how you can get it to work. The Following steps are necessary: - [Download](https://java.com/de/download/) and unzip of an up to date Java version and set the path to the unzipped folder: `PATH_TO_JAVA_UNZIP_FOLDER=Set_this_to_your_java_directory` - Symlink the new browser plugin on the new Java version: `mkdir -p ~/.mozilla/plugins` `cd ~/.mozilla/plugins` `if [ -f libnpjp2.so ]; then mv libnpjp2.so libnpjp2.so.old; fi` `ln -s $PATH_TO_JAVA_UNZIP_FOLDER/lib/amd64/libnpjp2.so` - Adjust the Java security settings: `cd $PATH_TO_JAVA_UNZIP_FOLDER/bin` `./ControlPanel` - In Tab Security set the security level to High - Afterwards that, insert the IP address of the IP-KVM device in to the whitelist. "Edit Site List" opens a window in which you can click on "Add" and insert the IP address. (please note that there has to be a "https://" in front of the IP address) - After these steps you need to restart the browser. Now you can start the KVM console and see an output. --- ## Basic authentication You can enable basic authentication for your application by using the `--basic-auth` flag of : ```bash --basic-auth Enable/Disable basic authentication for the application. $ nctl create app go-example --basic-auth ``` Once basic auth got enabled, a random password will be created. You can use the `--basic-auth-credentials` flag to show the username and password for your application: ```bash $ nctl get app go-example --basic-auth-credentials NAME USERNAME PASSWORD go-example go-example s823m92mMowd92eb ``` If you want to enable or disable basic authentication for an already existing application you can use the `--basic-auth` flag on the `nctl update app` command: ```bash $> nctl update app go-example --basic-auth=false ✓ updated Application "go-example" ⬆️ ``` ## Rotating credentials To rotate the credentials for basic authentication you can use : ```bash $ nctl update app go-example --change-basic-auth-password ✓ updated Application "go-example" ⬆️ ``` You can see the changed credentials by using: ```bash nctl get app go-example --basic-auth-credentials ``` ## Configuration layers You can also enable basic authentication on other [configuration layers](./deploio-configuration-layers). For example, to enable it for all applications in a project you can use: ```yaml $ nctl create config --basic-auth -p ``` --- ## Build customization To customize certain aspects of the build process, build environment variables can be used. You can specify them using the `--build-env` flag during creation or update of the application. ## Setting build environment variables on app creation You can use the `--build-env` flag of the `nctl create app` command to set build environment variables on application creation. ```bash --build-env=KEY=VALUE;... Environment variables which are passed to the app build process. $ nctl create app go-example --build-env='BP_GO_TARGETS=./cmd/app' ``` > depending on your shell, you may need to quote the arguments. Multiple build time env variables can be set by using the `--build-env` option multiple times or by adding the variables separated by a semicolon: ```bash $ nctl create app go-example --build-env='BP_GO_TARGETS=./cmd/app' --build-env='BP_KEEP_FILES=assets/*:public/*' ``` or ```bash $ nctl create app go-example --build-env='BP_GO_TARGETS=./cmd/app;BP_KEEP_FILES=assets/*:public/*' ``` ## Updating build environment variables To update build environment variables on existing applications you can use the `--build-env` flag of the `nctl update app` command. ```bash $ nctl update app go-example --build-env='BP_GO_TARGETS=./cmd/app' ``` ## More information All build environment variables are specific to the language and the buildpack that is being used in the backend to build your app. You can find the language specific supported build environment variables in the "Languages" section of the documentation. --- ## Buildpack Stacks A buildpack stack defines the set of buildpacks used to build your application. Deploio supports two stacks: `heroku` (the default) and `paketo`. ## Heroku Stack (Default) The heroku stack is the default and uses exclusively unchanged [Heroku Cloud Native Buildpacks](https://github.com/heroku/buildpacks). This ensures strict compatibility with standard Heroku CNB behavior and provides frequently updated runtime versions. ### Static Sites Automatic language detection for static sites is not available on the heroku stack, because that detection is handled by a nine-managed buildpack. When deploying a static site with the heroku stack using , you must explicitly set the language: ```bash nctl create application my-static-site \ --buildpack-stack=heroku \ --language=static \ --git-url=https://github.com/your-org/your-repo ``` ## Paketo Stack The paketo stack uses a combination of [Paketo buildpacks](https://paketo.io/), nine-managed buildpacks, and Heroku Cloud Native Buildpacks for some languages. This stack includes custom buildpacks that provide automatic language detection, including for static sites. ## Choosing a Stack The heroku stack is a good default for most applications. Consider switching to the paketo stack if: - You need automatic language detection for static sites, which is provided by a nine-managed buildpack not available in the heroku stack. - You need the specific features provided by Paketo buildpacks. ## Configuring the Buildpack Stack Use the `--buildpack-stack` flag when creating or updating an application: ```bash nctl create application my-app \ --git-url=https://github.com/your-org/your-repo ``` ```bash nctl create application my-app \ --buildpack-stack=paketo \ --git-url=https://github.com/your-org/your-repo ``` To switch an existing application between stacks: ```bash nctl update application my-app --buildpack-stack=heroku ``` ```bash nctl update application my-app --buildpack-stack=paketo ``` --- ## Configuration layers A Deploio application can be customized by multiple options. These options can be given at various configuration layers. All of the set configurations are merged in a specific order to create the final configuration for your application. The following sections will give an overview about the configuration system. ## Available configuration layers The following sources of configuration are available (ordered by precedence): - application configuration - git configuration via YAML file - project configuration - organization configuration - global default configuration The final configuration will be created by merging the configuration layers from top to bottom. Options set at a higher configuration layer overwrite those defined on a lower one. The final configuration will be attached to the latest release of the Deploio application. ![config merging](/img/deploio-config-merge.png) ### Application configuration The configuration which is set on the Deploio application itself is called "application configuration". It has the highest precedence, which means that all options are overwriting the corresponding options set on a lower configuration layer. You can view the configuration by using : ```bash $> nctl get app go -o yaml kind: Application apiVersion: apps.nine.ch/v1alpha1 metadata: name: go spec: forProvider: ... config: size: mini env: - name: MYDATABASE value: test port: null replicas: null ... ``` The application configuration can be seen at the yaml key `spec.forProvider.config`. In the above example an application size of "mini" is set. Additionally an environment variable was defined. As the "port" option is set to `null` it can still be defined on lower configuration levels. Same goes for the "replicas" value. You can do changes to the application configuration by using our CLI application (`nctl update app`) or use the web UI available at https://cockpit.nine.ch. ### Git configuration via yaml file The configuration can also be defined in a file named `.deploio.yaml` which you can store along with the application code in git. Our build system will check for the existence of such a file and read the contents of it. ```yaml title=".deploio.yaml" # Application size (micro, mini, standard-1, standard-2) size: micro # Port the app is listening on. port: 8080 ## Sets the amount of replicas of the running app. replicas: 1 # Env variables which are passed to the app at runtime. env: - name: RESPONSE_TEXT value: "Hello from a Go Deploio app!" # enables basic authentication for the application enableBasicAuth: true # A job that runs before a new release gets deployed. deployJob: name: "hello-go" command: echo "Hello from a Go Deploio app! # A job that runs in the background non-stop. workerJobs: - name: "sidekiq-worker" command: "bundle exec sidekiq -e production -C config/sidekiq.yml" # A job that is set to run at specific times. scheduledJobs: - name: "daily-backup" schedule: "0 3 * * *" command: /app/backup.sh ``` As already pointed out before, the settings specified directly in the application configuration take precedence over this file. An always up-to-date list of fields that can be used in the `.deploio.yaml` file is available in our . ### Project configuration Projects are a way to logically separate resources into different units. Besides resources like on-demand databases or Kubernetes clusters, projects can also contain Deploio applications. Additonally every project can have exactly one Deploio configuration which will apply to every application created in that project. This is called the "project configuration". You can create it by using `nctl create config`. ```bash $ nctl create config --env=RAILS_ENV=dev -p acme-dev ``` This creates a configuration in the project `acme-dev` which defines an environment variable _RAILS_ENV_ with the value "dev". All created applications in the project `acme-dev` will now have an environment variable _RAILS_ENV_ being set. The value of it can still be overwritten at higher configuration source layers. ### Organization configuration Every organization has a default project which has the same name as the organization itself. This project can not be removed. A Deploio configuration created in the default project is called "organization configuration" as it will be used by all Deploio applications, no matter in which project they are created. Creating a configuration at the organization level is very similar to creating one for a project. Just set the project name to the name of your organization. ```bash $ nctl create config --size=mini -p acme ``` The above example defines a Deploio configuration for the organization `acme` which defines an application size of "mini". This setting will be used by all Deploio applications (no matter in which project they are defined), except the size got overwritten at higher configuration layers. ### Global default configuration For every configuration field there is a global default value which will be used if no other configuration layer defined a value for it. You can see those global default values in our [API definitions](https://github.com/ninech/apis/blob/main/apps/v1alpha1/types.go). They are stored in the variable `DefaultConfig`. ### How merging works Single value options like "size" or "port" will be directly overwritten by higher configuration source layers. So a project configuration which specifies a size of "mini" for all applications in that project can directly be overwritten by specifying a size of "micro" in the application configuration (or via the yaml config file). Environment variables are merged in a different way. Variables defined in higher configuration source layers will be added to the ones defined in lower configuration source layers. If the same environment variable is defined on multiple layers, the definition of the higher layer will overwrite the one from the lower layer. It is currently not possible to remove environment variables defined at a lower configuration layer. ### Viewing the merged configuration If you want to see the final merged configuration, you can do so by getting the latest release for your Deploio application. ```bash $ nctl get app go -o yaml kind: Application apiVersion: apps.nine.ch/v1alpha1 metadata: name: go ... status: atProvider: latestRelease: finer-penance-sd78n ... ``` You can see the latest release at the yaml key `status.atProvider.latestRelease`. Now we can get the configuration for that release by using again. ```bash $ nctl get release finer-penance-sd78n -o yaml kind: Release apiVersion: apps.nine.ch/v1alpha1 metadata: name: finer-penance-sd78n ... spec: forProvider: configuration: size: size: mini origin: application env: - value: name: RESPONSE_TEXT value: Hello from a Go Deploio app! origin: git - value: name: MYDATABASE value: test origin: application port: value: 5678 origin: git replicas: value: 1 origin: git enableBasicAuth: value: false origin: default ... ``` You can see the final merged configuration at the yaml key `spec.forProvider.configuration`. For every field in the configuration you can see the originating configuration layer of it (field `origin`). In the above example the environment variable `RESPONSE_TEXT` was defined in the `.deploio.yaml` file colocated to the application source code, while the environment variable `MYDATABASE` was directly defined in the application configuration. Basic authentication was not configured specifically, so the global default value of `false` was used. --- ## Connecting to Services :::warning[Beta] This feature is currently in beta and only available through . Cockpit support is not yet implemented. ::: Deploio applications can connect to [On-Demand Services](../../on-demand-services/index.md) such as databases and key-value stores by declaring service references on the application. Nine automatically injects the connection credentials as environment variables into the app at runtime. Under the hood, Nine automatically creates and manages an encrypted [ServiceConnection](../../managed-kubernetes/nke/service-connections.md) in the target service's project. No manual networking setup is required. ## Supported Services The following service types can be referenced from a Deploio application: - [Key-Value Store](../../on-demand-services/key-value-store.md) - [MySQL](../../on-demand-services/mysql/index.md) - [PostgreSQL](../../on-demand-services/postgresql/index.md) - [OpenSearch](../../on-demand-services/opensearch.md) ## Add a Service Reference Each service reference requires a destination (the service to connect to). :::info[Coming soon] Managing service references in Cockpit is not yet available. We're currently working on the implementation. For now, please use . ::: When creating a new application, use `--service` with the format `name=kind/target-name`: ```bash nctl create application my-app \ --git-url=https://github.com/example/app.git \ --service cache=keyvaluestore/my-kvs ``` To add a service to an existing application: ```bash nctl update application my-app \ --service cache=keyvaluestore/my-kvs ``` To remove a service reference: ```bash nctl update application my-app \ --delete-service cache ``` ## Injected Environment Variables When service references are configured, Nine injects the connection details as environment variables into every instance of the application. Variable names use the format `NINE___`: - `` is a short code for the service type, such as `KVS`, `PG`, or `MYSQL`. Each table below shows the identifier for its service type. - `` is the **reference name** you assign in `--service =...` (the `name` part), uppercased with non-alphanumeric characters replaced by `_`. For example, the reference name `cache` becomes `CACHE`. - `` identifies the connection detail, such as `FQDN`, `PORT`, or `PASSWORD`. The variable names use the reference name you chose, not the target service's resource name. For example, `--service cache=keyvaluestore/my-kvs` produces variables like `NINE_KVS_CACHE_FQDN` — using `cache`, not `my-kvs`. :::info[A new release is required] Adding or changing a service reference does not update a running instance on its own. Nine injects the variables only when a new release is created. If your change doesn't already produce a new release, trigger one by using the `--retry-release` function of : ```bash nctl update app my-app --retry-release ``` A new release redeploys the application: Deploio rolls out new instances with the updated environment and replaces the running ones. Expect a deployment cycle, and plan the change accordingly. ::: The available variables depend on the service type: ### Verifying the Injected Variables :::info[Coming soon] Viewing the injected variables in Cockpit is not yet available. We're currently working on the implementation. For now, please use . ::: Verify the injected variables in two ways: - In the **release** resource (not the application resource). Releases have generated names, so list them first, then inspect the latest one. Its `serviceEnvVars` field lists the injected variable names (the values are kept in a referenced secret, so they aren't shown here): ```bash nctl get releases -p my-project nctl get releases -p my-project -o yaml ``` - From inside a running instance, by listing its environment. This shows both the names and the values: ```bash nctl exec app my-app -p my-project -- env ``` If the variables are missing, check if a new release was created after the service reference was added. --- ## Custom Host Names By default, every Deploio application gets a generated host name in the `deploio.app` domain. If you want to use your own domain, you can do so using the `--hosts` flag of when creating the application. ```shell-session --hosts=HOSTS,... Host names where the application can be accessed. If empty, the application will just be accessible on a generated host name on the deploio.app domain. $ nctl create app go-example --hosts=custom.host.example.com ``` With , you can also update your custom host names after the application was created: ```bash $ nctl update app go-example --hosts=custom.host.example.com,other.custom.host.example.com ``` ## Configuration Once you use custom host names for your application, you need to complete two additional steps: - Verify ownership of the custom host name's domain. - Point your custom host name to the Deploio infrastructure. For this, outputs a `TXT RECORD` and `DNS TARGET` once the first Build and Release was successfully created. You can also get the TXT record and CNAME content by using after you created the application by using the `--dns` flag: ```bash $ nctl get app test-app --dns NAME TXT RECORD DNS TARGET go-example deploio-site-verification=go-example-nine-46e4146 go-example.46e4146.deploio.app ``` Depending on the kind of host name you want to add, you either need the value of the `DNS TARGET` column or both values to fulfil the requirements. ## Creating CNAME Entries for Custom Hosts The easiest and recommended way to verify your custom host name and point it to the Deploio infrastructure in one step, is to create a CNAME record for it which points to the `DNS TARGET` given by . So, for example, when using the custom host name `custom.host.example.org` in the above application `go-example`, you would need to create the following CNAME record at your DNS provider: ``` custom.host.example.org CNAME go-example.46e4146.deploio.app ``` After the DNS record propagates (this might take some time), it verifies domain ownership and also points your custom host name to the Deploio infrastructure in one go. If your custom host name is an apex domain (a so-called 'apex entry'), such as `example.org` itself, you can't use a CNAME — see [Apex Domain DNS Entries](#apex-domain-dns-entries). For subdomains like `www.example.org`, creating the CNAME entry is sufficient. :::caution[CNAME Chaining] Deploio follows the full CNAME chain during domain verification, so your CNAME record can itself point to another CNAME. All domains in the chain must be under your control or explicitly trusted. If any intermediate domain expires and is re-registered by a third party, they could take over verification of your host name. To disable CNAME chain following, [contact our support](/docs/general/contact). ::: ## Apex Domain DNS Entries If your custom host name is an actual domain name (for example: `example.org`) rather than a subdomain, you will not be able to create a CNAME DNS record for it. The CNAME type is not permitted for so-called "apex DNS entries" (entries on the root level of your domain). Domain verification and pointing your host name to the Deploio infrastructure then requires creating two different DNS entries, as follows. ### Verification via TXT Record When using apex DNS host names, create a DNS `TXT record` which resolves to the value displayed in the `TXT RECORD` column of the output. The `TXT record` is used to verify domain ownership of the domain which hosts your custom host name. For example, to prove that you are permitted to use the custom host name 'example.org' for the above application `go-example`, create a TXT record at your DNS provider with the following content: ``` example.org TXT deploio-site-verification=go-example-nine-46e4146 ``` Deploio tries to verify this record periodically (which might take some time). Once Deploio verifies the record, the application starts listening for requests with the given custom host name. You can see all verified and unverified hosts by using : ```bash $ nctl get apps NAME HOSTS UNVERIFIED_HOSTS go-example go-example.46e4146.test.deploio.app example.org ``` ### Pointing Your Host Name to the Deploio Infrastructure There are several ways to point your custom host name to Deploio. The following sections explain each method. #### DNS ALIAS Record Some DNS providers allow to create so-called ALIAS DNS records. ALIAS records behave like CNAME records, but do appear as a normal DNS A record to the outside world. If your DNS provider supports ALIAS records, we recommend using them. Just create a DNS entry of type ALIAS which points to the value displayed in the [`DNS TARGET`](#configuration) column of . Again using the `go-example` application from above, the corresponding DNS entry needs to look like: ``` example.org ALIAS go-example.46e4146.deploio.app ``` Note that it might take some time until the record is fully propagated. #### DNS A Record If there is no support for ALIAS DNS records at your provider, create regular DNS A records for your custom host names. To achieve this, resolve the [`DNS TARGET`](#configuration) given by to an actual IP address. On Linux or macOS systems this can be done by using the `host` (or any equivalent) utility: ``` $ host test-app.edcd8e7.deploio.app test-app.edcd8e7.deploio.app is an alias for deploio.296dc8b.ingressnginx.nineapis.ch. deploio.296dc8b.ingressnginx.nineapis.ch has address 178.209.59.205 ``` In that case, create a DNS A record with the following content at your provider: ``` example.org A 178.209.59.205 ``` After the DNS entry fully propagates, you can access your application at `https://`. Propagation might take some time. --- ## Deploy Jobs A deploy job is a command that is executed before a new release gets deployed. The rollout of the release will only continue if the deploy job finished successfully. The command of the deploy job has access to the same environment and binaries as the app runtime. A common use-case for a deploy job is to run database schema migrations. It can either be specified on creation of the app or later using the `update` command. If a deploy job fails, the associated release will be set to failed and the previous release will continue to run if there was one to begin with. ## Specifying a deploy job on app creation You can specify a deploy job when creating an application by using various deploy job related flags on the `nctl create app` command. ```bash --deploy-job-command="rake db:prepare" Command to execute before a new release gets deployed. No deploy job will be executed if this is not specified. --deploy-job-name="release" Name of the deploy job. The deployment will only continue if the job finished successfully. --deploy-job-retries=3 How many times the job will be restarted on failure. Default is 3 and maximum 5. --deploy-job-timeout=5m Timeout of the job. Default is 5m, minimum is 1 minute and maximum is 30 minutes. $ nctl create app --deploy-job-command="rake db:prepare" ``` :::note The deploy job inherits its [resource limits from the deployed application](../). However, each deploy job operates with its own dedicated resource allocation, meaning its resources are not shared with the parent Deploio app. ::: The deploy job will run whenever the new release requires the app to be restarted. The following events will trigger the deploy job to run: - A new build is available - The configuration of the application changed (new environment variables, size change, etc) ## Specifying a deploy job on other configuration layers As deploy jobs are part of the configuration of a Deploio application, you can also specify them on other [configuration layers](configuration-layers.md). For example, you can specify a deploy job via the git configuration source file `.deploio.yaml`: ```yaml title=".deploio.yaml" deployJob: name: "release" command: "rake db:prepare" retries: 3 timeout: 5m ``` ## Viewing the status of a deploy job If a deploy job fails, the associated release will be set to failed and the previous release will continue to run if there was one to begin with. To see the detailed status of a deploy job you can get the full release: ```bash $ nctl get releases brave-scrambler-vc78b -o yaml [...] status: atProvider: deployJobStatus: exitTime: 2023-07-18T11:01:47Z name: brave-scrambler-vc78b-deploy-job reason: backoffLimitExceeded startTime: 2023-07-18T11:00:58Z status: failed releaseStatus: failure ``` At the bottom of the release you can see the status and it will show in detail when and how a deploy job failed. In addition to the status, the deploy job's log will be written to the normal app log and can be accessed using the `nctl logs app` command. --- ## Health Probe By default in Deploio, an application is considered ready as soon as the web server starts listening on its port. If it still needs extra startup work (e.g., load data, run migrations), traffic might arrive too early and cause temporary errors. A custom health probe URL fixes this by letting the platform poll your app until it reports itself as healthy - only then is traffic routed to it. This probe is optional. If you don't configure it, the platform will use its default health check behavior. ## What you configure We keep this simple - there are only two fields. The `--health-probe-period-seconds` flag controls how often the platform runs the health probe against your application. It is specified in seconds, with a default value of 10 and a minimum allowed value of 1. The `--health-probe-path` flag specifies the HTTP path on your application that the platform should request to determine whether the service is healthy, for example `/healthz`. Together, these flags let you define how frequently the health check runs and which endpoint should be used to signal when the application is ready and functioning properly. :::note Your app controls the response. During startup, return a non-success status to indicate it is not yet ready, and once healthy, return a success HTTP status code. Any code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure. Refer to the `deploio-examples` repository for additional code samples. For example, [the sample Go application](https://github.com/ninech/deploio-examples/tree/main/heroku-stack/go) defines a `/healthz` endpoint, which serves as a simple demonstration of a custom health probe. ::: ## Specifying a custom health probe during app creation You can configure a health probe when creating an application by using the health probe flags on the `nctl create app` command. ```bash nctl create app --health-probe-path="/healthz" --health-probe-period-seconds=9 ``` ## Removing a custom health probe Use this command if you want to remove a previously configured health probe and revert to the platform's default health check behavior: ```bash nctl update app --delete-health-probe ``` ## Specifying a health probe on other configuration layers As health probes are part of the configuration of a Deploio application, you can also configure them on other [configuration layers](configuration-layers.md). For example, you can set a health probe via the git configuration source file `.deploio.yaml`: ```yaml title=".deploio.yaml" healthProbe: httpGet: path: /healthz periodSeconds: 9 ``` ## Examples Refer to the [deploio-examples repository](https://github.com/ninech/deploio-examples) for additional code samples. For example, the sample Go application defines a `/healthz` endpoint, which serves as a simple demonstration of a custom health probe. --- ## Pausing an app You can pause an app in Deploio to stop all costs. This will scale down all replicas of your application and stop any underlying jobs (worker jobs, scheduled jobs, etc.). To pause an application, use the following nctl command: ```bash $ nctl update app --pause ``` --- ## Accessing private repositories There are 2 methods to access private git repositories. You can either use the [SSH](#using-ssh-to-access-your-repository) or [HTTPS](#using-https-to-access-your-repository) protocol to authenticate to the repository. Both ways are explained in the following sections. ## Using SSH to access your repository Using SSH to access a private git repository is our recommended approach. If you already have a ready SSH key pair to use, you can skip this section. To create a new SSH key pair you can use the `ssh-keygen` command. For example, on macOS or Linux based systems the following instruction creates a new key pair: ```bash ssh-keygen -t ed25519 -f ~/deploio.key -N '' ``` This will create a private SSH key in ed25519 format and place its content into the file `~/deploio.key`. The corresponding public part will be written to `~/deploio.key.pub`. The private key will not be password protected, which is important as otherwise Deploio couldn't read the content of it. In a future release, will automatically create the SSH key pair for you and output the public part ready for registering at your git provider (see below). When developing applications using , an automatically generated SSH key is already available for use. ### Registering the public key The public part of the just created key pair needs to be registered at your git provider so that Deploio can read the content of the repository. You will need to create a so called "deploy key" to accomplish this. You can find documentation links for various git providers in the table below. | Provider | Documentation | | --------- | -------------------------------------------------------------------------------------------------------------------------------- | | GitHub | [managing deploy keys](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#deploy-keys) | | Bitbucket | [ssh keys for system use](https://confluence.atlassian.com/bitbucketserver/ssh-access-keys-for-system-use-776639781.html) | | GitLab | [deploy keys](https://docs.gitlab.com/ee/user/project/deploy_keys/) | In the corresponding dialog for creating a new deploy key you can use any 'Title' or 'Name' when being asked for. Use something which indicates the client using this deploy key (e.g. "deploio"). You will then need to paste the content of the file which holds your public key (`~/deploio.key.pub` in above example) into the "Key" field. You don't need to grant write access for the deploy key as Deploio will just need to have read access. ### Configuring the Deploio application Once the key is registered, we can use to create the Deploio application. Here we are using GitHub as an example provider for your git repository, but you can replace the `git@github.com` part with the URL identifying your provider (e.g. `git@gitlab.com`). ```bash nctl create app \ --git-ssh-private-key-from-file=~/deploio.key \ --git-url=git@github.com:.git ``` You can also pass the content of the private key in a env variable called $GIT_SSH_PRIVATE_KEY or reference the file which contains the private key by using $GIT_SSH_PRIVATE_KEY_FROM_FILE. If the application already exists, you can use `nctl update app` to set SSH authentication. ## Using HTTPS to access your repository You can also use a username/password combination to let Deploio access your git repository via HTTPS. For this, it is best to create a repository scoped deploy token which can only be used to access a specific repo. We do not recommend to use a personalized deploy/access token as it will be directly associated with your user account and might have too broad permissions which are not needed by Deploio in the end. Please also do not set an expiration date on the deploy token. Currently, not all git providers have support for repository scoped deploy tokens. This is one of the reasons why we recommend to use deploy keys as they are generally better supported. Here are a few documentation links to set up repository scoped deploy tokens at various git providers: | Provider | Documentation | Notes | | --------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------- | | Bitbucket | [Repository Access Tokens](https://support.atlassian.com/bitbucket-cloud/docs/repository-access-tokens/) | Use `x-auth-token` as the username | | GitLab | [deploy tokens](https://docs.gitlab.com/ee/user/project/deploy_tokens/) | | ### Configuring the Deploio application Once you created a repository scoped deploy token, you can use to create an application: ```bash nctl create app \ --git-username= \ --git-password= \ https://.git ``` You can also pass the username and password by using the environment variables $GIT_USERNAME and $GIT_PASSWORD. If the application already exists, you can use `nctl update app` to set HTTPS authentication. --- ## Procfile The Procfile is something that Heroku introduced and has been adopted as somewhat of a standard for expressing what commands are run to start an application. In the case of Deploio you can use the Procfile to override the otherwise automatically generated `web` entrypoint that will be executed when starting your application. For example, when trying to run a [Docusaurus](https://docusaurus.io/) project on Deploio, the following Procfile is needed as by default it would try to start a development server: ```yaml title="Procfile" web: npm run serve -- -p $PORT ``` Other keys than `web` won't influence how your app is being run on Deploio. --- ## Scheduled Jobs A scheduled job is a command that runs at regular intervals based on a predefined schedule. The command of the scheduled job has access to the same environment and binaries as the app runtime. It can either be specified on creation of the app or later using the `update` command. ## Specifying a scheduled job on app creation You can specify a scheduled job when creating an application by using various scheduled job related flags on the `nctl create app` command. ```bash --scheduled-job-command="bundle exec rails runner" Command to execute to start the scheduled job. --scheduled-job-name=scheduled-1 Name of the scheduled job job to add. --scheduled-job-size=micro Size (resources) of the scheduled job (defaults to "micro"). --scheduled-job-schedule="* * * * *" Cron notation string for the scheduled job (defaults to "* * * * *"). $ nctl create app --scheduled-job-command="sleep 5; date" --scheduled-job-name="myjob-1" ``` The scheduled job will run depending on the schedulde parameter. The parameter is given in cron notation. You can use (this website)[https://crontab.guru/] to easier understand your schedule. If the scheduled job takes longer than the defined interval, the next execution will be skipped. Running scheduled jobs will not be interrupted. ## Specifying a scheduled job on other configuration layers As scheduled jobs are part of the configuration of a Deploio application, you can also specify them on other [configuration layers](configuration-layers.md). For example, you can specify a scheduled job via the git configuration source file `.deploio.yaml`: ```yaml title=".deploio.yaml" scheduledJobs: - command: sleep 60; date name: scheduled-1 retries: 0 schedule: "*/5 * * * *" size: micro timeout: 5m0s ``` ## Updating and deleting scheduled jobs You can use the same parameters as above with nctl to update your scheduled job: ```bash --scheduled-job-command="bundle exec rails runner" Command to execute to start the scheduled job. --scheduled-job-name=scheduled-1 Name of the scheduled job job to add. --scheduled-job-size=micro Size (resources) of the scheduled job (defaults to "micro"). --scheduled-job-schedule="* * * * *" Cron notation string for the scheduled job (defaults to "* * * * *"). $ nctl update app --scheduled-job-command="sleep 50; date" --scheduled-job-name="myjob-1" ``` To delete a scheduled job, you can also use the update command: ```bash --delete-scheduled-job=DELETE-SCHEDULED-JOB Delete a scheduled job by name $ nctl update app --delete-scheduled-job="myjob-1" ``` ## Viewing the status of a scheduled job If a scheduled job fails, the associated release will NOT be set to failed and continue running. To see the detailed status of a scheduled job you can get the full release: ```bash $ nctl get releases brave-scrambler-vc78b -o yaml [...] status: atProvider: scheduledJobStatus: - name: scheduled-1 replicaObservation: - replicaName: go-scheduled-scheduled-1-29038220 status: succeeded ``` At the bottom of the release you can see the status and it will show in detail the status of the scheduled job. In addition to the status, the scheduled job's log will be written to the normal app log and can be accessed using the `nctl logs app` command. --- ## Use a custom port In most cases you don't need to change the default port that Deploio chooses for your app. The chosen port will be injected as the env variable `$PORT` to the application at runtime. So in any case, for compiled languages you have to make sure the app is listening on the port defined in that env variable. For other languages such as Ruby, PHP, NodeJS or Python this will be automatically taken care of. ## Setting a custom port on app creation If you need to customize the port, you can configure it using the `--port` flag of : ```bash --port=8080 Port the app is listening on. $ nctl create app go-example --port=5678 ``` ## Updating the port of an application For already existing applications you can use `nctl update app` with the `--port` flag to update the port. ```bash $ nctl update app go-example --port=5678 ``` --- ## Setting environment variables You might want to configure extra environment variables that will be passed to the app at runtime, such as database credentials. ## Environment Variables Available by Default Every Deploio app is automatically provided with the following environment variables: For Dockerfile builds, variables that should be available at build time must be declared as an [`ARG`](https://docs.docker.com/reference/dockerfile/#arg) in your `Dockerfile`. See [Dockerfile Build](../dockerfile.md#build-arguments) for details. ## Set environment variables on app creation You can specify env variables on app creation with using the `--env` flag: ```bash --env=KEY=VALUE;... Environment variables which are passed to the app at runtime. $ nctl create app rails-example --env=RAILS_ENV=dev ``` Multiple environment variables can be added by using the `--env` option multiple times or by using a semicolon to separate them: ```bash $ nctl create app rails-example --env=RAILS_ENV=dev --env=MAIL_USERNAME=test $ nctl create app rails-example --env='RAILS_ENV=dev;MAIL_USERNAME=test' ``` ## Updating environment variables On existing applications you can use `nctl update app` to set new environment variables or change the values of already set ones. ```bash $ nctl update app rails-example --env=RAILS_ENV=prod ``` To delete existing environment variables you can use the `--delete-env` flag ```bash $ nctl update app rails-example --delete-env=RAILS_ENV ``` ## Listing the set environment variables To see the currently set environment variables, you can use the `nctl get app` command and specifying the `yaml` output format. ```bash $ nctl get app rails-example -o yaml kind: Application apiVersion: apps.nine.ch/v1alpha1 metadata: name: rails-example ... spec: forProvider: config: env: - name: MAIL_USERNAME value: test ... ``` --- ## Worker Jobs Worker jobs are separate processes which use the same image as the application, but with a different entry point. The worker has access to the same environment and binaries as the app runtime. One common use-case is to start a task scheduler which executes tasks on a regular base. It can either be specified on creation of the app or later when updating the app. ## Specifying a worker job on app creation You can specify a worker job when creating an application by using various worker job related flags on the `nctl create app` command. ```bash --worker-job-command="bundle exec sidekiq" Command to execute to start the worker. --worker-job-name=sidekiq Name of the worker job to add. --worker-job-size=micro Size of the worker (defaults to "micro"). $ nctl create app --worker-job-name=sidekiq --worker-job-command="bundle exec sidekiq" ``` :::note The size of a worker job can be set independently from its parent application. The same [sizing options](../) apply. ::: During creation, it's just possible to define a single worker job. You can add up to 3 worker jobs using the `nctl update app` command. The worker job will run together with the main app process. It will be restarted in the following cases: - A new build is available - The configuration of the application changed (new environment variables, size change, etc) The system ensures that only one instance of each worker job runs at the same time. A failing worker job won't impact the release of the main app and won't block it from starting. If the worker exits for any reason, it will be automatically restarted. ## Specifying a worker job on other configuration layers As worker jobs are part of the configuration of a Deploio application, you can also specify them on other [configuration layers](configuration-layers.md). For example, you can specify a worker job via the git configuration source file `.deploio.yaml`: ```yaml title=".deploio.yaml" workerJobs: - name: sidekiq command: "bundle exec sidekiq" size: mini ``` ## Viewing the status of a worker job The simplest way to view the status of a worker job with is to use the [`-o stats` command](../observing-your-app#stats). Additionally, the logs of the worker jobs can be accessed by viewing the [app logs](../observing-your-app#logs). --- ## Copying an App To copy an existing app you can use the `copy application` command. This creates a new app with the same configuration as your existing app with some minor exceptions that are explained below. In this example, the existing app named `go-example` in the currently active project is copied to a new app named `go-example-v2`. ```shell-session nctl copy application go-example --target-name=go-example-v2 ``` To copy an app to a different project, you can use the `--target-project` flag. ```shell-session nctl copy application go-example --target-name=go-example-v2 --target-project=nine-project2 ``` By default the copied app will be in paused state so you can review it and make adjustments before starting it. To override this behaviour you can pass the `--start` flag to the copy command to immediately start the copied app. Similarly, the copy won't include the custom hosts by default. To override this behaviour you can pass the `--copy-hosts` flag to the copy command to also copy over all hosts of the existing app. When copying hosts, they will need to be [verified again](./configuration/custom-hosts.md#configuration) and the hosts need to be removed from the old app, in case it still exists, before they become active on the new app. --- ## Dockerfile Build With [Dockerfile](https://docs.docker.com/build/concepts/dockerfile/) builds, Deploio can build any app that can be built using a Dockerfile. This is especially useful if your application uses a programming language or runtime that Deploio does not yet support natively. ## Getting Started Creating a Dockerfile application works like any other application in Deploio. The only requirements are that your repository contains a `Dockerfile` and that you specify the `--dockerfile` argument during creation (or enable the **Dockerfile Build** switch in Cockpit). We have a basic Dockerfile app in our [examples repository](https://github.com/ninech/deploio-examples/tree/main/dockerfile). 1. Open the [Create Application](https://cockpit.nine.ch/en/deploio/apps/applications/new) page. 2. Provide your repository details. 3. Enable the **Dockerfile Build** switch. You can deploy the example with : ```bash nctl create application dockerfile-rust \ --git-url=https://github.com/ninech/deploio-examples \ --git-revision=main \ --git-sub-path=dockerfile/rust \ --dockerfile ``` ## Configuration Deploio allows you to customize various aspects of your Dockerfile-based application: ### Dockerfile Path By default, the `Dockerfile` in your repository root is used. After enabling **Dockerfile Build**, you can specify the path in the **Dockerfile Path** field. Use the flag `--dockerfile-path` to specify a `Dockerfile` at a different location: ```bash --dockerfile-path="path/to/Dockerfile" ``` ### Build Context By default, the system sets the build context to your repository root. After enabling **Dockerfile Build**, you can specify the directory in the **Build Context** field. Use the flag `--dockerfile-build-context` to specify a different location: ```bash --dockerfile-build-context="path/to/build/context/" ``` ### Build Arguments You can pass [Dockerfile build arguments](https://docs.docker.com/build/building/variables/#build-arguments) to your build. Add your arguments in the **Build Environment Variables** section. Use the `--build-env` flag. For example, to define a `ARG` named `APP_VERSION` with a value of "v0.0.1": ```bash --build-env=APP_VERSION=v0.0.1 ``` Deploio also automatically provides certain variables as build arguments, such as `DEPLOIO_GIT_REVISION`. To use them during the build, declare them as `ARG` in your `Dockerfile`: ```dockerfile ARG DEPLOIO_GIT_REVISION RUN echo "Building revision $DEPLOIO_GIT_REVISION" ``` The following variables are automatically provided: ## Runtime and Health Checks The Deploio runtime uses the `ENTRYPOINT` and `CMD` specified in the Dockerfile to start your application. To serve traffic to your app, the runtime expects it to listen on a TCP socket at `0.0.0.0:$PORT`. The port defaults to `8080` if not specified, but you can configure it to any valid port number in the app definition. The runtime checks app health via a TCP probe to the configured port and traffic only flows to the app once the probe is successful. If the TCP probe fails at any point of the lifecycle, the runtime restarts the app. If the app exits for any reason, the runtime automatically restarts it. ## Optimize Image Size Images built with Deploio's Dockerfile build should not exceed 2 GiB uncompressed. There's a hard limit at 10 GiB for the whole build environment but with an image size of more than 2 GiB, the system won't be able to cache all the layers anymore and you'll notice degraded building performance. ## Best Practices Any best practices that apply to Dockerfiles in general also apply to Dockerfile builds on Deploio. For a comprehensive guide, see the [official Docker best practices](https://docs.docker.com/build/building/best-practices/). - Minimize your image size for faster builds, faster releases, and a decreased attack surface. - Use [multi-stage builds](https://docs.docker.com/build/building/multi-stage/) for compiled languages whenever possible. ## Restrictions Please note the following restrictions: - It is currently not possible to directly convert an existing application that is using buildpacks into a Dockerfile-based application. You must delete and recreate the application to use Dockerfile builds. The following script automates this process using and `jq` (both required). It deletes the existing application and recreates it with Dockerfile builds enabled. Pass `enable` to switch to Dockerfile builds or `disable` to switch back to buildpacks. :::warning This script deletes and recreates your application, causing temporary downtime. ::: ```bash #!/usr/bin/env bash set -euo pipefail if [[ $# -ne 3 ]] || [[ "$3" != "enable" && "$3" != "disable" ]]; then echo "Usage: $0 " >&2 echo " enable = Dockerfile build, disable = buildpacks" >&2 exit 1 fi PROJECT=$1 APP=$2 [[ "$3" == "enable" ]] && DOCKER_BUILD_ENABLED=true || DOCKER_BUILD_ENABLED=false TMP=$(mktemp "${TMPDIR:-/tmp}/${APP}.XXXXXX") RAW=$(mktemp "${TMPDIR:-/tmp}/${APP}.XXXXXX") trap 'rm -f "$TMP" "$RAW"' EXIT set +e nctl get applications "$APP" --project="$PROJECT" -o json > "$RAW" 2>&1 RC=$? set -e if [[ $RC -ne 0 ]]; then echo "nctl exited with status $RC. Output was:" >&2 cat "$RAW" >&2 exit 1 fi if ! jq empty "$RAW" 2>/dev/null; then echo "nctl did not return valid JSON. Output was:" >&2 cat "$RAW" >&2 exit 1 fi jq --argjson enabled "$DOCKER_BUILD_ENABLED" ' del(.metadata.creationTimestamp, .metadata.resourceVersion, .metadata.uid, .status) | .spec.forProvider.dockerfileBuild.enabled = $enabled | if $enabled then del(.spec.forProvider.language) else . end ' "$RAW" > "$TMP" [[ -s "$TMP" ]] || { echo "Empty config extracted, aborting." >&2; exit 1; } echo "Wrote config to: $TMP" >&2 cat "$TMP" echo "" >&2 echo "WARNING: Application '$APP' in project '$PROJECT' will be DELETED and recreated." >&2 echo "There will be TEMPORARY DOWNTIME until the new application is running." >&2 echo "" >&2 read -r -p "Press ENTER to continue or Ctrl+C to abort: " nctl delete application "$APP" --project="$PROJECT" --wait nctl create -f "$TMP" ``` --- ## Getting Started with Deploio Deploio is a fully managed app platform where you just bring the source code of your web application and it'll take care of building and deploying it continuously. Currently, the following languages are supported: - [Ruby](./languages/ruby.md) - [PHP](./languages/php.md) - [Node.js](./languages/nodejs.md) - [Go](./languages/go.md) - [Python](./languages/python.md) - [Static Sites](./languages/static.md) If your favorite language is missing and you would like to use Deploio, please let us know or activate the [Dockerfile feature](./dockerfile.md). We will add more languages based on demand. ## Deploying your first app Deploio is available via the , a CLI and an API. To install and setup the CLI, refer to the documentation of . You need at least version `v1.1.0` for using Deploio. You can check the installed version with `nctl --version`. Once is setup you can create your first application. In this guide we'll be deploying one of the example apps that we provide in our [deploio-examples](https://github.com/ninech/deploio-examples) repository. For the most minimal example, we just need to provide the git url containing the app source code and the subpath if the app is not in the root of the repository. Feel free to fork the [GitHub repository](https://github.com/ninech/deploio-examples) so you can make changes and see how they affect the app. ```bash $ nctl create app go-example --git-url=https://github.com/ninech/deploio-examples --git-sub-path=heroku-stack/go ``` ```bash Creating a new application ✓ created application "go-example" 🏗 ✓ waiting for build to start ⏳ ✓ building application 📦 ✓ releasing application 🚦 ✓ release available ⛺ Your application "go-example" is now available at: https://go-example.46e4146.deploio.app ``` After the creation has finished, you should be able to access the app: ```bash $ curl https://go-example.46e4146.deploio.app Hello from a Go Deploio app! ``` ### Using a private git repository In our example we used a public git repository that does not need any authentication for pulling the code. If your application code is hosted in a private repository, you can configure [git authentication in various ways](configuration/deploio-private-repository-access). ## Builds and Releases Now that we have an application deployed, we can dig into the details a bit and see the lifecycle of an application. During the creation there are two essential phases: - Build Phase: This is triggered by any source code change. - Release Phase: This is triggered by either an image change resulting from a build or a config change. Both of these are represented within the CLI with their respective commands. ```bash # get all builds $ nctl get builds nctl get builds NAME APPLICATION STATUS AGE go-example-build-1 go-example success 5m # get all releases $ nctl get releases NAME BUILDNAME APPLICATION SIZE REPLICAS STATUS AGE saved-hawkeye-2jcvs go-example-build-1 go-example micro 1 available 7m ``` To demonstrate a new release, we can increase the replicas from 1 to 2 by issuing the `nctl update app` command: ```bash $ nctl update app go-example --replicas=2 ✓ updated Application "go-example" ⬆️ $ nctl get releases NAME BUILDNAME APPLICATION SIZE REPLICAS STATUS AGE saved-hawkeye-2jcvs go-example-build-1 go-example micro 1 superseded 100m helping-aztec-l492k go-example-build-1 go-example micro 2 available 6s ``` As you can see another release has been created with 2 replicas. This change happens immediately as the app code itself did not change and the same build can be reused. If we were to change the source a new build would be triggered and after build success we would also see a new release using the new build. ## Accessing logs To view logs, troubleshoot issues and get an overview of the resources your app uses see [Observing your App](./observing-your-app). ## Configuring the app to your needs In our previous example we have demonstrated the most basic use-case for creating an app on Deploio. There are quite a few options, like using [custom host names](configuration/deploio-custom-hosts) or [setting environment variables](configuration/deploio-setting-environment-variables) which you can configure on a Deploio application to customize things to your needs. Please see the corresponding documentation in our "Configuration" section on the left to get more information about all possible options. ## Projects When you first start interacting with , everything you create will be scoped in your main project, which has the same name as your organization. To be able to separate different environments logically, you can create projects. Project names need to be prefixed with your organization name. So for example for the organization `acme`, a project could be named `acme-dev`. You are free to name it anything that fits the purpose, it just has to be prefixed with the organization name. ```bash $ nctl create project acme-dev Creating new project acme-dev for organization acme ✓ created project "acme-dev" 🏗 ✓ waiting for project to be ready ⏳ ✓ project ready 🛫 ``` Once you have created a new project, you can switch to default all commands in a project using `nctl auth set-project`. ```bash $ nctl auth set-project acme-dev ``` After that, any command that interacts with resources will be scoped inside said project. You can also use the `-p/--project` flag to e.g. get apps in a specific project without changing the default. To list all apps in a project you can use `-A/--all-projects` flag. ### Name vs Display Name Once a project is created, its name cannot be changed. But you can always change the display name. - Name: This is the project's internal name. Use it when interacting with the API and when using . - Display Name: This is the name displayed in the Cockpit. ## Maintenance Windows For more information, see the [Deploy.io weekly maintenance window](../general/weekly-maintenance-window.md#deploio). --- ## HTTP Headers Deploio apps are served by our TLS-termination proxy, which set the following HTTP headers to provide request information: ## Headers description - `X-Forwarded-Host`: Specifies the original domain that the user visited before being redirected to your server. - `X-Forwarded-Proto` - `X-Forwarded-Scheme`: Indicates whether the original request was made over a secure (HTTPS) or an insecure (HTTP) connection. - `X-Forwarded-Port`: Provides information about the port number used by the original request. - `X-Forwarded-For` - `X-Real-Ip`: Contains the IP address of the client that has connected to Deploio. The original chain of IP addresses can be found in the `X-Original-Forwarded-For` header. - `X-Original-Forwarded-For`: Contains the value of the X-Forwarded-For header set by a proxy in front of Deploio. This value should only be trusted if the proxy in front of Deploio can be trusted. ### Example ```yaml X-Forwarded-Host: http-headers.3a76d95.deploio.app X-Forwarded-Proto: https X-Forwarded-Scheme: https X-Real-Ip: 5.148.163.125 X-Forwarded-Port: 443 X-Forwarded-For: 5.148.163.125 ``` --- ## Deploio (PaaS) Deploio is a fully managed app platform where you just bring the source code of your web application and it takes care of building and deploying it continuously. To get started, see the [Getting Started with Deploio](./getting-started-with-deploio.md) guide. ## Pricing For database pricing, see [On-Demand Services pricing](../on-demand-services/index.md). ## Sizing Overview | Size | Standard RAM | CPU | Ephemeral Storage | | ---------- | ------------ | -------- | ----------------- | | micro | 256 MiB | 1/8 Core | 2 GiB | | mini | 512 MiB | 1/4 Core | 2 GiB | | standard-1 | 1 GiB | 1/2 Core | 2 GiB | | standard-2 | 2 GiB | 3/4 Core | 2 GiB | Each Deploio app, along with its corresponding jobs (for example, [deploy](./configuration/deploy-jobs.md) or [worker](./configuration/worker-jobs.md) jobs), receives a standard amount of resources (RAM, CPU, and ephemeral storage) when it runs. These resources are assigned individually and are not shared. If the app's or job's resource usage exceeds its standard limit, [Nine](https://nine.ch/) reserves the right to terminate the app. Every replica receives the documented amount of resources. The amount of resources is not shared between replicas. ## Community Join the [Deploio Community Slack Workspace](https://join.slack.com/t/deploiocommunity/shared_invite/zt-20tb3k93m-O4NEUs0RjZYGQNQoih8zkA) to get help. You can also open a support ticket (link top right) at any time. ## Supported Languages - [Ruby](./languages/ruby.md) - [PHP](./languages/php.md) - [Node.js](./languages/nodejs.md) - [Go](./languages/go.md) - [Python](./languages/python.md) - [Static Sites](./languages/static.md) If your language is missing, you can use the [Dockerfile feature](./dockerfile.md). --- ## Go The Deploio Go build environment makes use of the [Heroku Go Cloud Native Buildpack](https://github.com/heroku/buildpacks-go/) on both the paketo and heroku [buildpack stacks](../configuration/buildpack-stacks.md), so behavior is identical regardless of stack. ## Example App We have a basic Go app in our [examples repository](https://github.com/ninech/deploio-examples). You can deploy it with : ```bash nctl create application go \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/go ``` ```bash nctl create application go \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/go ``` ## App Requirements Any Go project that meets the following criteria should be buildable: - There is a go.mod at the root of the project. - The app compiles with go 1.16 or greater. - The app uses Go Modules for any dependency installation. ## Go version detection The Go version is read from the `go` line in `go.mod`. This is likely correct for most apps, but a different version may be selected using a [build directive in `go.mod`](https://github.com/heroku/buildpacks-go/tree/main?tab=readme-ov-file#go-version). ## Multiple Binaries The build process will build all main packages that it detects in the project. If you have multiple main packages, you might need to define the desired app entrypoint with a [`Procfile`](../configuration/procfile.md). For example, if your main.go file rests in a directory called `server`, the `Procfile` should look like this: ```yaml title="Procfile" web: server ``` This will result in the binary `server` being executed as the app entrypoint. If you want to only build selective packages, you can use a [directive in the `go.mod` file for that](https://github.com/heroku/buildpacks-go/tree/main?tab=readme-ov-file#package-installation). --- ## Node.js The Deploio Node.js build environment uses the [Paketo Node.js buildpack](https://paketo.io/docs/reference/nodejs-reference/) on the paketo stack, and the [Heroku Node.js Cloud Native Buildpack](https://github.com/heroku/buildpacks-nodejs/) on the heroku stack. See [Buildpack Stacks](../configuration/buildpack-stacks.md) for more information. ## Example App We have a basic Next.js app in our [examples repository](https://github.com/ninech/deploio-examples). You can deploy it with : ```bash nctl create application nextjs \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/nodejs/nextjs ``` ```bash nctl create application nextjs \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/nodejs/nextjs ``` ## Paketo Stack Configuration The build process offers a few env variables to adjust it to your use-case. See the [how to](https://paketo.io/docs/howto/nodejs/) section of the documentation for all available variables. ### Build an App in a Subdirectory To specify a subdirectory to be used as the root of the app, you can use the `BP_NODE_PROJECT_PATH` build variable. ```bash --build-env=BP_NODE_PROJECT_PATH="./node-app" ``` ## NextJS In specific scenarios, such as when utilizing the NextJS router, additional steps are essential to get your application to build. ### NODE_ENV When using the **paketo stack**, the `NODE_ENV` environment variable must be explicitly set to `production` to ensure a successful build. This requirement stems from an existing upstream issue that cannot be rectified. ```bash nctl create app next --git-url= --build-env=NODE_ENV="production" --env=NODE_ENV="production" ``` This is not required when using the heroku stack. --- ## PHP The Deploio PHP build environment uses the [Paketo PHP buildpack](https://paketo.io/docs/reference/php-reference/) on the paketo stack, and the [Heroku PHP Cloud Native Buildpack](https://github.com/heroku/buildpacks-php/) on the heroku stack. See [Buildpack Stacks](../configuration/buildpack-stacks.md) for more information. ## Example App We have a basic Symfony app in our [examples repository](https://github.com/ninech/deploio-examples). You can deploy it with : ```bash nctl create application symfony \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/php/symfony ``` ```bash nctl create application symfony \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/php/symfony ``` ## Using Extensions ### Paketo Stack The [Paketo PHP buildpack](https://paketo.io/docs/reference/php-reference/) includes the [Paketo php-dist buildpack](https://github.com/paketo-buildpacks/php-dist) which provides the PHP binary distribution. The built PHP binary distribution includes quite some extensions which can be used in your PHP application on Deploio. All of them are defined in separate yaml files per PHP version in the [Paketo php-dist buildpack](https://github.com/paketo-buildpacks/php-dist/tree/main/dependency/actions/compile/extensions-manifests). Currently it is not possible to use extensions which are not defined in the above mentioned files. It is important to mention that none of the pre-built extensions gets loaded by default (due to memory usage optimizations). You will either have to load them via requirements in Composer or via custom `*.ini` files. Both approaches will be explained in the following sections. #### Loading Extensions via Composer If you are using Composer as a package manager, you can specify extensions to load through the `composer.json` file. For example, to load the bz2, curl and zip extensions you can use the following content: ```json title="composer.json" { "require": { "php": ">=8.1", "ext-bz2": "*", "ext-curl": "*", "ext-zip": "*" } } ``` This is also documented in the [official Composer documentation](https://getcomposer.org/doc/articles/composer-platform-dependencies.md#composer-platform-dependencies). #### Loading Extensions via Custom .ini Files If you are not using Composer, you can load extensions via custom `*.ini` files located at `/.php.ini.d/*.ini` in your application source code repository. For example, to load the bz2, curl and zip extensions you could create a file `/.php.ini.d/custom-extensions.ini` with the following content: ```ini title=".php.ini.d/custom-extensions.ini" extension=bz2.so extension=curl.so extension=zip.so ``` #### Composer Platform Requirements As the build and runtime containers are different on Deploio you may run into issues where you cannot build a project successfully due to platform requirements not being fulfilled by the build-time container. You can ignore these requirements using either ```bash --build-env=BP_COMPOSER_INSTALL_OPTIONS="--ignore-platform-reqs" ``` which will ignore all build requirements, or you can scope it to specific extensions: ```bash --build-env=BP_COMPOSER_INSTALL_OPTIONS="--ignore-platform-req=ext-mysqli" ``` When doing this you will see from the logs that the buildpack will still validate that the extensions you require are available in the runtime image, but that the build will no longer fail due to the build container missing extensions. ### Heroku Stack Extensions are declared in `composer.json` using the `ext-` prefix in the `require` section. See the [Heroku documentation on managing PHP extensions](https://devcenter.heroku.com/articles/managing-php-extensions) for the full reference. ```json title="composer.json" { "require": { "ext-bcmath": "*", "ext-gmp": "*" } } ``` ## Paketo Stack Configuration The build process offers a few env variables to adjust it to your use-case. See the [how to](https://paketo.io/docs/howto/php/) section of the documentation for all available variables. ### Select a Web Server By default, the PHP built-in web server will be used. For production use-cases we recommend using Apache or NGINX: - PHP Built-in Web Server ```bash --build-env=BP_PHP_SERVER=php-server ``` - Apache HTTPD Web Server ```bash --build-env=BP_PHP_SERVER=httpd ``` - NGINX Web Server ```bash --build-env=BP_PHP_SERVER=nginx ``` Additionally, if required, the web server can be customized further by providing [your own server-specific config file](https://paketo.io/docs/howto/php/#provide-your-own-web-server-configuration-file). ### Configure the Web Directory Some frameworks put the `index.php` in a separate directory like `public` instead of the repository root. When the web server is HTTPD or NGINX, the web directory defaults to htdocs. In any case, you can override the web directory with a build env variable: ```bash --build-env=BP_PHP_WEB_DIR=public ``` ## Heroku Stack Web Server Configuration On the heroku stack, the web server is configured via a [`Procfile`](../configuration/procfile.md) in the root of your repository. See the [Heroku documentation](https://devcenter.heroku.com/articles/getting-started-with-php#define-a-procfile) for details. ## Symfony ### Paketo Stack For Symfony to build without failure, the auto-scripts currently [have to be disabled](https://github.com/paketo-buildpacks/php/issues/284): ```bash --build-env=BP_COMPOSER_INSTALL_OPTIONS=--no-scripts -o ``` --- ## Python The Deploio Python build environment uses the [Paketo Python Buildpack](https://paketo.io/docs/reference/python-reference/) on the paketo stack, and the [Heroku Python Cloud Native Buildpack](https://github.com/heroku/buildpacks-python/) on the heroku stack. See [Buildpack Stacks](../configuration/buildpack-stacks.md) for more information. ## Example App We have a basic Python Django app in our [examples repository](https://github.com/ninech/deploio-examples). You can deploy it with . The example application shows a random message on every page reload. The Django admin interface can be used to add messages. Just visit `https:///admin` to access it and use the credentials which you pass via the env variables below to login. Please also define the `SECRET_KEY` which is needed to secure signed data and should be kept secret. ```bash nctl create application django-example \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/python/django \ --env=DJANGO_SU_NAME=admin \ --env=DJANGO_SU_EMAIL=admin@example.com \ --env=DJANGO_SU_PASSWORD= \ --env=SECRET_KEY= ``` ```bash nctl create application django-example \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/python/django \ --env=DJANGO_SU_NAME=admin \ --env=DJANGO_SU_EMAIL=admin@example.com \ --env=DJANGO_SU_PASSWORD= \ --env=SECRET_KEY= ``` ## Paketo Stack Configuration There are just a few build environment variables supported by the Python buildpack. You can find them in the [Paketo documentation](https://paketo.io/docs/reference/python-reference/). ## Django Specifics The following configuration applies to both the paketo and heroku stacks. ### Procfile If you have a Django application, you will need to create a `Procfile` in the root of your app source code, which changes the default "web" entrypoint to a valid [wsgi project configuration file](https://docs.djangoproject.com/en/4.2/howto/deployment/wsgi/), which will be served by gunicorn. For example, the `Procfile` in our example app looks like: ```bash title="Procfile" web: gunicorn deploio.wsgi ``` ### Configuring ALLOWED_HOSTS The `ALLOWED_HOSTS` setting represents the permitted hostnames/domains which the Django site can serve. To allow the default Deploio URLs for your application, you can use the following entry in your `settings.py` file: ```python title="settings.py" ALLOWED_HOSTS = [".deploio.app"] ``` Please note that you will need to add all of your custom domain names to this list. So if you want your application to be served on `django-app.example.com` your `ALLOWED_HOSTS` should look like: ```python title="settings.py" ALLOWED_HOSTS = [ "deploio.app", "django-app.example.com", ] ``` ### Configuring the SECRET_KEY The `SECRET_KEY` parameter is used to secure signed data in Django. It should be kept secure and so not be stored alongside your application code. One way of specifying it is to load it from the environment. You can achieve this by using the following line in your `settings.py`: ```python title="settings.py" # The secret key can be passed via the env variable "SECRET_KEY" SECRET_KEY = os.environ.get('SECRET_KEY') if SECRET_KEY == None: raise ValueError("SECRET_KEY environment variable must be set") ``` You then need to specify the environment variable `SECRET_KEY` with as you can see in the [Example app section](#example-app). --- ## Ruby The Deploio Ruby build environment makes use of the [Ruby Heroku Cloud Native Buildpack](https://github.com/heroku/buildpacks-ruby/) on both the paketo and heroku [buildpack stacks](../configuration/buildpack-stacks.md), so behavior is identical regardless of stack. ## Example App We have a basic Rails app in our [examples repository](https://github.com/ninech/deploio-examples). You can deploy it with . This requires the `rails` command to be installed for the `SECRET_KEY_BASE`. If you don't have it, any long random string will do (127+ chars). ```bash nctl create application rails \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/ruby/rails-basic \ --env=SECRET_KEY_BASE=$(rails secret) ``` ```bash nctl create application rails \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/ruby/rails-basic \ --env=SECRET_KEY_BASE=$(rails secret) ``` ## Buildpack Behavior The full behavior of the buildpack is [documented here](https://github.com/heroku/buildpacks-ruby/blob/c8cdfd0be3a61f7b50d36cae12ec3d22f8068afc/docs/application_contract.md). ### Ruby Version Detection The buildpack will attempt to detect the desired ruby version from the `Gemfile.lock` in the app source. ### Node.js Runtime The Node.js runtime will only be installed if there is a package.json file present at the root of the repository. --- ## Static Sites If you have a site with purely static content, Deploio makes use of a combination of buildpacks to deploy a web server to serve your static files. :::note If you use the [heroku buildpack stack](../configuration/buildpack-stacks.md) with , you must explicitly set `--language=static` when creating or updating your application, as automatic static site detection is only available on the paketo stack. ::: Static sites are detected by looking for these files in your git repo: - `index.html` - `public/index.html` ## Example Apps We have two static sites in our [examples repository](https://github.com/ninech/deploio-examples). You can deploy them with : Plain `index.html`: ```bash nctl create application static-html \ --language=static \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/static/html ``` React app with `npm`: ```bash nctl create application static-react \ --language=static \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=heroku-stack/static/react ``` Plain `index.html`: ```bash nctl create application static-html \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/static/html ``` React app with `npm`: ```bash nctl create application static-react \ --buildpack-stack=paketo \ --git-url=https://github.com/ninech/deploio-examples \ --git-sub-path=paketo-stack/static/react ``` ## Paketo Stack Configuration To override the automatically detected paths mentioned above, you can specify the build env variable `BP_STATIC_WEBROOT=` to any directory within your Git repository. ## Heroku Stack Configuration When using the [heroku buildpack stack](../configuration/buildpack-stacks.md), the web server is configured via a `project.toml` file in the root of your repository instead of build environment variables. The default document root is the `public` directory. To change the document root or index file, add the following to your `project.toml`: ```toml title="project.toml" [com.heroku.static-web-server] root = "dist" index = "index.html" ``` The buildpack supports additional options such as custom response headers, error pages, redirects, and clean URLs — all configured through `project.toml`. See the [Heroku Static Web Server buildpack documentation](https://github.com/heroku/buildpacks-frontend-web/blob/main/buildpacks/static-web-server/README.md) for the full reference. ## NPM Frontend If you have Node modules that need to be installed during the build step, Deploio will detect this using the `package.json` file and run `npm` . In this case, the resulting files will end up in the directory `build` and it will serve the artifacts from there. --- ## Observing your App Whether your app is running as expected or not, there are various ways to find out what is wrong. ## Logs To view logs, you can either use Cockpit or . Various components of Deploio can emit logs: - **Build**: these are logs produced by the build process. Useful if there's an issue with building your app. - **App**: these logs contain what the app replicas are outputting to stdout/stderr during runtime. - **Worker Job**: these logs contain what the worker job is outputting to stdout/stderr during runtime. - **Deploy Job**: these logs contain what the deploy job is outputting to stdout/stderr during runtime. - **Scheduled Job**: these logs contain what the scheduled job is outputting to stdout/stderr during runtime. The logs are retained for 30 days. ### Examples Display all logs of an app. These examples all use an app named `go`: ```bash $ nctl logs app go 2024-10-22T16:07:55+02:00 {build="go-build-1", replica="go-build-1-build-pod"} Build successful 2024-10-22T16:07:59+02:00 {replica="go-59dcf7656f-9wr52"} 2024/10/22 14:07:59 starting HTTP server on :5678 ``` The labels in between the curly braces tells you which component each log line belongs to. The labels can be turned off with the option `--no-labels`. Display only the main app logs. ```bash nctl logs app go --type app ``` Display worker job logs of an app. ```bash nctl logs app go --type worker_job ``` Display deploy job logs of an app. ```bash nctl logs app go --type deploy_job ``` Display scheduled job logs of an app. ```bash nctl logs app go --type scheduled_job ``` Display all build logs of an app. ```bash nctl logs app go --type build ``` Alternatively, you can also view the logs of a specific build using the `logs build` command. ```bash nctl logs build go-build-1 ``` You can add the `-f` flag to keep following the logs as they are emitted. ```bash nctl logs app go -f ``` To get more lines, use the `-l/--lines` flag. ```bash nctl logs app go-example -l 1000 ``` To go further back in time, use the `-s/--since` flag. Durations can be specified using 1s (1 second), 1m (1 minute) and 1h (1 hour). ```bash nctl logs app go-example -s 48h ``` ## Stats To get an overview of the app replicas, their status and resource usage you can use the `-o stats` flag for the `nctl get app` command. ```bash $ nctl get app rails -o stats PROJECT NAME REPLICA STATUS CPU CPU% MEMORY MEMORY% RESTARTS LASTEXITCODE nine-staging rails rails-9f6bdb676-crlrf ready 1m 0.8% 121MiB 47.4% 0 nine-staging rails rails-9f6bdb676-2pfps ready 1m 0.8% 123MiB 48.4% 0 nine-staging rails rails-9f6bdb676-sd9mm ready 1m 0.8% 122MiB 48.0% 1 137 (Out of memory) ``` If you have worker jobs defined, they will also show up here. The help output of `nctl get app -h` describes all the columns: ```bash REPLICA: The name of the app replica. STATUS: Current status of the replica. CPU: Current CPU usage in millicores (1000m is a full CPU core). CPU%: Current CPU usage relative to the app size. This can be over 100% as Deploio allows bursting. MEMORY: Current Memory usage in MiB. MEMORY%: Current Memory relative to the app size. This can be over 100% as Deploio allows bursting. RESTARTS: The amount of times the replica has been restarted. LASTEXITCODE: The exit code the last time the replica restarted. This can give an indication on why the replica is restarting. ``` ## Metrics Some basic application metrics like CPU and RAM usage can be found directly in Cockpit. If you want more detailed metrics and want to visualize them with your own dashboards, install a [Grafana](../managed-kubernetes/nke/grafana) instance to access them. ### Dashboard We won't yet install a dashboard for you, but you can [download a pre-made dashboard](./observing-your-app-dashboard.json) and [import](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/import-dashboards/#import-a-dashboard) it into your instance. --- ## Troubleshooting and Advanced Topics ## How Deploio Works Understanding the backend process helps you troubleshoot deployment issues. During a successful app creation, Deploio performs these steps: - Clones the Git repository at the specified revision - Detects the app's language - Builds and packages the app into an OCI container with language-appropriate tools - Uploads the built app image to the Deploio image registry - Creates a release that executes the built image - Exposes the application on an HTTP endpoint The build and release steps are where most issues occur. If there's an error during cloning or building, shows the build logs to help with troubleshooting. If the build succeeds but a release doesn't become ready, check the app logs for an indication of the problem. If your app fails silently on start, the app log might be empty. ## Retry a Build Builds can fail for multiple reasons. Most of the time, a change to the application code or adding a missing environment/build variable fixes the issue. However, a temporary internal system issue might also cause the failure. In those cases, retry the build using : ```bash $ nctl update app go-example --retry-build ``` ## Retry a Release The retry release trigger lets you manually initiate a new release without code changes or rebuilding your application. This is especially useful when a release failed due to external factors, infrastructure issues, or temporary network failures. For example, if your release process failed due to a broken or incorrect database migration and you've resolved the issue at the database level, use the retry release feature to redeploy the existing build: ```bash $ nctl update app go-example --retry-release ``` ## Inspect and Run the Build Output Locally Since the build output is an OCI/Docker image, you can pull it to your local machine for inspection. This helps when the build succeeds but the release fails to start the app. You can only pull images for debugging; the registry doesn't accept pushes. ```bash $ nctl get build go-example-build-1 --pull-image Pulling image of build go-example-build-1 739cb80086f0: Download complete 1bfd697887fa: Download complete aea73d1e202c: Download complete e4fd482e7de7: Download complete ✓ Pulled image deploio.296dc8b.registry.nineapis.ch/nine/go-example 💾 ``` After downloading the image, you can interact with it using Docker or Podman (or anything that can interact with OCI images) to inspect and run the image locally. ```bash $ docker run deploio.296dc8b.registry.nineapis.ch/nine/go-example ``` ## Execute a Shell or Command in a Deploio Application :::note In some Deploio application images, especially those built using a Dockerfile from `distroless` or `scratch` base (for example, the [sample Dockerfile app](https://github.com/ninech/deploio-examples/tree/main/dockerfile)), the shell isn't included, preventing command execution. In such cases, use alternative debugging tools and techniques. ::: You can start a shell in your Deploio application after a successful release. Your application must run in a stable state for a shell to start. If your application constantly restarts because of an error, starting a shell isn't possible. Start a shell using . This starts a shell in the first available Deploio replica. ```bash $ nctl get app NAME HOSTS UNVERIFIED_HOSTS rails rails.7f0b629.deploio.app none $ nctl exec app rails cnb@rails-5fcc67cc-89fd2:/workspace$ ``` You can also start a single command by separating it with `--`. ```bash $ nctl exec app rails -- hostname rails-5fcc67cc-89fd2 ``` Every command you start counts towards your memory quota, which depends on your configured Deploio application size. If too much memory is used, Deploio terminates the entire application replica and you see an exit code of 137: ```bash nctl: error: command terminated with exit code 137 ``` Similarly, if your application exceeds its [local ephemeral storage limit](#local-ephemeral-storage-limitations), Deploio immediately terminates it. The session is forcibly cut off and the following error message appears: ```bash nctl: error: command terminated with exit code 137 ``` Any changes to local files made in a shell session are lost when the Deploio replica restarts. ## Let's Encrypt Certificates Deploio exposes applications via a randomly generated URL by default, letting you access your application during development without your own domain name. Deploio secures the default URL with a Let's Encrypt TLS certificate, generated automatically when you create an application. For Let's Encrypt certificates, Deploio uses the [HTTP-01 challenge type](https://letsencrypt.org/docs/challenge-types/#http-01-challenge). View the certificate status for the default URL using : ```bash nctl get app -o yaml kind: Application apiVersion: apps.nine.ch/v1alpha1 ... status: atProvider: ... defaultHostsCertificateStatus: Issued ``` When you add [custom host names](configuration/custom-hosts.md) to your Deploio application, Deploio issues a corresponding Let's Encrypt TLS certificate for all your custom host names. View the certificate status using : ```bash nctl get app -o yaml kind: Application apiVersion: apps.nine.ch/v1alpha1 ... status: atProvider: ... customHostsCertificateStatus: Pending ``` Because Deploio uses the Let's Encrypt HTTP-01 challenge type, the certificate is only issued once **all** of your custom hostnames point to the Deploio infrastructure. Deploio uses an optimized DNS resolving path to quickly react to DNS changes, but certificate issuance might still take a few minutes. This is especially important when migrating an application hosted elsewhere to Deploio. Since the transition can take several minutes, consider migrating during non-business hours. Let's Encrypt favors IPv6 DNS entries over IPv4. If you have DNS AAAA records for your custom hostnames, delete them when migrating to Deploio, as Deploio doesn't currently support IPv6. ## Local Ephemeral Storage Limitations Ephemeral storage is a temporary storage mechanism that your app's container uses while running. Deploio limits local ephemeral storage to 2 GiB per application. These limits prevent any single application from consuming excessive resources, ensuring fair distribution across the system. ### What Constitutes Ephemeral Storage? Containers use local ephemeral storage for scratch space, caching, and logs. Writing data through an [`exec` session](#execute-a-shell-or-command-in-a-deploio-application) into a running app's filesystem also counts toward total ephemeral storage usage. Exceeding the storage limit can cause the app to restart, resulting in the loss of all ephemeral storage data. ### Best Practices for Managing Ephemeral Storage To manage ephemeral storage effectively: - Minimize writes to the writable layer - Avoid excessive logging and unnecessary temporary file generation - Store critical data in persistent storage to prevent loss :::note The ephemeral storage limit does _not_ include the size of the Deploio application image. ::: ### What Happens When You Exceed Ephemeral Storage? If an application exceeds the ephemeral storage limit, Deploio terminates the container and returns exit code `137`. ### Impact on Exec Sessions If an [`exec` session](#execute-a-shell-or-command-in-a-deploio-application) is active inside a container that exceeds its ephemeral storage limit, Deploio immediately terminates the container. The session is forcibly cut off and the following error message appears: ```bash nctl: error: command terminated with exit code 137 ``` ## Writable Layer The writable paths and behavior in the app container depend on whether you built the application using [Buildpacks](https://buildpacks.io/) (default) or a Dockerfile. Different approaches result in different restrictions and permissions for writable directories. ### Applications Built with Buildpacks (Default) Buildpacks define a strict set of writable paths based on conventions designed to enhance security and reproducibility. In most Buildpack-built apps, writable paths are restricted to specific directories, commonly: ```bash /workspace /tmp ``` Buildpacks create immutable layers for application dependencies and base files. Modifications at runtime should only occur in ephemeral directories to maintain container integrity. The writable directories in a Buildpack-built container are bound by the app's [ephemeral storage limit](#local-ephemeral-storage-limitations), meaning excessive writing can exhaust available space. ### Applications Built with a Dockerfile When you build an application using a Dockerfile, writable paths and behavior depend on your choices and can vary widely based on application needs. Similar to Buildpacks, writable paths in a Dockerfile-built application are also limited by the app's ephemeral storage restrictions. ### Check Which User Runs the Container Before testing writability, determine which user runs the container by executing: ```bash id ``` Example output: ```bash uid=1000(cnb) gid=1000(cnb) groups=1000(cnb) ``` The user ID might differ between build time and runtime, depending on the Buildpack architecture. Some Buildpacks use different users during build and runtime, while others maintain the same user throughout. This can result in different runtime permissions. For more information, see [Cloud Native Buildpack policy](https://github.com/buildpacks/spec/blob/main/platform.md#build-image). Dockerfile-based builds give you full control over the user, and configurations can vary significantly. ### Determine Writable Directories in Your App First, [start a shell](#execute-a-shell-or-command-in-a-deploio-application) in your Deploio application. There is no direct command to list only writable directories, but exploration can help identify them. The following tips can help. To check for writable permissions: ```bash find / -writable -type d 2>/dev/null | sort ``` This command might not be available in all container images. If `find` is missing but `ls` is present, inspect directory permissions manually: ```bash ls -ld /path/to/directory ``` If you're unsure about writable directories, test them manually by attempting to create a file: ```bash touch /tmp/testfile && echo "/tmp is writable" || echo "/tmp is not writable" ``` If `touch` is unavailable, test writing to a file with `echo`: ```bash echo "test" > /tmp/testfile && echo "/tmp is Writable" || echo "/tmp is NOT Writable" rm -f /tmp/testfile ``` You can also try `mkdir`: ```bash mkdir /tmp/testdir && echo "/tmp is Writable" || echo "/tmp is NOT Writable" rmdir /tmp/testdir ``` --- ## Configure DNS Resolvers --- ## Edit your Hosts File to access your domain on another server/IP During the lifetime of a website, you will probably have the need to migrate the whole website to another host with another IP address. To test on your local desktop client if your website works before you change the productive DNS entries of your domain you can use the hosts file. ## How to change the local Hosts File In our example, we point the domain testdomain.ch with the subdomains (www, sub) to the new IP address 192.168.0.1. As soon as we open our local webbrowser, all requests regarding testdomain.ch and its subdomains will be sent to the new IP address 192.168.0.1. The usual DNS entries will be suppressed. :::warning Please do not forget to remove the entries after your testing! ::: ## Linux / macOS The file `/etc/hosts` contains all local host definitions. It can only be modified by the root user. Add the following entries: ```shell-session root@server:$ vi /etc/hosts 192.168.0.1 testdomain.ch www.testdomain.ch sub.testdomain.ch ``` ## Windows The file `c:\Windows\System32\Drivers\etc\hosts` contains all local host definitions. It can only be modified with administrator rights. Add the following entries using the notepad editor: ```shell-session 192.168.0.1 testdomain.ch www.testdomain.ch sub.testdomain.ch ``` --- ## How do I create a PTR record/reverse DNS? ## PTR record for a certain IP address You can define the PTR record (reverse DNS) by going to . 1. Log into 2. Go to the **Products** tab 3. Click on the **name of the product** or the **blue arrow** Now you can define the PTR record for the corresponding IP address. ## PTR record in one of your subnets Defining the PTR record for an IP within one of your IP ranges is also done via . 1. Log into 2. Go to the **Products** tab 3. Click on the **subnet name** or the **blue arrow** If you want to define the PTR record for a IPv4 address, click on the **edit icon** next to the address. Defining a PTR record for a IPv6 address is done by clicking on \*_adding an IP address_. --- ## How do I set up a domain? In the , you can enter domains on ns5.nine.ch and ns6.nine.ch name servers. Follow these steps to create a new domain: 1. Log into - Click the **DNS** tab - Click **Create domain** - Enter the domain name in the format `domain.tld`, e.g. `nine.ch`. - You don't have to specify a subdomain - You don't need to include `www` before the domain either - If you already know the server IP of your website, you can specify it in the field **Web IP** - The IP address of a server is found under the **Products** tab. Click on **Details and Statistics**. An example of a correctly entered domain looks like this: ![Uploaded Image](/img/b0f9332d9f16ac05ae1d.png) ![Uploaded Image](/img/b0f9332d9f16ac05ae1d.png) --- ## How do I use ns5/6.nine.ch as a second DNS? If you want to operate your own DNS server and configure the Nine DNS server to be a slave to prevent downtime, follow these steps: 1. Make sure your name server allows [AXFR zone transfer](http://en.wikipedia.org/wiki/DNS_zone_transfer) for the IPs `ns7.nine.ch` (`5.148.164.46`, IPv6: `2a02:418:400b:2::f`), `ns5.nine.ch` (`193.17.85.122`, IPv6 `2001:67c:2f98:5::5`), and `ns6.nine.ch` (`5.148.164.36`, IPv6: `2a02:418:400b:1::6`). - Login to with your customer login details and select the menu item **DNS** at the top - Click the **Add** button and enter a new secondary domain on `ns7.nine.ch` (ns7.nine.ch is the transfer and data master host of ns5/ns6.nine.ch) - Enter the domain and master server - Send a DNS notify to all of our name server For every change you make to your dns zones, you will have to send a DNS notify to `ns7.nine.ch`. --- ## SWITCH says that the domain is not functioning. What should I do? SWITCH will check whether your entered name servers are responsible for your domain. If this check fails, you will see the message "This domain name does not work from our end". Often this happens because SWITCH has an outdated result of a name server test. If the test result is outdated, perform the test again (contact SWITCH to get help on this topic). If you have created your domain using and have specified the name servers `ns5.nine.ch` and `ns6.nine.ch` for SWITCH, check in the DNS configuration that these entries are correct. If you encounter an error again, please contact our support at . --- ## Certifications ## Nine Internet Solutions AG Certification Scope Certificate **ISO 9001** Quality Management System [Download (PDF)](https://nine.ch/uploads/ISO-9001-Nine-Internet-Solutions-AG-2025.pdf) **ISO/IEC 27001** Information Security Management System (ISMS) [Download (PDF)](https://nine.ch/uploads/ISO-27001-Nine-Internet-Solutions-AG-2024.pdf) **ClimateCare** {/* prettier-ignore */} CO₂ compensation of our operations via certified myclimate projects (confirmation updated annually) [Confirmation (PDF)](https://nine.ch/uploads/myclimate_confirmation_2026.pdf) Customers can add the "ClimateCare" option to their product, to contribute to the costs of our annual CO₂ compensation payments to myclimate. They can then reference the [myclimate confirmation document](https://nine.ch/uploads/myclimate_confirmation_2026.pdf) in their own communications. This shows that we compensate our CO₂ emissions, and that ClimateCare customers contribute to that compensation. ## Our Data Centers The following certifications and standards are held by our data center operators. Each entry applies only to the specific data center listed in the table. A ☑️ means the data center itself does not hold this certification directly, but our customers still benefit from it through our own ISO certification and the use of our infrastructure. If you have any questions, we are happy to help. Certification Description {/* prettier-ignore */} NTT Zürich NTS / ColoZüri **Uptime Institute Tier IV** Highest availability standard: fully redundant systems, maintenance possible during live operation without any restrictions. ✅ ✅ **ISO 9001** Quality Management System ✅ ☑️ **ISO/IEC 27001** Information Security Management System (ISMS) ✅ ☑️ **ISO 22301** Business Continuity Management System ✅ ❌ **ISO 14001** Environmental Management System ✅ ❌ **ISO 45001** Occupational Health and Safety Management System ✅ ❌ **ISO 50001** Energy Management System ✅ ❌ **EN 50600** European standard for data center facilities and infrastructure ✅ ❌ **TIA-942:2010 – Annex G (Component A)** Telecommunications Infrastructure Standard for Data Centers ✅ ❌ **PCI DSS** Payment Card Industry Data Security Standard ✅ ❌ **ISAE 3402 Type II** Assurance report on controls at a service organization (equivalent to SOC 1). Available on request, subject to confidentiality requirements and potential costs. ✅ ❌ **ISAE 3000 Type 1 / FINMA** General assurance engagement; relevant for Swiss financial market outsourcing (FINMA Circular 2018/3). Available on request. ✅ ❌ **ISAE 3000 Type 2** Equivalent to SOC 2. Available on request, subject to confidentiality requirements and potential costs. ✅ ❌ --- ## Contact Nine You can reach our support team by email, phone, or through the Support Portal: ## Outside Office Hours :::warning If onsite service is required, for example to replace a hard drive, the drive to the data center is charged at flat rate. ::: In case of an emergency, you can call us outside of normal business hours. First, check our status page [status.nine.ch](https://status.nine.ch) for information about current incidents, planned maintenance, and third-party work that may affect the availability of our services. To contact our on-call staff, follow these steps: 1. Call our support number (). Outside office hours you will hear a recorded message. 2. Go to the Standby Service menu by pressing the 9 button (as suggested). 3. Confirm the terms of the service with button 1 (fixed rate of per call plus per hour, minimum 1 hour). 4. Our on-call staff will be notified of your call and will get back to you. --- ## Datacenter Locations These are the available datacenter locations and the products we offer in each of the locations: :::note Location IDs such as `nine-cz41`, `nine-cz42` and `nine-es34` are internal codenames, not [ISO 3166](https://en.wikipedia.org/wiki/ISO_3166) country codes. All locations listed below are physically located in Switzerland (Zürich / Rümlang). ::: ## Choosing a Location When choosing a location for services that communicate with each other, for example your Deploio application and a MySQL Business database, we recommend to to create these services in the same location to reduce package round trip times. Choosing different locations is advisable if your want to achieve a higher fault tolerance. --- ## Status Page You can access our status page at [status.nine.ch](https://status.nine.ch). There you will find information about current incidents, planned maintenance and third party work that may affect the availability of our services. As soon as a problem has been identified, a status message will appear on the relevant page and our technicians will be working on the problem. If you are experiencing any issues, please visit this page and let us know if the problem is not already mentioned on the page. --- ## Sustainable Electricity Our two datacenter locations use different electricity sources: - **NTS / Colozüri:** powered by 100 % renewable electricity. - **NTT / E-Shelter:** powered by CO₂-neutral electricity. ## Datacenters reports - NTS / Colozüri: [NTS offer page](https://nts.ch/en/Colocation/Offer) - NTT / E-Shelter: [NTT DATA EMEAL Sustainability Report, page 44](https://sustainabilityemealreport.com/wp-content/uploads/Sustainability_Report%20NTT_DATA_EMEAL_22_23.pdf) --- ## Weekly Maintenance Window A weekly maintenance window is defined for scheduled maintenance work. During this window, maintenance work such as the installation of security updates or the necessary restart of services or servers is carried out. Short service interruptions may therefore occur. We will announce more extensive maintenance work as early as possible and provide information on its progress on our Status Page. To ensure time-critical bug fixes and security updates, Nine reserves the right to update critical components outside the maintenance window as required. ## Managed Server & Services :::note On-Demand MySQL and PostgreSQL services are covered in this maintenance window. ::: There's one weekly maintenance window: - Tuesday, 00:00 to 02:00 (**Europe/Zurich Time**) ## Nine Kubernetes Engine (NKE) :::note On-Demand OpenSearch and Key-Value Store services are covered within these maintenance windows. ::: There are three weekly maintenance windows: - Tuesday, 00:00 to 04:00 (**UTC Time**) - Wednesday, 00:00 to 04:00 (**UTC Time**) - Thursday, 00:00 to 04:00 (**UTC Time**) Thanks to multiple maintenance windows, we're able to upgrade nodes in different clusters in a rolling fashion. ## Deploio There's one weekly maintenance window: - Tuesday, 00:00 to 04:00 (**UTC Time**) --- ## Why can't I access my server using SSH/FTP? If you have had multiple login attempts with invalid credentials, it is possible that your IP has been blocked for SSH, SFTP, FTPS and FTP login. We use this mechanism to protect your server against so called brute force attacks. You can check here whether your IP is blocked for this reason or not: If your IP address has been blocked, please let us know via our Support, so we can re-enable it. --- ## docs.nine.ch We have prepared a variety of information and support articles for our products for you. Should you still not be able to find the information you are looking for, we would be very pleased to receive your feedback. You can find more information about our products, system status and access to your support requests at the following links: [:globe_with_meridians: Nine Website](https://nine.ch/) [:white_check_mark: System Status](https://status.nine.ch/) [:email: Support Portal](https://portal.nine.ch/) > This app is proudly hosted on [Deploio](https://deplo.io/). ## Do you need further assistance? --- ## Acceptable Use Policy The provisions of the Acceptable Use Policy apply to all services offered by Nine Internet Solutions AG (hereinafter called "nine"). By using our products and services, you accept the following Terms and Conditions in full and without alteration. 1. While using Nine's services, the Customer may not store, publish or otherwise use any unlawful content. Unlawful content includes but is not limited to content that violates or prejudices the rights of nine or third parties, such as intellectual property rights in the broad sense (copyrights, patent rights, trademark rights, etc.), privacy rights (incl. data protection legislation), provisions of the Swiss Unfair Competition Act (UCA), or their business reputation. Unlawful content also includes all content or processes that constitute crimes (particularly in the areas of pornography, violent images, racism, trade secrets, defamation, money laundering, and fraud) or are otherwise illegal. The Customer is also prohibited from offering services that impede the prosecution of unlawful content (including open relays and VPN services with anonymization services). TOR exit nodes may only be used with the prior written consent of nine. 2. The Customer must investigate all reports concerning the misuse of his applications and software and must correct any misuse. 3. The Customer will refrain from taking any action that serves to circumvent user authentication (particularly access to the Customer Cockpit) or that may compromise the security of a server, a network, or the Customer's account (e.g., social engineering, password cracking, and scanning security gaps). Furthermore, the Customer will refrain from making any attempt to disturb the operation of servers or networks (e.g., denial of service attacks, flooding of networks, and intentionally attempting to overload services). 4. The Customer must keep the applications and software used by him at a level consistent with the state of the art, maintain them regularly, and update them regularly. The Customer must also adhere to the GTC, the Acceptable Use Policy, and any instructions issued by nine, particularly as regards maintaining, updating, or deleting software. 5. The Customer must immediately report to nine all faults and interruptions in Nine's services used by the Customer (including all cases of unlawful or non-contractual use of the service by third parties, e.g., hackers) and must assist nine, where possible, in remedying the fault. 6. The Customer is prohibited from sending emails against the stated or presumed wishes of the recipients (e.g. junk mail, spam, and mail bombing) or otherwise harassing, bothering, offending or disturbing them by sending them e-mails. The foregoing violations also include but are not limited to sending commercial advertisements, informational announcements, political writings, etc. The Customer may only send such material to recipients who have explicitly requested it. The Customer will also refrain from making any attempt to use Nine's accounts or services to collect replies to messages sent from another Internet Service Provider if the messages in question violate the present Acceptable Use Policy or that of the other provider. 7. Any disputes between joint holders of a customer account or between the Customer and third parties concerning the use of the customer account or the content distributed via the Customer Server are exclusively the responsibility of either the joint holders of the customer account or the Customer, as applicable. If nine receives inquiries/complaints from individual joint holders of a customer account or from third parties regarding a customer account or content provided via a customer account or the Customer Server, nine will forward the inquiry/complaint to the other joint holder(s) or the Customer for purposes of handling the issue. nine reserves the right to disclose the Customer's identity to third parties at the request of a court or government agency. 8. In case of violations of this Acceptable Use Policy, nine may immediately take any action in law or in fact against the perpetrator. This includes but is not limited to seeking injunctive relief, filing claims for damages, and reporting the violations to law enforcement agencies. 9. Upon a request from the responsible authorities for Nine's cooperation, nine will fully support any investigation of the types of behavior prohibited in this Acceptable Use Policy, even if nine is not directly affected by the same. This Acceptable Use Policy takes effect as of 10 August 2023 and supersedes all prior versions. Zurich, 10 August 2023 --- ## Commitment to Data Protection We understand that data protection is of the utmost importance. As your conscientious service provider, we do everything in our might to keep your data safe. Nine is dedicated and committed to ensuring data protection, as well as to continually assessing any measures taken. ## What data protection measures has nine taken? Here is a summary of our data protection roadmap and the steps we have taken on our journey: - Thorough examination of the areas of our products influenced by GDPR, customer relations and business partners - Assessment of the effects of the revised Federal Act on Data Protection (effective 1 September 2023, nFADP) on our services - Appointment of a data protection officer - Revision of the [Nine GTCs](./general-terms-and-conditions) - Development of a strategy to meet the requirements of the areas of our products affected by data protection - Implementing the necessary changes to our internal processes and procedures to achieve and maintain GDPR compliance Nine has also worked with external lawyers to understand the new legislation and counter its effects. We will continue this collaboration to be able to respond to any new developments that may arise in the future. ## What do nine customers have to consider? There are two things you have to do depending on your situation and jurisdiction. Below you will find the only changes we can foresee that could affect you through the use of Nine's infrastructure services: ### 1. Make sure your policies are up to date and understandable Ensure that your terms of use or privacy policy correctly communicate to your users how you use the services provided by nine (and other similar services) on your website or application. This requirement has always been part of Nine's Terms of Use, but data protection legislation (including the nFADP) can severely punish you if you have not clearly done so. We recommend that you make sure your policies are up to date and understandable to your readers. ### 2. Sign a DPA If you are in the European Union or process or manage data from customers in the EU, you will probably want to sign a data processing agreement with your customers. To also enable our Swiss clients to comply with data protection requirements, a corresponding agreement (DPA) forms an integral part of our General Terms and Conditions. ### 3. Terms and conditions You can see a copy of our terms and conditions here: https://docs.nine.ch/docs/legal-documents/general-terms-and-conditions. If you have any questions about the content, simply send an e-mail to info@nine.ch. ## I am new to GDPR and would like to know more details about what it is The EU's General Data Protection Act (GDPR) is considered the most important European data protection law introduced in the European Union (EU) in the last 20 years and will replace the 1995 Data Protection Directive. GDPR regulates the processing of personal data about persons in the European Union including their collection, storage, transmission or use. It is important that the term "personal data" is very broadly defined in the GDPR and includes all information relating to an identified or identifiable person (also called "data subject"). It gives data subjects more rights and control over their data by regulating how companies should handle and store the personal data they collect. GDPR also increases the commitment to compliance by increasing enforcement and imposing higher fines if the provisions of GDPR are violated. The DSGVO strengthens the privacy of EU citizens and obliges organisations to handle data. If you are a company outside the EU, you should be aware of this. The provisions of the GDPR apply to any organisation that processes personal data of individuals in the European Union, including the tracking of their online activities, whether or not the organisation has a physical presence in the EU. In summary, here are some of the most important changes that will come into force with GDPR: - Extended rights for individuals: GDPR provides for extended rights for individuals in the European Union, including the right to be forgotten and the right to request a copy of personal data stored in their context. - Compliance obligations: The GDPR requires companies to implement appropriate policies and security protocols that assess privacy impacts, keep detailed records of data activity and make written agreements with vendors. - Notification and security of data breaches: The GDPR stipulates that companies must report certain data protection violations to the data protection authorities and under certain circumstances to the persons concerned. The GDPR also places additional security requirements on organisations. - New requirements for profiling and monitoring: The GDPR provides for additional obligations for organisations involved in profiling or monitoring user behaviour of EU citizens. - Greater enforcement: According to the GDPR, the authorities can impose fines of up to 20 million euros or 4% of a company's worldwide annual turnover, depending on the severity of the violation and the damage caused. In addition, the GDPR provides a central enforcement body for organizations operating in several EU Member States by requiring companies to cooperate with a leading supervisory authority for cross-border data protection issues. --- ## Data Processing Agreement (DPA) between ``` Customer 'Data Controller' ``` and ``` Nine Internet Solutions AG Badenerstrasse 47 8004 Zürich 'Data Processor' ``` ## 1 Aim and Scope of Application (a) The Aim of this agreement on data processing ('**DPA**') is to ensure the compliance with the Federal Act on Data Protection ('**FADP**') and with  further data protection legislation (such as the General Data Protection Regulation of the European Union), insofar as it is applicable ('**Applicable Data Protection Regulations**'). Individual pieces of legislation shall only be applied within the scope in which they are applicable to the respective processing task. (b) This DPA pertains to the processing of personal data as described in appendix 1, and any terms defined in appendix 1 are deemed by this DPA as clearly defined terms. ## 2 Interpretation (a) Where this agreement uses terms which are defined in Applicable Data Protection Regulations, they carry the same meaning as in these regulations. (b) This DPA is to be read and interpreted in compliance with the requirements of Applicable Data Protection Regulations, in particular the FADP, within the scope in which these are pertinent. (c) These terms are not to be interpreted in a manner which is contrary to the rights and requirements stipulated by Applicable Data Protection Regulations, or which curbs the fundamental rights and liberties of affected persons. ## 3 Hierarchy (a) In such a case that there is a contradiction between this DPA and the stipulations made in another agreement between both parties, which is either in place when this DPA is signed or is arranged afterwards, this DPA takes precedence over such an agreement, unless an exception is explicitly stated in written form. ## 4 Description of Processing (a) The processing details, particularly the categories of personal data and the purposes for which this personal data is processed on behalf of the Data Controller, are listed in Appendix 1. ## 5 Duties of the Agreeing Parties ### 5.1 General (a) The Data Processor processes personal data only by written order given by the Data Controller, unless the Data Processor is subject to legislation which legally requires the processing of personal data. For the entire duration of personal data processing activities, the Data Controller has the right to give additional directives. Such directives must be documented at all times. The Data Controller agrees that the underlying agreement, this DPA, their updates, as well as, wherever possible, the Data Controller's technical configurations form the entire body of directives given by the Data Controller. (b) The Data Processor informs the Data Controller at once wherever the Data Processor deems the Data Controller's directives to be in violation of the FADP or further applicable legislation. ### 5.2 Purpose limitation (a) The Data Processor may process personal data solely for the processing purpose(s) listed in Appendix 1. ### 5.3 Erasing or Returning Data (a) Any data processing activities conducted by the Data Processor must not exceed the duration defined in Appendix 1. (b) Upon the cessation of services pertaining to the processing of personal data, or upon termination in accordance with Article 8, the Data Processor returns all personal data to the Controller and deletes any existing copies, unless Applicable Data Protection Regulations or other legislation stipulate the retention of this personal data. ### 5.4 Processing Security (a) The Data Processor takes the technical and organisational measures (TOM) outlined in Appendix 2 to ensure the security of personal data, including the safeguarding against accidental or illegal destruction, loss, alteration, unauthorised transfer of or access to this data (personal data breach). The Data Controller is responsible for the assessment of what constitutes an appropriate level of security, in particular with regard to the risks associated with data processing, the type of personal data as well as the type, scope, circumstances and purposes of processing this data. (b) In the case of a personal data breach pertaining to data which is processed by the Data Processor, the Data Processor notifies the Data Controller without delay, and no later than 48 hours after becoming aware of the breach. This notification must include information regarding a point of contact for further inquiries into the personal data breach, a description of the type of breach (including, wherever possible, the categories and approximate number of affected persons and data sets), its likely consequences and the measures taken or planned to litigate its possible negative repercussions. Should it not be possible to provide all information at once, the first notification includes any hitherto available information, while any further information is provided without unreasonable delay as soon as it is available. (c) The Data Processor will act in good faith and trust to work with and support the Data Controller in any way necessary to enable the Data Controller to notify, where appropriate, the relevant data protection authorities and affected persons, while taking into account the processing type and the information available to the Data Processor. (d) The Data Processor grants employees access to the data only insofar as it is strictly necessary for fulfilling, managing and monitoring the agreement between Data Processor and Data Controller. The Data Processor ensures that persons who are authorised to process the personal data obtained are obliged to confidentiality or are subject to an equivalent legal confidentiality obligation. ### 5.5 Documentation and Compliance (a) The parties must be able to prove their compliance with this DPA. (b) The Data Processor is obliged to immediately and duly answer any reasonable queries made by the Data Controller regarding processing within the scope of Applicable Data Protection Regulations. (c) The Data Processor provides the Data Controller with any information necessary to prove compliance with requirements defined by and directly resulting from Applicable Data Protection Regulations, enables the Data Controller upon request to assess data and records, or to conduct audits of the processing tasks which fall under the stipulations made herein and contributes to such audits, particularly when there are signs of non-compliance. (d) The audit can be conducted by the Data Controller, by an independent auditor on commission of and at the cost of the Data Controller, or the Data Controller can elect to accept an independent audit commissioned by the Data Processor. Should the Data Processor commission the audit, the cost of the independent auditor is carried by the Data Processor. The right of the Data Controller to auditing, access and inspection pertain solely to the records of the Data Processor (including, among others, records of data processing activities) and are not applicable to the physical premises of the Data Processor. Any assessment and information request must be limited to information necessary for the purposes of this DPA and must duly take into account the confidentiality obligations the Data Processor is subject to, as well the Data Processor's valid interest in the protection of business secrets. (e) The Data Processor and the Data Controller will supply the information outlined in this article, including the results of any audits, to the relevant regulatory body upon request, if and insofar as this is necessary under Applicable Data Protection Regulations. ### 5.6 Employment of Third-party Data Processors (a) The Data Processor has the agreement of the Data Controller to commission third-party data processors (subcontractors). A list of third-party data processors employed by the Data Processor can be found in Appendix 3. The Data Processor informs the Data Controller in writing (notification via email or another form of electronic communication is sufficient) at least 30 days before any planned changes to this list are made by adding or replacing third-party data processors, thereby giving the Data Controller the option to veto these changes before the employment of the relevant third-party data processor(s) begins. Such a veto must not be unwarranted. The parties keep the list up to date. (b) Where the Data Processor commissions a third-party data processor to undertake certain processing activities (on behalf of the Data Controller), this is carried out in accordance with an agreement which makes the third-party data processor subject to the same duties as the Data Processor under Applicable Data Protection Legislation. The Data Processor ensures that any third-party data processor complies with the same obligations to which the Data Processor is bound by this DPA and Applicable Data Protection Legislation. (c) The Data Processor remains accountable to the Data Controller for the fulfilment of obligations on the side of the third party data processor arising from the agreement between the Data Processor and the third party. The Data Processor informs the Data Controller, should the third-party data processor not fulfil the obligations set out in this agreement. ### 5.7 International Transfers (a) Any transfer of data to a 'Third Country' (any country outside Switzerland) or to an international organisation undertaken by the Data Processor is only authorised if it complies with Applicable Data Protection Regulations. Standard contractual clauses might need to be added and a data protection impact assessment might need to be conducted, which would create the necessity to add further requirements. (b) In cases where the Data Processor has commissioned a third party to take on certain processing activities (on behalf of the Data Controller) as stipulated in clause 5.6 in a Third Country and these processing activities include the transfer of personal data, the Data Controller agrees that the Data Processor and the third-party data processor use standard contractual clauses regarding data protection to fulfil the requirements set out by Applicable Data Protection Regulations, provided that the conditions for the use of such clauses are met. ## 6 Rights of Affected Persons (a) The Data Processor informs the Data Controller immediately about any requests made directly by an affected person. The Data Processor does not respond to this request unless authorised by the Data Controller to do so. (b) The Data Processor supports the Data Controller in fulfilling the Data Controller's data protection duties under Applicable Data Protection Regulations to respond to the requests made by the affected person in accordance with the person's rights. This includes in particular support in information, correction and data transfer requests. (c) In addition to the Data Processor's obligation to support the Data Controller as stipulated in article 6 (b), the Data Processor supports the Data Controller in fulfilling the following requirements, while taking into account the processing type and the information available to the Data Processor: ``` (1) The obligation to immediately notify any affected persons of a personal data breach, where this notification is necessary according to Applicable Data Protection Regulations; (2) The obligation to assess the impact of planned data processing activities on the protection of personal data ('Data Protection Impact Assessment'), where the data processing type likely carries a high risk for the rights and liberties of natural persons; (3) The obligation to consult the relevant regulatory body before data processing in case a Data Protection Impact Assessment shows that the processing activity would carry a high risk if the Data Controller did not take measures to litigate this risk. ``` (d) In Appendix 2, the contractual parties define the appropriate Technical and Organisational Measures (TOM) with which the Data Processor is obliged to support the Data Controller in fulfilling this clause, as well as the scope and extent of the necessary support. ## 7 Notification of a Personal Data Breach (a) In the case of a personal data breach, the Data Processor will act in good faith and trust to work with and support the Data Controller in any suitable way in fulfilling the Data Controller's obligation to undertake a Data Protection Impact Assessment, while taking into account the type of processing and the information available to the Data Processor. (b) The Data Processor supports the Data Controller in notifying the relevant regulatory body of the personal data breach. The Data Processor is obliged to help procure particularly the following information, which is to be included in the Data Controller's notification according to Applicable Data Protection Regulations. This information principally constitutes: ``` (1) The nature of the personal data including, wherever possible, the categories and approximate number of affected persons as well as the categories and approximate number of personal data sets; (2) The likely consequences of the personal data breach; (3) The measures taken or planned by the Data Controller to remedy the personal data breach, including, where applicable, measures to litigate possible negative consequences. ``` ## 8 Termination (a) The Data Controller has the right to terminate this DPA as well as the underlying agreement if: ``` (1) The Data Processor violates to a significant degree or permanently the applicable data protection laws (in particular the FADP) or the Data Processor's obligations according to the Applicable Data Protection Regulations, and the violation is not expected to be remedied;  (2) The Data Processor does not comply with a legally binding ruling made by a relevant court or a relevant regulatory body regarding the Data Processor's obligations in accordance with Applicable Data Protection Regulations. ``` (b) This DPA remains comprehensively in place as long as the contractual relationship entered by both parties is in place. ## 9 Liability and Indemnity (a) The liability stipulations of the underlying agreement are applicable. (b) The Data Controller must reimburse the Data Processor in full for expenses arising from any services within the scope of the aforementioned support obligations (Art. 5.5, 6 et seqq.). These services are payable at the Data Processor's standard hourly rates. ## 10 Jurisdiction and Applicable Law The place of jurisdiction is the place of business of the Data Processor. Swiss substantive law is applicable. ## 11 Other Matters Where no stipulation is made herein, the contractual clauses of the underlying agreement are applicable. ## Appendix 1 | | | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Processing Purpose** | Processing of personal data on behalf of the Data Controller based on a variety of service agreements (the 'underlying agreement') | | **Processing Duration** | As long as underlying agreements with the customer are in place | | **Categories of Affected Persons\*** | Customers, employees, suppliers | | **Categories of Personal Data\*** | Date of birth/age, contact information (email, phone no.), home address, IP address, name, nationality and passport/ID. Personal data which requires added protection (Special Category Data) may also be processed. | | **Storage and Processing Location** | The Data Processor's business address and the business address of agreed third-party data processors, as well as respective data centres, as outlined in this DPA | | **On-site Audits** | No | > `* Categories of Personal Data as well as of Affected Persons are defined by the Data Controller without the assistance of the Data Processor. The list is purely exemplary.` ## Appendix 2 A description of the technical and organisational security measures implemented by the Data Processor(s) can be found here: https://docs.nine.ch/docs/legal-documents/technical-and-organizational-measures-toms ## Appendix 3 ### Section A: Nine Group (All Products) No persistent storage of customer data takes place in Canada or outside of Nine's premises in Zurich, Switzerland. No data is intentionally processed or stored there outside of strictly limited support activities. Remote access from Canada is considered a cross-border data transfer. Appropriate safeguards are implemented to ensure an adequate level of data protection. | **Service Provider** | **Processing Location** | **Service Rendered** | **Purpose of Processing** | | -------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------- | | Nine Internet Solutions Ltd, Canada (100% owned by Nine) | Canada (remote access only, on Nine infrastructure) | Emergency support and maintenance activities outside business hours | Incident response and operational support | | Wittwer IT Services, Switzerland | Switzerland (on Nine infrastructure) | Email communication services and support requests via Jira | Email delivery, related services and support requests | #### Optional: Excluding Access for Nine Internet Solutions Ltd. Canada Customers may exclude Nine Internet Solutions Ltd employees from accessing their systems via SSH key management. Enabling this option results in service limitations that are defined in the corresponding service agreement or SLA. ### Section B: Third-Party Processors (Product-Specific) The following providers only apply if the respective products or services are used and are contracted as part of the customer's service agreement with Nine. | **Only Applies to Product** | **Service Provider** | **Processing Location** | **Service Rendered** | **Purpose of Processing** | **Transfer Safeguards** | | --------------------------- | ------------------------------- | ---------------------------------------- | --------------------------------------- | ------------------------------------------------------------------- | --------------------------------------------------- | | Nine Managed GKE | Google Cloud EMEA Ltd, Ireland | Switzerland (region-specific deployment) | Cloud infrastructure (GCP) | Compute, storage and networking resources for managed GKE workloads | Standard Contractual Clauses (SCC) where applicable | | Cloudflare | Cloudflare, Inc., United States | Worldwide (global network) | CDN, DDoS protection, security services | Traffic delivery, caching and application-layer security | Standard Contractual Clauses (SCC) where applicable | ### Notes on Processing Locations - Primary data processing for Nine services takes place in Switzerland. - Third-party providers (Section B) may process data in additional locations depending on the service configuration. - Where applicable, region-specific configurations can be used to restrict data processing to defined geographic regions. ### Sub-Processor Management According to [4.4 Order Control (Outsourcing to Third Parties)](/docs/legal-documents/technical-and-organizational-measures-toms/#44-order-control-outsourcing-to-third-parties), Nine ensures that all sub-processors are contractually bound to: - process data only on documented instructions - implement appropriate technical and organisational measures - provide adequate safeguards for international data transfers where required Customers will be informed about material changes to sub-processors in accordance with the applicable contractual terms. --- ## General Terms and Conditions (GTC) These General Terms and Conditions (GTC) apply to all services offered by Nine Internet Solutions AG (hereinafter called "nine"). By using our services, you accept the following Terms and Conditions in full and without alteration. The GTC form an integral component of the agreements made between nine and the Customer. ## 1. Scope and conclusion of agreement 1.1 Subject of these GTC is the use of services that nine provides or offers to its customers (hereinafter called the "Customer"). 1.2 The Customer consents to these GTC by entering an individual service agreement or by activating a customer account. From this point onwards, the consent applies to using our services. Using a service includes, in particular, activating a product made available to the Customer in nine's customer portal (hereinafter called the "Customer Cockpit"). 1.3 For purposes of using individual services, the Customer may be requested to consent to the GTC again by activating the appropriate checkbox. When delivering a customer-specific offer, nine makes these GTC available to the Customer along with the relevant documentation by mail or electronically (via email or link). In this case, the Customer gives his or her consent to the GTC by confirming the offer, using the service or paying the invoice; the agreement is formed through whichever of these actions occurs first in time. ## 2. Nine's services and rights 2.1 nine performs services related to the configuration or operation of servers. Unless otherwise specified in the agreement, nine makes storage space available to the Customer on an infrastructure (the "Customer Server") connected to the Internet. The server infrastructure may be operated by nine, as well as by a third party. The actual scope of services is indicated in the agreement between the Customer and nine. 2.2 Unless otherwise agreed, nine may perform the contractually agreed services directly or in cooperation with third parties or may have them performed entirely by third parties. nine remains responsible for performing the agreement in any case. 2.3 nine strives to offer its services around the clock without failures or interruptions as operational resources allow. However, maintenance work, troubleshooting, the expansion of services, steps taken to safeguard nine's infrastructure, etc. may make temporary operational interruptions necessary. The Customer will be notified of such operational changes in good time as circumstances allow. Where possible, scheduled interruptions will occur outside of office hours (unless otherwise specified in the agreement: Monday to Friday, 9:00 a.m. – 6:00 p.m., excluding national holidays in Switzerland and cantonal holidays in the Canton of Zurich). 2.4 nine may block access to the Customer Server in whole or in part and/or discontinue the services if (i) the requirements of the notice and take down process pursuant to Code of Conduct – Hosting ([www.swico.ch](https://www.swico.ch/de/verband/oeffentlichkeitsarbeit/eigenverantwortung/code-conducts-hosting/)) are met, (ii) nine is requested to do this by a court or government agency, (iii) nine itself might otherwise become liable under civil or criminal law, or (iv) if a random sample gives rise to concrete evidence or the suspicion of a breach of these GTC, the publication of illegal content or any other unlawful or non-contractual use of nine's products and services. nine also reserves the right to reject virus-infected emails. The Customer acknowledges that even desired messages may be filtered and could be lost. 2.5 nine may suspend its service or block access to the Customer Server if the user behavior of the Customer or third parties (e.g., a high number of simultaneous requests on the Customer Server involving DDoS attacks) in any way impairs the operation of the Customer Server or other servers/services of nine. nine will inform the Customer (as operational resources allow and where possible based on the actual circumstances) in advance or immediately after the fact regarding the block that is to be or has been implemented. 2.6 nine has the right to bill the Customer for expenses incurred by nine in relation to measures taken pursuant to sections 2.4 and 2.5. nine reserves the right to claim additional losses. nine may request the Customer to provide security as a precaution to cover the aforementioned expenses and any additional losses. If this security is not paid or if the Customer fails to comply with the requests made in relation to the measures taken, nine may suspend the services or terminate the agreement with the Customer without notice. ## 3. The Customer's rights and duties 3.1 The Customer has the right to make fair use/acceptable use of the services. 3.2 The calculation of the use of nine's services is based on the average use of nine's resources (fair use). The resources made available (in particular, storage space, traffic, CPU/RAM use and support) may only be utilized for the proper operation of the Customer Server. If the Customer uses nine's resources beyond the acceptable level, nine will offer an upgrade to a more powerful class of service. If the Customer does not agree to the offer, nine reserves the right to terminate the agreement (if necessary, without notice). 3.3 The Customer is responsible for content (language, images, sounds, computer programs, databases, audio/video files, etc.) that the Customer (and third parties who communicate with him) causes nine to transmit or that the Customer himself process, distributes, stores or makes available for retrieval. The Customer is also responsible for references (in particular, links) to content. nine has no duty to monitor any content published by the Customer. 3.4 The Customer must keep up to date the components and applications operated by him or her (by making regular updates, etc.) and is responsible for ensuring the system and network security of the same. If the Customer fails to fulfil his or her obligations, he or she is liable as specified in section 8. The Customer also acknowledges that, from time to time, nine performs software updates and upgrades and replaces the server hardware, particularly to ensure the security of the operating systems. Because this can result in an incompatibility with customer applications in use, these applications may need to be adapted under certain circumstances. nine may make a test system available to the Customer at no charge for up to two weeks prior to every upgrade or change of hardware so that the Customer can test the compatibility of his or her applications. 3.5 Any fault or interruption caused by the Customer or by the users attributable to him or her will be eliminated by nine at the Customer's expense. The work performed shall be charged at nine's prevailing rates at the time. 3.6 When placing an order, registering (particularly to set up a customer account for access to the Customer Cockpit), and using the services, the Customer is required to make truthful and transparent statements. In particular, the Customer is responsible for ensuring that the customer data stored in the Customer Cockpit (billing and administration contact, as well as technical contact) are current, complete, and accurate for the entire term of the agreement. nine has no obligation to consider customer data other than those stored in the Customer Cockpit nor to conduct its own inquiries with a view to correcting the same. However, nine has the right to correct or delete entries in the Customer Cockpit that are obviously incorrect or violate the rights of third parties. In the event of ambiguities as to the correctness of the Customer's data, nine may at any time suspend the services and bill the Customer for any resulting costs by analogy to section 2.6. If nine determines that the Customer has not made truthful or transparent statements about his or her identity (including addresses and contact information), nine may discontinue the service immediately and terminate the agreement without notice. 3.7 The Customer agrees to select passwords appropriately, store them carefully, and protect them from access by third parties. If the Customer determines that his or her account has been misused, he or she must inform nine in writing immediately (via email, followed by acknowledgement of receipt by nine). The passwords or other identification parameters shared with the Customer are intended for the personal use of the recipient and must be treated confidentially. nine may assume that the person using an identification parameter is entitled to take the action authorized thereby, particularly to enter into or terminate agreements by making the corresponding change in the Customer Cockpit. 3.8 For security reasons, the Customer has no access to the server rooms operated by nine or third parties engaged by nine. The foregoing will apply unless nine and the Customer have expressly agreed otherwise. 3.9 Furthermore, the Customer must comply with the terms and conditions of nine's Acceptable Use Policy. The current version of this policy may be viewed on the website www.nine.ch and via the Customer Cockpit. The policy is a component of these GTC. ## 4. Data backup 4.1 nine offers a variety of services aimed at safeguarding the Customer's databases, files, and emails. The frequency of backups and the duration of the availability of the backup copies created by nine vary depending on the option specified in the agreement. The Customer is responsible for selecting the data backup options corresponding to the required level of protection, the likelihood of occurrence, and the severity of the risks. 4.2 In his or her area of responsibility, the Customer must take the security measures necessary and appropriate to restore his or her information and data in the event of loss or unauthorized or unintentional alteration. This includes but is not limited to regularly verifying the readability of the backup copies created by nine, as well as other services of nine as specified in 4.1. nine advises its customers to back up their data regularly. 4.3 nine advises its customers that data are backed up at different times and at different time intervals depending on the type of data or on the service package selected by the Customer. Furthermore, in exceptional cases, for technical reasons, e.g., because of maintenance work, faults in the system, or if it has become necessary to replace parts of the server infrastructure, nine may not be able to perform any data backup or data recovery for specific hours. In any case, data recovery does not include volatile data, such as temporary files and emails redirected to a separate spam folder by a spam filter. The spam folder is not being saved but regularly deleted. ## 5. Invoicing and payment terms 5.1 The duty to pay for services begins upon entry into the agreement (cf. section 1). nine will normally bill the Customer in advance for the selected contractual term in each case. Unless the invoice form specifies otherwise, the invoice must be paid within 20 days, and the stated prices are net prices (excluding VAT). 5.2 If the Customer breaches the payment terms indicated above or stated on the invoice form, he or she is in default as of the due date of the receivable. If the Customer defaults in making a payment, nine has the right to charge 5% default interest and may also charge collection notice fees of (per notice letter) as of the second notice. Furthermore, nine may terminate the service as specified in section 11.2. Moreover, nine has the right to suspend the service if the second notice to the Customer proves unsuccessful. For purposes of restoring service, the Customer will be charged a fee of , and nine may require the Customer to pay for the regular billing period in advance. 5.3 Neither Party may offset its receivables against those of the other Party. ## 6. Warranties 6.1 nine is liable to the Customer without limitation for direct and proven losses resulting from wrongful intent or gross negligence on the part of nine. 6.2 In order for any warranties to take effect, nine must first receive a fault report from the Customer (email followed by confirmation of receipt by nine), including a clear description of the alleged defects. The Customer must grant nine a reasonable grace period of at least 30 days to correct the defects specified in the notice of defects. If the defects have not been corrected by the time the grace period has expired, the Customer may terminate the agreement immediately. Any fee paid previously will be reimbursed by nine to the Customer pro rata for the time period in which the Customer no longer uses the service because of the termination. Any other compensation is excluded, subject to section 7 of these GTC. ## 7. nine's liability 7.1 nine is unlimitedly liable to the customer for direct and proven damages caused by unlawful intent or gross negligence on the part of nine. 7.2 Liability for ordinary negligence on the part of nine and the third parties engaged by it is limited to the amount of ,000.00 per calendar year and to direct losses. nine explicitly excludes any liability for indirect or consequential losses. Consequential losses include but are not limited to lost profits, production losses, damage to reputation, damages resulting from data loss, and third-party claims. 7.3 nine is not liable for losses arising from any unlawful or non-contractual use of its services by the Customer or any third party. In particular, any liability for losses arising because third parties use nine's infrastructure or customer applications improperly or interfere with them without authorization is excluded. This includes but is not limited to interference by means of computer viruses or DDoS attacks, as well as changes made by hackers and sending e-mails without authorization. This disclaimer also covers losses that the Customer incurs because of measures that nine must take to defend against such interference by third parties (e.g., blocking access to the customer website to protect nine's infrastructure and other customers from DDoS attacks, and all other measures specified in sections 2.3 - 2.6). 7.4 The foregoing exclusions and restrictions of nine's liability do not apply in case of injury to life, physical or health, nor in the event of mandatory legal provisions, including the provisions of the Swiss Product Liability Act. ## 8. The Customer's liability 8.1 The Customer is liable for losses incurred by nine or any third party because of the fault of the Customer or his or her users. ## 9. Confidentiality and data protection 9.1 nine and the Customer mutually agree to preserve the confidentiality of all non-public information and data that become available to them in the course of preparing and implementing the agreement. This duty will remain in effect after the termination of the agreement, as long as a legitimate interest in this exists. 9.2 nine and the Customer will ensure data protection and security compliance within their respective areas of influence. nine takes reasonable organizational and technical measures to protect personal data against unauthorized processing. nine only uses personal data for the purpose of performing its own services. With certain services, it may be necessary to forward personal data to third parties in Switzerland or abroad (for instance, for purposes of accessing applications operated by third-party providers). nine also reserves the right to make personal data available to government agencies or third parties insofar as nine is required by law to do this. nine will preserve personal data only insofar and for as long as this is necessary for purpose of performing the services or nine is required by law to do this. 9.3 Where nine processes personal data as part of the work commissioned by the client (commissioned data processing), The Data Processing Agreement (DPA) forms an integral part of the agreement between the customer and nine. The DPA is found here: https://docs.nine.ch/docs/legal-documents/data-processing-agreement. Nine can adjust the DPA at any time. The customer will be informed of any such adjustments beforehand. 9.4 Furthermore, nine has the right to inform customers about ongoing developments and new services of nine and its partners. The Customer may at any time state that he or she does not wish to receive such information. ## 10. Intellectual property and ownership of hardware 10.1 For the duration of the agreement, customers are granted the non-transferable, non-exclusive right to use the product or service. All rights to intellectual property existing or arising at the time of performance of the agreement relating to nine's products or services (e.g., programs, templates, data, marks, patents, copyrights, etc.), will remain with nine or the third parties engaged by it. 10.2 Unless the Parties expressly agree otherwise, any components that nine utilizes to perform the services are owned exclusively by nine or the third parties engaged by it. ## 11. Contractual term and termination ### 11.1 Commencement and term 11.1.1 nine presents its range of services without obligation on the website www.nine.ch. nine may change its range of services at any time and restrict individual services and/or cease to provide the same. 11.1.2 The agreement between nine and the Customer becomes effective through confirmation of a customer-specific offer or through use of the services by the Customer (particularly upon activation of these services in the Customer Cockpit). 11.1.3 Unless otherwise specified, the agreement entered into by the Customer and nine is open ended. ### 11.2 Termination of the agreement 11.2.1 With open-ended agreements (unless otherwise agreed), each of the Parties may terminate the agreement at the end of the month in question by giving one month's prior notice. Fixed-term agreements may be terminated one month before the agreed contractual term expires. If notice of termination is not given within the specified period, the agreement renews automatically for the agreed contractual term in each instance. 11.2.2 Notice of termination must be given in writing by registered mail followed by confirmation of receipt by nine or, for certain services, by selecting the termination option in the Customer Cockpit online. nine also has the right to terminate the agreement via email to the e-mail address indicated by the Customer for contract-related notifications. 11.2.3 nine reserves the right to terminate the agreement for good cause at any time without notice. Good cause exists in cases including but not limited to the following: ``` * if the Customer breaches material contractual provisions (e.g., sections 2.5, 3.2 and 3.6), misuses services for unlawful purposes, or stores or publishes illegal content (cf. Acceptable Use Policy); * if the Customer is declared bankrupt or insolvent or it otherwise becomes clear that the Customer can no longer meet his or her payment obligations, and he or she fails to provide the appropriate security for at least one contractual term (see, in particular, section 5.2). ``` 11.2.4 If nine terminates the agreement without notice, the Customer must pay the fees accruing until the date of ordinary termination, as well as compensation for all additional costs accruing in relation to the termination of the agreement without notice. 11.2.5 Once the agreement has expired, nine has the right to delete the Customer's data. The Customer himself is responsible for backing up his or her data in a timely manner. ## 12. Change of contractual terms and conditions 12.1 nine strives to keep its infrastructure in line with a current standard that fulfils the security requirements and technical standard customary in the industry. The Customer acknowledges and agrees that new technical developments, security requirements, and/or changes in the range of services offered by contracting partners of nine or in the software utilized by nine may result in an expansion or restriction of the range of services offered and may also affect price developments. 12.2 Therefore, nine explicitly reserves the right to change the contractual terms and conditions, including these GTC, at any time. Changes to the GTC are made public on nine's website and become effective upon publication. If the Customer does not accept the changes, he or she has the option to notify nine within 30 days of receipt of the notice in writing via registered mail with confirmation of transmission or via the Customer Cockpit and terminate the agreement subject to the notice period specified in section 11.2.1. If such notice is not given within this period, the changes will be deemed to have been accepted by the Customer. ## 13. Additional provisions 13.1 Should any data which nine stores for the client become endangered through measures taken by third parties, such as seizure or requisition, bankruptcy or settlement proceedings, or any other events or measures effected by third parties, nine has to inform the client of this endangerment immediately. Nine will inform any parties responsible within this context that the data sovereignty and ownership is that of the client alone. 13.2 Subject to section 2.2, the rights and obligations arising from agreements entered into under these GTC may only be assigned to a third party with the written consent of the other Party. This provision does not include the assignment of the agreement by nine to a legal successor or affiliate. 13.3 If one or more provisions of these General Terms and Conditions prove to be void or invalid, this will not affect the remaining provisions hereof. These provisions will remain identical and remain valid. The void provision(s) must be replaced by lawful provisions that are as equivalent as possible to the void provision(s). 13.4 These GTC and any disputes arising from or in connection with the contractual relationship between nine and the Customer are exclusively subject to Swiss law, excluding its conflict-of-law provisions and the provisions of the UN Convention on Contracts for the International Sale of Goods (CISG). 13.5 The courts of ordinary jurisdiction of nine's principal place of business shall have exclusive jurisdiction. Alternatively, nine has the right to take legal action against the Customer at his or her domicile. 13.6 These General Terms and Conditions take effect as of 10 August 2023 and supersede all prior versions. Zurich, 10 August 2023 --- ## Technical and organizational measures (TOMs) Of the organization: Nine Internet Solutions AG Release: 10. August 2023 Organizations that collect, process or use personal data themselves or on behalf of others must take the technical and organizational measures necessary to ensure that the provisions of the data protection laws are implemented. Measures are only required if the effort required to implement them is proportionate to the intended protection purpose. The above organization meets this requirement through the following measures: ## 1. Confidentiality ### 1.1 Physical access control Measures that are suitable for preventing unauthorized persons from accessing data processing systems with which personal data is processed or saved. Nine Internet Solutions AG operates its systems in two independent data centers in Zurich (Switzerland): - NTS: NTS Colocation AG - NTT: NTT Global Data Centers Switzerland AG | Technical Measures | NTS | NTT | | :--------------------------------------------------------------- | :-: | :-: | | Personnel and goods lock with biometric access control | ✔️ | | | Locking system with keys and code lock in our storeroom | ✔️ | | | Bell system with camera | | ✔️ | | Badge system with prior identity verification by security guards | | ✔️ | | Alarm system and secured building shafts | ✔️ | ✔️ | | Video surveillance of the entrances | ✔️ | ✔️ | | Locking system for rack access with our own cylinders and keys | ✔️ | ✔️ | | Organizational Measures | NTS | NTT | | :------------------------------------------------------------------------------- | :-: | :-: | | Log of all entries on the personnel and goods lock | ✔️ | | | Security operations center with security guards | | ✔️ | | Careful selection of security guards | | ✔️ | | Log of all entries after identity verification at the security operations center | | ✔️ | | Key regulation / list of keys | ✔️ | ✔️ | | Employee and guest badges | ✔️ | ✔️ | | Guests without permanent access only when accompanied by authorized persons | ✔️ | ✔️ | | Careful selection of cleaning service employees | ✔️ | ✔️ | ### 1.2 Logical access control Measures that are suitable to prohibit virtual access to data processing systems by unauthorized persons. | Technical Measures | Organizational Measures | | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ Login with biometric authentication✔️ Login with SSH keys✔️ Login with username and password✔️ Anti-Virus-Software clients✔️ Firewall✔️ Intrusion Detection System (IDS)✔️ Intrusion Prevention System (IPS)✔️ Use of VPN for remote access✔️ Encryption of disks ✔️ Automatic desktop lock ✔️ Encryption of notebooks / tablets ✔️ Regular security scan routine | ✔️ Information security policy✔️ User Management✔️ Creation of user profiles✔️ Central password assignment✔️ Secure password policy✔️ Wipe / destroy policy✔️ Clean desk policy✔️ Mobile Device Policy | ### 1.3 Privilege control Measures that ensure that those authorized to use a data processing system can only access the data subject to their access authorization and that personal data while processing, using and after saving cannot be read, copied, changed or removed without authorization. | Technical Measures | Organizational Measures | | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ Paper shredder (security level P-4)✔️ Physical wiping of disks✔️ Logging of access to applications, especially during creation, change and removal of data | ✔️ Use of authorization concepts✔️ Minimum number of administrators✔️ Data protection vault✔️ Management of user rights by administrators | ### 1.4 Separation control Measures to ensure that data collected for different purposes can be processed separately. This can be ensured, for example by logically and physically separating the data. | Technical Measures | Organizational Measures | | :------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------- | | ✔️ Separation of production and test environment✔️ Multi-client capability of relevant applications | ✔️ Control via authorization concept✔️ Defining database rights | ### 1.5 Pseudonymization The processing of personal data in such a way that the data can no longer be assigned to a specific person without consulting additional information, provided that this additional information is stored separately and is subject to appropriate technical and organizational measures. | Technical Measures | Organizational Measures | | :---------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ In the case of pseudonymization: Separation of the assignment data and storage in a separate and secure system (encrypted) | ✔️ Internal instruction to anonymize and if possible pseudonymize personal data in the event of disclosure or after the statutory deletion period, respectively our preservation interest, has expired | ## 2. Integrity ### 2.1 Disclosure control Measures to ensure that personal data during electronic transmission or during their transport or while saving onto disks can not be unauthorized read, copied, changed or removed and that it can be checked and determined to which external parties a transfer of personal data through facilities for data transmission is intended. | Technical Measures | Organizational Measures | | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ Email-encryption (GPG)✔️ Email-Signature (GPG)✔️ Use of VPN✔️ Logging of accesses and retrievals✔️ Safe transport containers✔️ Sending over encrypted connections (SFTP, HTTPS)✔️ Usage of signature procedures | ✔️ Documentation and logging of the data recipients as well as the duration of the planned transfer or the deletion periods✔️ Overview of periodical retrieval and transmission processes✔️ Disclosure in anonymous or pseudonymised form if necessary✔️ Careful selection of transport staff and vehicles✔️ Personal delivery with protocol | ### 2.2 Input control Measures to ensure that it can be subsequently checked and determined whether and by whom personal data has been entered, changed or removed in data processing systems. | Technical Measures | Organizational Measures | | :--------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ Technical logging of creation, change and deletion of personal data✔️ Manual control of logs | ✔️ Overview of tools which are used to create, change or delete personal data✔️ Traceability of creation, modification and deletion of data by individual usernames (not user groups)✔️ Assignment of rights to create, modify or deletion of personal data based on an authorization concept✔️ Clear responsibilities for deletions | ## 3. Availability and resilience ### 3.1 Availability control Measures to ensure that personal data is protected against accidental destruction or loss. | Technical Measures | Organizational Measures | | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ✔️ Redundant emergency power systems with diesel generators and batteries✔️ Fire and smoke alarm systems✔️ Gas fire extinguishing system✔️ Fire extinguisher server room✔️ Server room monitoring temperature and moisture✔️ Server room redundantly air-conditioned✔️ UPS✔️ Protective power strips server room✔️ Privacy safe✔️ RAID system✔️ Video surveillance server room✔️ Alarm message in the event of unauthorized access to the server room | ✔️ Backup & recovery concept✔️ Control of the backup process✔️ Regular data recovery tests and logging of results✔️ Storage of the backup media in a safe place outside the server room✔️ No sanitary connections in or above the server room✔️ Existence of an emergency plan | ## 4. Procedures to periodically review, assess and evaluate ### 4.1 Privacy management | Technical Measures | Organizational Measures | | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ Central documentation of all procedures and regulations with access options for employees as required / authorized✔️ ISO 27001 Information security certification✔️ ISO 9001 Quality management certification✔️ The effectiveness of the technical protective measures is checked at least once a year | ✔️ Internal data protection officer✔️ Employees trained and committed to confidentiality / data secrecy✔️ Regular security awareness training of employees at least once a year✔️ Internal information security officer✔️ The data protection impact assessment is carried out if necessary✔️ The organization complies with the information obligations under Art. 13 and 14 GDPR✔️ Formalized process for processing requests for personal data from those affected | ### 4.2 Incident-Response-Management Security breach response assistance | Technical Measures | Organizational Measures | | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ Use of firewall with regular updates✔️ Use of spam filter with regular updates✔️ Use of virus scanner with regular updates✔️ Intrusion Detection System (IDS)✔️ Intrusion Prevention System (IPS) | ✔️ Documented process for detecting and reporting security incidents / data breaches (also with regard to the obligation to report to the supervisory authority)✔️ Documented procedure for handling security incidents✔️ Involvement of Information security officer and data protection officer in security incidents and data breaches✔️ Documentation of security incidents and data breaches using a ticket system✔️ Formal process and responsibilities for post-processing of security incidents and data breaches | ### 4.3 Privacy friendly presets Privacy by design / Privacy by default | Technical Measures | | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✔️ No more personal data is collected than is required for the respective purpose✔️ Simple exercise of the data subject's right of withdrawal through technical measures | ### 4.4 Order control (Outsourcing to third parties) Measures to ensure that personal data processed on behalf of the client can only be processed in accordance with the client's instructions. In addition to data processing on behalf, this item also includes the performance of maintenance and system support work both on site and via remote maintenance. If the Contractor uses service providers in the sense of commissioned processing, the following points must always be regulated with them. | Organizational Measures | | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ✔️ Prior verification of the safety measures taken by the contractor and their documentation✔️ Selection of the contractor under due diligence aspects (especially with regard to data protection and data security)✔️ Conclusion of the necessary agreement on commissioned processing or if need be EU standard contractual clauses✔️ Written instructions to the contractor✔️ Obligation of the contractor's employees to maintain data secrecy✔️ Obligation to appoint a data protection officer by the contractor if the obligation to appoint exists✔️ Agreement on effective control rights vis-à-vis the contractor✔️ Regulation on the use of further subcontractors✔️ Ensuring the destruction of data after the completion of the order✔️ In the case of longer cooperation: Ongoing review of the contractor and its level of protection | #### Third-Party Processors with Access to Customer Data For the list of third-party data processors that may access customer data as part of service delivery, refer to [Appendix 3 of the Data Processing Agreement (DPA)](/docs/legal-documents/data-processing-agreement#appendix-3). #### Operational Vendors and Partners Nine Internet Solutions AG works with the following vendors and partners in the context of internal operations, infrastructure, and service delivery. These parties do not have access to customer data as part of their engagement with Nine. - [**ActiveCampaign, LLC**](https://www.activecampaign.com/) - [**ALSO Schweiz AG**](https://www.also.ch/) - [**Anthropic**](https://trust.anthropic.com/) - [**Brevo**](https://www.brevo.com/) - [**Canonical Group Limited**](https://canonical.com/company) - [**Dalco AG**](https://www.dalco.ch/company/) - [**Digitec Galaxus AG**](https://www.galaxus.ch/en/wiki/528) - [**Elektrizitätswerk der Stadt Zürich (ewz)**](https://www.ewz.ch/) - [**Freexian SARL**](https://www.freexian.com/) - [**Fusion Trade, Inc.**](https://www.fusionww.com/) - [**GAS&COM AG**](https://www.gas-com.ch/en/company-profile/) - [**GitLab Inc.**](https://about.gitlab.com/company/) - [**Hewlett Packard (Schweiz) GmbH**](https://www.hp.com/ch-de/home.html) - [**Infinigate Schweiz AG**](https://www.infinigate.com/about/) - [**Init7 (Schweiz) AG**](https://www.init7.net/en/init7/) - [**iWay AG**](https://www.iway.ch/ueber-iway/) - [**Livit AG**](https://www.livit.ch/en/livit-ag) - [**Nine Internet Solutions Ltd. (Canada)**](https://www.nine.ch/) - [**NTS Colocation AG**](https://nts.ch/en/colocation) - [**NTT Global Data Centers Switzerland AG**](https://www.nttdata.com/global/en/services/global-data-centers) - [**Nutanix**](https://www.nutanix.com/what-we-do) - [**OpenVox**](https://github.com/openvoxproject) - [**Qualinet Consulting AG**](https://www.qualinet.swiss/) - [**Renuo AG**](https://www.renuo.ch/en/story) - [**Ripe NCC**](https://www.ripe.net/about-us/) - [**Slack Technologies Limited**](https://slack.com/about) - [**Sunrise GmbH**](https://www.sunrise.ch/en/corporate/home) - [**Wallee**](https://en.wallee.com/) - [**Zenduty**](https://www.zenduty.com/) --- ## Application Monitoring Application Monitoring is a component of nine Managed GKE that allows you to monitor your applications in a self-service way. ## Video Guide Checkout our video guide series for Application Monitoring. ## Details With application monitoring, nine provides a complete monitoring solution for you with the following features: - managed Prometheus instance - integrated exporters - service discovery and Prometheus rules can be configured in a self-service way - managed alertmanager to send out notifications - alertmanager configuration can be changed in a self-service way - integrated grafana datasource for Prometheus ## Availability Application monitoring is charged separately from the nine managed GKE base platform. To order the needed components for application monitoring please [create a support ticket](/docs/general/contact). ## Prerequisites Nine nodes need to be upgraded to n1-standard-2 nodes atleast, as the n1-standard-1 don't have enough power. If you order Application Monitoring, we will upgrade the nodes to n1-standard-2. ## Usage Please see the following sections for an explanation on how to use the application monitoring solution. ### General information about the setup The application monitoring solution is based on the [prometheus-operator project](https://github.com/coreos/prometheus-operator). When ordering the application monitoring product, nine will: - run one (or more) Prometheus instance(s) (backed by GCP regional storage) for you on the nine node pool - run an instance of the Prometheus-operator on the nine node pool - run 2 instances of alertmanager on the nine node pool - pre-configure some metric exporters in the customer Prometheus - create a grafana datasource for the customer Prometheus Running all components on the nine node pool will leave all available resources of your node pools to your applications. Moreover, by using GCP regional storage for Prometheus, the instance can failover to another GCP compute zone (high availability). With the help of the prometheus-operator project you can then use the following resources to create scraping configurations and recording/alerting rules: - ServiceMonitors - PodMonitors - PrometheusRules It is possible to run multiple Prometheus instances in your cluster if needed. Every Prometheus instance gets a name which you can see on [runway](https://runway.ninegcp.ch). This name must be used in the **nine.ch/prometheus** label of all corresponding resources so that it will be picked up by the prometheus instance. Please have a look on ["Adding customer application metrics to Prometheus"](#adding-application-metrics-to-prometheus) and ["Adding alerting rules to Prometheus"](#adding-rules-to-prometheus) for more information about how to use these resources. The provided alertmanager will be sending out notifications in case of triggering Prometheus rules. The alertmanager instances can be configured by providing a special named kubernetes secret in a certain namespace. Please see ["Configure alertmanager"](#configuring-alertmanager) for further information. ### Accessing the web UI The Prometheus and alertmanager web UI URLs can be found on [runway](https://runway.ninegcp.ch). ### Instrumenting your application Before Prometheus can scrape metrics from your application, you will need to instrument your application to export metrics in a special given format. You can find information about how to do this in the [official Prometheus documentation](https://prometheus.io/docs/instrumenting/clientlibs/). ### Adding application metrics to Prometheus Once your application supports metrics, you can use `ServiceMonitors` or `PodMonitors` to let Prometheus scrape your applications metrics. [ServiceMonitors](https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#servicemonitor) will scrape all pods which are targeted by one or more services. This resource needs to be used in most of the cases. You need to define a label selector in the `ServiceMonitor` which will be used to find all the wanted services. The `ServiceMonitor` should be created in the same namespace as the service(s) it selects. Next to the label selector your `ServiceMonitor` should also have the label **nine.ch/prometheus** set with the name of your Prometheus instance (can be found on [runway](https://runway.ninegcp.ch). Consider the following example `ServiceMonitor` and `Service` definition: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: my-app namespace: my-app labels: nine.ch/prometheus: myPrometheus spec: selector: matchLabels: app: my-app endpoints: - port: web ``` ```yaml kind: Service apiVersion: v1 metadata: name: my-app-service namespace: my-app labels: app: my-app spec: selector: application: example-app ports: - name: web port: 8080 ``` The given `ServiceMonitor` definition will select the service "my-app-service" because the label "app: my-app" exists on that service. Prometheus will then search for all pods which are targeted by this service and starts to scrape them for metrics on port 8080 (the `ServiceMonitor` defines the port in the _endpoints_ field). [PodMonitors](https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#podmonitor) will scrape all pods which are selected by the given label selector. It works very similar to the `ServiceMonitor` resource (just without an actual `Service` resource). You can use the `PodMonitor` resource if your application does not need a `Service` resource (like some exporters) for any other reason. The pods should run in the same namespace as the `PodMonitor` is defined. Here is an example for a `PodMonitor` with a corresponding pod: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: my-pods namespace: my-app labels: nine.ch/prometheus: myPrometheus spec: selector: matchLabels: application: my-app endpoints: - port: web ``` ```yaml apiVersion: v1 kind: Pod metadata: labels: application: my-app name: my-app namespace: my-app spec: containers: - image: mycompany/example-app name: app ports: name: web containerPort: 8080 ``` Based on the given `PodMonitor` resource the prometheus-operator will generate a scrape config which scrapes the shown pod "my-app" on port 8080 for metrics. Prometheus will create a _job_ For every `ServiceMonitor` or `PodMonitor` resource you define. It will also add a _job_ label to all scraped metrics which have been gathered in the corresponding job. This can be used to find out from which services or pods a given metric has been scraped. ### Querying for metrics You can use [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) to query for metrics. There are some [examples](https://prometheus.io/docs/prometheus/latest/querying/examples/) on the official Prometheus page. Querying can be done by either using the Prometheus web UI or by using grafana in the explore view. When using grafana please make sure to select the data source matching your Prometheus instance. The data source name will be **prometheus-\**. ### Adding rules to Prometheus Prometheus supports two kinds of rules: _recording rules_ and _alerting rules_. Both have a similar syntax, but a different use case. [Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules) can be used to calculate new metrics from already existing ones. This can be useful if you use computationally expensive queries in dashboards. To speed them up you can create a recording rule which will evaluate the query in a defined interval and stores the result as a new metric. You can then use this new metric in your dashboard queries. [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) allow you to define alert conditions (based on PromQL). When those conditions are true, Prometheus will send out an alert to the connected alertmanager instances. Alertmanager will then send notifications to users about alerts. When creating alerting or recording rules, please make sure to add the **nine.ch/prometheus** label with the name of your Prometheus instance. This will assign the created rule to your Prometheus instance. The following example alerting rule will alert once a job can not reach the configured pods (targets) anymore: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: nine.ch/prometheus: myPrometheus role: alert-rules name: jobs-check spec: groups: - name: ./example.rules rules: - alert: InstanceDown expr: up == 0 for: 5m labels: severity: Critical annotations: summary: "Instance {{ $labels.instance }} down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes." ``` This alerting rule definition will trigger an alert once a _up_ metric gets a value of 0. The _up_ metric is a special metric as it will be added by Prometheus itself for every job target (pod). Once a pod can not be scraped anymore, the corresponding _up_ metric will turn to 0. If the _up_ metric is 0 for more than 5 minutes (in this case), Prometheus will trigger an alert. The specified "labels" and "annotations" can be used in alertmanager to customize your notification messages and routing decisions. For the full `PrometheusRule` spec, see the [prometheus-operator API reference](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusrule). Here is an example of a recording rule: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: nine.ch/prometheus: myPrometheus role: recording-rules name: cpu-per-namespace-recording spec: groups: - name: ./example.rules rules: - record: namespace:container_cpu_usage_seconds_total:sum_rate expr: sum(rate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!="", container!="POD"}[5m])) by (namespace) ``` This recording rule will create a new metric called _namespace:container_cpu_usage_seconds_total:sum_rate_ which shows the sum of used CPU of all containers per namespace. This metric can easily be shown in a grafana dashboard to have an overview about the CPU usage of all pods per namespace. The [kubernetes-mixins project](https://github.com/kubernetes-monitoring/kubernetes-mixin) contains sample alerts and rules for various exporters. It is a good place to get some inspiration for alerting and recording rules. Some of those rules and alerts have already been integrated into your instance of Prometheus. ### Checking website availability You can easily check the HTTP return code of any HTTP endpoint which is reachable via a Kubernetes ingress. All you need to do is to add a label named **nine.ch/prometheus** with the name of your Prometheus instance set as the value to your Kubernetes ingress. The name of your Prometheus instance can be found on [runway](https://runway.ninegcp.ch). Prometheus will then start to check your HTTP endpoint. This is done by utilising the [Prometheus blackbox exporter](https://github.com/prometheus/blackbox_exporter) which we run as a [GCP Cloud Function](https://cloud.google.com/functions) alongside your GCP project. This instance is only accessible from within your nine Managed GKE cluster. To further customize the check we support the following annotations on the ingress resource: | annotation | description | example | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | blackbox-exporter.nine.ch/valid_status_codes | Allow the check to be successful if the return code matches one of the given comma separated status codes. Shortcuts like _2xx_ can be used to specify the corresponding full range of http status codes. If this annotation is not present, the default expected status codes are 200-299 (2xx). | blackbox-exporter.nine.ch/valid_status_codes: 2xx, 3xx, 401 | | blackbox-exporter.nine.ch/expect_regexp | Mark the probe as successful if the given regular expression was found in the body of the response. | blackbox-exporter.nine.ch/expect_regexp: status=\[oO\]\[kK\] | | blackbox-exporter.nine.ch/fail_on_regexp | Mark the probe as unsuccessful if the given regular expression was found in the body of the response. | blackbox-exporter.nine.ch/fail_on_regexp: status=(\[fF\]ailed|\[eE\]error) | You can query your prometheus instance for the relevant metrics which are returned by this check with the following query: ``` {job="ingress-check"} ``` To get the returned status code you can use: ``` probe_http_status_code{job="ingress-check",namespace=,ingress=} ``` To see if your check returned successfully you can leverage the "probe_success" metric: ``` probe_success{job="ingress-check",namespace=,ingress=} ``` The metric "probe_duration_seconds" will show how long it took to check the HTTP endpoint. It contains a "phase" label which helps to identify the correspondiung probe duration of a specific HTTP connection stage. ### Configuring Alertmanager Alertmanager is the component responsible to send out notifications in case of Prometheus alerts. Alertmanager support various channels for notifications, like Slack, Email, Hipchat, PagerDuty, etc. Please have a look at the [official documentation](https://prometheus.io/docs/alerting/configuration/#configuration-file) for detailed information about the configuration. We also supply [example configurations](#alertmanager-configuration-examples). The default alertmanager instances, which are created by us, do not have any notification receivers configured by default. To configure them, we provide a way to supply a complete alertmanager configuration. You can create a secret which has to be named **alertmanager** in a namespace called **alertmanager-config**. The secret needs to have a key called **alertmanager.yaml** which contains a full alertmanager configuration. Furthermore [special annotations](#mandatory-annotations) have to be set. The contained alertmanager configuration will be checked for validity on creation and update of the secret. If the configuration contains errors or the mandatory annotations of the secret are missing, the secret will not be accepted. In that case you will receive an error message which describes the issue. As the configuration can be given as a secret, we recommend to use a [sealed secret resource](./sealed-secrets) in combination with gitops techniques to create and maintain it. With this way, you do not store confidential information in your configuration git. #### Mandatory annotations Please make sure that your **alertmanager** secret contains the following annotations: ```yaml replicator.v1.mittwald.de/replication-allowed: "true" replicator.v1.mittwald.de/replication-allowed-namespaces: "nine-alertmanager-customer" ``` #### Manual steps to create the configuration 1. create the namespace **alertmanager-config** ```bash $> kubectl create namespace alertmanager-config ``` 1. create a local directory where all needed configuration files go in. The directory needs to have at least a file called **alertmanager.yaml** which contains a valid alertmanager configuration. Have a look at [our examples](#alertmanager-configuration-examples). You can also put template files into this directory (extension should be **.tmpl**). 1. create a secret called **alertmanager** with the mandatory annotations in the above created namespace **alertmanager-config** ```bash $> export AMDIR= $> kubectl create secret generic alertmanager --from-file=$AMDIR --dry-run -o yaml -n alertmanager-config | \ kubectl annotate -f- --dry-run --local -o yaml \ replicator.v1.mittwald.de/replication-allowed=true \ replicator.v1.mittwald.de/replication-allowed-namespaces=nine-alertmanager-customer | \ kubectl apply -f- ``` If you use template files in your configuration, please make sure to include the following line into your **alertmanager.yaml** file: ```yaml templates: - "/etc/alertmanager/config/*.tmpl" ``` #### Using sealed secrets for configuration If you want to use gitops to control the alertmanager configuration, we recommend to use a [SealedSecret](./sealed-secrets) to define your configuration. This way you do not expose access credentials in git. You can use [runway](https://runway.ninegcp.ch) to generate a secret with the name **alertmanager** in the namespace **alertmanager-config**. The secret needs to have at least a key called **alertmanager.yaml** which contains a valid alertmanager configuration. It might have additional keys which define template names (like 'slack.tmpl' for example). To make sure that the sealed secrets controller adds the mandatory annotations please add a 'template' section to the generated secret as shown below: example secret generated by runway: ```yaml apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: alertmanager namespace: alertmanager-config spec: encryptedData: alertmanager.yaml: ``` secret after adding the mandatory _template_ section: ```yaml apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: alertmanager namespace: alertmanager-config spec: encryptedData: alertmanager.yaml: template: metadata: annotations: replicator.v1.mittwald.de/replication-allowed: "true" replicator.v1.mittwald.de/replication-allowed-namespaces: "nine-alertmanager-customer" ``` ##### Overwriting an existing secret If you already created a kubernetes secret **alertmanager** manually and want to overwrite it now with the use of a sealed secret, you first need to add the following annotation to the existing secret: ```yaml sealedsecrets.bitnami.com/managed: "true" ``` This can be achieved with the following command: ```bash kubectl annotate secret alertmanager sealedsecrets.bitnami.com/managed="true" -n alertmanager-config ``` If this annotation is missing, the sealed secrets controller will refuse to overwrite the already existing secret. #### Alertmanager configuration examples **1.** Send all alerts via email ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m route: receiver: "email" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: [] receivers: - name: "email" email_configs: - to: "monitoring-alerts-list@your-domain.ch" send_resolved: true # when using STARTTLS (port 587) this needs to be 'true' require_tls: false from: "alertmanager@your-domain.ch" smarthost: smtp.your-domain.ch:465 auth_username: "alertmanager@your-domain.ch" auth_password: "verysecretsecret" headers: { Subject: "[Alert] Prometheus Alert Email" } ``` **2.** Send all critical alerts via slack. All other severities will be sent out via email. Please make sure to add a `severity` label to your alerts. ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m route: # this specifies the default receiver which will be used if no route matches receiver: "email" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: - receiver: "slack" match_re: severity: "[cC]ritical" receivers: - name: "email" email_configs: - to: "monitoring-alerts-list@your-domain.ch" send_resolved: true # when using STARTTLS (port 587) this needs to be 'true' require_tls: false from: "alertmanager@your-domain.ch" smarthost: smtp.your-domain.ch:465 auth_username: "alertmanager@your-domain.ch" auth_password: "verysecretsecret" headers: { Subject: "[Alert] Prometheus Alert Email" } - name: "slack" slack_configs: - send_resolved: true api_url: https://hooks.slack.com/services/s8o3m2e0r8a8n2d/8snx2X983 channel: "#alerts" ``` **3.** Send all alerts of the production environment via slack. Drop all other alerts. Please make sure to define the label 'environment' in your alerts. ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m route: # this specifies the default receiver which will be used if no route matches receiver: "devnull" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: - receiver: "slack" match: environment: production receivers: - name: "slack" slack_configs: - send_resolved: true api_url: https://hooks.slack.com/services/s8o3m2e0r8a8n2d/8snx2X983 channel: "#alerts" - name: devnull ``` **4.** Use templates to customize your notifications and send all alerts via slack. Here we define some templates in a file called 'slack.tmpl'. ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m # THIS LINE IS VERY IMPORTANT AS OTHERWISE YOUR TEMPLATES WILL NOT BE LOADED templates: - "/etc/alertmanager/config/*.tmpl" route: receiver: "slack" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: [] receivers: - name: "slack" slack_configs: - send_resolved: true api_url: https://hooks.slack.com/services/s8o3m2e0r8a8n2d/8snx2X983 channel: "#alerts" pretext: "{{ .CommonAnnotations.description }}" text: '{{ template "slack.myorg.text" . }}' ``` ```yaml title="slack.tmpl" {{ define "slack.myorg.text" -}} {{ range .Alerts -}} *Alert:* {{ .Labels.alertname }} - `{{ .Labels.severity }}` *Description:* {{ .Annotations.description }} *Details:* {{ range .Labels.SortedPairs -}} • *{{ .Name }}:* `{{ .Value }}` {{ end -}} {{ template "slack.default.text" . }} {{ end -}} {{ end -}} ``` ## Documentation and Links - [Prometheus documentation](https://prometheus.io/docs/) - [prometheus-operator project](https://github.com/coreos/prometheus-operator) - [screencast about application monitoring](https://youtube.com/coolvideo) - [Video guide to Application Monitoring Part 1](https://www.youtube.com/watch?v=AvoD4LeR020) - [Video guide to Application Monitoring Part 2](https://www.youtube.com/watch?v=4ai1qDSRqic) - [Video guide to Application Monitoring Part 3](https://www.youtube.com/watch?v=HjUMs6S6SZc) --- ## Argo CD Argo CD is a service of nine Managed GKE that allows to continuously deploy applications to the GKE cluster by using a gitops workflow. ## Details For customers who need to continuously deploy application code, Argo CD provides: - declarative and version controlled application deployments - automation and traceability via gitOps workflow - support for helm, kustomize and jsonnet application declarations - a web UI for visualizing kubernetes resources - webhook integration to fully automate deployments on git operations - a command line interface application - audit trails for application events and API calls - parameter overrides of helm/ksonnet declarations (simplifies development deployments) - a grafana metrics dashboard ## Availability Argo CD is available as standard with nine Managed GKE ## Usage Before starting to use Argo CD it is important to understand how a typical workflow should look in the end. Argo CD supports a continuous deployment by utilizing a gitops workflow. For that to work it recommends to separate application code from application configuration (helm charts, kustomize files, etc...). The separation should happen by using 2 different git repositories. Although it is technically possible to use 1 git repository, [best practises](https://argoproj.github.io/argo-cd/user-guide/best_practices/) advise strongly against doing so. When using 2 separate git repositories, one possible production deployment workflow with Argo CD could look like: 1. a developer creates a pull request/merge request to get some application code changes merged into the master branch 1. after merging of the changes happened (and all tests passed), a tag will be created by the developer signaling that a new productive version of the application should be build 1. a CI pipeline starts. It executes the following steps: - it builds, tags and pushes a new application container image. - it creates a commit in the configuration git repository, specifying the new image version to be used (for example by changing the content of the values.yaml in a helm chart) - it pushes the commit 1. (optional) a git webhook signalises Argo CD to check for new commits in the configuration repository 1. Argo CD deploys the new image version of the container Argo CD is not connected to the application source code repsoitory in any way. It only connects to the configuration git repository (read only permissions are sufficient). If there are any problems with the deployed version, a rollback can be initiated by reverting the commit in the configuration git repository. ArgoCD will then deploy the previous version of the image. For a further separation of access it is also possible to not directly commit to the configuration git repository within the pipeline. Instead a pull request/merge request will be created which needs to be approved before the new image version should be deployed. With this it is possible to give developers access to the code repository without granting permissions in the configuration repository. ### Requirements To be able to use Argo CD (for production deployments) with a gitops workflow you will need at least the following: - the URL to your Argo CD installation (see [Login](#web-ui)) - a kubernetes namespace where Argo CD can deploy to (see [Namespace Creation](#namespace-creation)) - a git repository with the configuration of your application (called the _config repo_). This can be: - a [kustomize](https://kustomize.io/) application - a [helm chart](https://helm.sh/) - a directory of plain yaml manifests - a [jsonnet](https://jsonnet.org/) application - a CI tool/service for: - automatically building container images - doing changes to the configuration repository (optional) We at nine are preferring helm charts as we are using them in the company ourselves. An example helm application configuration which deploys a guestbook application can be found [in the argo project github namespace](https://github.com/argoproj/argocd-example-apps.git) in the _helm-guestbook_ directory. ### Permissions The current authorization concept permits all configured nine Managed GKE users with one of the following roles with full access to all Argo CD applications and projects in their installation of Argo CD: - admin - user Users with the role "view" are only permitted to see configured Argo CD applications and projects, but are not authorized to change them. ### Login Argo CD provides a web user interface as well as a cli application to interact with it. #### Web UI You can find the URL for visiting the web UI in [runway](https://runway.ninegcp.ch). You can login with your ninegcp.ch user account credentials after clicking on **Login via keycloak** in the web UI. #### CLI The CLI application can be downloaded on the _help_ page in the [Argo CD web interface](#web-ui) (you will find a link to the help page in the navigation menu on the left side). To login via the cli application, please follow these steps: 1. execute `argocd login --sso` locally in a terminal on your machine 1. argocd will open a browser page so that you can enter your ninegcp.ch credentials 1. after a successful authentication you can use `argocd` locally on the cli as an authenticated user Argo CD opens a local port (8085 by default) on your machine to be able to authenticate via single sign on. If that port is already in use by another application, please choose a different port by the using the `--sso-port` argument. ### Configuration resources in Argo CD Argo CD introduces 2 kubernetes resources: _Applications_ and _Projects_. **Applications**: The Application CRD is the Kubernetes resource object representing a deployed application instance in an environment. It is defined by two key pieces of information: - a _source_ reference to the desired state in the configuration Git (repository, revision, path, environment) - a _destination_ reference to the target cluster and namespace. It basically describes which configuration state should be deployed to which namespace in your nine Managed GKE cluster. **Projects**: The AppProject CRD is the Kubernetes resource object representing a logical grouping of applications. It is defined by the following key pieces of information: - a _sourceRepos_ reference to the configuration repositories that applications within the project can pull manifests from - a _destinations_ reference to clusters and namespaces that applications within the project can deploy into - a roles list of entities with definitions of their access to resources within the project Both CRDs can be used in git configuration repositories as well. Specifying the Argo CD applications/projects itself in git (versus creating them via Web UI or cli application) can be used to make use of the ["App of Apps Principle"](https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#app-of-apps). Please note that the namespace of your Argo CD installation is **nine-argocd** and not **argocd** as used in the examples. ### Deploying your application with Argo CD #### Namespace creation To be able to deploy your application you will need a kubernetes namespace where your application should be deployed to. By default Argo CD does not have the permissions to deploy to any namespace in your nine Managed GKE cluster. You explicitly need to annotate the deployment namespace where your application needs to be deployed with `nine.ch/argo-admin="true"`. The following example creates the namespace `my-application` and annotates it to permit access with Argo CD: ```bash kubectl create namespace my-application kubectl annotate namespace/my-application nine.ch/argo-admin="true" ``` Argo CD now has permissions to deploy into that namespace: ```bash $> kubectl describe rolebinding namespace-admins -n my-application Name: namespace-admins <...> Subjects: Kind Name Namespace ---- ---- --------- ServiceAccount argocd-server nine-argocd ServiceAccount argocd-application-controller nine-argocd <...> ``` #### Creating an Argo CD application An Argo CD application basically describes which application configuration should be deployed to which namespace of your nine managed gke cluster. You should first configure the git configuration repositpory via Argo CD. Afterwards you can use it when configuring the Argo CD application. You can configure the application and the repository either via the web UI or by using the CLI application. The following documentation describes how to create the application via the web UI. ##### A note about SSH Although it is possible to connect your config git repository via SSH, we recommend to use HTTPS. If you want to use SSH you need to add the public host key of your git provider to Argo CD as [described in the Argo CD documentation](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#unknown-ssh-hosts). This is not needed when using the HTTPS protocol. Please make sure that you are using a trusted TLS certificate when using HTTPS. ##### Steps Follow these steps to configure a repository via the web UI: 1. Login to the web UI (see [Login](#web-ui)) 1. Click on the _gear icon_ in the menu on the left ("Manage your repositories,projects,settings") 1. Click on _Repositories_ 1. Click on _Connect repo using https_ 1. You can now enter the repository details | Item | Description | Example | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Repository URL | the https URL to your config git repository | [https://gitlab.com/example/my-application-config](https://gitlab.com/example/my-application-config) | | Username | the username to access the repository (please use [access tokens](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#access-token) and not personal credentials) | argocd | | Password | the password to access the repository (please use [access tokens](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#access-token) and not personal credentials | 3macm32449asdnf243rt | | TLS client certificate | an optional TLS client certificate in PEM format which you use for authentication with your git repository | | | TLS client certificate key | an optional TLS client certificate key in PEM format which you use for authentication with your git repository | | | skip server verification | check this box if Argo CD should not verify the TLS certificate of your HTTPS connection | | | Enable LFS support | check this if you used [git large file support](https://github.com/git-lfs/git-lfs/) in your repository | | After you registered the git repository you can now configure your Argo CD application 1. Click on _New Application_ (upper left of the screen) on the main page of Argo CD 1. You can now enter the application details | Item | Description | Example | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Application Name | the name of your application | my-application | | Project | the project your application is part of (see [Projects](#projects) for more information) | default | | Sync Policy | choose between _Manual_ or _Automatic_ synchronisation. With _Manual_ synchronisation you will have to trigger a sync manual via Web UI or CLI. _Automatic_ synchronisation will check your git repo every 3 minutes (or immediately when using [webhooks](#webhooks) from your git provider) | Automatic | | Source | the source of the configuration git repository (you should be able to select the created repository from the first step) | [https://gitlab.com/example/my-application-config](https://gitlab.com/example/my-application-config) | | Revision | this specifies either the branch, tag or commit you want to use in the configuration repository. This can be used to create different environments of your application. | development | | Path | please specify _._ here if all your files (helm chart, kustomize files, etc) are in the root of the git repository. Otherwise you can specify the sub directory. | **.** | | Cluster | the cluster where to deploy (only "in-cluster" is currently possible) | in-cluster (https://kubernetes.default.svc) | | Namespace | the namespace where all the resources should be deployed to (see [Namespace creation](#namespace-creation)) | my-application | | Type | the type of the configuration (plain yaml files, kustomize, helm, etc). | Helm | | include subdirectories | if subdirectories should also be included | | If you use helm charts as configuration type it is possible to set multiple _value.yaml_ files which will be merged in the given order. #### Parameter overrides Sometimes a separate configuration repository is not really needed or just too much effort. This might be the case in development/testing environments where one wants to: - have faster iteration cycles - use upstream helm charts without forking them into an own git repository You might also want to set secrets directly in Argo CD without committing them into the configuration git. For those use cases Argo CD provides so called [parameter overrides](https://argoproj.github.io/argo-cd/user-guide/parameters/). Parameter overrides are only possible for applications which use helm charts or ksonnet configurations. By overriding parameters (for example in a helm chart) we are providing configuration information directly to Argo CD, without committing to a configuration repository. One possible workflow for a helm configuration application would look like: 1. a developer creates an Argo CD application which uses a upstream helm chart (either hosted in a git repository or in a helm chart repository) 1. development of application code happens in a feature branch 1. once the developer pushes new changes to the application code git repository, a CI pipeline starts which executes the following tasks: - build and push a new container image - tell Argo CD to use the new container image by using the `argocd` cli command to set a parameter override in the _values.yaml_ of the upstream helm chart - sync the application configuration state by executing `argocd app sync` for your application Parameter overrides for an application can be set via the cli application or via the web interface. ### Advanced topics #### Projects Projects in Argo CD are used to logical group applications. You can find more information about them in the [Argo CD documentation](https://argoproj.github.io/argo-cd/user-guide/projects/). Due to some restrictions in nine Managed GKE it is currently not possible to define RBAC rules for projects. One use case for projects is to create [roles](https://argoproj.github.io/argo-cd/user-guide/projects/#project-roles). With roles for example you can permit access to Argo CD applications from a CI/CD pipeline (to sync for example), by using the JWT token assigned to a role. Here is an example to create a role called _cicd_ allowed to sync all applications in the default project: ```bash argocd proj role create default cicd argocd proj role create-token default cicd # save this token somewhere argocd proj role add-policy default cicd -a sync -o '*' -p 'allow' ``` In your pipeline you can then sync applications with ```bash argocd app sync --auth-token ``` #### Webhooks With webhooks your git provider can immediately notify Argo CD about changes in the configuration git repository. Without webhooks Argo CD will check for new commits every 3 minutes. You need to create the webhook in your [git providers settings](https://argoproj.github.io/argo-cd/operator-manual/webhook/). The URL and predefined secrets can be found on the [runway](https://runway.ninegcp.ch) page. ## Documentation and links - [Argo CD Architecture](https://argoproj.github.io/argo-cd/#architecture) - [Argo CD documentation for developers](https://argoproj.github.io/argo-cd/user-guide/) --- ## Automatic TLS Certificates Automated TLS certificate provisioning is a service of nine Managed GKE that allows you to automate the lifecycle of Let's Encrypt certificates for ingress. ## Details For customers who need to have https ingress our cert-manager service provides an open source solution for provisioning and managing TLS certificates in Kubernetes clusters. ## Availability cert-manager is available as standard with nine Managed GKE. ## Usage To use cert-manager on your ingress object you simply need to add an annotation for the cluster issuer and a TLS block to indicate that a certificate should be created and stored in a secret: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: # add an annotation indicating the issuer to use. cert-manager.io/cluster-issuer: name: myIngress namespace: myIngress spec: # this is optional since the haproxy class is the default ingressClassName: haproxy rules: - host: myingress.com http: paths: - path: / backend: service: name: myservice port: number: 80 tls: # < placing a host in the TLS config will indicate a cert should be created - hosts: - myingress.com secretName: myingress-cert # < cert-manager will store the created certificate in this secret. ``` for the `cert-manager.io/cluster-issuer` value you may choose between `letsencrypt-prod` and `letsencrypt-staging`. For information about the difference between these please see the [letsencrypt documentation](https://letsencrypt.org/docs/staging-environment/). ## Documentation and Links - [cert-manager Source](https://github.com/jetstack/cert-manager) - [cert-manager Documentation](https://docs.cert-manager.io/en/latest/) - [cert-manager Ingress Shim](https://github.com/jetstack/cert-manager/blob/master/docs/tasks/issuing-certificates/ingress-shim.rst#how-it-works) - [Let's Encrypt Documentation](https://letsencrypt.org/docs/) --- ## Autoscaling your GKE workload Autoscaling enables you to worry less about capacity planning and ensures the uptime of your services during load peaks. All this while you only pay what resources are needed at any given moment. ## Details With autoscaling configured, GKE automatically adds new node(s) to your cluster if you've created new Pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized and its Pods can be run on other nodes, GKE can delete the node. Keep in mind that when resources are deleted or moved in the course of autoscaling your cluster, your services can experience some disruption. For example, if your service consists of a controller with a single replica, that replica's Pod might be restarted on a different node if its current node is deleted. Before enabling autoscaling, ensure that your services can tolerate potential disruption or that they are designed and configured so that downscaling does not disrupt Pods that cannot be interrupted. ## Availability All nine Managed GKE clusters come with cluster autoscaling enabled. But there are a few things that have to be configured in order to automatically scale your workload. ## Usage ### Scaling your workloads horizontally 1. Let us know your maximum node count By default we won't just scale your cluster to an infinite amount of nodes to guard you from unexpected costs. We have defined a minimum count of 3 nodes and a variable count of maximum nodes. Let us know what your preferred maximum node count is and we will set it for your cluster. 1. [Set CPU requests on your pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit) The cluster autoscaler is using this as a base to know how much capacity a node has. Without setting CPU requests the cluster autoscaler does not function. Plus this is good practice regardless if you make use of the autoscaler or not. 1. [Setup a Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) To scale your pods with the incoming load you can setup a Horizontal Pod Autoscaler (HPA) to scale the pods on the CPU utilization. As soon as your nodes are full this will in turn trigger the cluster autoscaler to add more nodes. The Kubernetes documentation has a great [walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) to help you setup a HPA. #### Scaling on custom metrics There is the possibility to scale horizontally by leveraging an optional managed installation of [keda](https://keda.sh). Keda allows to get metrics from various backends (called 'scalers' in keda terms) and to scale based on them. You can find the [available scalers](https://keda.sh/docs/2.4/scalers/) in keda's documentation. Keda also allows to scale on own custom metrics, by providing a self written [external scaler](https://keda.sh/docs/2.4/scalers/external/). If you are interested in a managed installation of keda or if you just want to know if your use case is supported by it, please send us a message. ### Scaling your workloads vertically To have kubernetes schedule your workloads properly you need to set resource requests on your containers (see [How pods with resource requests are scheduled](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-requests-are-scheduled)). This will tell kubernetes how many resources your container will use. The numbers you set are actually independent of what your container will really use when it is running. In reality the 'requested resources' differ a lot from what the pod is really using. As it is hard to estimate 'resource requests' (and as they also might change over time), the [vertical pod autoscaler project](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler) aims to provide sane settings for you. It monitors the resource usage of your pods when they are running and provides recommendations for 'resource requests'. Depending on the _updateMode_ setting, it will also apply those recommendations to your running pods, by evicting and restarting them with updated 'resource requests'. Please make use of [Pod Disruption Budgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) when using VerticalPodAutoscaler resources, as otherwise the eviction of pods might lead to disruptions. You should also always use at least 2 replicas. #### How to create a VerticalPodAutoscaler resource A VerticalPodAutoscaler resource can be used for kubernetes resources which control/manage pods themselves like `Deployments`, `StatefulSets`, `ReplicaSets`, etc. Here is an example for a VerticalPodAutoscaler which targets a Deployment called 'my-app'. The Deployment needs to be in the same namespace as the VerticalPodAutoscaler resource itself. ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: my-app-vpa spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: my-app updatePolicy: updateMode: "Auto" ``` This resource can be created with the usual `kubectl create -f -n ` command. The vertical pod autoscaler will check the usage metrics of the pods created by the Deployment. After some time, the VerticalPodAutoscaler resource will be updated with the found recommendations. As the _updateMode_ was set to "Auto" it will also replace the pods and restart them with the recommended values set. You can set the _updateMode_ to "Off" to just get the recommendations without replacing pods. Please have a look at this [tutorial from Google](https://cloud.google.com/kubernetes-engine/docs/how-to/vertical-pod-autoscaling#getting_resource_recommendations) which also provides information about how to disable recommendations for certain containers. #### Known Limitations For known limitations, see [vertical-pod-autoscaler known limitations](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#known-limitations). Most notably you shouldn't use a VerticalPodAutoscaler in combination with a HorizontalPodAutoscaler when scaling on cpu or memory. ## Documentation and Links - [GKE cluster autoscaler details](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler) --- ## Backup and Restore Backup & Restore is a service of nine Managed GKE allowing for regular backups and recovery of cluster data and configuration. ## Details Customers of nine Managed GKE need peace of mind that their cluster configuration and Persistent Volume Claim (PVC) data is backed up and can be made available when needed, for security and disaster recovery. Therefore nine regularly creates automated backups and on customer request engages in recovery and deployment of those backups (schedule and retention plan may be SLA dependent). Data of Persistent Volumes is snapshotted and backed up in a [multi-regional location](https://cloud.google.com/compute/docs/disks/snapshots), until the functionality of creating regional snapshots becomes available. Kubernetes resource backups are saved in a GCS Bucket and spread across the zones (i.e. a, b and c) of the cluster region. For example, if the cluster is in Zürich, the resource backups stay in Zürich. Non-Cluster Services (CloudSQL, nine Managed, etc.) have their own backup plans which are not covered by this service. ## Availability Backup/Restore is available as standard with nine Managed GKE. ## Usage - Backups of data and configuration will be automatically be taken nightly - Backups are retained for 30 days by defaults - You can restore namespaces [yourself](#restoring-namespaces) ### File Storage (NFS) PVC Restoration Please note that this paragraph is about the deprecated File Storage (NFS) add-on provided by Nine. For information about the Google Filestore product, see the [Google Filestore documentation](./filestore). When requesting a data restoration from an NFS PVC your entire NFS drive will be mounted to a new location. You will be given access to this backup via a pod. If it is important to your security implementation that if a disaster occurs only specific PVC's data is exposed you will need to implement specific storage classes for these volumes. Please contact to discuss this. ### Restoring namespaces To create restores, the [velero utility](https://velero.io/docs/v1.6/basic-install/#install-the-cli) needs to be installed locally. Make sure that your current kubecontext targets your GKE cluster by following the [cluster login steps](./#cluster-login). You can restore a complete namespace yourself. First you need to find the backup from which you want to restore. You can list all backups by executing the following command: ```shell-session velero backup get -n nine-velero ``` Once you found the backup from which you want to restore, you can restore the whole content of one namespace into another one by using: ```shell-session velero restore create -n nine-velero --from-backup --include-namespaces --namespace-mappings : ``` If you want to restore the content of an existing namespace, you either delete the target namespace before restoring it or you delete all existing resources which would be restored by velero. Velero does not overwrite any existing resources. ```shell-session velero restore create -n nine-velero --from-backup --include-namespaces ``` --- ## Centralized Logging with Loki Centralized Logging allows you to view and query logs of your containers using Grafana Loki. ## Details Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream. The logs are persisted for 90 days. ## Availability Centralized Logging is available as standard with nine Managed GKE. ## Usage Loki can be accessed by using the Grafana Web UI. The login details are provided on [runway](https://runway.ninegcp.ch). ### Labelling your pods If your pod is part of a deployment, statefulset or another controller, it will automatically be picked up by Loki, no matter what labels are set. We recommend using these [common labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) to easily find your logs. If you run a single pod, you will need to set one of these labels to ensure Loki will pick up your logs. - `app` - `name` ### Querying Logs with LogQL The query language used in Loki is called _LogQL_. To start querying your logs, head to the Grafana UI and click on _Explore_ in the sidebar or use the direct link provided on [runway](https://runway.ninegcp.ch). A LogQL query consists of two parts: log stream selector, and a search expression. A stream is selected by supplying one or more labels, for example: ```bash {app="nginx", name=~"frontend.+"} ``` To search for a certain string in the results, you can use a search expression. This can be just text matching by using `|=` or a regex expression by using `|~`. And by using a `!` instead of the pipe, the expression can be negated. Here are some examples: ```bash {app="nginx"} |= "GET" {app="nginx"} |~ "200|201|202" {app="nginx"} != "GET" {app="nginx"} !~ "200|201|202" ``` For more details, please refer to the [Loki documentation](https://grafana.com/docs/features/datasources/loki/#querying-logs). ### Pushing custom Logs If you have pods which store logs in files rather than writing them to `STDOUT`, you can use any [Loki client](https://github.com/grafana/loki/tree/v0.4.0/docs/clients) to push logs to it. Below, there's an example what this could look like. In the example we are using fluent-bit with the Loki plugin as a sidecar to an Nginx container to send logs to Loki. Please make sure to replace `` with the address found on [runway](https://runway.ninegcp.ch). The log path, format and labels are passed to fluent-bit as environment variables defined in the pod spec. [More information about Fluent Bit Loki plugin](https://github.com/grafana/loki/blob/v0.4.0/cmd/fluent-bit/README.md). ```yaml apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-loki data: fluent-bit.conf: |- [INPUT] Name tail Path ${LOG_PATH} [Output] Name loki Match * Url http://:3100/loki/api/v1/push BatchWait 1 BatchSize 1001024 Labels {app="${APP_LABEL}",pod="${POD_NAME}",namespace="${POD_NAMESPACE}"} LineFormat ${LOG_FORMAT} LogLevel info --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: volumes: - name: fluent-bit-config configMap: name: fluent-bit-loki - name: logs emptyDir: {} containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 volumeMounts: - name: logs mountPath: /var/log/nginx - name: fluent-bit-loki image: grafana/fluent-bit-plugin-loki:v0.4.0-amd64 volumeMounts: - name: fluent-bit-config mountPath: /fluent-bit/etc - name: logs mountPath: /var/log/nginx env: - name: APP_LABEL value: nginx - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LOG_PATH value: /var/log/nginx/*.log - name: LOG_FORMAT value: key_value ``` ### Pushing external logs If you want to push logs from external systems (like a external kubernetes cluster) to loki, we can create a basic auth secured ingress resource which will forward traffic to your loki instance. You can then use fluent-bit or promtail to push logs. Please [contact support](/docs/general/contact) to enable that feature. --- ## Cloud Pub/Sub Cloud Pub/Sub is an asynchronous service-to-service messaging system provided by GCP designed to be flexible and reliable. ## Availability Cloud Pub/Sub is NOT available as standard with nine Managed GKE. However, you can create a support-ticket for the activation. ## Ticket Content We will need following information when creating your Topic: ### Required - **Topic Name** - This will be the ID for your topic - **ServiceAccounts** - Let us know what setup you want for ServiceAccounts. We recommend to have a ServiceAccount to publish and one to subscribe. We can also setup ServiceAccounts that have permission to do both, or have access to multiple topics or subscriptions, if this is a requirement. Per Subscription: - **Subscription Name(s)** - This will be the ID(s) for you to subscribe on your topic. Multiple subscriptions per topic are possible. - **Type** - either push- or pull-subscriptions. - **Push Endpoint** - only for push-subscriptions, where to push messages to. Needs to be an HTTPS endpoint with valid certificates. ### Optional - **Allowed Persistence Regions** - You may choose if the messages should be stored in a different region than your GKE cluster's. Default is the same as the GKE cluster, europe-west6 (Switzerland). Per Subscription: - **Acknowledge Deadline** - The maximum time to allow for acknowledging a received message. If this deadline is exceeded without an acknowledgement by the receiver, the message is considered to be sent again. - **Message Retention Duration** - How long to retain unacknowledged messages in the subscriptions backlog, from the moment the message is published. Default: 7 days. If Retain Acked Messages is enabled, it also defines how long acknowledged messages will be retained. - **Retain Acked Messages** - Wether to retain acknowledged messages or not. ## Usage See the [Cloud Pub/Sub documentation](https://cloud.google.com/pubsub/docs/quickstart-client-libraries#publish_messages) for examples. The names of the Topics and Subscriptions are listed on [runway](https://runway.ninegcp.ch/). ### Login The ServiceAccounts will be listed on [runway](https://runway.ninegcp.ch/). ## Documentation and Links - [Cloud Pub/Sub](https://cloud.google.com/pubsub/) - [Cloud Pub/Sub Docs](https://cloud.google.com/pubsub/docs) --- ## Cloud SQL Cloud SQL is a service of GCP that allows customers to run mySQL or Postgres databases. ## Details For customers who need a managed database to store application data Cloud SQL provides this, and is fully integrated with your nine Managed GKE cluster. ## Availability To order a Cloud SQL database please contact info@nine.ch ## Usage Once you have ordered a Cloud SQL database you can find the address and credentials for access on [runway](https://runway.ninegcp.ch). You can use these to configure your application as normal. ### Backup and Maintenance Window Backups are stored in the same region the cluster is in. For example, if the cluster is in Zürich, the backups stay in Zürich. The backups are also spread across zones (i.e. a, b and c) across europe-west6 (Zürich). Your Cloud SQL has the following backup schedule: - Backup window starts at 00:00 UTC every day. - Maintenance window starts at 02:00 UTC every monday. - Restores are currently available by contacting - A restore can be provided either by directly restoring your backup over the current version, or by providing the data under a new db name, allowing you to manually restore necessary tables/rows. - automatic created backups will be stored for 7 days ### Access your CloudSQL instance As all Cloud SQL instances will have only private IPs assigned, you can not connect directly from your local computer to an instance out of the box (only access from the nine Managed GKE cluster is possible). There are a few ways to access your instances externally though: - Spawn a webapplication in your nine Managed GKE cluster to manage your databases (e.g. phpmyadmin, pgadmin, etc). - We can create a VPN connection from your location to the google cloud VPC. This allows for direct access to your Cloud SQL instance from your local computer. Please be aware that you will need VPN hardware available. - Start a proxy pod in your nine Managed GKE cluster and use it with `kubectl port-forward` to access your Cloud SQL instance. - We can add a public IPv4 to your Cloud SQL instance and also restrict access from certain networks if required. Please [contact us](/docs/general/contact) if you want this setup on your instance. #### Proxy pod example With kubectl's local port forwarding feature (which encrypts traffic by default) and a reverse proxy/bouncer deployed in your MGKE cluster you can access your Cloud SQL instance. Here is a sample of a Kubernetes manifest that contains a deployment and a configuration map resource for the reverse proxy which connects to 2 Cloud SQL mysql DBMS. This solution is based on [Gobetween](http://gobetween.io/), but other TCP reverse proxies can also be used. Please replace the values in the "discovery" section of the configuration. TCP ports within the bind section are just a mere suggestion, just make sure to use port numbers higher than 1024. Container ports should be the same. The private IP addresses of your DB instances can be found on [Runway](https://runway.ninegcp.ch/). If you are running a Cloud SQL postgresql instance, then you need to edit the target ports in the "discovery" section. - Create a file _manifest.yaml_ with the following content ```yaml title="manifest.yaml" apiVersion: v1 kind: ConfigMap metadata: name: sql-connect-cm data: config.toml: |- [servers.db-prod] bind = "0.0.0.0:9991" protocol = "tcp" [servers.db-prod.discovery] kind = "static" static_list = [ ":3306", ] [servers.db-staging] bind = "0.0.0.0:9992" protocol = "tcp" [servers.db-staging.discovery] kind = "static" static_list = [ ":3306", ] --- apiVersion: apps/v1 kind: Deployment metadata: name: sql-connect labels: app: sql-connect spec: replicas: 1 selector: matchLabels: app: sql-connect template: metadata: labels: app: sql-connect spec: containers: - name: sql-connect ports: - containerPort: 9991 - containerPort: 9992 volumeMounts: - name: config-volume mountPath: /etc/gobetween/conf image: yyyar/gobetween volumes: - name: config-volume configMap: name: sql-connect-cm items: - key: config.toml path: gobetween.toml --- apiVersion: v1 kind: Service metadata: name: access-db-prod spec: selector: app: sql-connect ports: - protocol: TCP port: 3306 targetPort: 9991 --- apiVersion: v1 kind: Service metadata: name: access-db-staging spec: selector: app: sql-connect ports: - protocol: TCP port: 3306 targetPort: 9992 ``` - Apply the manifest ```bash kubectl create ns kubectl create -f manifest.yaml -n ``` - Create a local port forward ```bash # prod DB: kubectl port-forward -n svc/access-db-prod 3306 & # staging DB: kubectl port-forward -n svc/access-db-staging 3307:3306 & ``` Remember, the port-forwarding is not persistent. It must be recreated if the machine is rebooted. - Access prod DBMS ```bash mysql -u --host 127.0.0.1 -p ``` - Access staging DBMS ```bash mysql -u --port 3307 --host 127.0.0.1 -p ``` ## Documentation and Links - [Cloud SQL](https://cloud.google.com/sql) - [Cloud SQL Documentation](https://cloud.google.com/sql/docs/) --- ## Cluster Monitoring Cluster Monitoring is a component of nine Managed GKE that allows you to quickly see the current state of your cluster and deployments. ## Details For customers who need to visualise the state of their cluster, nine provides a set of dashboards, built with Grafana, which allow this. You are also free to create your own dashboards, manage them in folders to create different hierarchies and add alerts to them. ## Availability Cluster Monitoring is available as standard with nine Managed GKE. ## Usage Nine provides a standardised set of dashboards which you may use to view your cluster. These include views showing: - Overall cluster health (resource usage, cluster scaling state, etc...) - Current state and data of the cluster ingress controller - cloudSQL status (if cloudSQL is purchased) Login details to your grafana instance are provided on [runway](https://runway.ninegcp.ch). ### Custom Dashboards and Alerts To create your own dashboards, have a look at the [official Grafana Documentation](https://grafana.com/docs/guides/getting_started/). Please note that Nine Managed Grafana comes with certain restrictions: - Currently, it is not possible to add your own datasources. - Editing the pre-deployed dashboards is not allowed, however, you can copy the JSON-definition of panels and use it in your own dashboard. - It is not possible to configure alerts via email - Alerts can only be defined on Dashboards & Panels that you have created ## Migration Guide for AngularJS-Based Panels With the upcoming [deprecation of AngularJS support](https://grafana.com/docs/grafana-cloud/whats-new/2024-03-04-angularjs-plugin-warnings-in-dashboards/) in Grafana, it is essential to migrate all AngularJS-based panels to the supported frameworks to avoid future disruptions. Grafana 10.4 offers a Migration Wizard that makes this process straightforward. In your dashboard, AngularJS-based panels are flagged with an exclamation mark (!), helping you quickly identify the panels that require migration. Simply open each identified panel in edit mode and follow the steps in the Migration Wizard to complete the conversion process. ### Tips and Common Issues During Migration #### Migration from `Graph (old)` to `Time Series` Panel _Problem_: The graph is not visible, but some data can be seen by hovering mouse pointer over the render area. This means that while the data is present, it is not being properly displayed in the graph visualization. _Solution_: This issue is often caused by incorrect or invalid threshold configurations. To verify if this is the root cause, access the `Dashboard Configuration` (ensure you have `Edit` mode enabled in the `Dashboard settings`), select the `JSON Model` tab, and locate the relevant panel by searching for its name in the JSON data structure source code. Check if each `color` setting (under `thresholds`) has a `value` assigned (note that `"value": "nil"` is a valid entry). If any threshold configuration is missing a `value` definition, for example, if the configuration for `red` is invalid, simply remove the `red` object from `steps` to resolve the issue: ```json ... { "fieldConfig": { "defaults": { "custom": {...}, "thresholds": { "mode": "absolute", "steps": [ { "color": "transparent", "value": null }, { "color": "red" } ] }, "unit": "none" }, "overrides": [...] }, "gridPos": {...}, "options": {...}, "pluginVersion": "10.4.7", "targets": [...], "title": "Average CPU Usage", "type": "timeseries" }, ... ``` Once you remove this invalid step, the graph should render correctly. Afterward, you can redefine the threshold settings, such as specifying the `red` configuration again, but ensure to use an updated format to prevent the initial rendering problem from recurring. #### Migration from `Table (old)` to `Table` Panel _Problem_: After migrating, some columns may be missing, or there could be changes to data types, formatting, or threshold configurations. _Solution_: These issues often stem from `Overrides` configurations. Follow these steps, which may resolve the issue: - Change Field Overrides. In the panel properties, tab `Overrides`, update the `Fields with name` setting to a selectable option from the dropdown list (e.g., `Time`). If it changes to something like `Time (not found)`, use the dropdown again and select an appropriate field (for example `Field`). - Edit `Display Name` and `Units`. Navigate to the relevant text box for `Change the field or series name` and manually input the correct name. Similarly, in the `Unit` text box, experiment with available types — for instance, try selecting `Misc > String`. - Correct additional columns. Repeat the process for the remaining columns as needed. For instance, for the second column, set the `Fields with name` option to `Last *` and make sure you select the correct name and unit. This should help resolve formatting issues and ensure the threshold display renders colors correctly. ## Documentation and Links - [Grafana Source](https://github.com/grafana/grafana) - [Grafana Documentation](https://grafana.com/docs/) - [Runway](https://runway.ninegcp.ch) --- ## Container Registry Container registry is a service of Nine Managed GKE that allows you to securely store your container images on Google infrastructure. ## Details Customers using Nine Managed GKE will need to store their container images somewhere. Nine provides the integrated service Artifact Registry for this, giving you the peace of mind that your images and deployment pipelines are end to end secured. ## Availability Container registry is available as standard with Nine Managed GKE. We can enable some optional features: - [Cleanup policies](https://cloud.google.com/artifact-registry/docs/repositories/cleanup-policy#create) - [Automatic vulnerability scanning](https://cloud.google.com/artifact-analysis/pricing) Please [contact us](/docs/general/contact) to configure these optional features. ## Usage You can find the address and credentials to access your container registry on [runway](https://runway.ninegcp.ch). To deploy an image simply use its name and tag in your deployment configurations `europe-west6-docker.pkg.dev/{GCP_PROJECT_ID}/gke/{image_name}:{your_image_tag}`, your cluster will be configured to have access to this by default. ## Documentation and Links - [Artifact Registry](https://cloud.google.com/artifact-registry/) - [Artifact Registry documentation](https://cloud.google.com/artifact-registry/docs) --- ## Filestore Filestore is a GCP service that allows you to run high-performance, fully managed file storage. ## Details For customer who need to persist data or have stateful applications nine offers file storage with this service. ## Availability Filestore is charged separately from the nine managed GKE base platform. To order Filestore please contact info@nine.ch. ## Request Content We will need following information when creating your instance: Required: - Instance name - Unique. This will be your instance ID aswell. - Tier - Standard or High availability. - Size - In TB. (1TB min. / 2.5TB min for high availability) - Fileshare name - Name of the fileshare. Optional: - Description - Default: empty ## Usage To provision file storage, after purchase, you have to use your storage class. You can find it on [runway](https://runway.ninegcp.ch/) und your Filestore instance Please be aware that there are different access modes for the PVC: - ReadWriteOnce – the volume can be mounted as read-write by a single node - ReadOnlyMany – the volume can be mounted read-only by many nodes - ReadWriteMany – the volume can be mounted as read-write by many nodes Example: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-disk spec: storageClassName: filestore-instance1 accessModes: - ReadWriteMany resources: requests: storage: 10Gi ``` Filestore is provided as a pool of space which you are able to divide up into logical units via kubernetes PVC objects. We do not enforce the size of a NFS Persistent Volume Claim by default. This means that every NFS Persistent Volume Claim can use all the available space on Filestore. This makes it easier to expand the storage of Filestore if ever needed. Please inform us if you need hard quotas set on NFS Persistent Volume Claims. Please see the official documentation, linked below, for more information on the concepts of storage in Kubernetes. ## Backup There is no automated backup provided by Google when using Filestore. The creation and deletion of backups needs to be done manually. The easiest solution for now, is to create a Kubernetes cronjob resource which executes the gcloud commands listed in the [official Filestore documentation](https://cloud.google.com/filestore/docs/backup-restore). Please contact for help to create a Google service account with the corresponding permissions. To restore a created backup, please contact . We will then restore the content of the backup into a new Filestore instance, which can be mounted in a Kubernetes pod to access individual files. ## Documentation and Links - [Filestore](https://cloud.google.com/filestore) - [Filestore Docs](https://cloud.google.com/filestore/docs) - [Filestore Backup](https://cloud.google.com/filestore/docs/backups) - [k8s Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) - [GCP storage classes](https://cloud.google.com/storage/docs/storage-classes) --- ## Google Cloud Storage Google Cloud Storage(GCS) is the object storage service of GCP. ## Details For customers who need reliable and secure object storage. ## Availability GCS is charged separately from the Nine managed GKE base platform. To order GCS buckets please [contact our support](/docs/general/contact). ## Usage For a good introduction on how to use GCS, we recommend to read through the official [introduction](https://cloud.google.com/storage/docs/introduction). ### Object versioning To support the retrieval of objects that are deleted or replaced, Cloud Storage offers the Object Versioning feature. By default, Nine does not enable versioning, since high frequent changes of objects will increase the cost of the bucket. If you would like to enable versioning for your bucket, please [contact us](/docs/general/contact). More information can be found here: [versioning](https://cloud.google.com/storage/docs/object-versioning) ### Object Lifecycle Management To support common use cases like setting a Time to Live (TTL) for objects, retaining noncurrent versions of objects, or "downgrading" storage classes of objects to help manage costs, Cloud Storage offers the Object Lifecycle Management feature. By default, Nine does not enable any lifecycle rules. If you would like to enable versioning for your bucket, please [contact us](/docs/general/contact). More information can be found here: [lifecycle](https://cloud.google.com/storage/docs/lifecycle) ## Documentation and Links - [introduction](https://cloud.google.com/storage/docs/introduction) - [versioning](https://cloud.google.com/storage/docs/object-versioning) - [lifecycle](https://cloud.google.com/storage/docs/lifecycle) --- ## Helm and Chartmuseum Helm and Chartmuseum are a set of services that provide the industry standard for packaging Kubernetes deployments. ## Details For customer who need to package and deploy their applications the combination of Helm and Chartmuseum provides a way to easily manage this. By using helm charts you can define, install and upgrade applications running on Kubernetes. Helm is the industry standard package manager, supported by the CNCF. There are many existing helm packages available to help install applications in your cluster. Chart museum is an open source helm chart repository, where you can store your helm charts, for easy deployment from your pipelines or IaC systems. ## Availability Helm and Chartmuseum is available as standard with nine Managed GKE. ## Usage You can use helm locally, or in your pipelines to deploy applications. For an introduction on how to use helm please see the application documentation linked below. ### Chart Museum Because you will need to store your charts somewhere convenient for deployment, nine provides you with an instance of chart museum, an open source helm chart repository server. You can find the address and credentials to access Chartmuseum on [runway](https://runway.ninegcp.ch). For more information about chart museum please see the documentation below. ## Documentation and Links - [Helm Quickstart](https://helm.sh/docs/using_helm/#quickstart) - [Helm Documentation](https://helm.sh/docs/) - [Official Helm Charts](https://github.com/helm/charts) - [Chart Museum](https://chartmuseum.com/) --- ## Google Kubernetes Engine (GKE) nine managed Kubernetes is a platform, based on Google's Kubernetes Engine, with a Swiss location and additional services that let you focus on your application development. ## Details Running containers in production isn't easy, it's not enough just to have a Kubernetes cluster running when you need to ensure reliability and resilience. Covering this complexity on behalf of the Customer is at the heart of Nine's managed GKE. Nine's Managed GKE offering helps customers focus on their core business value by allowing them to focus on their applications and not on the services around them. ## Usage ### Getting started You will need both the gcloud SDK and kubectl to start using your kubernetes cluster #### GCloud SDK - Download the [Google Cloud SDK CLI tool](https://cloud.google.com/sdk/) - Install the SDK by [following the documentation](https://cloud.google.com/sdk/docs/) - During the `init` step please log in using the details provided in your sign-up email #### Kubectl - After installing and initialising Google's Cloud SDK tools you should install kubectl for direct control of your cluster - You can do this directly with the gcloud tool by running the following command ```bash gcloud components install kubectl ``` Please see the [documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl/#download-as-part-of-the-google-cloud-sdk) for more information about installing kubectl #### Cluster login After you have completed the gcloud installation you can use it to login to your Kubernetes cluster. Follow these steps to do so: ```bash # Find your project ID $ gcloud projects list PROJECT_ID NAME PROJECT_NUMBER nine-example-478153 nine-example-478153 667903848739 # Switch to that project $ gcloud config set project nine-example-478153 # Find your cluster $ gcloud container clusters list NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS example-cluster europe-west6 1.12.8-gke.10 32.63.127.187 n1-standard-1 1.12.8-gke.10 6 RUNNING # Login to your cluster $ gcloud container clusters get-credentials example-cluster --region=europe-west6 # Use kubectl to interact with your cluster $ kubectl cluster-info Kubernetes master is running at https://32.63.127.187 ``` ### Authentication information and secrets Information regarding credentials and endpoints for your cluster can be found on [runway](https://runway.ninegcp.ch). Alternatively the same information can be found in a secure GCP bucket. Your bucket URL has the following format: `gs://credentials-`. You can find your project number on [runway](https://runway.ninegcp.ch) or by using the gcloud utility. ```bash # To get your project number gcloud projects describe $(gcloud config get-value project) --format="value(projectNumber)" # To see all secrets in your bucket gsutil ls -r gs://credentials- # To view the contents of a secret gsutil cat gs://credentials-//info.json # Full example including parsing json output (requires jq - https://stedolan.github.io/jq/) gsutil cat gs://credentials-/cloudsql/my-cluster-288835/info.json | jq . { "data": { "credentials": { "address": "10.224.129.7", "database_version": "POSTGRES_9_6", "password": "1234", "username": "admin" } }, "meta": { "description": "Instance: my-cluster-288835", "name": "Cloud SQL PostgreSQL", "support_url": "/a/w0dWXF-xIFc" } } ``` ### Node Pools Your cluster is configured with groups of machines, called node pools. A node pool consists of 1-n machines of the same GCP type. The default node pool that nine creates will split the 3 nodes of your cluster between the 3 availability zones of GCP's swiss infrastructure, to ensure your setup is highly available. When upgrading or changing your cluster configuration it is possible to request node pools changes in three different ways: #### Expand the existing node pool It is possible to simply add more machines of the same type to your existing node pool. #### Creating a new node pool It is possible to create a completely new node pool for your cluster. When requesting this your new node pool will be set up, and then your existing nodes will be drained with the existing deployments moved to the new pool. #### Create an additional node pool If you require additional compute power, that is not the same machine type as the existing nodes, it is possible to add another node pool to your cluster. When adding an additional node pool it is advised to have a minimum of 3 nodes so that they can be spanned across all possible availability zones. ### SLA Nine offer two SLA options for your nine Managed GKE cluster, sold as an additional service. You can find more details of the SLA terms and conditions on our [website](https://nine.ch) ## Further Information For further information or sales please contact info@nine.ch For support please contact --- ## Ingress The ingress system of kubernetes is specifically designed to route external HTTP and HTTPS traffic into the cluster. It is composed of the ingress resource itself and an ingress controller which implements the needed logic. We deploy the HAProxy ingress controller by default in every nine Managed GKE cluster. You can control various features by adding annotations to your ingress object. ## Details The HAProxy ingress controller allows to route external HTTP/HTTPS traffic into the cluster. ## Availability The HAProxy ingress controller is available as standard with nine Managed GKE. :::warning The nginx ingress controller is deprecated and will be removed in a future release. Please migrate your Ingress resources to use the HAProxy ingress controller. Refer to the [HAProxy Ingress Features](#haproxy-ingress-features) section for the available annotations. ::: ## Usage The basic usage and structure of a ingress resource is documented [in the official kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/). To use automatic generated Lets Encrypt certificates for [TLS termination](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) please refer to [automatic TLS certificates](./automatic-tls-certificates). ### DNS Setup #### Wildcard DNS domain We provide an automatic created DNS wildcard "apps" domain for you. It is meant for quick application tests in development. You can use any hostname of that wildcard zone in your ingress resources. DNS is already set up. You will find the application wildcard domain in the Ingress info on [runway](https://runway.ninegcp.ch). #### Ingress DNS We also provide a DNS name which will always point to your HAProxy ingress controller's IP. You can use it to point your own domain hostnames to nine Managed GKE. It can be found at the same place as your wildcard apps domain in the Ingress info on [runway](https://runway.ninegcp.ch). To use it just create a CNAME record in your own domain and point it to our provided ingress DNS. ### IngressClass To make use of our ingress controller, you can set the `ingressClassName` field in your `Ingress` resource to `haproxy`. Alternatively you can also omit the field, since it is set as the default class. The deprecated nginx ingress controller is still available using `ingressClassName: nginx` but will be removed in a future release. ### Access Logs The access logs of your Ingress requests can be viewed in your Grafana Instance in the Loki Explore view. The Ingress logs are available under the label `app_kubernetes_io_name="haproxy-ingress"`. To only get the logs of a specific Ingress instance, you can filter by using the additional label `ingress`. The label is in the schema of `--`. Here's an example query to get all the logs of the Ingress `frontend` with the port `80` in the namespace `shop-prod`: ```bash {app_kubernetes_io_name="haproxy-ingress", ingress="shop-prod-frontend-80"} ``` Additionally the Ingress logs can be filtered by these labels: - `method` the HTTP method of a request - `status` the HTTP status code of the request For more information on the usage of Loki, refer to the [specific support article](./centralized-logging-with-loki). ### HAProxy Ingress Features The HAProxy ingress controller provides many features like rate limiting, IP whitelisting, temporary or permanent redirects, etc. All of the configuration keys which can be used to control those features can be found [in the official HAProxy ingress controller documentation](https://haproxy-ingress.github.io/docs/configuration/keys/). Documentation for the most used features can be found below. #### Basic authentication You can add basic authentication to your ingress resource by providing the credentials in a kubernetes secret. Here are some instructional steps: 1. set some env variables for easier processing ```bash USERNAME= SECRET_NAMESPACE= INGRESS_NAMESPACE= INGRESS= ``` 1. create the kubernetes secret which contains the credentials for basic auth. It can also be created in a different namespace than your ingress resource is stored. You will need the `mkpasswd` tool installed locally (can be found in the `whois` package in Debian/Ubuntu). ```bash kubectl create secret generic basic-auth-secret --namespace=$SECRET_NAMESPACE --from-literal=auth=$USERNAME:$(mkpasswd -m sha-512) ``` 1. add some annotations to your ingress object ```bash kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-realm='Authentication required' ``` #### Rate limiting You have various ways of putting rate limits on your ingresses. You can limit requests per second using the `haproxy-ingress.github.io/limit-rps` annotation or limit concurrent connections using the `haproxy-ingress.github.io/limit-connections` annotation. All available options are documented in [the official HAProxy ingress docs](https://haproxy-ingress.github.io/docs/configuration/keys/#limit). #### Temporary and persistent redirects To enable a redirect to another URL for your ingress you can use the following annotation: ```yaml haproxy-ingress.github.io/redirect-to: ``` The redirect will use the HTTP status code of 302 (temporary) by default. If you want to change the status code, for example to 301 for a permanent redirect, use: ```yaml haproxy-ingress.github.io/redirect-to-code: "301" ``` #### HTTPS redirect If TLS is enabled for the given ingress, the HAProxy ingress controller will automatically redirect to the equivalent HTTPS URL of the ingress. To disable this redirect use: ```yaml haproxy-ingress.github.io/ssl-redirect: "false" ``` #### IP whitelisting You can whitelist the IP addresses which are allowed to connect to your ingress resource. You can specify them in CIDR notation in the following annotation: ```yaml haproxy-ingress.github.io/allowlist-source-range: ``` #### Custom default backend The default backend is responsible for showing a 404 error page if a request arrives on the HAProxy ingress controller for which no ingress rule was specified. You can create your own custom default backend (+ kubernetes service) and refer to it on your ingress object. The default backend only has 2 requirements: - it needs to serve a 404 page/code on the path / - it needs to serve a 200 HTTP code on the path /healthz Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress: ```yaml haproxy-ingress.github.io/default-backend: ``` ### SLI Probe You may find a service at `sli-probe.apps-customer..ninegcp.ch`. This service is responsible for monitoring the provided ingress instance from an outside perspective, helping to detect ingress failures as early as possible. ## Deprecated: Nginx Ingress Controller {/* #nginx-ingress */} :::warning The nginx ingress controller is deprecated and will be removed in a future release. Please migrate your Ingress resources to the HAProxy ingress controller documented above. :::
Show deprecated nginx ingress documentation ### IngressClass To use the deprecated nginx ingress controller, set the `ingressClassName` field in your `Ingress` resource to `nginx`. ### Access Logs The nginx ingress logs are available under the Loki label `app_kubernetes_io_name="ingress-nginx"`. Example query: ```bash {app_kubernetes_io_name="ingress-nginx", ingress="shop-prod-frontend-80"} ``` ### Nginx Ingress Features All available annotations can be found [in the official nginx ingress controller documentation](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/). #### Basic authentication ```bash kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-type=basic kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-realm='Authentication required' ``` #### Rate limiting All available options are documented in [the official nginx ingress docs](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rate-limiting). #### Temporary and persistent redirects ```yaml # Temporary redirect (HTTP 302) nginx.ingress.kubernetes.io/temporal-redirect: # Permanent redirect (HTTP 301) nginx.ingress.kubernetes.io/permanent-redirect: ``` #### HTTPS redirect ```yaml nginx.ingress.kubernetes.io/ssl-redirect: "false" ``` #### IP whitelisting ```yaml nginx.ingress.kubernetes.io/whitelist-source-range: ``` #### Caching ```yaml nginx.ingress.kubernetes.io/proxy-buffering: "on" nginx.ingress.kubernetes.io/configuration-snippet: | proxy_cache static-cache; proxy_cache_valid 10m; proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504; proxy_cache_bypass $http_x_purge; add_header X-Cache-Status $upstream_cache_status; ``` #### Custom default backend ```yaml nginx.ingress.kubernetes.io/default-backend: ``` #### Custom error pages ```yaml nginx.ingress.kubernetes.io/custom-http-errors: # for example: "404,415,503" ``` More information can be found [in the official documentation](https://kubernetes.github.io/ingress-nginx/user-guide/custom-errors/).
## Documentation and Links - [the official kubernetes ingress documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/) - [all available configuration keys of the HAProxy ingress controller](https://haproxy-ingress.github.io/docs/configuration/keys/) - [all available annotations of the nginx ingress controller (deprecated)](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/) --- ## Kubernetes Dashboard The Kubernetes dashboard is a service for nine Managed GKE that provides a web interface for your cluster. ## Details For anyone who need to view the status of their cluster without using the command line the Kubernetes dashboard provides a secure web accessible interface. ## Availability The Kubernetes dashboard is available as standard with nine Managed GKE. ## Usage You can find the address to access the dashboard on [runway](https://runway.ninegcp.ch). With this it will be possible to sign in and view that state of your cluster and namespaces. ### View only permissions To allow you to audit changes to your cluster, and to provide the maximum security, nine disables the Kubernetes dashboard from anything except read permissions in each namespace. Although we strongly suggest that you do not change this setting it is possible to grant the dashboard admin permissions, with namespace granularity, by adding the following annotation to your namespace object ```yaml nine.ch/dashboard-admin: "true" ``` Please note that if this step is performed, EVERY user of the dashboard will gain admin permissions in this namespace. This feature is subject to change in a future version of nine Managed GKE. ## Documentation and Links - [Dashboard Documentation](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) --- ## Log Forwarding Log Forwarding allows you to send all logs of your containers running on GKE to an existing centralized logging solution. ## Details We deploy a Fluent Bit instance that forwards all your pod logs to a predefined address. On this address you need to run a receiver that supports the [Fluentd Forward Protocol](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1). Typically this would be [Fluent Bit](https://fluentbit.io/) for simple use-cases or [Fluentd](https://www.fluentd.org/) for more flexibility. Usually this receiver should run within the GKE cluster. If you have the need to forward the logs to an external host, [contact us](/docs/general/contact) to discuss the options. ## Availability Log Forwarding is available as an additional service with nine Managed GKE. ## Usage Let us know to which address we should forward the logs to. After we complete the installation of the service, the details will be shown on [runway](https://runway.ninegcp.ch). ### Example receiver with Fluentd Here's a basic example for a possible receiver that uses Fluentd and sends the logs to stdout. This configuration is just for debugging purposes to show that logs are coming in properly. In a real world scenario the logs should be forwarded to something like a [Elasticsearch instance](https://docs.fluentd.org/output/elasticsearch). This example creates a ConfigMap, a Service and a Deployment in the namespace `logging`. All the logs that we send will start with the tag `kube.` so you have to create a `match kube.**` to select all the logs. With this configuration the receiver address results in `fluentd.logging.svc:24224`, which is what you would need to tell us so we can forward the logs to that address. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: logging data: fluent.conf: |- @type forward port 24224 bind 0.0.0.0 # here you would usually configure something like elasticsearch @type stdout --- apiVersion: v1 kind: Service metadata: name: fluentd namespace: logging spec: selector: app: fluentd ports: - protocol: TCP port: 24224 targetPort: 24224 --- apiVersion: apps/v1 kind: Deployment metadata: name: fluentd namespace: logging spec: replicas: 1 selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd annotations: # we exclude this pod so we don't create a logging loop fluentbit.io/exclude: "true" spec: volumes: - name: fluentd-config configMap: name: fluentd-config containers: - name: fluentd image: fluent/fluentd:v1.9 ports: - containerPort: 24224 volumeMounts: - name: fluentd-config mountPath: /fluentd/etc ``` ## Documentation and Links - [Fluentd forward input](https://docs.fluentd.org/input/forward) - [Fluent Bit forward input](https://docs.fluentbit.io/manual/input/forward#getting-started) --- ## Memorystore Memorystore is a GCP service that allows customers to run a Redis instance. ## Details For customers who would like to use Redis within the GCP. Redis is an in-memory data structure project implementing a distributed, in-memory key-value database with optional durability. Redis supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams, and spatial indexes. ## Availability Memorystore is NOT available as standard with nine Managed GKE. However, you can create a ticket for the activation. ## Ticket Content We will need following information when creating your instance: Required: - Instance name - This will be your instance ID aswell. Optional: - Display name - Default: empty - Memory Size in GB - Default: 1 GB - Redis version - Default: 4.0 - Tier - Default: Standard ## Usage Every Redis instance receives a private IP. This means you can NOT connect to it from your local computer. Only access form within the nine Managed GKE cluster is possible. The IP you can find on [runway](https://runway.ninegcp.ch/). ### Requirements No special requirements needed. ### Login Memorystore does not require any password by default. ## Documentation and Links - [Memorystore](https://cloud.google.com/memorystore/) - [Memorystore Docs](https://cloud.google.com/memorystore/docs/redis/) --- ## Sealed Secrets _Sealed Secrets_ encrypts Kubernetes _Secrets_ so you can store them in git without any worries. ## Details Usually the content of Kubernetes _Secret_ definitions is unencrypted which means it is not recommended to store them alongside other Kubernetes definitions in version control or anywhere that is not a secured environment. This adds manual and error-prone steps to your application deployment. As a solution to this, we are running [a controller](https://github.com/bitnami-labs/sealed-secrets) that will take care of decrypting your _Sealed Secrets_ and turning them into normal _Secret_ objects. ## Availability _Sealed Secrets_ are available as standard with nine Managed GKE. ## Scopes The _Scope_ is nothing but the context of a sealed secret within a Kubernetes cluster. The _Scope_ of a _Sealed Secret_ relates to where and how the _Sealed Secret_ can be decrypted and used within your cluster. These are the possible _Scopes_: - `strict` (default): the secret must be sealed with exactly the same name and namespace. These attributes become part of the encrypted data and thus changing name and/or namespace would lead to a decryption error. - `namespace-wide`: you can freely rename the _Sealed Secret_ within a given namespace. - `cluster-wide`: the secret can be unsealed in any namespace and can be given any name. By default, `strict` _Scope_ is selected unless you pass the `--scope` flag to kubeseal CLI with a different value. It's also possible to request a _Scope_ via `annotations` in the input secret you pass to kubeseal. Please refer to [Scopes documentation](https://github.com/bitnami-labs/sealed-secrets?tab=readme-ov-file#scopes) for more details. ## Usage ### Strict Scope (default) The easiest way to create a strict scoped _Sealed Secret_ is to use our generator on [runway](https://runway.ninegcp.ch). 1. Generate a new _Sealed Secret_ by filling out the form in the _Secrets Generator_ Tab. 1. Download the _Sealed Secret_. 1. Apply it via `kubectl`. ```bash $ kubectl apply -f ~/Downloads/cloudsql-prod.yaml sealedsecret.bitnami.com/cloudsql-prod created ``` 1. Read back the _Secret_ resource that the controller created for us. ```bash $ kubectl get secret cloudsql-prod --template={{.data.password}} | base64 -d s#g{eJJ#O)p~VCHVNt26*WGD3 ``` To delete the _Secret_ again, you can just delete the _Sealed Secret_ and the controller will also remove the _Secret_ object. ```bash $ kubectl delete sealedsecret cloudsql-prod sealedsecret.bitnami.com "cloudsql-prod" deleted ``` Note that in a production scenario we do not recommend you to apply the _Sealed Secret_ locally with `kubectl`, but instead store it in your configuration repository and let [Argo CD](./argo-cd) take care of creating it. ### Cluster-wide Scope The procedure is quite similar to the case of [strict scope](#strict-scope-default). However, our [runway](https://runway.ninegcp.ch) generator tool will not work here. In order to create a cluster-wide _Sealed Secret_, you need to install the CLI-utility `kubeseal` which is part of the [sealed-secrets](https://github.com/bitnami-labs/sealed-secrets#installation) project. After you installed `kubeseal` for your OS you can start to encrypt secrets locally. 1. Define your normal unencrypted secret in a local file named `secret.yaml`. ```yaml title="secret.yaml" apiVersion: v1 kind: Secret metadata: name: example type: Opaque stringData: password: verysecure ``` 1. Use kubeseal to generate an encrypted _SealedSecret_ resource. Note that you need to pass `--scope cluster-wide` to kubeseal CLI (or use `annotations`). Please refer to the [Cluster login documentation](./#cluster-login) to learn how to get your ``. ```bash $ kubeseal --cert https://sealed-secrets.apps..ninegcp.ch/v1/cert.pem --scope cluster-wide < secret.yaml > sealed-secret.json ``` 1. Apply it via `kubectl`. ```bash $ kubectl apply -f sealed-secret.json sealedsecret.bitnami.com/example created ``` 1. Read back the _Secret_ resource that the controller created for us. ```bash $ kubectl get secret example -o jsonpath='{.data.password}' | base64 -d verysecure ``` To delete the _Secret_ again, you can just delete the _SealedSecret_ and the controller will also remove the _Secret_ object. ```bash $ kubectl delete sealedsecret example sealedsecret.bitnami.com "example" deleted ``` ## Documentation and Links - [Sealed Secrets Documentation](https://github.com/bitnami-labs/sealed-secrets#overview) --- ## Storage Storage is a service of nine Managed GKE that allows you to persist data in your cluster. ## Details For customer who need to persist data or have stateful applications, nine offers both block and file storage, with both storage types available on rotational disks and ssds. ## Availability Storage is charged separately from the nine Managed GKE base platform. Block storage can be provisioned direct by the customer, to order file storage please contact info@nine.ch. ## Usage Nine offers different types of storage that can be used with Kubernetes. For information about backups and restore of your storage please see the appropriate documentation, listed below. ### Block storage Block storage is available to be provisioned directly by the customer, in a pay-as-you-go model. Storage provisioned will automatically be added to your bill, and you will be charged by usage as per the gcp billing model. To provision block storage you are able to use the following storage classes: ``` standard (default) standard-regional standard-late-binding ssd ssd-regional balanced balanced-regional ``` The default block storage class is `standard` (rotational disks in a single zone). This means that as default your data is guaranteed to reside in Switzerland, but will not have high-availability by being replicated across multiple zones in the region. To ensure that your data is replicated across two zones use a storage class with a `-regional` suffix. The Storage-Classes reflect Google's [Disk Types](https://cloud.google.com/compute/docs/disks/#introduction). All storage classes except `standard` support [Volume Expansion](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/volume-expansion) and are ["Late Binding"](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode). Late Binding is important for zonal volumes, as these can only be used within one zone. We thus recommend to not use the default `standard` storage class and use one of the aforementioned alternatives. Here is an example on how to create a Block storage volume using a Persistent Volume Claim: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-disk spec: storageClassName: standard-late-binding accessModes: - ReadWriteOnce resources: requests: storage: 10Gi ``` #### Block Storage Backup and Restore Block storage is backed up daily via snapshots. These snapshots can be restored on request. See the [backup](./backup-and-restore) documentation for more information. ### File Storage (NFS) (deprecated) NFS file storage is an add-on product, which must be requested from info@nine.ch. You will need to request either ssd or rotational disks when ordering file storage. Each have different performance characteristics. Nine uses regional disks when provisioning your NFS storage cluster, to ensure your setup is highly available. Due to this the minimum possible NFS cluster size is limited to 200GB. To provision file storage, after purchase, you are able to use the following storage class: ``` nfs ``` Please be aware that the actual `accessMode` for NFS needs to be `ReadWriteMany`: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-disk spec: storageClassName: nfs accessModes: - ReadWriteMany resources: requests: storage: 10Gi ``` File storage is provided as a pool of space which you are able to divide up into logical units via kubernetes PVC objects. We do not enforce the size of a NFS Persistent Volume Claim by default. This means that every NFS Persistent Volume Claim can use all the available space on the NFS server. This makes it easier to expand the storage of the NFS server if ever needed. Please inform us if you need hard quotas set on NFS Persistent Volume Claims. Please see the official documentation, linked below, for more information on the concepts of storage in Kubernetes. #### Migration to Filestore To migrate from the deprecated NFS solution, nine provides a tool called `relokator`. The [relokator tool](https://github.com/ninech/relokator) is available on GitHub. The usage is described in the README of the repository. Should you encounter any issues migrating your data, please [contact us](/docs/general/contact). #### Performance Please note that write throughput is limited by the vCPU and disk size of your file storage solution. Please see [Google's documentation](https://cloud.google.com/compute/docs/disks/performance#size_price_performance) on this for more information. Additionally GCP's [Network Egress cap](https://cloud.google.com/compute/docs/disks/performance?hl=en#egress_performance_cap) may limit performance. Therefore performance sensitive applications will need to factor in vCPU's and disk size when calculating the appropriate configuration for your storage nodes. nine strongly suggests using ssd's where possible for your file storage. For support and consulting on this topic please contact info@nine.ch ##### Performance limits of rotational persistent disks Machine types with less than 4 vCPU cores will limit your write speed to standard persistent disks. - Read throughput: Up to 240 MB/s at a 2 TB disk size. - Write throughput: Up to 240 MB/s at a 2 TB disk size. - Read IOPS: Up to 3,000 IOPS at a 4 TB disk size. - Write IOPS: Up to 15,000 IOPS at a 10 TB disk size. ###### FIO benchmark of rotational NFS storage - data amount 4MB block size: 50GB - data amount 16KB block size: 20GB - persistent disk size: 200GB - client is of type n1-standard-1 | Server | write speed sequential (4MB) | write speed random (4MB) | write IOPS sequential (16KB) | write IOPS random (16KB) | read speed sequential (4MB) | read speed random (4MB) | read IOPS sequential (16KB) | read IOPS random (16kb) | | ------------- | ---------------------------- | ------------------------ | ---------------------------- | ------------------------ | --------------------------- | ----------------------- | --------------------------- | ----------------------- | | n1-standard-2 | 63 MB/s | 65 MB/s | 1777 IOPS | 715 IOPS | 20.5 MB/s | 19.5 MB/s | 1604 IOPS | 258 IOPS | ##### Performance limits of SSDs The performance of SSD's relies heavily depends on the number of vCPUs and the size of the persistent disk in the machine [please see the documentation](https://cloud.google.com/compute/docs/disks/performance#ssd-pd-performance) ###### FIO benchmark of SSD NFS storage - data amount 4MB block size: 50GB - data amount 16KB block size: 20GB - persistent disk size: 200GB - client is of type n1-standard-1 | Server | write speed sequential (4MB) | write speed random (4MB) | write IOPS sequential (16KB) | write IOPS random (16KB) | read speed sequential (4MB) | read speed random (4MB) | read IOPS sequential (16KB) | read IOPS random (16kb) | | ------------- | ---------------------------- | ------------------------ | ---------------------------- | ------------------------ | --------------------------- | ----------------------- | --------------------------- | ----------------------- | | n1-standard-2 | 64.2 MB/s | 63.8 MB/s | 1518 IOPS | 1629 IOPS | 103.2 MB/s | 96.3 MB/s | 4965 IOPS | 5080 IOPS | #### File Storage Backup and Restore Your File Storage (NFS) data will be backed up once a day. It is possible to request access to these backups for disaster recovery and file restoration actions. See the [backup](./backup-and-restore) documentation for more information. ## Documentation and Links - [Backup and Restore](./backup-and-restore) - [k8s Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) - [GCP storage classes](https://cloud.google.com/storage/docs/storage-classes) --- ## Users & Permissions For maximum security nine controls user access and authentication to your cluster. If you wish to add new user you must request this from . ## User Roles Nine Managed GKE offers different roles for user accounts, which need to be specified when ordering a new user. These roles are mostly related to GKE, although they also influence what a user can see on Runway, which is explained [later](#runway). The following roles exist: | role name | permissions | | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | viewer | Can view all contents of all namespaces, except secrets | | user | Can create and delete owned namespaces, but does not have access to any other namespace. Can view secrets in owned namespaces | | admin | Has full access to all namespaces | Further, one may combine the `viewer` and `user` role, which would result in a user that can view all namespaces and their contents, and can create own namespaces, where the user has full access to. ### Granular access to namespaces By default, the user that creates a namespace will also be Admin in it. Further, all users with the `admin` role will have full access, and users with the `viewer` role read-only access to that namespace too. It is also possible to give an otherwise unprivileged user (with `viewer` or `user` permissions) access to a namespace. This is possible by binding the user to a specific Kubernetes role. Every GKE cluster comes with the following predefined cluster roles which can be used in namespaces: - cluster-admin - admin - edit - view The following table lists the differences between those cluster roles when used in namespaced rolebindings: | cluster role name | permissions when used in rolebinding | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **cluster-admin** | This role gives full control over every resource in the rolebinding's namespace, including the namespace itself. | | **admin** | Allows admin access, intended to be granted within a namespace. It allows read/write access to most resources in a namespace, including the ability to create roles and rolebindings within the namespace. It does not allow write access to resource quota or to the namespace itself | | **edit** | Allows read/write access to most objects in a namespace. It does not allow viewing or modifying roles or rolebindings. | | **view** | Allows read-only access to see most objects in a namespace. It does not allow viewing roles or rolebindings. It does not allow viewing secrets, since those are escalating. | To assign one of the predefined cluster roles to users in namespaces, the `kubectl` application can be used. Here are some examples: ```bash # creating a namespace admin with full privileges kubectl create rolebinding custom-admins-full-privileges --clusterrole=cluster-admin --user=@ninegcp.ch [--user ...] --namespace= # creating a normal namespace admin kubectl create rolebinding custom-admins --clusterrole=admin --user=@ninegcp.ch [--user ...] --namespace= # granting normal edit permissions kubectl create rolebinding custom-editors --clusterrole=edit --user=@ninegcp.ch [--user ...] --namespace= # granting view only permissions kubectl create rolebinding custom-viewers --clusterrole=view --user=@ninegcp.ch [--user ...] --namespace= ``` To edit the created rolebindings later on the `kubectl edit rolebinding -n ` command can be used. ### Runway Runway lists Service Accounts, their credentials, and connection secrets to CloudSQL instances, for example. Because these service accounts may have higher privileges than a user's account, only users with the `admin` role can view those secrets on Runway. However, if you need an account that should not be admin in a GKE project, but should still be able to download the Service Account credentials or read CloudSQL connection credentials, you can also order a user plus the additional permission `credentials-viewer`. Please keep in mind that this will potentially give users access to higher-privileged service accounts, which is a form of privilege escalation. ## Password reset If you need a new password for your nine Managed GKE account, please contact support at . Please be aware that you have to be registered as a technical contact in our system. --- ## Airlock Microgateway [Airlock Microgateway](https://docs.airlock.com/microgateway/latest/) is a Kubernetes-native web application firewall (WAF) that protects your workloads against common web attacks. On the Nine Kubernetes Engine we run the Airlock Microgateway operator for you and expose one or more Kubernetes Gateways that you attach your applications to using the standard [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/). :::info Airlock Microgateway on NKE is in an early stage. To get it set up on your cluster, [contact our support](mailto:support@nine.ch). This article covers the basics and will be extended over time. ::: ## Availability Airlock Microgateway is available as an optional service for NKE clusters. It is not self-service yet: contact us and we will install the operator and create the Gateways on your cluster. The Airlock Microgateway operator is a cluster-wide component, so there is one operator per NKE cluster. That single operator can back multiple Gateways, so you can request as many Gateways as you need on the same cluster. ## How It Works Once you request Airlock Microgateway for a cluster, we: - Install the Airlock Microgateway operator on your NKE cluster. - Create one or more **Gateways** in the Nine-managed `nine-system` namespace. Each Gateway gets its own load balancer with a public IP address and a stable hostname (for example `production..airlockgateway.nineapis.ch`) that you can point your own domains to. - Set up [Let's Encrypt](https://letsencrypt.org/) `ClusterIssuer` resources so that certificates for your domains are issued and renewed automatically. You then attach your own Gateway API resources (`ListenerSet`, `HTTPRoute`) in your application namespaces to route traffic through the Gateway. ## Why the Gateway API Airlock Microgateway is configured exclusively through the Kubernetes Gateway API. It does not work with the [Ingress](./ingress) resources used elsewhere on NKE. The Gateway API deliberately splits responsibilities: Nine owns the shared infrastructure (the Gateway and its load balancer), while you own the per-app routing and security resources (`ListenerSet`, `HTTPRoute`, and the Airlock policies) in your own namespaces. This split is what lets us run and secure the Gateway for you while you keep full control over how your traffic is routed and protected. ## Usage The following examples assume a Gateway named `airlock-production` was created for you in the `nine-system` namespace, and that your application runs in the `my-app` namespace. ### Add an HTTPS Listener With Automatic TLS Create a `ListenerSet` in your application namespace to add an HTTPS listener to the Gateway. When the `cert-manager.io/cluster-issuer` annotation is present, cert-manager automatically issues and renews the certificate for the given hostname. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: ListenerSet metadata: name: my-listeners namespace: my-app annotations: cert-manager.io/cluster-issuer: airlock-letsencrypt-production spec: parentRef: name: airlock-production namespace: nine-system listeners: - name: https hostname: "myapp.example.com" port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - name: myapp-tls namespace: my-app allowedRoutes: namespaces: from: Same ``` Use the `airlock-letsencrypt-staging` issuer while testing to avoid the Let's Encrypt rate limits, then switch to `airlock-letsencrypt-production` for real certificates. ### Route Traffic to a Backend Attach an `HTTPRoute` to the listener to forward requests to your backend service: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: my-app-route namespace: my-app spec: parentRefs: - group: gateway.networking.k8s.io kind: ListenerSet name: my-listeners namespace: my-app sectionName: https hostnames: - "myapp.example.com" rules: - backendRefs: - name: my-app-service port: 8080 ``` ### Plain HTTP Traffic Each Gateway comes with a built-in HTTP listener on port 80, which is also used to solve Let's Encrypt HTTP01 challenges. For plain-HTTP traffic you can attach an `HTTPRoute` directly to the Gateway, without a `ListenerSet`: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: my-app-http namespace: my-app spec: parentRefs: - name: airlock-production namespace: nine-system sectionName: http hostnames: - "myapp.example.com" rules: - backendRefs: - name: my-app-service port: 8080 ``` You can test reachability before setting up DNS by sending the hostname as a `Host` header: ```bash curl -H "Host: myapp.example.com" http:/// ``` ### Point Your Domain at the Gateway To serve production traffic, create a CNAME record for your domain that points to the Gateway's hostname: ```dns myapp.example.com. CNAME production..airlockgateway.nineapis.ch. ``` We provide the exact hostname for your Gateway when it is set up. ### Customize Error Responses Airlock Microgateway can replace the responses your clients see, for example the error page returned when a request is blocked or when your backend returns an error. First define the content with a `CustomResponse`: ```yaml apiVersion: microgateway.airlock.com/v1alpha1 kind: CustomResponse metadata: name: custom-404 namespace: my-app spec: statusCode: 404 content: - contentType: text/html body: value: "

Page not found

" ``` Then attach a `CustomResponsePolicy` to your `HTTPRoute` to serve that response for matching status codes: ```yaml apiVersion: microgateway.airlock.com/v1alpha1 kind: CustomResponsePolicy metadata: name: my-app-custom-responses namespace: my-app spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: my-app-route policies: local: - responses: - statusCodeCondition: matcher: exact: 404 customResponseRef: name: custom-404 ``` The `CustomResponse`, the `CustomResponsePolicy`, and the `HTTPRoute` they reference must all be in the same namespace. Airlock offers many more policies, such as traffic filtering, header rewriting, and rate limiting. See the [Airlock Microgateway CRD reference](https://docs.airlock.com/microgateway/latest/index/api/crds/index.html) for the full list. ### View Access Logs The Gateway logs every request as structured JSON. Each entry includes a request ID (the `http.request.id` field) that you can use to trace a single request end to end, along with the matched route and whether Airlock allowed or blocked it: ```json { "http": { "request": { "id": "4cddf510-516c-4cda-9066-3809dad0b249" } }, "airlock": { "summary": { "action": "allowed" } } } ``` When [Loki](./loki) is enabled on your cluster, these access logs are collected automatically. In Grafana, use the following LogQL query to see all access logs for a specific Gateway, replacing `` with the name of your Gateway (for example `airlock-production`): ```bash {app="airlock-"} ``` To trace a single request across the logs, extract the request ID field and filter by its value: ```bash {app="airlock-"} | json request_id="http.request.id" | request_id="4cddf510-516c-4cda-9066-3809dad0b249" ``` ## Further Reading - [Airlock Microgateway documentation](https://docs.airlock.com/microgateway/latest/) - [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) --- ## Alertmanager Alertmanager is a component of NKE that allows you to monitor your applications. ## Availability Alertmanager is available as an optional service for NKE and it can be deployed using Cockpit. ## Usage ### Configuring Alertmanager Alertmanager is the component responsible to send out notifications in case of Prometheus alerts. Alertmanager supports various channels for notifications, like Slack, Email, Hipchat, PagerDuty, etc. Please have a look at the [official documentation](https://prometheus.io/docs/alerting/configuration/#configuration-file) for detailed information about the configuration. We also supply [example configurations](#configuring-alertmanager). When an Alertmanager instance is created, it does not have any notification receivers configured by default. You will have to create a full Alertmanager configuration and send it to us. The best way would be to create a secret in your cluster and populate it with the desired config. We will then add the Alertmanager configuration to the instance. Note that this process is only temporary, and you will soon be able to configure the Alertmanager on your own in Cockpit. #### Alertmanager configuration examples **1.** Send all alerts via email ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m route: receiver: "email" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: [] receivers: - name: "email" email_configs: - to: "monitoring-alerts-list@your-domain.ch" send_resolved: true # when using STARTTLS (port 587) this needs to be 'true' require_tls: false from: "Alertmanager@your-domain.ch" smarthost: smtp.your-domain.ch:465 auth_username: "Alertmanager@your-domain.ch" auth_password: "verysecretsecret" headers: { Subject: "[Alert] Prometheus Alert Email" } ``` **2.** Send all critical alerts via slack. All other severities will be sent out via email. Please make sure to add a `severity` label to your alerts. ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m route: # this specifies the default receiver which will be used if no route matches receiver: "email" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: - receiver: "slack" match_re: severity: "[cC]ritical" receivers: - name: "email" email_configs: - to: "monitoring-alerts-list@your-domain.ch" send_resolved: true # when using STARTTLS (port 587) this needs to be 'true' require_tls: false from: "Alertmanager@your-domain.ch" smarthost: smtp.your-domain.ch:465 auth_username: "Alertmanager@your-domain.ch" auth_password: "verysecretsecret" headers: { Subject: "[Alert] Prometheus Alert Email" } - name: "slack" slack_configs: - send_resolved: true api_url: https://hooks.slack.com/services/s8o3m2e0r8a8n2d/8snx2X983 channel: "#alerts" ``` **3.** Send all alerts of the production environment via slack. Drop all other alerts. Please make sure to define the label 'environment' in your alerts. ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m route: # this specifies the default receiver which will be used if no route matches receiver: "devnull" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: - receiver: "slack" match: environment: production receivers: - name: "slack" slack_configs: - send_resolved: true api_url: https://hooks.slack.com/services/s8o3m2e0r8a8n2d/8snx2X983 channel: "#alerts" - name: devnull ``` **4.** Use templates to customize your notifications and send all alerts via slack. Here we define some templates in a file called 'slack.tmpl'. ```yaml title="alertmanager.yaml" global: resolve_timeout: 5m # THIS LINE IS VERY IMPORTANT AS OTHERWISE YOUR TEMPLATES WILL NOT BE LOADED templates: - "/etc/alertmanager/config/*.tmpl" route: receiver: "slack" group_by: ["alertname"] group_wait: 30s group_interval: 5m repeat_interval: 1h routes: [] receivers: - name: "slack" slack_configs: - send_resolved: true api_url: https://hooks.slack.com/services/s8o3m2e0r8a8n2d/8snx2X983 channel: "#alerts" pretext: "{{ .CommonAnnotations.description }}" text: '{{ template "slack.myorg.text" . }}' ``` ```yaml title="slack.tmpl" {{ define "slack.myorg.text" -}} {{ range .Alerts -}} *Alert:* {{ .Labels.alertname }} - `{{ .Labels.severity }}` *Description:* {{ .Annotations.description }} *Details:* {{ range .Labels.SortedPairs -}} • *{{ .Name }}:* `{{ .Value }}` {{ end -}} {{ template "slack.default.text" . }} {{ end -}} {{ end -}} ``` ## Documentation and Links - [Prometheus documentation](https://prometheus.io/docs/) - [prometheus-operator project](https://github.com/coreos/prometheus-operator) ## Video Guide Checkout our video guide series for GKE Application Monitoring. While the videos are done on our GKE product, the concepts are the same. --- ## ArgoCD Argo CD is a service that allows you to continuously deploy applications to NKE clusters by using a gitops workflow. ## Details For customers who need to continuously deploy application code, Argo CD provides: - declarative and version controlled application deployments - automation and traceability via gitOps workflow - support for Helm, kustomize and jsonnet application declarations - a web UI for visualizing kubernetes resources - webhook integration to fully automate deployments on git operations - a command line interface application - audit trails for application events and API calls - parameter overrides of Helm/ksonnet declarations (simplifies development deployments) - a grafana metrics dashboard ## Availability Argo CD is available as an optional service for NKE. It can be deployed using Cockpit and can connect with any number of configured NKE clusters. ## Usage Before starting to use Argo CD it is important to understand how a typical workflow should look in the end. Argo CD supports a continuous deployment by utilizing a gitops workflow. For that to work it recommends to separate application code from application configuration (Helm charts, kustomize files, etc.). The separation should happen by using 2 different git repositories. Although it is technically possible to use 1 git repository, [best practises](https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/) advise strongly against doing so. When using 2 separate git repositories, one possible production deployment workflow with Argo CD could look like: 1. a developer creates a pull request/merge request to get some application code changes merged into the master branch 1. after merging of the changes happened (and all tests passed), a tag will be created by the developer signaling that a new productive version of the application should be build 1. a CI pipeline starts. It executes the following steps: - it builds, tags and pushes a new application container image. - it creates a commit in the configuration git repository, specifying the new image version to be used (for example by changing the content of the values.yaml in a Helm chart) - it pushes the commit 1. (optional) a git webhook signalises Argo CD to check for new commits in the configuration repository 1. Argo CD deploys the new image version of the container Argo CD is not connected to the application source code repository in any way. It only connects to the configuration git repository (read only permissions are sufficient). If there are any problems with the deployed version, a rollback can be initiated by reverting the commit in the configuration git repository. ArgoCD will then deploy the previous version of the image. For a further separation of access it is also possible to not directly commit to the configuration git repository within the pipeline. Instead a pull request/merge request will be created which needs to be approved before the new image version should be deployed. With this it is possible to give developers access to the code repository without granting permissions in the configuration repository. ### Requirements To be able to use Argo CD (for production deployments) with a gitops workflow you will need at least the following: - the URL to your Argo CD installation (see [Login](#web-ui)) - a kubernetes namespace where Argo CD can deploy to (see [Namespace Creation](#namespace-creation)) - a git repository with the configuration of your application (called the _config repo_). This can be: - a [kustomize](https://kustomize.io/) application - a [Helm chart](https://helm.sh/) - a directory of plain yaml manifests - a [jsonnet](https://jsonnet.org/) application - a CI tool/service for: - automatically building container images - doing changes to the configuration repository (optional) We at Nine are preferring Helm charts as we are using them in the company ourselves. An example Helm application configuration which deploys a guestbook application can be found [in the argo project github namespace](https://github.com/argoproj/argocd-example-apps.git) in the _helm-guestbook_ directory. ### Permissions The current authorization concept permits all configured NKE users with one of the following roles with full access to all Argo CD applications and projects in their installation of Argo CD: - admin - user Users with the role "view" are only permitted to see configured Argo CD applications and projects, but are not authorized to change them. ### Login Argo CD provides a web user interface as well as a cli application to interact with it. #### Web UI You can find the URL in [Cockpit](https://cockpit.nine.ch/nke). You can login with your Cockpit account credentials after clicking on _Login via Nine_ in the web UI. #### CLI The CLI application can be downloaded on the _help_ page in the [Argo CD web interface](#web-ui) (you will find a link to the help page in the navigation menu on the left side). To login via the cli application, please follow these steps: 1. execute `argocd login --sso` locally in a terminal on your machine 1. argocd will open a browser page so that you can enter your Nine Cockpit credentials 1. after a successful authentication you can use `argocd` locally on the cli as an authenticated user Argo CD opens a local port (8085 by default) on your machine to be able to authenticate via single sign on. If that port is already in use by another application, please choose a different port by the using the `--sso-port` argument. ### Configuration resources in Argo CD Argo CD introduces 3 kubernetes resources: _Applications_, _Projects_ and _ApplicationSets_. **Applications**: The Application CRD is the Kubernetes resource object representing a deployed application instance in an environment. It is defined by two key pieces of information: - a _source_ reference to the desired state in the configuration Git (repository, revision, path, environment) - a _destination_ reference to the target cluster and namespace. It basically describes which configuration state should be deployed to which namespace in your cluster. **Projects**: The AppProject CRD is the Kubernetes resource object representing a logical grouping of applications. It is defined by the following key pieces of information: - a _sourceRepos_ reference to the configuration repositories that applications within the project can pull manifests from - a _destinations_ reference to clusters and namespaces that applications within the project can deploy into - a roles list of entities with definitions of their access to resources within the project **ApplicationSets**: ApplicationSets allow you to automate the creation of applications and improve multi-cluster support. ApplicationSets can only be created via the CLI. Check the official [documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/application-set/) on how to use ApplicationSets. If you have problems getting ApplicationSets to work, please [contact support](/docs/general/contact). ### App of Apps principle Both CRDs (Applications and Projects) can be used in git configuration repositories directly. Specifying the Argo CD applications/projects itself in git (versus creating them via Web UI or cli application) can be used to make use of the ["App of Apps Principle"](https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#app-of-apps). In a "app of apps" deployment you specify one "root" application in git which points to other ArgoCD application definitions. Those definitions may then either point again to ArgoCD applications or create other Kubernetes resources like Deployments, etc. Please be aware that your ArgoCD installation is running on an external cluster and not on your NKE cluster. When deploying ArgoCD application custom resources you therefore need to use a special target namespace in which your generated ArgoCD applications will be stored. This special namespace is the project of your ArgoCD instance. You can find it in the "Access information" pane of your ArgoCD instance in cockpit. Here is an example of a root ArgoCD application which points to a directory "apps" in a ArgoCD example repository. This "apps" directory contains a helm chart which itself renders ArgoCD Applications. The URL for the destination NKE cluster can be found in the UI of your ArgoCD instance at "Settings" -> "Clusters". ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root spec: destination: name: in-cluster project: default source: path: apps repoURL: https://github.com/ninech/argocd-example-apps targetRevision: HEAD helm: values: | config: argocdNamespace: spec: destination: server: source: repoURL: https://github.com/ninech/argocd-example-apps targetRevision: master applications: - name: guestbook # the namespace sets the installation namespace in the target NKE # cluster namespace: guestbook ``` You can store this definition in a file and apply it with the ArgoCD CLI. Make sure you follow the [CLI login steps](#cli). You will also need to precreate the namespace on your target NKE cluster as [documented](#namespace-creation). After a successful CLI login you can then create the root application. ```bash argocd app create --file= ``` After the root application has been created, it will create a new "guestbook" application which will install the "guestbook" software onto your NKE cluster. ### Deploying your application with Argo CD #### Namespace creation To be able to deploy your application you will need a kubernetes namespace where your application should be deployed to. By default Argo CD does not have the permissions to deploy to any namespace in cluster. You explicitly need to annotate the deployment namespace where your application needs to be deployed with `nine.ch/argo-admin="true"`. The following example creates the namespace `my-application` and annotates it to permit access with Argo CD: ```bash kubectl create namespace my-application kubectl annotate namespace/my-application nine.ch/argo-admin="true" ``` Argo CD now has permissions to deploy into that namespace: ```bash $> kubectl describe rolebinding namespace-admins -n my-application Name: namespace-admins <...> Subjects: Kind Name Namespace ---- ---- --------- ServiceAccount argocd-c3182374-mcj28sd default <...> ``` #### Creating an Argo CD application An Argo CD application basically describes which application configuration should be deployed to which namespace of your cluster. You should first configure the git configuration repositpory via Argo CD. Afterwards you can use it when configuring the Argo CD application. You can configure the application and the repository either via the web UI or by using the CLI application. The following documentation describes how to create the application via the web UI. ##### A note about SSH Although it is possible to connect your config git repository via SSH, we recommend to use HTTPS. If you want to use SSH you need to add the public host key of your git provider to Argo CD as [described in the Argo CD documentation](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#unknown-ssh-hosts). This is not needed when using the HTTPS protocol. Please make sure that you are using a trusted TLS certificate when using HTTPS. ##### Steps Follow these steps to configure a repository via the web UI: 1. Login to the web UI (see [Login](#web-ui)) 1. Click on the _gear icon_ in the menu on the left ("Manage your repositories,projects,settings") 1. Click on _Repositories_ 1. Click on _Connect repo using https_ 1. You can now enter the repository details | Item | Description | Example | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Repository URL | the https URL to your config git repository | [https://gitlab.com/example/my-application-config](https://gitlab.com/example/my-application-config) | | Username | the username to access the repository (please use [access tokens](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#access-token) and not personal credentials) | argocd | | Password | the password to access the repository (please use [access tokens](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#access-token) and not personal credentials | 3macm32449asdnf243rt | | TLS client certificate | an optional TLS client certificate in PEM format which you use for authentication with your git repository | | | TLS client certificate key | an optional TLS client certificate key in PEM format which you use for authentication with your git repository | | | skip server verification | check this box if Argo CD should not verify the TLS certificate of your HTTPS connection | | | Enable LFS support | check this if you used [git large file support](https://github.com/git-lfs/git-lfs/) in your repository | | After you registered the git repository you can now configure your Argo CD application 1. Click on _New Application_ (upper left of the screen) on the main page of Argo CD 1. You can now enter the application details | Item | Description | Example | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Application Name | the name of your application | my-application | | Project | the project your application is part of (see [Projects](#projects) for more information) | default | | Sync Policy | choose between _Manual_ or _Automatic_ synchronisation. With _Manual_ synchronisation you will have to trigger a sync manual via Web UI or CLI. _Automatic_ synchronisation will check your git repo every 3 minutes (or immediately when using [webhooks](#webhooks) from your git provider) | Automatic | | Source | the source of the configuration git repository (you should be able to select the created repository from the first step) | [https://gitlab.com/example/my-application-config](https://gitlab.com/example/my-application-config) | | Revision | this specifies either the branch, tag or commit you want to use in the configuration repository. This can be used to create different environments of your application. | development | | Path | please specify _._ here if all your files (Helm chart, kustomize files, etc) are in the root of the git repository. Otherwise you can specify the sub directory. | **.** | | Cluster | the cluster where to deploy | your-cluster-name | | Namespace | the namespace where all the resources should be deployed to (see [Namespace creation](#namespace-creation)) | my-application | | Type | the type of the configuration (plain yaml files, kustomize, Helm, etc). | Helm | | include subdirectories | if subdirectories should also be included | | If you use Helm charts as configuration type it is possible to set multiple _value.yaml_ files which will be merged in the given order. #### Parameter overrides Sometimes a separate configuration repository is not really needed or just too much effort. This might be the case in development/testing environments where one wants to: - have faster iteration cycles - use upstream Helm charts without forking them into an own git repository You might also want to set secrets directly in Argo CD without committing them into the configuration git. For those use cases Argo CD provides so called [parameter overrides](https://argoproj.github.io/argo-cd/user-guide/parameters/). Parameter overrides are only possible for applications which use Helm charts or ksonnet configurations. By overriding parameters (for example in a Helm chart) we are providing configuration information directly to Argo CD, without committing to a configuration repository. One possible workflow for a Helm configuration application would look like: 1. a developer creates an Argo CD application which uses a upstream Helm chart (either hosted in a git repository or in a Helm chart repository) 1. development of application code happens in a feature branch 1. once the developer pushes new changes to the application code git repository, a CI pipeline starts which executes the following tasks: - build and push a new container image - tell Argo CD to use the new container image by using the `argocd` cli command to set a parameter override in the _values.yaml_ of the upstream Helm chart - sync the application configuration state by executing `argocd app sync` for your application Parameter overrides for an application can be set via the cli application or via the web interface. ### Advanced topics #### Projects Projects in Argo CD are used to logical group applications. You can find more information about them in the [Argo CD documentation](https://argoproj.github.io/argo-cd/user-guide/projects/). Due to some restrictions in NKE it is currently not possible to define RBAC rules for projects. One use case for projects is to create [roles](https://argoproj.github.io/argo-cd/user-guide/projects/#project-roles). With roles for example you can permit access to Argo CD applications from a CI/CD pipeline (to sync for example), by using the JWT token assigned to a role. Here is an example to create a role called _cicd_ allowed to sync all applications in the default project: ```bash argocd proj role create default cicd argocd proj role create-token default cicd # save this token somewhere argocd proj role add-policy default cicd -a sync -o '*' -p 'allow' ``` In your pipeline you can then sync applications with ```bash argocd app sync --auth-token ``` #### Webhooks With webhooks your git provider can immediately notify Argo CD about changes in the configuration git repository. Without webhooks Argo CD will check for new commits every 3 minutes. You need to create the webhook in your [git providers settings](https://argoproj.github.io/argo-cd/operator-manual/webhook/). The URL and predefined secrets can be found in . #### Excluded Resources As it conflicts with our backup management system and the way backups get created, we exclude the following resources from syncing by default: - `snapshot.storage.k8s.io/VolumeSnapshot` - `snapshot.storage.k8s.io/VolumeSnapshotContent` If you use one of these resources in your ArgoCD application, it won't be synced to the cluster. If this affects your use case, please [contact us](/docs/general/contact). ## Documentation and links - [Argo CD Architecture](https://argoproj.github.io/argo-cd/#architecture) - [Argo CD documentation for developers](https://argoproj.github.io/argo-cd/user-guide/) --- ## Audit Logging The Audit Log feature enables Kubernetes auditing for NKE. Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself. Auditing shows who did what on your NKE cluster, e.g. User X send a get request on secret Y. Note: The log level is set to `Metadata`. Request and response contents are not being logged. See the official [Kubernetes documentation for more information](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/). ## Availability Audit Log is available as an optional service for NKE and it for now only be deployed by API/kubectl. ## Usage To enable Audit Log, you will need a running [Loki](./loki) instance. The audit logs will be pushed to that instance and you can then view them with either `logcli` or a `Grafana` instance. For now you can only enable the Audit Log feature via the API using curl or kubectl. For authentication, please check the . **Kubectl:** ```bash kubectl patch kubernetescluster -n --type='merge' -p ' spec: forProvider: nke: auditLog: targets: - group: observability.nine.ch/v1alpha1 kind: Loki name: ' ``` To view the log you can either do it via LogCLI or Grafana: **LogCLI:** ```bash logcli --username "username" --password "password" --addr "" --tls-skip-verify query '{log_type="audit"}' --from="" ``` The date should be in the format `2024-08-16T12:00:00Z`. **Grafana:** Go to `Explore` in the Grafana menu, select your Loki instance in the datasource and set the query: `{log_type="audit"}`. --- ## Autoscaling your NKE workload Autoscaling enables you to worry less about capacity planning and ensures the uptime of your services during load peaks. All this while you only pay what resources are needed at any given moment. ## Details With autoscaling configured, NKE automatically adds new node(s) to your cluster if you've created new Pods that don't have enough capacity to run. Conversely, if a node in your cluster is underutilized and its Pods can be run on other nodes, the autoscaler can delete the node. Keep in mind that when resources are deleted or moved in the course of autoscaling your cluster, your services can experience some disruption. For example, if your service consists of a controller with a single replica, that replica's Pod might be restarted on a different node if its current node is deleted. Before enabling autoscaling, ensure that your services can tolerate potential disruption or that they are designed and configured so that downscaling does not disrupt Pods that cannot be interrupted. ## Availability Autoscaling can be configured on a node pool level in Cockpit. But there are a few more things that have to be configured in order to automatically scale your workload. ## Usage ### Scaling your workloads horizontally 1. Configure the minimum and maximum node count: By default we won't just scale your cluster to an infinite amount of nodes to guard you from unexpected costs. You can set the min and max count of nodes on a per node pool basis and the autoscaler will scale within these boundaries. 1. [Set CPU requests on your pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit): The cluster autoscaler is using this as a base to calculate how much free capacity a node has. Without setting CPU requests the cluster autoscaler does not function. Plus this is good practice regardless if you make use of the autoscaler or not. 1. [Setup a Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/): To scale your pods with the incoming load you can setup a Horizontal Pod Autoscaler (HPA) to scale the pods on the CPU utilization. As soon as your nodes are full this will in turn trigger the cluster autoscaler to add more nodes. The Kubernetes documentation has a great [walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) to help you setup a HPA. ### Scaling on custom metrics There is the possibility to scale horizontally by leveraging an optional managed installation of [keda](https://keda.sh). Keda allows to get metrics from various backends (called 'scalers' in keda terms) and to scale based on them. You can deploy Keda on your cluster using Cockpit. You can find the [available scalers](https://keda.sh/docs/2.8/scalers/) in keda's documentation. For example if your cluster also has [Prometheus](./prometheus) deployed, you can make use of the [Prometheus scaler](https://keda.sh/docs/2.8/scalers/prometheus/). Keda also allows to scale on own custom metrics, by providing a self written [external scaler](https://keda.sh/docs/2.8/scalers/external/). --- ## Backup and Restore(Nke) Backup & Restore is a service of NKE allowing for regular backups and recovery of cluster data and configuration. ## Details Customers of NKE need peace of mind that their cluster configuration and Persistent Volume Claim (PVC) data is backed up and can be made available when needed, for security and disaster recovery. Therefore Nine regularly creates automated backups and on customer request engages in recovery and deployment of those backups. Data of Persistent Volumes is snapshotted and stored redundantly across our virtualization infrastructure. Kubernetes resource backups are saved in an Object Storage Bucket. ## Availability Backup/Restore is available as standard with NKE. ## Usage - Backups of data and configuration will automatically be taken nightly - Backups are retained for 30 days by default - You can restore namespaces [yourself](#restoring-namespaces) ### Restoring namespaces To create restores, the [velero utility](https://velero.io/docs/v1.7/basic-install/#install-the-cli) needs to be installed locally. Please make sure that your current kubecontext contains your NKE cluster by following the [documented cluster login steps](cluster-login). You can restore a complete namespace yourself. First you need to find the backup from which you want to restore. Execute the following command to list all backups: ```shell-session velero backup get -n nine-system ``` Once you found the backup from which you want to restore, you can restore the whole content of one namespace into another one by using: ```shell-session velero restore create -n nine-system --from-backup --include-namespaces --namespace-mappings : ``` If you want to restore the content of an existing namespace, you either delete the target namespace before restoring it or you delete all existing resources which would be restored by velero. Velero does not overwrite any existing resources. ```shell-session velero restore create -n nine-system --from-backup --include-namespaces ``` ### Backups of Read Write Many (RWX files) volumes :::caution The backups of RWX Persisten Volume Claims (PVC) are coupled to the PVC itself. If you accidentally delete the PVC, the corresponding backups will also be removed. To prevent an accidental deletion you can make use of our [deletion protection feature](deletion-protection). ::: For RWX volumes, the restore process is slightly different. These volumes use an integrated snapshotting mechanism that runs outside of the usual Velero schedule. The schedule is fixed and defined as follows: - Hourly: a new snapshot is created every hour and kept for 24h. - Daily: a new snapshot is created daily and kept for 7 days. - Weekly: a new snapshot is created every week on Sunday and kept for 4 weeks. - Monthly: a new snapshot is created every first day of the month and kept for 3 months. The snapshot data can be accessed from within the filesystem of the volume in in a directory named `.snapshot` in the root of the volume. To restore file(s) from older snapshots, you can either exec into an existing pod that mounts the volume or create a temporary pod to do the restore. ```bash # create a temporary pod, change the to the PVC you want to access. $ kubectl apply -f - < EOF $ kubectl exec -ti files-snapshot sh # all the PVC data is mounted at /data, so you can now list all your snapshots. $ ls -l /data/.snapshot total 14 drwxrwxrwt 2 root root 3 Nov 14 15:00 afs-auto-snap_hourly-2023-11-15-1100 drwxrwxrwt 2 root root 3 Nov 14 15:00 afs-auto-snap_hourly-2023-11-15-1200 drwxrwxrwt 2 root root 3 Nov 14 15:00 afs-auto-snap_hourly-2023-11-15-1300 ``` Once your temporary pod is up and running you can simply copy files from an older snapshot to `/data` or use the `rsync` command to sync larger directories. --- ## Cluster Login ## Prerequisites - [Install kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) - [Install nctl](/docs/nctl/) There are two different methods for logging into your Kubernetes cluster, depending on your use-case: 1. Login with your Cockpit account for use on your personal machine 1. Login with a service account for automation purposes The instructions differ slightly for these two methods. ## Cockpit Account Login First you will need to login to your Cockpit account with the CLI. The `auth login` command will automatically open your browser where you can login interactively. ```bash $ nctl auth login ✓ added nineapis.ch to kubeconfig 📝 ✓ logged into cluster nineapis.ch 🚀 ``` Now you can authenticate with any cluster within your organization using the `auth cluster` command: ```bash $ nctl auth cluster ✓ added / to kubeconfig 📝 ``` Alternatively you can also download the _kubeconfig_ from Cockpit when viewing your Kubernetes cluster and use that directly instead of letting create the config. Now you are ready to use `kubectl` as usual. will take care keeping you logged in at all times. ## Service Account Login Using a service account does not require any additional tooling. - Create a new _Account_ in Cockpit using the _Access Management_ tab - Create a _Cluster Role Binding_ and attach it to your previously created _Account_ - Go to the _Account_ and download the _kubeconfig_ Now you can use the _kubeconfig_ as you would any other. In case you want to selectively add permissions to this service account, you can do so using normal [rbac](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions). This service account will map to a service account in your clusters default namespace with the full name `system:serviceaccount:default:`. --- ## Container (OCI) Registry _Registry_ is a service for storing private container images and Helm charts. ## Availability _Registry_ is available as an optional service for NKE. It can be deployed using Cockpit and be used from any number of NKE clusters. ## Pushing Container Images To push container images, you will need to login through the URL and username and password combination that is provided in Cockpit. ```bash $ docker login Username: Password: Login Succeded ``` Afterwards, images can be tagged with `docker tag` and pushed with `docker push`: ```bash $ docker tag /: $ docker push /: The push refers to repository [] ... ``` ## Pushing Helm charts > Note: To push Helm charts to the registry, Helm v3.8.0 or newer is required. Prior to v3.8.0, OCI support was considered experimental and needs to be explicitly enabled by setting the environment variable `HELM_EXPERIMENTAL_OCI=1`. For more information, see [Enabling OCI support](https://helm.sh/docs/topics/registries/#enabling-oci-support) in the Helm documentation. To upload a Helm chart to the registry, you will need to login through the URL and username and password combination that is provided in Cockpit. ```bash $ helm registry login -u Password: Login Succeeded ``` After successful authentication, a chart can be pushed through the `helm push` command: ```bash $ helm push oci:// Pushed: ... Digest: ... ``` The URL needs to be prefixed with `oci://` instead of `https://` in order to work correctly. > Note: The `push` command can only be used against `.tgz` files created ahead of time using `helm package`. For more information about using Helm with the registry, please see the [official Helm documentation](https://helm.sh/docs/topics/registries/#commands-for-working-with-registries). ## Pulling Container Images in your Kubernetes Cluster In order to use your private registry in your Kubernetes cluster, a container image pull secret has to be created and referenced in your deployment manifests. To create the secret, you can fetch the `.dockerconfigjson` from Cockpit and use `kubectl` to create it in your cluster: ```bash kubectl create secret docker-registry \ --from-file=.dockerconfigjson= ``` This secret can then be referenced in the Pod's `imagePullSecrets` list. See the [official Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) for more information regarding using private registries. --- ## Deletion Protection Deletion protection allows to prevent specific Kubernetes resources from being accidentally deleted. ## Details On NKE clusters, you can protect Kubernetes "Namespaces" and "PersistentVolumeClaims" (PVCs) from being accidentally deleted. This is an additional safety net to prevent productive applications and their data to be deleted by mistake. ## Availability The deletion-protection feature is enabled by default on every NKE cluster. ## Usage To prevent a Kubernetes Namespace and/or PersistentVolumeClaim from being deleted, you can add the `nine.ch/deletion-protection: "true"` annotation. As long as this annotation exists, the resource can not be deleted. Please make sure to use a value of `"true"` to activate the feature. To add the annotation on a Kubernetes Namespace, you can use the following command: ```bash kubectl create ns foo kubectl annotate ns foo nine.ch/deletion-protection=true ``` To test if the annotation prevents an accidental deletion you can use the `--dry-run=server` option of `kubectl`: ```bash kubectl delete --dry-run=server ns foo Error from server (Forbidden): admission webhook "deletion-protection.nine-controllers.nine.ch" denied the request: preventing deletion because of nine.ch/deletion-protection annotation ``` To disable the deletion protection, you can either remove the annotation completely or use a value of `"false"`: ```bash kubectl annotate --overwrite ns foo nine.ch/deletion-protection=false ``` An additional deletion test should confirm the deactivation of the feature: ```bash kubectl delete --dry-run=server ns foo namespace "foo" deleted (server dry run) ``` Please note that, using a value different to `"true"` or `"false"` will lead to an error on deletion of the resource. Furthermore, please be aware that to fully protect a "PersistentVolumeClaim" from being deleted, the annotation should be set on the corresponding PVC(s) and the Kubernetes Namespace which contains the PVC(s). --- ## External Secrets [External Secrets Operator](https://external-secrets.io/latest/) (ESO) is a Kubernetes operator designed to integrate with various external secret management systems such as AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, and CyberArk Conjur, among others. This operator retrieves information from external APIs and injects the values into Kubernetes `Secret`s automatically. ## Availability External Secrets is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit or . ## Overview ESO consists of custom API resources: `ExternalSecret`, `SecretStore`, and `ClusterSecretStore`, which offer a user-friendly abstraction for managing and synchronizing secrets through external APIs. The `SecretStore` references a collection of key/value pairs, which may correspond to different external APIs like an Azure KeyVault instance or an AWS Secrets Manager in a specific AWS account and region. An `ExternalSecret` defines the data to be fetched and also serves as a blueprint for creating Kubernetes `Secrets`. The resource model can be visualized as follows: ```mermaid flowchart LR secretStore("kind=SecretStore") --> externalAPI("`External API provides k/v pairs ------------------ user: billy mymap: {#quot;foo#quot;: #quot;bar#quot;} baz: 1234`") kye("`auth 🔑`") --> secretStore externalSecret1("`kind=ExternalSecret key=user`") -->secretStore externalSecret2("`kind=ExternalSecret key=mymap property=foo`") -->secretStore externalSecret1 -->|create|secret1("kind=Secret") externalSecret2 -->|create|secret2("kind=Secret") ``` ### SecretStore The `SecretStore` resource aims to distinguish between authentication/access concern and the actual secrets and configurations needed for workloads. The `ExternalSecret` defines what data to retrieve, while the `SecretStore` specifies how to access that data. The `SecretStore` contains references to credentials required to access the external API. This resource is namespaced. ```yaml apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: secretstore-sample spec: provider: aws: service: SecretsManager region: eu-central-2 auth: secretRef: accessKeyIDSecretRef: name: awssm-secret key: access-key secretAccessKeySecretRef: name: awssm-secret key: secret-access-key ``` ### ExternalSecret An `ExternalSecret` specifies the data to be fetched and includes a reference to a `SecretStore`, which knows how to access the data. The controller uses the `ExternalSecret` as a blueprint to create the Kubernetes `Secret`. ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: example spec: refreshInterval: 1h secretStoreRef: name: secretstore-sample kind: SecretStore target: name: secret-to-be-created creationPolicy: Owner data: - secretKey: secret-key-to-be-managed remoteRef: key: mymap property: foo # Or use `dataFrom` to have all key properties automatically mapped to the k8s secret. dataFrom: - extract: key: mymap ``` ### ClusterSecretStore The `ClusterSecretStore` is a cluster-wide `SecretStore` that can be referenced from any namespace. It acts as a central gateway to the secret provider for the entire cluster. ## Usage Implementation details and the range of supported features vary based on the selected backend provider. For a comprehensive list of currently supported providers, an overview of feature support across different providers, and advanced examples, please refer to the [Stability and Support documentation](https://external-secrets.io/latest/introduction/stability-support/). ### Hashicorp Vault Provider Example External Secrets Operator integrates with [HashiCorp Vault](https://www.vaultproject.io/) for secret management. The [KV Secrets Engine](https://www.vaultproject.io/docs/secrets/kv) is the only one supported by this provider. First, create a `SecretStore` with a vault backend. For the sake of simplicity we'll use a static token `root`: ```yaml apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: vault-backend spec: provider: vault: server: "http://my.vault.server:8200" path: "secret" # Version is the Vault KV secret engine version. # This can be either "v1" or "v2", defaults to "v2" version: "v2" auth: # points to a secret that contains a vault token # https://www.vaultproject.io/docs/auth/token tokenSecretRef: name: "vault-token" key: "token" --- apiVersion: v1 kind: Secret metadata: name: vault-token data: token: cm9vdA== # "root" ``` :::note In case of a `ClusterSecretStore`, Be sure to provide `namespace` for `tokenSecretRef` with the namespace of the secret that we just created. ::: Then create a simple k/v pair at path `secret/foo`: ```bash vault kv put secret/foo my-value=s3cr3t ``` You can check the `kv` version using the command below and check the `Options` column, it should show `[version:2]`: ```bash vault secrets list -detailed ``` If you are using version: 1, just remember to update your SecretStore manifest appropriately Now create a `ExternalSecret` that uses the above `SecretStore`: ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: vault-example spec: refreshInterval: "15s" secretStoreRef: name: vault-backend kind: SecretStore target: name: example-sync data: - secretKey: foobar remoteRef: key: foo property: my-value --- # will create a secret with: kind: Secret metadata: name: example-sync data: foobar: czNjcjN0 ``` For additional HashiCorp Vault examples and advanced usage instructions, please refer to the [official provider documentation](https://external-secrets.io/latest/provider/hashicorp-vault/). --- ## Grafana Grafana allows you to quickly see the current state of your cluster and deployments. ## Details For customers who need to visualise the state of their NKE cluster, nine provides a default dashboard, built with Grafana. You are also free to create your own dashboards, manage them in folders to create different hierarchies and add alerts to them. ## Availability Grafana is available as an optional service for NKE and it can be deployed using Cockpit. ## Restrictions Please note the following restrictions: - **Grafana Alerting**: Grafana Alerting is currently not supported when using the [metrics agent](./metrics-agent). Please use [Alertmanager](./alertmanager) instead. ## Usage You can log in with your Cockpit account credentials. ### Custom Dashboards and Alerts To create your own dashboards, have a look at the [official Grafana Documentation](https://grafana.com/docs/guides/getting_started/). By default, Nine Managed Grafana comes with certain restrictions: - Currently, it is not possible to add your own datasources. - Editing the pre-deployed dashboards is not allowed, however, you can copy the JSON-definition of panels and use it in your own dashboard. - It is not possible to configure alerts via email - Alerts can only be defined on Dashboards & Panels that you have created Some of these restrictions can be removed by enabling admin access. ### Admin Access :::warning We advise you to be careful with the admin permissions, as you could potentially break your Grafana instance. Nine reserves the right to remove admin permissions at any moment if the admin access is deemed a security or an operational risk. ::: Enabling admin access grants the Grafana Admin role to all users in your organization, replacing the default Editor role. This removes most of the default restrictions listed above and gives you full control over your Grafana instance, including the ability to add datasources and manage users. Set `EnableAdminAccess` to `true` in the Grafana parameters in Cockpit. Enable or disable admin access using : ```bash nctl update grafana --admin-access nctl update grafana --no-admin-access ``` ### Local Users By default, all Grafana login attempts are redirected to the nine.ch OIDC provider. Once you have enabled admin access, you can create local Grafana users via the built-in user management. To allow those local users to reach the login form, enable `AllowLocalUsers` using . Without this setting, local users are immediately redirected to the OAuth login page and cannot sign in. ```bash nctl update grafana --local-users nctl update grafana --no-local-users ``` :::warning When `AllowLocalUsers` is enabled, OIDC users are no longer auto-redirected and must manually click **Sign in with OAuth** on the login form. ::: ## Migration Guide for AngularJS-Based Panels With the upcoming [deprecation of AngularJS support](https://grafana.com/docs/grafana-cloud/whats-new/2024-03-04-angularjs-plugin-warnings-in-dashboards/) in Grafana, it is essential to migrate all AngularJS-based panels to the supported frameworks to avoid future disruptions. Grafana 10.4 offers a Migration Wizard that makes this process straightforward. In your dashboard, AngularJS-based panels are flagged with an exclamation mark (!), helping you quickly identify the panels that require migration. Simply open each identified panel in edit mode and follow the steps in the Migration Wizard to complete the conversion process. ### Tips and Common Issues During Migration #### Migration From `Graph (old)` to `Time Series` Panel _Problem_: The graph is not visible, but some data can be seen by hovering mouse pointer over the render area. This means that while the data is present, it is not being properly displayed in the graph visualization. _Solution_: This issue is often caused by incorrect or invalid threshold configurations. To verify if this is the root cause, access the `Dashboard Configuration` (ensure you have `Edit` mode enabled in the `Dashboard settings`), select the `JSON Model` tab, and locate the relevant panel by searching for its name in the JSON data structure source code. Check if each `color` setting (under `thresholds`) has a `value` assigned (note that `"value": "nil"` is a valid entry). If any threshold configuration is missing a `value` definition, for example, if the configuration for `red` is invalid, simply remove the `red` object from `steps` to resolve the issue: ```json ... { "fieldConfig": { "defaults": { "custom": {...}, "thresholds": { "mode": "absolute", "steps": [ { "color": "transparent", "value": null }, { "color": "red" } ] }, "unit": "none" }, "overrides": [...] }, "gridPos": {...}, "options": {...}, "pluginVersion": "10.4.7", "targets": [...], "title": "Average CPU Usage", "type": "timeseries" }, ... ``` Once you remove this invalid step, the graph should render correctly. Afterward, you can redefine the threshold settings, such as specifying the `red` configuration again, but ensure to use an updated format to prevent the initial rendering problem from recurring. #### Migration From `Table (old)` to `Table` Panel _Problem_: After migrating, some columns may be missing, or there could be changes to data types, formatting, or threshold configurations. _Solution_: These issues often stem from `Overrides` configurations. Follow these steps, which may resolve the issue: - Change Field Overrides. In the panel properties, tab `Overrides`, update the `Fields with name` setting to a selectable option from the dropdown list (e.g., `Time`). If it changes to something like `Time (not found)`, use the dropdown again and select an appropriate field (for example `Field`). - Edit `Display Name` and `Units`. Navigate to the relevant text box for `Change the field or series name` and manually input the correct name. Similarly, in the `Unit` text box, experiment with available types — for instance, try selecting `Misc > String`. - Correct additional columns. Repeat the process for the remaining columns as needed. For instance, for the second column, set the `Fields with name` option to `Last *` and make sure you select the correct name and unit. This should help resolve formatting issues and ensure the threshold display renders colors correctly. ## Documentation and Links - [Grafana Source](https://github.com/grafana/grafana) - [Grafana Documentation](https://grafana.com/docs/) --- ## Nine Kubernetes Engine (NKE) Nine Kubernetes Engine or NKE is a fully managed Kubernetes offering running in Nine-operated data centers in Switzerland. Supplemented by additional services, letting you completely focus on your application development. ## Getting started To get started with NKE you need a login to access our . Then simply select _Managed Kubernetes_ from our products overview and use _Add Cluster_ to create your first cluster. The initial creation can take a few minutes until all services are deployed. To access the cluster, have a look at our [cluster login article](./cluster-login). ## Locations NKE is available in the following locations: _All locations are physically located in Switzerland — see [Datacenter Locations](/docs/general/datacenter-locations) for details._ ## Subnets in each Location All Nine Kubernetes Engine cluster nodes will have public IPs by default. Each Pod running on a node will use the public IP of the node it is running when initiating connections to cluster external systems (Kubernetes default networking model). The IP subnets which will be used depend on the location of the Nine Kubernetes Engine cluster. You can find a current list of subnets in JSON format by using our **insight service**: | Location | API Endpoint | | ----------- | --------------------------------------------- | | All Subnets | https://insight.nineapis.ch/subnets | | nine-es34 | https://insight.nineapis.ch/subnets/nine-es34 | | nine-cz41 | https://insight.nineapis.ch/subnets/nine-cz41 | _All locations are physically located in Switzerland — see [Datacenter Locations](/docs/general/datacenter-locations) for details._ Please note that new subnets might be added at any point. ### Example You can use the `curl` command line tool to request a current list: ```bash curl https://insight.nineapis.ch/subnets {"data":{"nine-cz41":["94.230.211.128/26"],"nine-es34":["178.209.59.64/26","185.88.237.64/26","5.148.184.0/26"]}} ``` ## Node Pools Like many Kubernetes offerings, NKE manages nodes in pools. A pool is defined by a name, machine type, disk size and an amount of nodes. Node pools simplify isolating different workloads on a node level from each other. Also, they can be individually scaled up and down. NKE uses three `nine-standard-2` as control plane nodes. Depending on the services configured and the specific usage of the cluster, it may be necessary to adjust the machine type of these nodes. If such a change is required, we will reach out to you proactively. ## Machine Types NKE offers the following machine types: | Name | vCPU | RAM (GiB) | Price per Month | | --------------- | ---- | --------- | ------------------------------------------ | | nine-standard-1 | 1 | 4 | | | nine-standard-2 | 2 | 8 | | | nine-standard-4 | 4 | 16 | | | nine-highmem-2 | 2 | 16 | | | nine-highmem-4 | 4 | 32 | | | nine-highcpu-2 | 2 | 4 | | | nine-highcpu-4 | 4 | 8 | | | nine-highcpu-8 | 8 | 16 | | The resulting monthly costs can be calculated using the [Price Calculator](https://calculator.nine.ch/?category=Nine+Kubernetes+Engine&segment=2%29+Worker+Nodes+%26+Storage) or the table below. | Additional Resources | Price per Month | | ------------------------ | ---------------------------------------------- | | Per vCPU | | | Per 1 GiB RAM | | | Per 10 GiB storage space | | ## Load Balancing NKE supports services with the type load balancer out of the box. After creating your service it will be assigned a public IPv4 address for the lifetime of the service. Simply follow the [Kubernetes documentation to create a new load balancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/). Though for most use cases an _Ingress_ might be the simpler alternative, if you just need to expose HTTP(S) traffic. Head over to our [ingress documentation](./ingress) for that. ## Persistent Storage Persistent Storage can be requested by creating a [_PersistentVolumeClaim_](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reserving-a-persistentvolume). Our controller will take care of creating a _PersistentVolume_ and mounting it to your pod. ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: example namespace: example spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard ``` Our controller supports online expansion of a _PVC_. Simply edit the _PVC_ in question and increase the storage requests to your liking. All volumes are secured with Data Encryption at Rest (DARE), using AES-256 encryption to protect stored data from unauthorized access. ### Read Write Many (RWX) > ⚠ Note: RWX is currently only available in the location nine-es34. If you have a use case for RWX storage, you can create such volumes with the storage class `files`. This will create a NFS-backed volume that can be mounted in multiple pods at once. In most cases we recommend against using RWX and instead make use of [object storage](/docs/object-storage/manage-buckets-and-users) where possible. ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: example namespace: example spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi storageClassName: files ``` ## Backups All NKE clusters are backed up by default. For more details see our [article on backups](./backup-and-restore). ## Maintenance Windows Security and software updates are generally performed during the NKE [maintenance windows](../../general/weekly-maintenance-window.md#nine-kubernetes-engine-nke). Therefore, short service interruptions may occur during this maintenance window without prior notice. ## Limitations NKE clusters have restricted cluster-admin permissions to ensure the stability and security of managed components. For more details on these restrictions and how they affect Custom Resource Definitions (CRDs), see the [Security Concepts](./security-concepts#cluster-admin) page. ## Additional Services To supplement NKE, we offer a number of services to make a fully featured application platform. You can see all available services in , [pricing on our website](https://www.nine.ch/en/kubernetes) and more details in other support articles. ## Further Information For further information or sales please contact info@nine.ch and for support [contact us](/docs/general/contact). --- ## Ingress(Nke) The ingress system of Kubernetes is specifically designed to route external HTTP and HTTPS traffic into the cluster. It is composed of the ingress resource itself and an ingress controller which implements the needed logic. We offer a managed controller in the form of the HAProxy ingress controller that you can choose to deploy to your NKE cluster in Cockpit. You can control various features by adding annotations to your ingress object. ## Availability The HAProxy ingress controller is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit. :::warning The nginx ingress controller is deprecated and we will be completely shutting down support and monitoring for all Nginx ingresses on **October 1st**. Please migrate your Ingress resources to use the HAProxy ingress controller before that date. Follow the [step-by-step migration guide](#migrating-from-nginx-to-haproxy). ::: ## Usage The basic usage and structure of an ingress resource is documented [in the official Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/). ### Ingress DNS Name We provide a DNS name which will always point to your HAProxy ingress controller's IP. You can use it to point your own domain hostnames to the ingress. DNS is already set up, you will find the name in the HAProxy ingress view in Cockpit. To use it just create an Alias or CNAME record in your own domain and point it to our provided ingress DNS. ### Wildcard DNS Name Additionally we also create a wildcard DNS record `*.`. It is meant for quick application tests in development. You can use any hostname of that wildcard zone in your ingress resources to quickly get something up and running without requiring to mess with DNS records. ### IngressClass To make use of our ingress controller, set the `ingressClassName` field in your `Ingress` resource to the ingress class of your HAProxy controller, which defaults to `haproxy`. You can change the ingress class name when you deploy the controller in Cockpit, and you can also configure whether it should be the cluster's default ingress class. If you marked the controller as the default class, you can omit the `ingressClassName` field. It is possible to deploy multiple HAProxy ingress controllers per NKE cluster, each with its own ingress class. Use the `ingressClassName` field to select which controller handles a given ingress resource. The deprecated nginx ingress controller is still available and uses the ingress class that was configured when it was deployed. It will be removed in a future release. ### Automatic TLS Certificates NKE ships with [cert-manager](https://cert-manager.io/) and [Let's Encrypt](https://letsencrypt.org/) preconfigured, so securing an ingress with free TLS certificates is as simple as adding an annotation. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: # this tells cert-manager to issue a certificate for # all hosts specified in the tls section kubernetes.io/tls-acme: "true" name: hello-ingress spec: ingressClassName: haproxy rules: - host: app.example.org http: paths: - path: / pathType: Prefix backend: service: name: hello-ingress port: number: 80 tls: - hosts: - app.example.org secretName: hello-ingress ``` ### Access Logs The access logs of your ingress requests can be viewed in your Grafana instance in the Loki Explore view. The access logs are written by a dedicated container of the ingress controller and are available under the labels `app="haproxy-ingress"` and `container="access-logs"`: ```bash {app="haproxy-ingress", container="access-logs"} ``` To narrow the logs down to a specific backend, use a [line filter](https://grafana.com/docs/loki/latest/query/log_queries/#line-filter-expression). Each access log line contains the target backend in the schema `__`. Here's an example query to get all the logs of the service `frontend` with the port `80` in the namespace `shop-prod`: ```bash {app="haproxy-ingress", container="access-logs"} |= "shop-prod_frontend_80" ``` The HTTP method, status code, request path and timings are part of the log line itself, so you can filter for them with additional line filters (for example `|= " 503 "` to find requests that returned a `503`). For more information on the usage of Loki, refer to the [specific support article](./loki). ### HAProxy Ingress Features The HAProxy ingress controller provides many features like rate limiting, IP whitelisting, temporary or permanent redirects, etc. All of the configuration keys which can be used to control those features can be found [in the official HAProxy ingress controller documentation](https://haproxy-ingress.github.io/docs/configuration/keys/). Documentation for the most used features can be found below. #### Basic authentication You can add basic authentication to your ingress resource by providing the credentials in a Kubernetes secret. Here are some instructional steps: 1. set some env variables for easier processing ```bash USERNAME= SECRET_NAMESPACE= INGRESS_NAMESPACE= INGRESS= ``` 1. create the Kubernetes secret which contains the credentials for basic auth. It can also be created in a different namespace than your ingress resource is stored. You will need the `mkpasswd` tool installed locally (can be found in the `whois` package in Debian/Ubuntu). ```bash kubectl create secret generic basic-auth-secret --namespace=$SECRET_NAMESPACE --from-literal=auth=$USERNAME:$(mkpasswd -m sha-512) ``` 1. add some annotations to your ingress object ```bash kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-realm='Authentication required' ``` #### Rate limiting You have various ways of putting rate limits on your ingresses. You can limit requests per second using the `haproxy-ingress.github.io/limit-rps` annotation or limit concurrent connections using the `haproxy-ingress.github.io/limit-connections` annotation. All available options are documented in [the official HAProxy ingress docs](https://haproxy-ingress.github.io/docs/configuration/keys/#limit). #### Temporary and persistent redirects To enable a redirect to another URL for your ingress you can use the following annotation: ```yaml haproxy-ingress.github.io/redirect-to: ``` The redirect will use the HTTP status code of 302 (temporary) by default. If you want to change the status code, for example to 301 for a permanent redirect, use: ```yaml haproxy-ingress.github.io/redirect-to-code: "301" ``` #### HTTPS redirect If TLS is enabled for the given ingress, the HAProxy ingress controller will automatically redirect to the equivalent HTTPS URL of the ingress. To disable this redirect use: ```yaml haproxy-ingress.github.io/ssl-redirect: "false" ``` #### IP whitelisting You can allowlist the IP addresses which are allowed to connect to your ingress resource. You can specify them in CIDR notation in the following annotation: ```yaml haproxy-ingress.github.io/allowlist-source-range: ``` #### Custom default backend The default backend is responsible for showing a 404 error page if a request arrives on the HAProxy ingress controller for which no ingress rule was specified. You can create your own custom default backend (+ kubernetes service) and refer to it on your ingress object. The default backend only has 2 requirements: - it needs to serve a 404 page/code on the path / - it needs to serve a 200 HTTP code on the path /healthz Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress: ```yaml haproxy-ingress.github.io/default-backend: ``` ## Static IPs for ingress and egress The IP of every ingress controller is static. This means all ingress resources using that controller will share that IP. In addition, it is possible to deploy multiple ingress controllers per NKE cluster and you can choose the controller with the `ingressClassName` field in the ingress resource. While ingress IPs remain static, it's important to note that, by default, egress IPs are currently dynamic due to operational considerations. Specifically, during node replacements, such as underlying OS version upgrades, NKE initiates the creation of a new node. The existing workloads are seamlessly moved to the new node, which then inherits a new IP address. As a result, the egress IPs are subject to change in this process. If you need a static egress IP for your cluster workloads, you can make use of our [static-egress feature](./static-egress.md). ## IP ranges for egress See [Subnets in each Location](./#subnets-in-each-location) for the current egress IP ranges. ## Deprecated: Nginx Ingress Controller {/* #nginx-ingress */} :::warning The nginx ingress controller is deprecated and will be removed in a future release. Please migrate your Ingress resources to the HAProxy ingress controller documented above. :::
Show deprecated nginx ingress documentation ### IngressClass To use the deprecated nginx ingress controller, set the `ingressClassName` field in your `Ingress` resource to its configured ingress class, that is, the class that was set when the controller was deployed. ### Access Logs The nginx ingress logs are available under the Loki label `app="ingress-nginx"`. Example query: ```bash {app="ingress-nginx",ingress="shop-prod-frontend-80"} ``` ### Nginx Ingress Features All available annotations can be found [in the official nginx ingress controller documentation](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/). #### Basic authentication ```bash kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-type=basic kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-realm='Authentication required' ``` #### Rate limiting All available options are documented in [the official nginx ingress docs](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rate-limiting). #### Temporary and persistent redirects ```yaml # Temporary redirect (HTTP 302) nginx.ingress.kubernetes.io/temporal-redirect: # Permanent redirect (HTTP 301) nginx.ingress.kubernetes.io/permanent-redirect: ``` #### HTTPS redirect ```yaml nginx.ingress.kubernetes.io/ssl-redirect: "false" ``` #### IP whitelisting ```yaml nginx.ingress.kubernetes.io/whitelist-source-range: ``` #### Caching The nginx ingress controllers allows to enable basic caching of backend resources, which can be particularly useful for static content. You can enable caching on specific ingress resources by setting the following annotations in your ingress definition (`metadata.annotations`): ```yaml nginx.ingress.kubernetes.io/proxy-buffering: "on" nginx.ingress.kubernetes.io/configuration-snippet: | proxy_cache static-cache; proxy_cache_valid 10m; proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504; proxy_cache_bypass $http_x_purge; add_header X-Cache-Status $upstream_cache_status; ``` In this example, the cache is invalidated after 10 minutes, and only HTTP status codes 200, 301 and 302 are cached. If another behaviour should be desired, `proxy_cache_valid` can also take a list of status codes in front of the time: ``` proxy_cache_valid 404 1m; ``` would cache 404 responses for one minute. There is also the special code `any`, which can be specified to cache any responses. Additionally, multiple `proxy_cache_valid` statements can be added on one ingress to specify different cache times for different status codes. The default cache size is 100MB. Should you have other requirements, please do not hesitate to get in contact with us. To check that the caching worked, you can use `cURL` to inspect the header of the returned resource. For that, you will need to execute the command twice, and at the second request, the resource should be cached: ```bash curl --head ``` should print the header `x-cache-status: HIT`. If you want to enable caching only for a sub-path of your application (for example an endpoint `/static`), you will need to create two separate ingress resources. To further configure caching, you can use the `proxy_cache` options that are valid for `location` blocks. See the [nginx `proxy_cache` module documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_key) for available options. #### Custom default backend The default backend is responsible for showing a 404 error page if a request arrives on the nginx ingress controller for which no ingress rule was specified. You can create an own custom default backend (+ kubernetes service) and refer to it on your ingress object. The default backend only has 2 requirements: - it needs to serve a 404 page/code on the path / - it needs to serve a 200 HTTP code on the path /healthz The implementation of the default nginx ingress controller backend can be found in the [ingress-gce 404-server source](https://github.com/kubernetes/ingress-gce/tree/master/cmd/404-server). Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress: ```yaml nginx.ingress.kubernetes.io/default-backend: ``` ##### Additional custom error pages To be able to additionally display custom error pages on the default backend (for example if the service your ingress points to is not available) you can use the following annotation: ```yaml nginx.ingress.kubernetes.io/custom-http-errors: # for example: "404,415,503" ``` The nginx ingress controller will forward error information via certain HTTP headers to your default backend, which then can return the best possible error representation. More information about this can be found [in the official documentation](https://kubernetes.github.io/ingress-nginx/user-guide/custom-errors/). An example of a default backend which can display custom error pages can be found in the [ingress-nginx custom-error-pages repository](https://github.com/kubernetes/ingress-nginx/tree/master/images/custom-error-pages).
## Migrating From Nginx to HAProxy You can run the nginx and HAProxy ingress controllers side by side. This lets you migrate one ingress at a time and verify each application on HAProxy before sending production traffic to it, so no maintenance window is required. The migration is a per-ingress process: for each nginx `Ingress` resource you add an equivalent HAProxy `Ingress`, verify it, switch DNS, and then remove the old one. ### Migrating Annotations From Nginx If your ingress resources use nginx-specific annotations, you need to update them to their HAProxy equivalents. Here are the most common ones: | Nginx annotation | HAProxy annotation | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `nginx.ingress.kubernetes.io/ssl-redirect` | `haproxy-ingress.github.io/ssl-redirect` | | `nginx.ingress.kubernetes.io/auth-secret` | `haproxy-ingress.github.io/auth-secret` | | `nginx.ingress.kubernetes.io/auth-realm` | `haproxy-ingress.github.io/auth-realm` | | `nginx.ingress.kubernetes.io/whitelist-source-range` | `haproxy-ingress.github.io/allowlist-source-range` | | `nginx.ingress.kubernetes.io/temporal-redirect` | `haproxy-ingress.github.io/redirect-to` (defaults to status code 302) | | `nginx.ingress.kubernetes.io/permanent-redirect` | `haproxy-ingress.github.io/redirect-to` + `haproxy-ingress.github.io/redirect-to-code: "301"` | | `nginx.ingress.kubernetes.io/limit-rps` | `haproxy-ingress.github.io/limit-rps` | 1. **Deploy the HAProxy ingress controller** for your cluster in Cockpit. Note its ingress class name (defaults to `haproxy`) and its ingress DNS name, both shown in the HAProxy ingress view. See [Availability](#availability). 2. **Add a second `Ingress` resource** for your application that uses the HAProxy ingress class and the equivalent HAProxy annotations, and leave the existing nginx ingress in place for now. See [Migrating Annotations From Nginx](#migrating-annotations-from-nginx) for the annotation mapping. To find the ingress resources you still need to migrate, list them across all namespaces. The `CLASS` column shows which controller each one uses: ```bash kubectl get ingress --all-namespaces ``` For example, this nginx ingress: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" spec: ingressClassName: nginx rules: - host: app.example.org http: paths: - path: / pathType: Prefix backend: service: name: hello port: number: 80 ``` gets an HAProxy counterpart with the translated `ingressClassName` and annotations. While testing, point it at a hostname under the HAProxy [wildcard DNS name](#wildcard-dns-name) so you don't have to change your production DNS yet: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello-haproxy annotations: haproxy-ingress.github.io/ssl-redirect: "true" spec: ingressClassName: haproxy rules: # a test hostname under the HAProxy wildcard DNS name - host: hello. http: paths: - path: / pathType: Prefix backend: service: name: hello port: number: 80 ``` 3. **Verify** that your application is reachable through the HAProxy ingress using the test hostname: ```bash curl http://hello./ ``` 4. **Switch your DNS records** to the HAProxy ingress. Update the CNAME or Alias record for your domain so it points to the HAProxy ingress DNS name instead of the nginx one. You can then replace the test hostname in the HAProxy ingress with your production hostname. See [Ingress DNS Name](#ingress-dns-name). 5. **Remove the old nginx `Ingress` resource** once all traffic is served through HAProxy: ```bash kubectl --namespace= delete ingress ``` 6. **Remove the old nginx ingress controller** in Cockpit once no `Ingress` resources reference its ingress class anymore. If the nginx controller was your cluster's default ingress class, set the HAProxy controller as the new default in Cockpit so ingresses without an explicit `ingressClassName` keep working. ## Documentation and Links - [the official kubernetes ingress documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/) - [all available configuration keys of the HAProxy ingress controller](https://haproxy-ingress.github.io/docs/configuration/keys/) - [all available annotations of the nginx ingress controller (deprecated)](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/) --- ## Kubernetes Cluster backed by vcluster A vcluster is an alternative to [NKE](./)-backed Kubernetes clusters with different use-cases and limitations. It makes use of [vcluster](https://www.vcluster.com) to provide Kubernetes clusters at a lower price point with some drawbacks in terms of availability guarantees and managed add-ons. As vclusters are running on top of NKE, it has access to the same compute infrastructure. This comparison table should give an overview of most of the differences: | | NKE | vcluster | | -------------------------------- | --- | -------- | | Service type load balancer | ✓ | ✓ | | Persistent storage (RWO/RWX) | ✓ | ✓ | | Ingress | ✓ | ✓ | | Autoscaling | ✓ | ✓ | | Argo CD integration | ✓ | ✓ | | NKE machine types | ✓ | ✓ | | Dedicated worker nodes | ✓ | ✓ | | Dedicated HA control-plane nodes | ✓ | ✗ | | Cluster add-ons | ✓ | ✗ | | Automatic backup | ✓ | ✗ | | Uptime guarantees (SLA) | ✓ | ✗ | | Cluster fee | ✓ | ✗ | | Fast creation time (< ~2 min) | ✗ | ✓ | | Cluster admin | ✗ | ✓ | ## Getting started To get started with vclusters you need a login to access our . Then simply select _Managed Kubernetes_ from our products overview and use _Add Cluster_ and choose the _vcluster_ option. To access the cluster, have a look at our [cluster login article](./cluster-login). For more information on the available node types, load balancing and storage refer to the [main NKE documentation](./). ## Use cases for a vcluster - **CI/CD:** Because of the fast creation and deletion times of a vcluster, as well as their low cost, they lend themselves to be used in CI/CD pipelines to test deployments of your apps end-to-end. - **Testing new Kubernetes API versions:** We try to always provide the latest Kubernetes releases within vcluster so you can test your apps against new API versions early. - **Well isolated and cost effective environments:** Staging and development environments can use their own vcluster to be better isolated from production instead of using multiple namespaces on a single NKE cluster. ## ArgoCD If you want to use ArgoCD for deploying onto your vcluster, the steps documented [in the ArgoCD article](./argocd#namespace-creation) are not enough. Since vclusters are not fully managed clusters, you will need to give permissions to the ArgoCD service account yourself. First, you will need to find the name of the ArgoCD service account: ```bash $ kubectl get serviceaccounts -n default argocd-xyz 1 188d ``` Then you will need to create a clusterrolebinding and give the ArgoCD cluster-admin permissions: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: argocd roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: argocd-xyz namespace: default ``` ## Ingress Because of the design of vcluster, the Ingress controller of the host cluster can be reused. This means there is no need to deploy an Ingress controller in your vcluster. You can find the `IngressClass` and `Host` of the predefined Ingress in the cluster details in Cockpit or in the status of the API. This also makes it possible to get automatic [Let's Encrypt](https://letsencrypt.org/) TLS certificates when using the host ingress class. For more information on how to use Ingress, refer to our [dedicated documentation](./ingress#usage). ### Installing custom Ingress Controllers If you don't want to use the predefined Ingress controller, you can also deploy your own within the vcluster. Just note that you need to use a different `IngressClass` name that does not conflict with the predefined one. --- ## Loki Loki allows you to view and query logs of your containers using Grafana Loki. ## Details Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream. The logs are persisted by default for 30 days, but you can change that to whatever you want. ## Availability Loki is available as an optional service for NKE and it can be deployed using Cockpit. In order to ship logs from an NKE cluster to Loki you will also need to create a Promtail instance on the Kubernetes Clusters page pointing to your Loki instance. ## Usage Loki can be accessed by using the Grafana Web UI. For usage instructions, see [Grafana](./grafana#usage). ### Labelling your pods If your pod is part of a deployment, statefulset or another controller, it will automatically be picked up by Loki, no matter what labels are set. We recommend using these [common labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) to easily find your logs. If you run a single pod, you will need to set one of these labels to ensure Loki will pick up your logs. - `app` - `name` ### Querying Logs with LogQL The query language used in Loki is called _LogQL_. To start querying your logs, head to the Grafana UI and click on _Explore_ in the sidebar. A LogQL query consists of two parts: log stream selector, and a search expression. A stream is selected by supplying one or more labels, for example: ```bash {app="nginx", name=~"frontend.+"} ``` To search for a certain string in the results, you can use a search expression. This can be just text matching by using `|=` or a regex expression by using `|~`. And by using a `!` instead of the pipe, the expression can be negated. Here are some examples: ```bash {app="nginx"} |= "GET" {app="nginx"} |~ "200|201|202" {app="nginx"} != "GET" {app="nginx"} !~ "200|201|202" ``` For more details, please refer to the [Loki documentation](https://grafana.com/docs/features/datasources/loki/#querying-logs). ### Pushing custom Logs If you have pods which store logs in files rather than writing them to `STDOUT`, you can use any [Loki client](https://grafana.com/docs/loki/latest/clients/) to push logs to it. Below, there's an example what this could look like. In the example we are using fluent-bit with the Loki plugin as a sidecar to a Nginx container to send logs to Loki. Please make sure to replace `` with your specific address. The log path, format and labels are passed to fluent-bit as environment variables defined in the pod spec. [More information about Fluent Bit Loki plugin](https://grafana.com/docs/loki/latest/clients/fluentbit). ```yaml apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-loki data: fluent-bit.conf: |- [INPUT] Name tail Path ${LOG_PATH} [Output] Name loki Match * Url http://:3100/loki/api/v1/push BatchWait 1 BatchSize 1001024 Labels {app="${APP_LABEL}",pod="${POD_NAME}",namespace="${POD_NAMESPACE}"} LineFormat ${LOG_FORMAT} LogLevel info --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: volumes: - name: fluent-bit-config configMap: name: fluent-bit-loki - name: logs emptyDir: {} containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 volumeMounts: - name: logs mountPath: /var/log/nginx - name: fluent-bit-loki image: grafana/fluent-bit-plugin-loki:2.5.0-amd64 volumeMounts: - name: fluent-bit-config mountPath: /fluent-bit/etc - name: logs mountPath: /var/log/nginx env: - name: APP_LABEL value: nginx - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LOG_PATH value: /var/log/nginx/*.log - name: LOG_FORMAT value: key_value ``` ### Pushing external logs If you want to push logs from external systems (like an external Kubernetes cluster) to your Loki instance, please [open a support ticket](/docs/general/contact) and we will provide you with the required data. --- ## Metrics Agent ProductPrice, ProductDescription, } from "@site/src/components/ProductPrice" Metrics Agent is a component of NKE that monitors applications. It succeeds [Prometheus](../nke/prometheus.md). ## Usage NKE deploys a new Metrics Agent instance in the `nine-system` namespace upon creation. The pods run on control-plane nodes, so node pools remain fully available for applications. The Metrics Agent is based on [Victoria Metrics](https://github.com/VictoriaMetrics/VictoriaMetrics). Manage configuration via [prometheus-operator project](https://github.com/coreos/prometheus-operator) resources. Use the following resource types to configure scraping, alerting, and recording rules: - `ServiceMonitors` - `PodMonitors` - `PrometheusRules` The agent collects metrics and sends them to an external Metrics Cluster. This offloads resources from the NKE cluster control-plane compared to the previous Prometheus product. ### Migrating from Prometheus :::tip To ensure a seamless migration, name your Metrics Agent the same as your existing Prometheus instance. This eliminates the need to update labels on `ServiceMonitors` and `PodMonitors`. ::: If you're currently using Prometheus and want to migrate to Metrics Agent, the process is straightforward: 1. **Select your Kubernetes Cluster** in [Cockpit](https://cockpit.nine.ch/en/nine/nke/infrastructure/kubernetes_clusters/), open the **Metrics** tab, and note your current Prometheus name. 1. Click on **Add Metrics Agent** and define the **same name** as your existing Prometheus instance, so that Metrics Agent automatically picks up all existing `ServiceMonitors` and `PodMonitors` without any configuration changes. **If you use a different name**, you need to update the label on your existing `ServiceMonitors` and `PodMonitors` to match the new Metrics Agent name. For example, if your Prometheus instance was named `prometheus` and you create a Metrics Agent named `metrics-agent`, change the label from `prometheus.nine.ch/prometheus: scrape` to `prometheus.nine.ch/metrics-agent: scrape`. 1. Give Metrics Agent some time to discover the monitors and start scraping metrics. 1. **Update the data source in Grafana** to point to the new Metrics Agent. 1. **After you verify that metrics collection is working correctly, delete the old Prometheus instance.** ### Exporters and Metrics The agent automatically collects a set of default metrics. You can optionally enable additional metrics from the following exporters: - CertManager - IngressNginx - NodeExporter - Kubelet - Kubelet cAdvisor - KubeStateMetrics - Velero Please [contact us](/docs/general/contact) to enable specific exporters. Future updates will allow self-service activation via Cockpit. :::note The number of active series and data points you collect directly corresponds to your bill. See [Pricing](#pricing) for details. ::: ### Visualizing Metrics with Grafana :::note Metrics Agent does not currently support Grafana Alerting. Please use [Alertmanager](./alertmanager) instead. ::: Create a [Grafana instance](./grafana.md) to visualize metrics. If you create the Grafana instance in the default project (same name as the organization), all metrics in the organization are visible. If you create the instance in any other project, only metrics of the same project are visible. ### Instrumenting Your Application To enable metric scraping, you must instrument the application to export metrics in a supported format. For details, see the [official Prometheus documentation](https://prometheus.io/docs/instrumenting/clientlibs/). After you add metrics support, use `ServiceMonitors` or `PodMonitors` to configure scraping. [ServiceMonitors](https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#servicemonitor) scrape all pods targeted by one or more services. Use this resource in most cases. Define a label selector in the `ServiceMonitor` to find the desired services. Create the `ServiceMonitor` in the same namespace as the service(s) it selects. In addition to the label selector, set the label **prometheus.nine.ch/\: scrape** with the name of the Metrics Agent instance. Consider the following example `ServiceMonitor` and `Service` definition: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: my-app namespace: my-app labels: prometheus.nine.ch/mymetricsagent01: scrape spec: selector: matchLabels: app: my-app endpoints: - port: web ``` ```yaml kind: Service apiVersion: v1 metadata: name: my-app-service namespace: my-app labels: app: my-app spec: selector: application: example-app ports: - name: web port: 8080 ``` The given `ServiceMonitor` definition selects the service "my-app-service" because the label "app: my-app" exists on that service. Metrics Agent then searches for all pods targeted by this service and scrapes them for metrics on port 8080 (the `ServiceMonitor` defines the port in the _endpoints_ field). [PodMonitors](https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#podmonitor) scrape all pods selected by the given label selector. This works similarly to the `ServiceMonitor` resource (but without an actual `Service` resource). Use the `PodMonitor` resource if the application does not need a `Service` resource (like some exporters). Run the pods in the same namespace where you define the `PodMonitor`. Here is an example for a `PodMonitor` with a corresponding pod: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: my-pods namespace: my-app labels: prometheus.nine.ch/mymetricsagent01: scrape spec: selector: matchLabels: application: my-app endpoints: - port: web ``` ```yaml apiVersion: v1 kind: Pod metadata: labels: application: my-app name: my-app namespace: my-app spec: containers: - image: mycompany/example-app name: app ports: name: web containerPort: 8080 ``` Based on the given `PodMonitor` resource, the Metrics Agent generates a scrape config which scrapes the shown pod "my-app" on port 8080 for metrics. Metrics Agent creates a _job_ for every `ServiceMonitor` or `PodMonitor` resource defined. The agent adds a _job_ label to all scraped metrics gathered in the corresponding job. Use this label to identify from which services or pods a given metric was scraped. ### Scraping External Targets {/* #external-target */} Use the [ScrapeConfig CRD](https://prometheus-operator.dev/docs/developer/scrapeconfig/) to scrape targets outside the Kubernetes cluster or to create scrape configurations not achievable with higher-level resources such as `ServiceMonitor` or `PodMonitor`. Currently, `ScrapeConfig` supports a limited set of service discovery mechanisms. Although numerous options are available (for a comprehensive list, refer to the [API documentation](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1alpha1.ScrapeConfig)), only `static_config` and `http_sd` configurations are currently supported. The CRD continually evolves (for now at the `v1alpha1` stage), and regularly adds new features and support for additional service discoveries. #### `static_config` Example The following example provides basic configuration and does not cover all supported options. For example, to scrape the target located at `http://metricsagent.demo.do.metricsagent.io:9090`, use the following configuration: ```yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: ScrapeConfig metadata: name: my-static-config namespace: my-namespace labels: prometheus.nine.ch/mymetricsagent01: scrape spec: staticConfigs: - labels: job: metricsagent targets: - metricsagent.demo.do.metricsagent.io:9090 ``` :::note Specify the target as a hostname, not as an HTTP(S) URL. For instance, to scrape the target located at `http://metricsagent.demo.do.metricsagent.io:9090`, enter `metricsagent.demo.do.metricsagent.io:9090` in the `targets` field. ::: For further details, refer to the [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config) and the [API documentation](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1alpha1.StaticConfig). #### `http_sd` Example HTTP-based service discovery provides a generic way to configure static targets and serves as an interface to plug in custom service discovery mechanisms. It fetches targets from an HTTP endpoint containing a list of zero or more `static_config`s. The target must meet the following requirements: - Reply with an `HTTP 200` response - Set the HTTP header `Content-Type` to `application/json` - Always respond with a `JSON` array, - Format the response in UTF-8 - If no targets exist, also emit HTTP 200 with an empty list `[]` - Target lists are unordered See [Requirements of HTTP SD endpoints](https://prometheus.io/docs/prometheus/latest/http_sd/#requirements-of-http-sd-endpoints) for more information. In general, the content of the answer is as follows: ```json [ { "targets": [ "", ... ], "labels": { "": "", ... } }, ... ] ``` Example response body: ```json [ { "targets": ["metricsagent.demo.do.metricsagent.io:9090"], "labels": { "job": "metricsagent", "__meta_test_label": "test_label1" } } ] ``` :::note The HTTP SD URL is not secret. Pass authentication and any API keys with the appropriate authentication mechanisms. Metrics Agent supports TLS authentication, basic authentication, OAuth2, and authorization headers. ::: The endpoint is queried periodically at the specified refresh interval. Return the whole list of targets on every scrape. Metrics Agent does not support incremental updates. A Metrics Agent instance does not send its hostname and SD endpoints cannot determine if the SD request is the first one after a restart. Each target has a meta label `__meta_url` during the [relabeling phase](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). The value contains the URL from which the target was extracted. A simple example: ```yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: ScrapeConfig metadata: name: my-http-sd namespace: my-namespace labels: prometheus.nine.ch/mymetricsagent01: scrape spec: httpSDConfigs: - url: http://my-external-api/discovery refreshInterval: 15s ``` Metrics Agent caches target lists and continues to use the current list if an error occurs while fetching an updated one. However, restarts do not preserve the targets list. Therefore, monitor HTTP service discovery (HTTP SD) endpoints for downtime. During a Metrics Agent restart, which may occur during regular maintenance windows, the agent clears the cache. If the HTTP SD endpoints are also down at this time, you may lose the endpoint target list. For more information, refer to the [Requirements of HTTP SD endpoints](https://prometheus.io/docs/prometheus/latest/http_sd/#requirements-of-http-sd-endpoints) documentation. For further details, refer to the [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) and the [API documentation](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1alpha1.HTTPSDConfig). ### Querying Metrics Use [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) to query for metrics. See [examples](https://prometheus.io/docs/prometheus/latest/querying/examples/) on the official Prometheus page. Query metrics using Grafana in the explore view. When using Grafana, select the data source matching your Metrics Agent instance. The data source name will be **\/**. ### Adding Rules Metrics Agent supports two kinds of rules: _recording rules_ and _alerting rules_. Both have a similar syntax, but a different use case. [Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules) calculate new metrics from existing ones. This is useful for computationally expensive queries in dashboards. To speed them up, create a recording rule that evaluates the query at a defined interval and stores the result as a new metric. Use this new metric in dashboard queries. [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) define alert conditions (based on PromQL). When those conditions are true, Metrics Agent sends an alert to the connected Alertmanager instances. Alertmanager then sends notifications to users about alerts. When creating alerting or recording rules, add the **prometheus.nine.ch/\: scrape** label with the name of the Metrics Agent instance. This assigns the created rule to the Metrics Agent instance. The following example alerting rule will alert once a job can not reach the configured pods (targets) anymore: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus.nine.ch/mymetricsagent01: scrape role: alert-rules name: jobs-check spec: groups: - name: ./example.rules rules: - alert: InstanceDown expr: up == 0 for: 5m labels: severity: Critical annotations: summary: "Instance {{ $labels.instance }} down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes." ``` This alerting rule definition triggers an alert once an _up_ metric gets a value of 0. The _up_ metric is special because Metrics Agent itself adds it for every job target (pod). When a pod can no longer be scraped, the corresponding _up_ metric turns to 0. If the _up_ metric is 0 for more than 5 minutes (in this case), Metrics Agent triggers an alert. Use the specified "labels" and "annotations" in Alertmanager to customize notification messages and routing decisions. See the full spec for the `PrometheusRule` definition in the [prometheus-operator documentation](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusrule). Here is an example of a recording rule: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus.nine.ch/mymetricsagent01: scrape role: recording-rules name: cpu-per-namespace-recording spec: groups: - name: ./example.rules rules: - record: namespace:container_cpu_usage_seconds_total:sum_rate expr: sum(rate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!="", container!="POD"}[5m])) by (namespace) ``` This recording rule creates a new metric called _namespace:container_cpu_usage_seconds_total:sum_rate_ which shows the sum of used CPU of all containers per namespace. This metric can easily be shown in a Grafana dashboard to have an overview about the CPU usage of all pods per namespace. The [kubernetes-mixins project](https://github.com/kubernetes-monitoring/kubernetes-mixin) contains sample alerts and rules for various exporters. It is a good place to get some inspiration for alerting and recording rules. ## Pricing Metrics Agent pricing consists of a fixed fee per instance, plus usage-based billing for the metrics it collects. Usage is measured using two components: - **Active series**: the number of unique time series currently receiving data. - **Data points per minute (DPM)**: the number of individual data points sent per minute. You are billed for the higher of the two values, active series or DPM, once usage exceeds the volume included with your instance. | Component | Price | Description | | ---------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------- | | Metrics Agent instance | | | | Usage, per metric | | | For exact pricing and included volume, see the [Price Calculator](https://calculator.nine.ch/?category=Nine+Kubernetes+Engine&segment=3%29+Services). ### Remote Write Metrics must reach Metrics Agent by being scraped, using `ServiceMonitors`, `PodMonitors`, or [`ScrapeConfigs`](#external-target) for external targets. Direct `remote_write` to nine's metrics platform is not supported - your Metrics Agent instance is the only supported ingestion point for metrics associated with your account. This also ensures your usage-based billing accurately reflects everything routed through your instance. ## Limitations Metrics Agent is limited to 100,000 unique time series per day. This limit prevents our Global Metrics cluster from becoming congested. ## Documentation and Links - [PrometheusRule documentation](https://prometheus.io/docs/) - [prometheus-operator project](https://github.com/coreos/prometheus-operator) ## Video Guide Check out our video guide series for GKE Application Monitoring. While the videos are done on our GKE product with Prometheus, the concepts are the same. --- ## Prometheus Prometheus is a component of NKE that allows you to monitor your applications. ## Availability :::warn The successor to Prometheus is [Metrics Agent](./metrics-agent) and it should be used instead for new setups. We are in transition towards Metrics Agent and will eventually sunset the Prometheus product. ::: Prometheus is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit. ## Usage Please see the following sections for an explanation on how to use Prometheus. ### General information about the setup When Prometheus is ordered, a new Prometheus instance with two replicas will be deployed in your NKE cluster in the `nine-system` namespace. The pods will run on the control-plane nodes, leaving your node pools fully available for your applications. Additionally, a new Grafana datasource will be created and automatically registered in your [Grafana](./grafana) instance (if you have one deployed). The Prometheus instance is based on the [prometheus-operator project](https://github.com/coreos/prometheus-operator). Therefore, you can use the following resources to create scraping configurations and recording/alerting rules: - ServiceMonitors - PodMonitors - PrometheusRules It is possible to run multiple Prometheus instances in your cluster if needed. ### Exporters and Metrics Also, Prometheus comes with some pre-configured metrics exporters: - CertManager - IngressNginx - NodeExporter - Kubelet - Kubelet cAdvisor - KubeStateMetrics - NineControllers - Velero You will need to tell us which of these exporters you want to enable. In the future, you will be able to enable them yourself in Cockpit. Note that enabling all metrics of an exporter will increase the resources required for Prometheus to run. To solve this, you can also limit the amount of metrics to track by explicitly giving us a list of the wanted metrics. To summarise, we recommend the following workflow: - Tell us which exporters to enable and we will enable all metrics for you of said exporter - Create your dashboards/rules - See which metrics you need and tell us. We will limit the scrape configuration to only scrape your needed metrics. ### API preparation steps {/* #api-preparation-steps */} In some of the following paragraphs you might need to use and `kubectl` to work with the nine Prometheus API resource. Before doing that, please make sure you are logged in to the API and selected a Prometheus instance to work on. After a successful login you will be able to use the **nineapis.ch** kubeconfig context to execute `kubectl` commands against the API. #### Login to the API 1. Make sure that you have `kubectl` and installed. 1. Authenticate with our API using : ```bash nctl auth login ``` 1. Now you can use `kubectl` commands ```bash kubectl --context nineapis.ch get prometheus.observability.nine.ch ``` #### Select a Prometheus resource All API commands will need a Prometheus resource name and project they can be applied to. We will export the name and project via environment variables for easier access in the examples. 1. List all your Prometheus instances and select one to work with. ```bash $ nctl get all --kinds=Prometheus -A PROJECT NAME KIND GROUP acme example Prometheus observability.nine.ch acme-prod sample Prometheus observability.nine.ch ``` 1. Export the name and project of the desired Prometheus instance via environment variables. ```bash export PROMNAME=example PROMPROJECT=acme ``` ### Accessing metrics There are different ways to view the collected metrics of your Prometheus instance which are explained in the following sections. #### Grafana This is our recommended way of viewing the metrics of your Prometheus instance(s). Just create a [Grafana instance](./grafana.md) and all your created Prometheus instances will automatically be configured as data sources in it. #### Prometheus Web-UI Every Prometheus instance provides a web UI which is accessible from outside of the cluster and secured by basic authentication. You will need to gather the connection details before accessing the web UI. 1. Make sure to execute the [API preparation steps](#api-preparation-steps). 1. Check if a connection secret reference was already set for your instance. A referenced connection secret will expose the URL, the basic auth username and the basic auth password. ```bash kubectl get prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ -o template --template={{.spec.writeConnectionSecretToRef}} ``` If you see an output like ``, you will need to set a connection secret reference. Otherwise you can just retrieve the URL and credentials as described further down. 1. If you need to set a connection secret reference, you can use `kubectl` for this. In the following example, we will use a secret 'my-prometheus-connection-details' for this, but you can choose your own name. Please make sure that the chosen name does not exist yet (you can use `kubectl --context=nineapis.ch get secret -n $PROMPROJECT` to check). ```bash kubectl patch prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ --type=merge -p '{"spec":{"writeConnectionSecretToRef":{"name":"my-prometheus-connection-details","namespace":"'$PROMPROJECT'"}}}' ``` 1. Once the connection secret reference was set, we can gather the connection details from the referenced secret. Please make sure to use the name of your chosen connection secret reference. ```bash kubectl get secret my-prometheus-connection-details \ --context nineapis.ch \ -n $PROMPROJECT \ -o jsonpath='{.data.url}' | base64 --decode ``` ```bash kubectl get secret my-prometheus-connection-details \ --context nineapis.ch \ -n $PROMPROJECT \ -o jsonpath='{.data.basicAuthUsername}' | base64 --decode ``` ```bash kubectl get secret my-prometheus-connection-details \ --context nineapis.ch \ -n $PROMPROJECT \ -o jsonpath='{.data.basicAuthPassword}' | base64 --decode ``` You can then access the Prometheus Web UI in a browser. Support in Cockpit is still in development. Documentation will be added soon. #### In-cluster access to Prometheus ##### Enable In-cluster access If you want to access the Prometheus instance from pods running within your NKE cluster, you will need to enable and configure internal cluster access first. As the internal cluster access will not use any authentication, it is disabled by default. One use case for in-cluster access is the operation of a self-managed Grafana instance for example. 1. Make sure to execute the [API preparation steps](#api-preparation-steps). 1. Enable and configure internal access to Prometheus. You can check first if the internal access was already enabled. ```bash $ kubectl get prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ -o jsonpath='{.spec.forProvider.access.internal}' {"enabled":false,"namespaceSelector":{},"podSelector":{}} ``` As you can see in above output, internal access was not yet enabled ("enabled" is set to `false`). 1. Enabling internal access will, by default, allow every pod of the NKE cluster to connect to Prometheus. You can restrict which pods are allowed to connect via the "namespaceSelector" and "podSelector" fields. They contain label key-value pairs which select the permitted namespace(s) and/or pods based on the labels set on these namespaces/pods. An empty "namespaceSelector" selects pods from all namespaces. An empty "podSelector" field selects all pods from the namespaces which have been selected by the "namespaceSelector" field. Further information for selectors and labels can be found in the [official Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). For example, to allow access from all pods running in a namespace "grafana" you can make use of the special Kubernetes label `kubernetes.io/metadata.name` which gets automatically attached to every Kubernetes namespace and contains the name of the namespace as a value. Here are some examples for enabling the internal access. ```bash # just enable access for all pods kubectl patch prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":true}}}}}' ``` ```bash # enable access for all pods in the namespace 'grafana' kubectl patch prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":true,"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"grafana"}}}}}}}' ``` ```bash # enable access for all pods which have a 'app:grafana' label in the namespace 'grafana' kubectl patch prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":true,"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"grafana"}},"podSelector":{"matchLabels":{"app":"grafana"}}}}}}}' ``` 1. Once you enabled the internal access, you can find the internal URL to access Prometheus in the status of the Prometheus resource itself. ```bash $ kubectl get prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ -o jsonpath='{.status.atProvider.internalURL}' http://prometheus-internal-3343791.nine-system.svc.cluster.local:9090 ``` You can then use this URL to access Prometheus from the permitted pods in your cluster. Support in Cockpit is still in development. Documentation will be added soon. ##### Disable in-cluster access to Prometheus In-cluster access can be disabled by using `kubectl`. 1. Make sure to execute the [API preparation steps](#api-preparation-steps). 1. Disable access to Prometheus. ```bash kubectl patch prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":false}}}}}' ``` Support in Cockpit is still in development. Documentation will be added soon. ##### Disable external access to Prometheus You might have reasons to switch off the external access to the Prometheus web UI. For example, if you run a self managed Grafana instance in your cluster, you will not access Prometheus from outside of the cluster. In these cases you can disable access to Prometheus from external sources. Please note, that [nine managed Grafana instances](./grafana.md) will also not have any access to your Prometheus instance once external access was disabled. 1. Make sure to execute the [API preparation steps](#api-preparation-steps). 1. Disable external access to Prometheus. ```bash kubectl patch prometheus.observability.nine.ch $PROMNAME \ --context nineapis.ch \ -n $PROMPROJECT \ --type=merge -p '{"spec":{"forProvider":{"access":{"noExternalAccess":true}}}}' ``` Support in Cockpit is still in development. Documentation will be added soon. ### Instrumenting your application Before Prometheus can scrape metrics from your application, you will need to instrument your application to export metrics in a special given format. You can find information about how to do this in the [official Prometheus documentation](https://prometheus.io/docs/instrumenting/clientlibs/). ### Adding application metrics to Prometheus Once your application supports metrics, you can use `ServiceMonitors` or `PodMonitors` to let Prometheus scrape your application's metrics. [ServiceMonitors](https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#servicemonitor) will scrape all pods which are targeted by one or more services. This resource needs to be used in most of the cases. You need to define a label selector in the `ServiceMonitor` which will be used to find all the wanted services. The `ServiceMonitor` should be created in the same namespace as the service(s) it selects. Next to the label selector your `ServiceMonitor` should also have the label **prometheus.nine.ch/\: scrape** set with the name of your Prometheus instance. Consider the following example `ServiceMonitor` and `Service` definition: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: my-app namespace: my-app labels: prometheus.nine.ch/myprom01: scrape spec: selector: matchLabels: app: my-app endpoints: - port: web ``` ```yaml kind: Service apiVersion: v1 metadata: name: my-app-service namespace: my-app labels: app: my-app spec: selector: application: example-app ports: - name: web port: 8080 ``` The given `ServiceMonitor` definition will select the service "my-app-service" because the label "app: my-app" exists on that service. Prometheus will then search for all pods which are targeted by this service and starts to scrape them for metrics on port 8080 (the `ServiceMonitor` defines the port in the _endpoints_ field). [PodMonitors](https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#podmonitor) will scrape all pods which are selected by the given label selector. It works very similar to the `ServiceMonitor` resource (just without an actual `Service` resource). You can use the `PodMonitor` resource if your application does not need a `Service` resource (like some exporters) for any other reason. The pods should run in the same namespace as the `PodMonitor` is defined. Here is an example for a `PodMonitor` with a corresponding pod: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: my-pods namespace: my-app labels: prometheus.nine.ch/myprom01: scrape spec: selector: matchLabels: application: my-app endpoints: - port: web ``` ```yaml apiVersion: v1 kind: Pod metadata: labels: application: my-app name: my-app namespace: my-app spec: containers: - image: mycompany/example-app name: app ports: name: web containerPort: 8080 ``` Based on the given `PodMonitor` resource the prometheus-operator will generate a scrape config which scrapes the shown pod "my-app" on port 8080 for metrics. Prometheus will create a _job_ for every `ServiceMonitor` or `PodMonitor` resource you define. It will also add a _job_ label to all scraped metrics which have been gathered in the corresponding job. This can be used to find out from which services or pods a given metric has been scraped. ### Use ScrapeConfig to scrape an external target {/* #external-target */} The [ScrapeConfig CRD](https://prometheus-operator.dev/docs/developer/scrapeconfig/) can be employed to scrape targets outside the Kubernetes cluster or to create scrape configurations that are not achievable with higher-level resources such as `ServiceMonitor` or `PodMonitor`. Currently, `ScrapeConfig` supports a limited set of service discovery mechanisms. Although numerous options are available (for a comprehensive list, refer to the [API documentation](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1alpha1.ScrapeConfig)), we currently only support `static_config` and `http_sd` configurations. The CRD is continually evolving (for now at the `v1alpha1` stage), with new features and support for additional service discoveries being added regularly. We need to carefully determine which fields will be useful and need to be maintained in the long term. :::note Adding targets to your Prometheus instance can impact resource usage. As the number of targets or the cardinality of metrics increases, it may become necessary to scale up the resources or number of management nodes in your cluster to accommodate the increased resource demands. ::: #### `static_config` example The following example provide basic configuration and do not cover all supported options. For example, to scrape the target located at `http://prometheus.demo.do.prometheus.io:9090`, use the following configuration: ```yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: ScrapeConfig metadata: name: my-static-config namespace: my-namespace labels: prometheus.nine.ch/myprom01: scrape spec: staticConfigs: - labels: job: prometheus targets: - prometheus.demo.do.prometheus.io:9090 ``` :::note The target must be specified as a hostname, not as an HTTP(S) URL. For instance, to scrape the target located at `http://prometheus.demo.do.prometheus.io:9090`, you should enter `prometheus.demo.do.prometheus.io:9090` in the `targets` field. ::: For further details, refer to the [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config) and the [API documentation](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1alpha1.StaticConfig). #### `http_sd` example HTTP-based service discovery provides a more generic way to configure static targets and serves as an interface to plug in custom service discovery mechanisms. It fetches targets from an HTTP endpoint containing a list of zero or more `static_config`s. The target must reply with an HTTP 200 response. The HTTP header `Content-Type` must be `application/json`, and the body must be valid JSON. The answer must be UTF-8 formatted. If no targets should be transmitted, HTTP 200 must also be emitted, with an empty list `[]`. Target lists are unordered. See [Requirements of HTTP SD endpoints](https://prometheus.io/docs/prometheus/latest/http_sd/#requirements-of-http-sd-endpoints) for more information. In general, the content of the answer is as follows: ```json [ { "targets": [ "", ... ], "labels": { "": "", ... } }, ... ] ``` Example response body: ```json [ { "targets": ["prometheus.demo.do.prometheus.io:9090"], "labels": { "job": "prometheus", "__meta_test_label": "test_label1" } } ] ``` :::note The URL to the HTTP SD is not considered secret. The authentication and any API keys should be passed with the appropriate authentication mechanisms. Prometheus supports TLS authentication, basic authentication, OAuth2, and authorization headers. ::: The endpoint is queried periodically at the specified refresh interval. The whole list of targets must be returned on every scrape. There is no support for incremental updates. A Prometheus instance does not send its hostname and it is not possible for a SD endpoint to know if the SD requests is the first one after a restart or not. Each target has a meta label `__meta_url` during the [relabeling phase](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). Its value is set to the URL from which the target was extracted. A simple example: ```yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: ScrapeConfig metadata: name: my-http-sd namespace: my-namespace labels: prometheus.nine.ch/myprom01: scrape spec: httpSDConfigs: - url: http://my-external-api/discovery refreshInterval: 15s ``` Prometheus caches target lists and continues to use the current list if an error occurs while fetching an updated one. However, the targets list is not preserved across restarts. Therefore, it is crucial to monitor your HTTP service discovery (HTTP SD) endpoints for downtime. During a Prometheus restart, which may occur during our regular maintenance window, the cache will be cleared. If the HTTP SD endpoints are also down at this time, you may lose the endpoint target list. For more information, refer to the [Requirements of HTTP SD endpoints](https://prometheus.io/docs/prometheus/latest/http_sd/#requirements-of-http-sd-endpoints) documentation. For further details, refer to the [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) and the [API documentation](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1alpha1.HTTPSDConfig). ### Querying for metrics You can use [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) to query for metrics. There are some [examples](https://prometheus.io/docs/prometheus/latest/querying/examples/) on the official Prometheus page. Querying can be done by using Grafana in the explore view. When using Grafana please make sure to select the data source matching your Prometheus instance. The data source name will be **\/\/prometheus**. ### Estimating Metrics Agent Usage If you are considering migrating to [Metrics Agent](./metrics-agent), you can estimate your expected usage and cost beforehand by running the following queries against your Prometheus instance. Metrics Agent bills the higher of two values, **Active Series** and **Data Points per Minute (DPM)**. See [Metrics Agent Pricing](./metrics-agent#pricing) for how usage translates into cost. **Active Series** shows the number of currently active time series: ```promql max(max_over_time(prometheus_tsdb_head_series[1h])) ``` **DPM** shows the average number of samples ingested per minute: ```promql avg_over_time( ( sum(rate(prometheus_tsdb_head_samples_appended_total[5m])) * 60 )[$__range:5m] ) ``` Run both queries in the [Grafana](#grafana) explore view, where `$__range` automatically adapts to the selected time range. ### Adding rules to Prometheus Prometheus supports two kinds of rules: _recording rules_ and _alerting rules_. Both have a similar syntax, but a different use case. [Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules) can be used to calculate new metrics from already existing ones. This can be useful if you use computationally expensive queries in dashboards. To speed them up you can create a recording rule which will evaluate the query in a defined interval and stores the result as a new metric. You can then use this new metric in your dashboard queries. [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) allow you to define alert conditions (based on PromQL). When those conditions are true, Prometheus will send out an alert to the connected Alertmanager instances. Alertmanager will then send notifications to users about alerts. When creating alerting or recording rules, please make sure to add the **prometheus.nine.ch/\: scrape** label with the name of your Prometheus instance. This will assign the created rule to your Prometheus instance. The following example alerting rule will alert once a job can not reach the configured pods (targets) anymore: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus.nine.ch/myprom01: scrape role: alert-rules name: jobs-check spec: groups: - name: ./example.rules rules: - alert: InstanceDown expr: up == 0 for: 5m labels: severity: Critical annotations: summary: "Instance {{ $labels.instance }} down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes." ``` This alerting rule definition will trigger an alert once an _up_ metric gets a value of 0. The _up_ metric is a special metric as it will be added by Prometheus itself for every job target (pod). Once a pod can not be scraped anymore, the corresponding _up_ metric will turn to 0. If the _up_ metric is 0 for more than 5 minutes (in this case), Prometheus will trigger an alert. The specified "labels" and "annotations" can be used in Alertmanager to customize your notification messages and routing decisions. For the full `PrometheusRule` spec, see the [prometheus-operator API reference](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusrule). Here is an example of a recording rule: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus.nine.ch/myprom01: scrape role: recording-rules name: cpu-per-namespace-recording spec: groups: - name: ./example.rules rules: - record: namespace:container_cpu_usage_seconds_total:sum_rate expr: sum(rate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!="", container!="POD"}[5m])) by (namespace) ``` This recording rule will create a new metric called _namespace:container_cpu_usage_seconds_total:sum_rate_ which shows the sum of used CPU of all containers per namespace. This metric can easily be shown in a Grafana dashboard to have an overview about the CPU usage of all pods per namespace. The [kubernetes-mixins project](https://github.com/kubernetes-monitoring/kubernetes-mixin) contains sample alerts and rules for various exporters. It is a good place to get some inspiration for alerting and recording rules. ## Documentation and Links - [Prometheus documentation](https://prometheus.io/docs/) - [prometheus-operator project](https://github.com/coreos/prometheus-operator) ## Video Guide Checkout our video guide series for GKE Application Monitoring. While the videos are done on our GKE product, the concepts are the same. --- ## Quota System Since all resources in and around NKE are self-serviceable, there is a quota system in place to safeguard against accidental creation of many costly resources and to help Nine with resource planning. The quota system applies to all services that can be created in Cockpit and also some resources which can be directly consumed on an NKE cluster, such as load balancers and persistent storage. In case you run into an error provisioning a resource because of a quota limitation, please [contact us](/docs/general/contact) and we will look into increasing the quota for you. --- ## Sealed Secrets(Nke) _Sealed Secrets_ encrypts Kubernetes _Secrets_ so you can store them in git without any worries. ## Details Usually the content of Kubernetes _Secrets_ definitions is unencrypted which means it is not recommended to store them alongside other Kubernetes definitions in version control or anywhere that is not a secured environment. This adds manual and error-prone steps to your application deployment. As a solution to this, we are running [a controller](https://github.com/bitnami-labs/sealed-secrets) that will take care of decrypting your _Sealed Secrets_ and turning them into normal _Secrets_ objects. ## Availability _Sealed Secrets_ is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit. ## Scopes The _Scope_ is nothing but the context of a sealed secret within a Kubernetes cluster. The _Scope_ of a _Sealed Secret_ relates to where and how the _Sealed Secret_ can be decrypted and used within your cluster. These are the possible _Scopes_: - `strict` (default): the secret must be sealed with exactly the same name and namespace. These attributes become part of the encrypted data and thus changing name and/or namespace would lead to a decryption error. - `namespace-wide`: you can freely rename the _Sealed Secret_ within a given namespace. - `cluster-wide`: the secret can be unsealed in any namespace and can be given any name. By default, `strict` _Scope_ is selected unless you pass the `--scope` flag to kubeseal CLI with a different value. It's also possible to request a _Scope_ via `annotations` in the input secret you pass to kubeseal. Please refer to [Scopes documentation](https://github.com/bitnami-labs/sealed-secrets?tab=readme-ov-file#scopes) for more details. ## Usage ### Strict Scope (default) In order to use sealed-secrets, you need to install the CLI-utility `kubeseal` which is part of the [sealed-secrets](https://github.com/bitnami-labs/sealed-secrets#installation) project. After you installed `kubeseal` for your OS you can start do encrypt secrets locally. 1. Define your normal unencrypted secret in a local file named `secret.yaml`. ```yaml title="secret.yaml" apiVersion: v1 kind: Secret metadata: name: example namespace: dev type: Opaque stringData: password: verysecure ``` 1. Use kubeseal to generate an encrypted _SealedSecret_ resource. ```bash $ kubeseal --controller-namespace nine-system < secret.yaml > sealed-secret.json ``` > Note: the current kube context needs to be set to the NKE cluster, > alternatively the kubeconfig and context can be provided with additional > options to kubeseal. 1. Apply it via `kubectl`. ```bash $ kubectl apply -f sealed-secret.json sealedsecret.bitnami.com/example created ``` 1. Read back the _Secret_ resource that the controller created for us. ```bash $ kubectl get secret example -o jsonpath='{.data.password}' | base64 -d verysecure ``` To delete the _Secret_ again, you can just delete the _SealedSecret_ and the controller will also remove the _Secret_ object. ```bash $ kubectl delete sealedsecret example sealedsecret.bitnami.com "example" deleted ``` Note that in a production scenario we do not recommend you to apply the _SealedSecret_ locally with `kubectl`, but instead store it in your configuration repository and let [Argo CD](./argocd#web-ui) take care of creating it. ### Cluster-wide Scope The procedure is exactly the same as in the case of the [Strict Scope](#strict-scope-default). Just pass `--scope cluster-wide` to kubeseal CLI (or use `annotations`). ```bash $ kubeseal --scope cluster-wide --controller-namespace nine-system < secret.yaml > sealed-secret.json ``` ## Documentation and Links - [Sealed Secrets Documentation](https://github.com/bitnami-labs/sealed-secrets#overview) --- ## Security Concepts ## Kubernetes Distribution An NKE cluster is based on a [Rancher Kubernetes Engine 2 (RKE2)](https://docs.rke2.io/) cluster. RKE2 is a CNCF-certified Kubernetes distribution which eases the installation and update of the whole Kubernetes cluster. ## Operating System Nine uses [Flatcar OS](https://www.flatcar.org/) as the underlying Linux operating system on each cluster node. From the [FAQ](https://www.flatcar.org/faq#why-use-a-container-linux-instead-of-a-general-purpose-linux-distribution) of Flatcar OS: > The OS image shipped by Flatcar Container Linux includes just the minimal > amount of tools to run container workloads. This means that the attack surface > is significantly reduced. On top of this, as the OS image is immutable (/usr is > a read-only partition and there's no package manager to install packages), > which means there's less chance of both accidental and intentional breakage. ### Upgrades Nine provides periodic upgrades of new operating system images on NKE cluster nodes. These upgrades are automatically rolled out in staged phases on all NKE clusters in the [weekly maintenance window](../../general/weekly-maintenance-window). ## Networking Nine uses [Cilium](https://cilium.io/) as the networking provider in NKE clusters. Cilium supports Kubernetes `NetworkPolicy` resources to secure inbound and outbound network traffic. ### Firewall NKE cluster nodes have a publicly reachable IP address assigned by default. Nine restricts access to certain services running on the nodes of an NKE cluster. This includes SSH access, which is only permitted via special VPN servers managed by Nine. ## Permissions ### Authentication Nine provides central authentication for managed applications. Services, like [Grafana](./grafana), [Argo CD](./argocd) or the Kubernetes API server itself, are secured via OIDC. In addition to centralized management for user access, OIDC also allows you to set up two-factor authentication (2FA). ### Authorization Nine provides default cluster-wide [RBAC (role-based access control)](https://en.wikipedia.org/wiki/Role-based_access_control) roles which can be assigned to users or service accounts: | Name | Description | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | admin | Specifies admin permissions to a cluster, meaning that the subject will be able to create new namespaces, update these and also delete these user created namespaces and all resources contained in them. Access to certain namespaces cannot be revoked. | | viewer | Specifies viewer permissions to a cluster, meaning that the user will be able to view all resources on the cluster, except secrets. More permissions to specific namespaces can be granted through RBAC. | | user | Specifies user permissions to a cluster, meaning that the user can create namespaces, delete these owned namespaces and can view secrets in owned namespaces. | ### Cluster Admin Administrators in NKE have restricted permissions compared to a full `cluster-admin`. All permissions are given in namespaces only. Administrators can create, update, and delete all namespaces which are not owned by Nine, but they do **not** have unrestricted cluster wide `cluster-admin` access. For example the creation, modification and deletion of Kubernetes `ClusterRoleBindings` is not allowed. #### Custom Resource Definitions (CRDs) Installing CRDs is currently not possible on NKE clusters for the following reasons: - **Conflicts:** CRDs are cluster-wide resources, which creates a risk of conflicts with pre-installed CRDs or those used by managed add-ons. - **Security Permissions:** CRDs are often paired with Controllers or Operators requiring broad cluster-wide permissions (such as reading all secrets). Granting these permissions could compromise the security of Nine's management components (e.g., alerting systems) running on the cluster. - **Managed Approach:** We aim to provide popular services requiring CRDs as managed add-ons to handle maintenance and updates for you. We recognize the growing importance of CRDs and the requirement for custom controllers. If your use case requires specific CRDs, please reach out to us at so we can better understand your needs. ## Managed Applications Nine provides certain managed applications which help to increase the security when using NKE. Examples are: - [Container (OCI) Registry](./container-oci-registry) for storing private container images and Helm charts. - [Sealed Secrets](./sealed-secrets) for securely storing secrets in a Git repository. - [Automatic TLS Certificates](./ingress#automatic-tls-certificates) provided by [cert-manager](https://cert-manager.io/docs/). - [Audit Logging](./audit-logging-on-nke) enables Kubernetes auditing for NKE. --- ## Service Connections A `ServiceConnection` creates an encrypted, authenticated tunnel between a source and a destination service. It allows applications running in NKE clusters or Deploio to reach [On-Demand Services](../../on-demand-services/index.md) over a private network instead of the public internet. ## Sources The following sources are supported: - [NKE Kubernetes Clusters](./index.md) (`kubernetescluster`) - [Deploio](../../deplo-io) applications (`application`) ## Destinations The following destinations are supported: - [Key-Value Store](../../on-demand-services/key-value-store.md) (`keyvaluestore`) - [MySQL](../../on-demand-services/mysql/index.md) (`mysql`, `mysqldatabase`) - [PostgreSQL](../../on-demand-services/postgresql/index.md) (`postgres`, `postgresdatabase`) The destination must be in the same project as the service connection. ## Create a Service Connection {/* TODO add instructions on enabling service connectiosn for NKE clusters. */} Service Connections are not yet available in Cockpit. Please use for now. Use `nctl create serviceconnection` with `--source` and `--destination` in the format `kind/name`: ```bash nctl create serviceconnection my-connection \ --source=kubernetescluster/my-cluster \ --destination=keyvaluestore/my-kvs ``` ## Connect to a Service over a Service Connection After configuring a service connection, you can connect to the destination using the private FQDN of the service: Service Connections are not yet available in Cockpit. Please use for now. ```bash nctl get keyvaluestore my-kvs \ -o yaml ``` Make note of the `privateNetworkingFQDN` field in the output. You can use this FQDN to connect to the service. ## Restrict Access When using a [NKE Kubernetes Cluster](./index.md) (`kubernetescluster`) as a source, you can restrict access to specific pods or namespaces: ### Restrict Access by Pod By default, all pods in the source cluster can use the connection. To restrict access to specific pods, provide a label selector: Service Connections are not yet available in Cockpit. Please use for now. Use `--source-pod-selector`: ```bash nctl create serviceconnection \ --source=kubernetescluster/my-cluster \ --destination=keyvaluestore/my-kvs \ --source-pod-selector='app=my-app,env=production' ``` ### Restrict Access by Namespace To limit which namespaces the connection is available in, provide a namespace label selector: In the **Create Service Connection** form, enter a label selector in the **Namespace Selector** field, for example: `kubernetes.io/metadata.name=production` Use `--source-namespace-selector`: ```bash nctl create serviceconnection \ --source=kubernetescluster/my-cluster \ --destination=keyvaluestore/my-kvs \ --source-namespace-selector='kubernetes.io/metadata.name=production' ``` When both selectors are set, only pods matching the pod selector within namespaces matching the namespace selector can use the connection. ## Service Connections for Deploio Applications If you use Deploio, Nine can create and manage service connections automatically when you configure [service references](../../deplo-io/configuration/connecting-to-services.md). --- ## Static Egress The static egress feature allows your Pods to have a static, predictable outgoing IP address. This IP address can then be used in remote systems to allow/deny traffic emitted from the selected Pods. ## Details By default, Pods running in NKE clusters will use the IP address of the node they are running on as source IP when initiating connections to external (out of cluster) systems. As the IP addresses of nodes can change and also because Pods can be scheduled onto different nodes, restricting traffic from Pods at external systems is very hard. The only possibility so far was to allow traffic from all NKE subnets which are in use. The static egress feature overcomes this limitation. It allows Pods, which have a specific label set, to use a static egress IP. All egress traffic made by these Pods will be routed via one node in the NKE cluster and be emitted with a static source IP. All other Pods (which don't have the specific label set) in the cluster will continue to use the IP of the node they are running on for egressing traffic. This way, traffic from specific Pods in the cluster can be allowed/denied on external systems (firewalls, etc). If the node which was selected for the static egress traffic is getting replaced during our maintenance window, a new node will be automatically selected and static egress traffic will be routed over the new node. Only nodes from the "nine node pool" will be selected for static egress traffic. ## Availability The static egress feature is currently available for the following products: - NKE clusters - vClusters - Deploio applications ## Usage The configuration of the static egress feature differs slightly, depending on the product in use. ### NKE clusters To enable the feature on NKE clusters, you will first need to enable the static egress runtime on the NKE cluster itself. The runtime will select the egressing node from the "nine" node pool and prepare all network configurations. Please find the corresponding commands for enabling the runtime below. Enabling the runtime is a requirement for static egress to work in NKE clusters. #### Enabling the Static Egress Runtime {/* #enable-static-egress-runtime */} 1. Make sure that you have `kubectl` and installed. 1. Authenticate with our API using : ```bash nctl auth login ``` 1. List all your clusters to find your desired NKE cluster. ```bash nctl get clusters -A ``` 1. Once you found the name of your NKE cluster you can use `kubectl` with the **nineapis.ch** context to enable the runtime: ```bash kubectl --context=nineapis.ch patch kubernetescluster -n \ --type=merge -p '{"spec":{"forProvider":{"nke":{"staticEgress":{"enabled":true}}}}}' ``` Go to your Kubernetes Cluster and then to the `Static Egress` tab. You can then enable Static Egress for the cluster by toggling the checkbox slider `Activate Static Egress`. #### Create a Static Egress Resource {/* #create-static-egress-resource */} After the runtime got enabled, a static egress resource which targets your cluster can be created. This will result in the automatic creation of a static egress IP and a corresponding Pod label. You can then add this label to all the Pods which should make use of the static egress IP. 1. Make sure that you followed the API authentication steps from above and have `kubectl` and installed. 1. List all your clusters. ```bash nctl get clusters -A ``` 1. Create the following static egress resource definition in a `static-egress.yaml` file and make sure to replace the content of the placeholders with your corresponding values. ```yaml title="static-egress.yaml" apiVersion: networking.nine.ch/v1alpha1 kind: StaticEgress metadata: name: my-static-egress namespace: spec: forProvider: disabled: false target: group: infrastructure.nine.ch kind: KubernetesCluster name: ``` 1. Use to apply the file to the API. ```bash nctl apply -f static-egress.yaml ``` 1. After a few seconds an IP address and a Pod label should be selected. ```bash $> kubectl --context nineapis.ch get staticegress my-static-egress -n -o yaml ... status: atProvider: address: selectionLabel: name: networking.nine.ch/static-egress value: ``` After activating Static Egress, a `Create` button will appear, and you can create your Static Egress via the form. 1. Make sure that you have installed. 1. Authenticate with the API: ```bash nctl auth login ``` 1. Set the project that contains the target cluster: ```bash nctl auth set-project ``` 1. Create a static egress resource targeting your cluster: ```bash nctl create staticegress my-static-egress --cluster ``` 1. After a few seconds, list the static egress resources to see the assigned IP address: ```bash nctl get staticegress my-static-egress ``` #### Assigning the Static Egress Label to Pods {/* #assign-static-egress-label */} Lets assume the static egress configuration selected the label `networking.nine.ch/static-egress: production-egress` for static egress pod selection. If you already have a Kubernetes Deployment called "production" which deploys your production application, you can attach the label to all running pods of that Deployment by: 1. creating a file `label-patch.yaml` with the following content: ```yaml title="label-patch.yaml" spec: template: metadata: labels: networking.nine.ch/static-egress: production-egress ``` 2. and patching your Kubernetes Deployment "production": ```shell-session kubectl patch deploy production --patch-file=label-patch.yaml ``` This will rollout new pods which will all have the special static egress label attached. All egressing traffic of those Pods will now make use of the static egress IP. If you remove the label again, the pods will be using the default node IPs instead. ### vClusters The usage of static egress for vClusters is very similar to the NKE cluster one. The only difference is that you do not need to enable the static egress runtime (as the runtime gets automatically enabled). You can just [create static egress resources](#create-static-egress-resource) and attach the selected Kubernetes label to your Pods as [described in the example](#assign-static-egress-label). ### Prometheus When using Prometheus [to scrape external targets](./prometheus#external-target), it can be useful when the requests are coming from a static IP Address. First, make sure to enable the [static egress runtime](#enable-static-egress-runtime) on the cluster. After that you can create the static egress pointing to your Prometheus instance. ```yaml apiVersion: networking.nine.ch/v1alpha1 kind: StaticEgress metadata: name: my-static-egress namespace: spec: forProvider: disabled: false target: group: observability.nine.ch kind: Prometheus name: ``` Static Egress for Prometheus is not yet configurable in Cockpit. ### Deploio applications The static-egress configuration for Deploio applications is very easy. Just create a static egress resource and your Deploio application will use the selected IP for egress communication. #### Create a Static Egress Resource {/* #deploio-create-static-egress-resource */} 1. Make sure that you have `kubectl` and installed. 1. Authenticate with the API using : ```bash nctl auth login ``` 1. List all your Deploio applications. ```bash nctl get apps -A ``` 1. Create the following static egress resource definition in a `deploio-static-egress.yaml` file and make sure to replace the content of the placeholders with your corresponding values. ```yaml title="deploio-static-egress.yaml" apiVersion: networking.nine.ch/v1alpha1 kind: StaticEgress metadata: name: my-deploio-static-egress namespace: spec: forProvider: disabled: false target: group: apps.nine.ch kind: Application name: ``` 1. Use to apply the file to the API. ```bash nctl apply -f deploio-static-egress.yaml ``` 1. After a few seconds an IP address should be selected and your Deploio application will use it automatically for egress traffic (no further configuration required). ```bash $> kubectl --context nineapis.ch get staticegress my-deploio-static-egress -n -o yaml ... status: atProvider: address: ``` Go to your application and then to the `Static Egress` tab. You can then create a Static Egress resource by clicking the `Activate Static Egress` button. 1. Make sure that you have installed. 1. Authenticate with the API: ```bash nctl auth login ``` 1. Set the project that contains the target application: ```bash nctl auth set-project ``` 1. Create a static egress resource targeting your application: ```bash nctl create staticegress my-deploio-static-egress --application ``` 1. After a few seconds, list the static egress resources to see the assigned IP address: ```bash nctl get staticegress my-deploio-static-egress ``` ### Disable a Static Egress Resource {/* #disable-a-static-egress-resource */} You can temporarily disable a static egress resource. Contrary to the [deletion of a static egress resource](#remove-a-static-egress-resource), this will not delete the automatically selected egress IP address which can be reused after you activated the static egress again. 1. Make sure that you have `kubectl` and installed. 1. Authenticate with the API using : ```bash nctl auth login ``` 1. List all your static egress resources to select the one which you want to temporarily disable. ```bash nctl get staticegress ``` 1. Patch the static egress resource with `kubectl`. ```bash kubectl --context=nineapis.ch patch staticegress -n \ --type=merge -p '{"spec":{"forProvider":{"disabled":true}}}' ``` **For Kubernetes Cluster and vcluster:** On the Kubernetes Cluster page, go to the `Static Egress` tab, choose your Static Egress resource and click the `Edit` button. You can toggle the `Disabled` checkbox slider, in order to disable the Static Egress resource. Click `Save` to apply the changes. **For Deplo.io applications:** On the Application page, go to the `Static Egress` tab. You can toggle the `Disabled` checkbox slider. The changes are applied automatically. 1. Make sure that you have installed. 1. Authenticate with the API: ```bash nctl auth login ``` 1. Set the project that contains the static egress resource: ```bash nctl auth set-project ``` 1. List the static egress resources in your current project: ```bash nctl get staticegress ``` 1. Disable the static egress resource: ```bash nctl update staticegress --disabled ``` To re-enable a disabled static egress resource: ```bash nctl update staticegress --no-disabled ``` ### Remove a Static Egress Resource {/* #remove-a-static-egress-resource */} If you want to remove the static egress feature again, you can just delete the corresponding static egress resource from the API. This will also delete the automatically selected egress IP address. 1. Make sure that you have `kubectl` and installed. 1. Authenticate with the API using : ```bash nctl auth login ``` 1. List all your static egress resources to select the one which you want to delete. ```bash nctl get staticegress ``` 1. Delete the static egress resource with `kubectl`. ```bash kubectl --context nineapis.ch delete staticegress -n ``` In order to remove the Static Egress, simply click the `Delete` button on the Static Egress page. 1. Make sure that you have installed. 1. Authenticate with the API: ```bash nctl auth login ``` 1. Set the project that contains the static egress resource: ```bash nctl auth set-project ``` 1. List the static egress resources in your current project: ```bash nctl get staticegress ``` 1. Delete the static egress resource: ```bash nctl delete staticegress ``` --- ## Tempo [Tempo](https://grafana.com/oss/tempo/) is a distributed tracing backend which allows you to store traces emitted by your applications. A direct integration into Grafana allows to query and search for specific traces. ## Details For customers who want to use distributed tracing in their applications, we provide Tempo as a storage backend. Application code can be instrumented by using the [OpenTelemetry framework](https://opentelemetry.io). Traces can be sent via gRPC or HTTPS to the deployed Tempo instance. All stored traces/spans can then be queried with the help of a deployed [Grafana](grafana.md) instance. ### Architecture ## Availability Tempo is available as an optional service and can be deployed using Cockpit. It can be used to store traces sent by applications which are running on Nine Kubernetes Engine (NKE) or are deployed externally. When deploying a [nine managed Grafana instance](grafana.md), a corresponding Tempo data source will be automatically configured in it. It is still possible to query the Tempo instance from an external running Grafana. ## Usage If your application code is not instrumented to emit traces yet, please visit the [OpenTelemetry getting started guides](https://opentelemetry.io/docs/getting-started/dev/), which provide an excellent introduction to the API and SDK for different programming languages. You will need to use an [OTLP exporter](https://opentelemetry.io/docs/reference/specification/protocol/exporter/) to send traces to Tempo. The configuration depends on where the application is running. ### Sending traces from applications running on NKE Applications running on NKE can make use of the automatically deployed [OpenTelemetry collector](https://opentelemetry.io/docs/collector/). The collector allows to collect all traces before sending them batched to the Tempo instance. Tempo authentication information is injected automatically on the collector. You only need to configure the OTLP endpoint by pointing it to the collector URL displayed in Cockpit. You can use the [predefined OTEL_EXPORTER_OTLP_ENDPOINT environment variable](https://opentelemetry.io/docs/reference/specification/protocol/exporter/#configuration-options) for this. ### Sending traces from applications not running on NKE Applications which are not deployed on a NKE cluster need to send traces directly to the deployed Tempo instance. You can send traces either via gRPC or HTTPS. The Tempo URL can be found in Cockpit. Please make sure to integrate the basic authentication credentials for your Tempo instance which are also shown in Cockpit. Here is an example on how to configure an OTLP gRPC exporter in golang: ```go package example import ( "context" "crypto/tls" "encoding/base64" "fmt" "go.opentelemetry.io/otel/exporters/otlp/otlptrace" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" "google.golang.org/grpc" "google.golang.org/grpc/credentials" ) func grpcExporter(ctx context.Context, username, password, host string) (*otlptrace.Exporter, error) { conn, err := grpc.DialContext( ctx, fmt.Sprintf("dns:%s", host), grpc.WithBlock(), // we use a default tls.Config, which enforces TLS verification. grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})), ) if err != nil { return nil, fmt.Errorf("error dialing gRPC: %w", err) } auth := username + ":" + password return otlptracegrpc.New( ctx, otlptracegrpc.WithGRPCConn(conn), otlptracegrpc.WithHeaders( map[string]string{ "Authorization": "Basic " + base64.StdEncoding.EncodeToString([]byte(auth)), }, ), ) } ``` ### Viewing traces in Grafana To view the sent traces in Grafana you will need to: 1. Select the "explore" mode in the left navigation 2. Select the Tempo data source at the top 3. Select the "Search" query type 4. Further select traces by filtering them with the available fields ("Service Name", "Span Name", etc) 5. Click "Run Query" in the upper right corner 6. Select a trace ID to display the full trace and all spans for it ![grafana-steps-to-view-traces](/img/tempo-grafana-integration.png) ## Documentation and Links - [OpenTelemetry getting started](https://opentelemetry.io/docs/getting-started/dev) - [OTLP exporter configuration](https://opentelemetry.io/docs/reference/specification/protocol/exporter/#configuration-options) - [OpenTelemetry collector](https://opentelemetry.io/docs/collector/) --- ## Run Ruby Applications This article explains how to run Ruby apps on Nine's managed servers. ## Requirements For the example below, we assume that your application supports a Ruby web server like [puma](https://puma.io/), [thin](https://github.com/macournoyer/thin), etc. Rails comes pre-bundled with `puma`. Therefore we'll use that in this example. ## 1. Install the Ruby Version Manager We recommend using a version manager like [`rbenv`](https://github.com/rbenv/rbenv) to be able to use different Ruby versions than provided by the distribution release of Ubuntu. It also allows you to use different versions per project. To install `rbenv`: ```shell-session curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash echo 'eval "$(rbenv init -)"' >> ~/.bashrc source ~/.bashrc ``` ## 2. Install Ruby Version The desired Ruby version can now be installed: ```shell-session rbenv install --list # list installable versions rbenv install 3.2.0 ``` Update your Bundler to the latest version: ```shell-session rbenv exec bundle update --bundler ``` ## 3. Deploy Application Deploying your application is different for each project. With Rails, you need to compile assets, upload them and the code to the server, and install Ruby dependencies: > Run these commands on your local machine Compile assets: ```shell-session RAILS_ENV=production bundle exec rake assets:precompile ``` Copy files to server: ```shell-session rsync -a -v --delete --exclude='node_modules/*' --exclude='tmp/*' --exclude='vendor/*' --exclude='.git/*' ./ www-data@server.nine.ch:app/current ``` Ensure dependencies are up-to-date: ```shell-session ssh www-data@server.nine.ch 'cd ~/app/current && BUNDLER_WITHOUT="development test" rbenv exec bundle install' ``` ## 4. Setup systemd Service Use a systemd user service to keep a service running if it fails or on a server restart. This example loads environment variables from `~/app/env` and then executes `rails server -b localhost --log-to-stdout` in the directory `~/app/current`. The commands depend on your setup. The above example assumes that you're running a Rails application. Create the service file in `~/.config/systemd/user/rails-app.service`. ```systemd title="~/.config/systemd/user/rails-app.service" [Unit] Description=Application [Service] Type=simple WorkingDirectory=%h/app/current Environment=RAILS_ENV=production Environment=PORT=3000 EnvironmentFile=%h/app/env ExecStart=%h/.rbenv/bin/rbenv exec bundle exec rails server -b localhost --log-to-stdout TimeoutSec=15 Restart=on-failure PrivateTmp=yes ProtectSystem=full [Install] WantedBy=default.target ``` And then start the service: ```shell-session touch ~/app/env # ensure environment file exists systemctl --user daemon-reload systemctl --user enable rails-app.service systemctl --user start rails-app.service ``` Your application should now be registered as service and already running: ``` $ systemctl --user status rails-app.service ● rails-app.service - Application Loaded: loaded (/home/www-data/.config/systemd/user/rails-app.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-07-16 10:13:36 CEST; 5s ago Main PID: 3615317 (ruby) CGroup: /user.slice/user-33.slice/user@33.service/rails-app.service └─3615317 puma 5.3.2 (tcp://localhost:3000) [current] Jul 16 10:13:37 server rbenv[3615317]: => Run `bin/rails server --help` for more startup options Jul 16 10:13:37 server rbenv[3615317]: Puma starting in single mode... Jul 16 10:13:37 server rbenv[3615317]: * Puma version: 5.3.2 (ruby 3.0.2-p107) ("Sweetnighter") Jul 16 10:13:37 server rbenv[3615317]: * Min threads: 5 Jul 16 10:13:37 server rbenv[3615317]: * Max threads: 5 Jul 16 10:13:37 server rbenv[3615317]: * Environment: production Jul 16 10:13:37 server rbenv[3615317]: * PID: 3615317 Jul 16 10:13:37 server rbenv[3615317]: * Listening on http://127.0.0.1:3000 Jul 16 10:13:37 server rbenv[3615317]: * Listening on http://[::1]:3000 Jul 16 10:13:38 server rbenv[3615317]: Use Ctrl-C to stop ``` ### Troubleshooting ## 5. Configure Webserver ## Automate Deployment To automate the deployment of your Ruby application, you can use [Capistrano](https://capistranorb.com/), a remote server automation tool. Below are the steps to set up and use Capistrano for deployment: ### Install Capistrano Add Capistrano to your `Gemfile`: ```ruby group :deployment do gem 'capistrano', require: false gem 'capistrano-rbenv', require: false gem 'capistrano-rbenv-install', require: false gem 'capistrano-rails', require: false gem 'capistrano-systemd-multiservice', require: false end ``` Then run: ```shell-session bundle install cap install ``` This will create several default configuration files for Capistrano in your project. ### Configure Capistrano Edit the `Capfile` to include the necessary Capistrano plugins: ```ruby title="Capfile" require 'capistrano/setup' require 'capistrano/deploy' require 'capistrano/scm/git' install_plugin Capistrano::SCM::Git require 'capistrano/rbenv' require 'capistrano/rbenv_install' require 'capistrano/bundler' require 'capistrano/rails' require "capistrano/systemd/multiservice" install_plugin Capistrano::Systemd::MultiService.new_service("app", service_type: "user") # Load custom tasks from `lib/capistrano/tasks` if you have any defined Dir.glob("lib/capistrano/tasks/*.rake").each { |r| import r } ``` Edit `config/deploy.rb` to define the deployment settings: ```ruby title="config/deploy.rb" lock "~> 3.16" set :application, "your_app_name" set :repo_url, "git@example.com:me/my_repo.git" set :branch, ENV.fetch("CI_COMMIT_REF_NAME", "main") set :deploy_to, "/home/#{fetch :user}/#{fetch :application}" set :keep_releases, 5 set :rbenv_type, :user set :rbenv_ruby, File.read(".ruby-version").strip set :rbenv_path, "/home/#{fetch :user}/.rbenv" append :linked_files, "env" append :linked_dirs, "log", "tmp/pids", "tmp/cache", "tmp/sockets", "vendor/bundle", "public/system" before "systemd:app:validate", "systemd:app:setup" after "deploy:publishing", "systemd:app:restart" ``` Set up the server details in `config/deploy/production.rb`: ```ruby title="config/deploy/production.rb" server 'server.nine.ch', user: 'www-data', roles: %w{app db web} ``` Define the Ruby-Version in a `.ruby-version` file: ```text title=".ruby-version" 3.2.0 ``` And finally, create a systemd service template in `config/systemd/app.service.erb`: ```systemd title="config/systemd/app.service.erb" [Unit] Description=<%= fetch :application %> Rails Application [Service] Type=simple WorkingDirectory=<%= current_path %> Environment=RAILS_SERVE_STATIC_FILES=true EnvironmentFile=<%= shared_path %>/env ExecStart=%h/.rbenv/bin/rbenv exec bundle exec rails server -b localhost --log-to-stdout TimeoutSec=15 Restart=on-failure PrivateTmp=yes ProtectSystem=full [Install] WantedBy=default.target ``` ### Deploy with Capistrano Now you can deploy your application using Capistrano with the following command: ```shell-session bundle exec cap production deploy ``` Capistrano handles the details of deployment, including Ruby version management, dependencies, asset compilation, file copying, and managing the systemd service. This approach ensures that your deployment process is streamlined and less prone to errors, thereby enabling more efficient and reliable releases. --- ## Sending Emails from Applications Postfix, the [Mail Transfer Agent](https://en.wikipedia.org/wiki/Message_transfer_agent) pre-installed on Nine systems, is not offered as a Managed Service. There is no monitoring of the email queue or of [rejected messages (bounces)](https://en.wikipedia.org/wiki/Bounce_message). To ensure that sending of emails works smoothly, we recommend using a specialized service provider for business-critical applications. This offers several advantages: 1. [Reputation of IP addresses](https://en.wikipedia.org/wiki/Cold_email#Bad_server_IP_reputation): No effort is required to establish and maintain the reputation. In addition, the IP address of the Managed Server remains unaffected by blocklists. 1. Sending via SMTP or HTTP: Many applications and frameworks offer configuration options for SMTP servers. Most email providers also offer an HTTP interface, which can simplify sending from self-developed applications. 1. Compliance and adaptation to changing standards: In addition to common [email authentication methods](https://en.wikipedia.org/wiki/E-Mail_authentication) such as SPF, DKIM and DMARC, this also includes more specific, content-related standards such as the required [List-Unsubscribe Header](https://certified-senders.org/wp-content/uploads/2017/07/CSA_one-click_list-unsubscribe.pdf) when sending newsletters according to [RFC 8058](https://www.rfc-editor.org/rfc/rfc8058.html). 1. Evaluation: Deliverability and statistics in general can be visualized using dashboards. 1. Support with technical questions regarding email delivery. ## Why Does Nine Recommend Using a Specialized Provider? Sending emails over the Internet is being shaped by a large number of standards. These standards are primarily intended to ensure confidentiality, integrity and authenticity. Compliance with these standards ultimately also increases the deliverability to the designated recipient of a message. The application of the standards is in the hands of large players such as Microsoft, Yahoo and Google. For example, Google has announced [a stricter approach](https://blog.google/products/gmail/gmail-security-authentication-spam-protection/) towards senders of more than 5,000 emails per day from February 2024. Comparable measures are only rarely communicated by large providers and are often implemented in very different ways. It is difficult for smaller providers and their users who do not specialize in sending emails to always meet all requirements. This can lead to systems or their IP addresses being placed on blocklists, which means that delivery to the designated recipient is no longer possible. While sending transactional emails for order confirmations or resetting a password usually causes fewer problems, sending mass emails can quickly become a challenge. ## Examples of Established Email Providers - [Brevo](https://www.brevo.com/products/transactional-email/) - [Mailgun](https://www.mailgun.com/products/send/) - [SendGrid](https://sendgrid.com/en-us/solutions/email-api) - [SMTP2GO](https://www.smtp2go.com/) You might find additional providers on [european-alternatives.eu/category/transactional-email-service](https://european-alternatives.eu/category/transactional-email-service). ## Sending Emails from a Managed Server If you nevertheless decide to send emails directly from a managed server, please consider the following configuration to avoid being marked as SPAM or being blacklisted: 1. Configure a sender address under your own domain. You mustn't send emails from a `*.nine.ch` subdomain e.g. the server name. Ensure that both the email envelope `MAIL FROM` (RFC 5321) and the email header `From` (RFC 5322) addresses are correct. 1. Allow the server to send emails from your domain by configuring a [correct SPF](http://www.open-spf.org/Why/) record. --- ## Installing phpMyAdmin in userspace Installing phpMyAdmin in user space will come with severals benefits, including: - Allows for custom URLs. - Let's Encrypt TLS certificates - .htaccess Files for Basic Auth enablement and/or ip access restrictions ## phpMyAdmin Installation Download the latest version of phpMyAdmin from [phpmyadmin.net/downloads](https://www.phpmyadmin.net/downloads/) and extract it to a folder of your choice: **Please adapt the version mentioned in the following example to the newest version available.** ```shell-session $ export VERSION=5.2.0 $ mkdir -p ~/phpmyadmin $ cd ~/phpmyadmin $ curl -O https://files.phpmyadmin.net/phpMyAdmin/$VERSION/phpMyAdmin-$VERSION-all-languages.tar.gz # download $ tar -zxf phpMyAdmin-$VERSION-all-languages.tar.gz # extract ``` ## Symbolic Link Create a symbolic link so that in the future you can easily update the phpMyAdmin installation without adjusting the webserver configuration: ```shell-session $ ln -vsf phpMyAdmin-$VERSION-all-languages current ``` ## Create the Virtual Host Using our tool `nine-manage-vhosts`, you can create a virtual host for phpMyAdmin. We reference the formerly created symbolic link: ```shell-session $ sudo nine-manage-vhosts virtual-host create --webroot=/home/www-data/phpmyadmin/current ``` Additional details about `nine-manage-vhosts` can be found in its own [support article](../webserver/nine-manage-vhosts/manage-virtualhosts-with-nine-manage-vhosts). ## Create a TLS Certificate We recommend to secure the web accesses to phpMyAdmin installations using a TLS certificate. On our "Managed Server" products you can use `nine-manage-vhosts` to use our Let's Encrypt integration. For details about the setup and usage of Let's Encrypt, check the corresponding [support article](../webserver/nine-manage-vhosts/nine-manage-vhosts-with-lets-encrypt). If you would like to use your own, or a purchased TLS certificate for the site, please contact us for purchase and/or installation at . ## Optional Access Restriction Using a ".htaccess" file, the access to the newly created phpMyAdmin instance can be restricted. ### Restrict by IP To restrict access by static IP addresses, create `~/phpmyadmin/current/.htaccess`: ```apacheconf title="~/phpmyadmin/current/.htaccess" Require ip 122.122.122.122 Require ip 123.123.123.123 ``` ### Restrict with Basic Auth To restrict access with basic auth, first create a user like so: ```shell-session $ htpasswd -c ~/phpmyadmin/.htpasswd USERNAME ``` And then create `~/phpmyadmin/current/.htaccess`: ```apacheconf title="~/phpmyadmin/current/.htaccess" AuthType Basic AuthName "Restricted Content" AuthUserFile /home/www-data/phpmyadmin/.htpasswd Require valid-user ``` ## Configure phpMyAdmin We recommend using a basic default configuration. To create it automatically, run the following command: ```shell-session cat << EOF > ~/phpmyadmin/current/config.inc.php `. 2. To use a Let's Encrypt certificate and HTTPS, set the template `proxy_letsencrypt_https` instead of `proxy`. 3. To automatically redirect HTTP to HTTPS using a Let's Encrypt certificate, use the template `proxy_letsencrypt_https_redirect`. The dedicated [documentation](../webserver/nine-manage-vhosts/manage-virtualhosts-with-nine-manage-vhosts) contains more examples and details about the usage of `nine-manage-vhosts`. ::: ## Starting the Tomcat Instance Automatically We provide a default systemd template to start/stop your Tomcat instances. To ensure the instance launches on server boot, run: ```bash systemctl --user enable user-tomcat@test-server.ch ``` The following commands manually start or stop the Tomcat instance named `test-server.ch`: ```bash systemctl --user start user-tomcat@test-server.ch systemctl --user stop user-tomcat@test-server.ch ``` To check the current status of the Tomcat instance, you can use the following command: ```bash systemctl --user status user-tomcat@test-server.ch ``` :::info 1. The first part of the service name remains `user-tomcat@`. The part after @ is your Tomcat instance name. 2. If you have multiple Tomcat versions installed, you can specify the desired version in the service file like this `user-tomcat10@test-server.ch`: ::: For more information on managing services with systemd, refer to [Manage Daemons as a User With Systemd](./manage-daemons-as-a-user-with-systemd). ## Deploying Your Application In this section, you will deploy your Java application to the Tomcat instance. Below are a few things to know about the deployment process: ### Application Format Applications are typically packaged as `.war` or `.jar` files. ### Deployment Location The application should be located in `/home/www-data//ROOT`. To run multiple applications in one virtual host, place each application in its own directory under `/home/www-data//webapps`. Applications are then accessible at `http://servername.com/name_of_the_folder_in_webapps/`. ### Example To deploy test-application.war to the test-server.ch instance, this could be used: ```bash rsync --progress test-application.war www-data@server.nine.ch:~/test-server.ch/webapps/ ``` Tomcat then extracts it to: ```text /home/www-data/test-server.ch/webapps/test-application ``` The application is now accessible at http://test-server.ch/test-application, and logs are stored in /home/www-data/test-server.ch/logs/catalina.out. ## Logging If your application logs to standard output, logs are written to `/home/www-data//logs/catalina.out`. :::warning The `catalina.out` log file can grow quickly. Follow **[this guide](../operations/rotate-log-files)** to set up scheduled log rotation and avoid disk space issues. ::: ## Environment Variables Tomcat's behavior cab be modified by setting environment variables in `~//bin/setenv.sh`. ### Examples - Using a different Java version: ```text JAVA_HOME="/opt/java/production" ``` - Changing log location: ```text CATALINA_OUT="logs/custom.out" ``` - Adjusting memory limits: ```text JAVA_OPTS="-Djava.awt.headless=true -Xmx512M ${JAVA_OPTS}" ``` :::info The default memory limit for the instance is set to 128MB. When adjusting this value, ensure that the new limit is a multiple of 1024 and is at least 2MB. ::: ## Systemd Settings Some parameters can not be set by Tomcat but rather need to be set by systemd. Those settings can be modified by so-called _systemd drop-ins_. To apply settings for all instances, place the file in: ```text ~/.config/systemd/user/user-tomcat@.service.d ``` To modify settings for a single instance, place the file in: ```text ~/.config/systemd/user/user-tomcat@.service.d ``` For example, to set the Max Open Files limit, create the following file: ```text title="~/.config/systemd/user/user-tomcat@test-server.ch.service.d/limits.conf" [Service] LimitNOFILE=2048 ``` --- ## Run Node.js Applications This article explains how to run Node.js apps on Nine's managed servers. ## 1. Setup the Node Version Manager We recommend using a version manager like [`nvm`](https://github.com/creationix/nvm) to be able to use different `node` versions than provided by the distribution release of Ubuntu. It also allows you to use different versions per project and to install "global" npm packages. To install `nvm`: ```shell-session curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash ``` This will install nvm and add it to your `.bashrc`. To use nvm via the CLI, you need to read in your `.bashrc` again. ```shell-session source .bashrc ``` ## 2. Install the Node Version We can now install the desired Node version: ```shell-session nvm ls-remote # show available versions nvm install 'lts/*' # install the latest LTS release nvm install 22 # install the latest 22.x release nvm alias default 'lts/*' # set LTS as default version to use ``` ### Manage Different Versions Use a [.nvmrc](https://github.com/nvm-sh/nvm#nvmrc) file to define different versions per project.: ```gitignore title=".nvmrc" lts/* ``` ## 3. Install Process Manager We recommend the usage of systemd for process management, but in the case of Node it should be used together with [`pm2`](https://github.com/Unitech/pm2). While systemd is excellent for startup management and service recovery, `pm2` simplifies process management with its built-in load balancing (Node only runs as a single process without `pm2`), monitoring, and zero-downtime restarts. So we'll use systemd to ensure `pm2` is always started after a reboot and `pm2` to manage all of a user's node applications. Install `pm2` with the following command: ```shell-session npm install -g pm2 ``` You can now use it to manage your Node.js applications. More details on how to do this can be found in the [official documentation](https://pm2.keymetrics.io/docs/usage/quick-start/). ## 4. Start `pm2` after Reboot with systemd To start `pm2` as a user service when the system is restarted, create a systemd unit file (`~/.config/systemd/user/pm2.service`): ```systemd title="~/.config/systemd/user/pm2.service" [Unit] Description=PM2 process manager Documentation=https://pm2.keymetrics.io/ [Service] Type=forking LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity Environment=NODE_ENV=production Environment=NODE_VERSION=14 Environment=PM2_HOME=%h/.pm2 PIDFile=%h/.pm2/%p.pid ExecStart=%h/.nvm/nvm-exec pm2 resurrect ExecReload=%h/.nvm/nvm-exec pm2 reload all ExecStop=%h/.nvm/nvm-exec pm2 kill [Install] WantedBy=default.target ``` Enable and start the `pm2` service: ```shell-session systemctl --user enable pm2 systemctl --user start pm2 ``` ### Troubleshooting ### Logrotate It is recommended to also set up [log rotation](../operations/rotate-log-files) for pm2 to ensure, not too much space is used by log files. To enable logrotate, follow the instructions in the ["Rotate Log Files"](../operations/rotate-log-files) and add the following configuration: ```logrotate title="~/.logrotate.conf" ~/.pm2/logs/*.log { daily rotate 14 notifempty compress copytruncate compresscmd /usr/bin/zstdmt compressoptions -18 -T0 --rm -qq compressext .zst uncompresscmd /usr/bin/unzstd } ``` ## 5. Configure Webserver --- ## Run Python applications with uWSGI This article explains how to run Python apps on Nine's managed servers with uWSGI in userspace. ## Requirements For the example below, we assume that your application supports [wsgi](https://www.fullstackpython.com/wsgi-servers.html). If you use Django or Flask, this is already the case. ## 1. Install Python Your managed server already comes with Python pre-installed, but the available version is fixed based on the Ubuntu version. | Ubuntu | Python | | ---------------- | ------- | | 22.04 (Jammy) | 3.10.\* | | 24.04 (Noble) | 3.12.\* | | 26.04 (Resolute) | 3.14.\* | If you require a different version, we recommend the use of [`pyenv`](https://github.com/pyenv/pyenv). 1. Execute their install script with the following command: ```bash curl https://pyenv.run | bash ``` 2. Run the following command to extend your `~/.bashrc`: ```bash cat << 'EOF' >> ~/.bashrc export PATH="$HOME/.pyenv/bin:$PATH" eval "$(pyenv init --path)" eval "$(pyenv virtualenv-init -)" EOF ``` 3. Reload your shell: ```bash source .bashrc ``` 4. Install the desired Python version: ```bash pyenv install 3 # install latest v3 pyenv global 3 # set as default ``` ## 2. Setup Virtual Environment (`venv`) To separate your Python applications from each other, it's best practice to use [virtual environments](https://docs.python.org/3/library/venv.html#creating-virtual-environments) or short, `venv`. 1. Create and activate a new environment: ```bash mkdir ~/exampleapp/ && cd ~/exampleapp/ python -m venv .venv ``` 2. Install uWSGI inside the virtual environment: ```bash source .venv/bin/activate (.venv) pip install uwsgi ``` ## 3. Application Deployment Deploying your application is different for each project. With Django, for example, you'll need to install the Django package, then upload all files to the server and install the dependencies: 1. Install the Django CLI: ```bash (.venv) pip install Django ``` 2. Copy the application files to your server: ```bash rsync -Hav --delete --exclude='node_modules/*' --exclude='logs/*' --exclude='.git/*' --exclude='.venv/*' ./local_project www-data@server.nine.ch:exampleapp/ ``` 3. Install your dependencies: ```bash (.venv) pip install -r requirements.txt ``` ## 4. Run Application as a Service To ensure a service keeps running after an application crash or system reboot, we highly recommend using a [systemd user service](./manage-daemons-as-a-user-with-systemd). The configuration for such a systemd unit depends on your application. The above example assumes that you're running a Django app. 1. Create the service file in `~/.config/systemd/user/exampleapp.service`: ```systemd title="~/.config/systemd/user/exampleapp.service" [Unit] Description=exampleapp [Service] WorkingDirectory=%h/exampleapp Restart=always KillSignal=SIGQUIT Type=notify NotifyAccess=all Environment=PYTHON_VERSION=3.9.11 Environment=PATH=%h/.pyenv/versions/$PYTHON_VERSION/bin:$PATH EnvironmentFile=%h/exampleapp/.env ExecStart=%h/exampleapp/.venv/bin/uwsgi \ --module=exampleapp.wsgi:application \ --master \ --http=127.0.0.1:3000 \ --enable-threads \ --threads=2 \ --processes=4 \ --harakiri=20 \ --max-requests=5000 \ --vacuum \ --static-map /static=%h/exampleapp/static \ --home=%h/exampleapp/.venv [Install] WantedBy=default.target ``` _These settings will vary depending on your system and application. We suggest consulting the [official documentation](https://uwsgi-docs.readthedocs.io/en/latest/index.html) for the configuration parameters and best practices._ 2. Start the service: ```bash touch ~/exampleapp/.env # ensure environment file exists systemctl --user daemon-reload systemctl --user enable exampleapp.service systemctl --user start exampleapp.service systemctl --user status exampleapp.service ``` ### Troubleshooting ## 5. Configure Webserver --- ## Data Backups and Restore Nine creates daily backups of the file system on your managed server. These backups are encrypted and transferred to our backup system at two separate data centers and kept for 7 days. Using the user 'www-data' available to you, you can access the backups and restore data as needed. ## What Is Included in the Backup? The backup contains all user data of your environments. This includes the data of all `www-*` users. In addition, daily database backups are also transferred to our backup systems. Common cache and temporary directories are excluded from the backup: ``` home/www-*/**/wp-content/cache/* home/www-*/**/wp-content/temp/* home/www-*/**/cache/com_content/* home/www-*/**/cache/com_modules/* home/www-*/**/cache/com_languages/* home/www-*/**/cache/page/* home/www-*/**/generated/metadata/* home/www-*/**/generated/code/* home/www-*/**/var/cache/* home/www-*/**/var/page_cache/* home/www-*/**/typo3temp/Cache/* ``` We are happy to define other exceptions upon request. ## How Can I Access the Backups? You can use the `sudo nine-backup snapshots` command to display the backups of your server: ``` www-data@nine1:~ $ sudo nine-backup snapshots repository afe45457 opened successfully, password is correct ID Time Host Tags Paths ---------------------------------------------------------------- bf8f7d8c 2022-01-25 05:36:03 nine01-test nine01-test / /boot bccdd6dd 2022-01-26 05:36:03 nine01-test nine01-test / /boot d62431f7 2022-01-27 05:36:02 nine01-test nine01-test / /boot 80d625a6 2022-01-28 05:36:03 nine01-test nine01-test / /boot 1eb6a19f 2022-01-29 05:36:02 nine01-test nine01-test / /boot 5e801eaf 2022-01-30 05:36:03 nine01-test nine01-test / /boot abdb5f95 2022-01-31 05:36:03 nine01-test nine01-test / /boot ---------------------------------------------------------------- 7 snapshots ``` The command `sudo nine-backup mount-repository` connects to our backup system and makes the individual backups available under the path `/mnt/restore.XYZ` . ``` www-data@nine01-test:~ $ sudo nine-backup mount-repository repository afe45457 opened successfully, password is correct Now serving the repository at /mnt/restore.Eh9lZt0Rxh When finished, quit with Ctrl-c or umount the mountpoint. ``` Because this command remains in the foreground, the shell environment in which you run this command cannot be used for any other work. Therefore, to access the backups, you must make another connection to the system. ## Access by Snapshot ID, Date or Tag After executing `mount-repository` the temporary directory is displayed where the backup is now available with read permission: `Now serving the repository at /mnt/restore.Eh9lZt0Rxh`. ``` www-data@nine01-test:/mnt/restore.Eh9lZt0Rxh $ ls -la total 0 dr-xr-xr-x 1 root root 0 Jan 31 14:50 . drwxr-xr-x 6 root root 86 Jan 31 14:50 ... dr-xr-xr-x 1 root root 0 Jan 31 14:50 hosts dr-xr-xr-x 1 root root 0 Jan 31 14:50 ids dr-xr-xr-x 1 root root 0 Jan 31 14:50 snapshots dr-xr-xr-x 1 root root 0 Jan 31 14:50 tags ``` In the directory `ids` you can find all existing snapshots by the ID as it was displayed in `nine-backup snapshots`. For everyday use, the `snapshots` directory is the easiest to use. This contains folders named after the creation date of the backups. ``` www-data@nine01-test:/mnt/restore.Eh9lZt0Rxh/snapshots $ ls -la total 0 dr-xr-xr-x 1 root root 0 Jan 31 14:50 . dr-xr-xr-x 1 root root 0 Jan 31 14:50 ... dr-xr-xr-x 2 root root 0 Jan 25 05:36 2022-01-25T05:36:03+01:00 dr-xr-xr-x 2 root root 0 Jan 26 05:36 2022-01-26T05:36:03+01:00 dr-xr-xr-x 2 root root 0 Jan 27 05:36 2022-01-27T05:36:02+01:00 dr-xr-xr-x 2 root root 0 Jan 28 05:36 2022-01-28T05:36:03+01:00 dr-xr-xr-x 2 root root 0 Jan 29 05:36 2022-01-29T05:36:02+01:00 dr-xr-xr-x 2 root root 0 Jan 30 05:36 2022-01-30T05:36:03+01:00 dr-xr-xr-x 2 root root 0 Jan 31 05:36 2022-01-31T05:36:03+01:00 lrwxrwx 1 root root 0 Jan 31 05:36 latest -> 2022-01-31T05:36:03+01:00 ``` You have access to all data that is also available to you in the current production environment of the system. ## Restore Data Starting from the displayed directories, you can move down the directory tree to the desired directory and perform a check of the contents. ``` www-data@nine01-test:/mnt/restore.Eh9lZt0Rxh/snapshots $ cd latest/home/www-data/wp-dev/ www-data@nine01-test:/mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/01 $ ls -la total 170 drwxr-xr-x 2 www-data www-data 0 Apr 12 2021 . drwxr-xr-x 2 www-data www-data 0 Jan 27 08:55 ... -rw-r--r-- 1 www-data www-data 235 Aug 28 2019 .htaccess -rw-r--r-- 1 www-data www-data 420 Dec 1 2017 index.php -rw-r--r-- 1 www-data www-data 19935 Jun 19 2019 license.txt -rw-r--r-- 1 www-data www-data 8538 Jun 19 2019 readme.html -rw-r--r-- 1 www-data www-data 7447 Dec 18 2019 readme.html drwxr-xr-x 2 www-data www-data 0 Aug 28 2019 .well-known -rw-r--r-- 1 www-data www-data 6919 Jan 12 2019 wp-activate.php drwxr-xr-x 2 www-data www-data 0 Jun 19 2019 wp-admin -rw-r--r-- 1 www-data www-data 369 Dec 1 2017 wp-blog-header.php -rw-r--r-- 1 www-data www-data 2283 Jan 21 2019 wp-comments-post.php -rw-rw- 1 www-data www-data 3721 Aug 28 2019 wp-config.php -rw-r--r-- 1 www-data www-data 3396 Jun 19 2019 wp-config-sample.php drwxr-xr-x 2 www-data www-data 0 Nov 21 2019 wp-content -rw-r--r-- 1 www-data www-data 3847 Jan 9 2019 wp-cron.php drwxr-xr-x 2 www-data www-data 0 Jun 19 2019 wp-includes -rw-r--r-- 1 www-data www-data 2502 Jan 16 2019 wp-links-opml.php -rw-r--r-- 1 www-data www-data 3306 Dec 1 2017 wp-load.php -rw-r--r-- 1 www-data www-data 39551 Jun 10 2019 wp-login.php -rw-r--r-- 1 www-data www-data 8403 Dec 1 2017 wp-mail.php -rw-r--r-- 1 www-data www-data 18962 Mar 28 2019 wp-settings.php -rw-r--r-- 1 www-data www-data 31085 Jan 16 2019 wp-signup.php -rw-r--r-- 1 www-data www-data 4764 Dec 1 2017 wp-trackback.php -rw-r--r-- 1 www-data www-data 3068 Aug 17 2018 xmlrpc.php ``` All operations that do not make changes to the files of the backups can be applied to the data from the backup. This makes it easy to find differences between a backup and the current version: ``` diff -w /home/www-data/wp-dev/.htaccess /mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/wp-dev/.htaccess 1c1 < # This is a restore test entry --- > ``` ``` md5sum /home/www-data/wp-dev/.htaccess /mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/wp-dev/.htaccess 929e2c784c0f52033099b21751342cae /home/www-data/wp-dev/.htaccess 093cc2d0ddff4121504ec86c32943a28 /mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/wp-dev/.htaccess ``` To restore data, the `cat`, `cp` or `rsync` commands can be used, depending on the data type or amount. In our example we recommend the use of `cat` or `cp`: ``` cat /mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/wp-dev/.htaccess > /home/www-data/wp-dev/.htaccess ``` or ``` cp /mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/wp-dev/.htaccess /home/www-data/wp-dev/.htaccess ``` If whole folders or directory structures are to be restored, it is recommended to use `rsync`. The parameter `-n` performs a dry run, making it possible to check whether the command performs the desired operation. For test purposes the contents of the directory `wp-dev` were deleted. ``` rsync -n -avg /mnt/restore.Eh9lZt0Rxh/snapshots/latest/home/www-data/wp-dev/ /home/www-data/wp-dev/ sending incremental file list index.php license.txt readme.html readme.html wp-activate.php wp-blog-header.php .. .. wp-includes/widgets/class-wp-widget-rss.php wp-includes/widgets/class-wp-widget-search.php wp-includes/widgets/class-wp-widget-tag-cloud.php wp-includes/widgets/class-wp-widget-text.php sent 56,042 bytes received 6,447 bytes 124,978.00 bytes/sec total size is 45,005,601 speedup is 720.22 (DRY RUN) ``` If the output check gives the desired result, the `-n` parameter can be removed from the `rsync` call. ## Completion of the Work After you have completed the data synchronization or recovery, the active `mount-repository` process can be terminated. First, change back to the user's root directory in the _other_ shell you opened: ``` www-data@nine01-test:/mnt/restore.Eh9lZt0Rxh/snapshots/latest $ cd www-data@nine01-test:~ $ ``` After that the `mount-repository` process can be terminated: ``` www-data@nine01-test:~ $ sudo nine-backup mount-repository repository afe45457 opened successfully, password is correct Now serving the repository at /mnt/restore.Eh9lZt0Rxh When finished, quit with Ctrl-c or umount the mountpoint. signal interrupt received, cleaning up ``` If you do not perform this step, you may be left with an inaccessible mountpoint: `cannot access '/mnt/restore.Eh9lZt0Rxh': Transport endpoint is not connected``. There is no need to worry here, the non-functional mount point is automatically removed from the system and has no effect on the backup of the system or the access to the data within the backup. ## Backup Retention Options In case the default 7 day retention period is too short for your requirements and you want to keep backups for a longer period of time, you can find additional retention options in our [Product Overview](./#additional-options). --- ## MySQL Backup and Restore ## Backups Nine creates backups of your servers MySQL databases every day between 02:00 and 03:00. These backups are stored locally for 10 days, and are also backed up to our backup servers daily. You can access these local backups as the user `www-data`, to either archive them at a place of your choice, use them on your local machine, or restore them on the system itself. The backups are stored in the directory `/home/database-backup/mysql/` and revisioned in directories with the following timestamp schema: `2021-12-06-0134` The symbolic link `/home/database-backup/mysql/latest/` always points to the newest backups. You'll find the backups of all your databases in the sub-directory `customer`. If you're looking for the database structure only, find them in the sub-directory `structure`. ## Creating Additional Backups To allow you an easy method to create backups with the same parameters as we do, we're allowing access to our backup script with `sudo` for the user `www-data`: ``` www-data@nine01:~ $ sudo nine-mysql-backup 2021-12-06T09:54:19+01:00 Dumped and compressed database 'nmd_frontend_production' in 53 seconds 2021-12-06T09:55:04+01:00 Dumped and compressed database 'nmd_frontend_staging' in 45 seconds ``` ## Restore Within each daily revision you'll find the file `restore.sh`, for example at `/home/database-backup/mysql/2021-12-06-0134/restore.sh`. All backed up databases are listed here including the full path to the files. You can copy these and modify them to suit your needs. The commented lines are meant for a restoration into the source database. ``` ... # Single database restores # zstdcat /home/database-backup/mysql/2021-12-06-0134/customer/nmd_frontend_production/*.zst | mysql "nmd_frontend_production" # zstdcat /home/database-backup/mysql/2021-12-06-0134/customer/nmd_frontend_staging/*.zst | mysql "nmd_frontend_staging" ``` The `mysql` commands need to be expanded with the password and username option, f.e.: `mysql -p -u nmd_frontend_production` `mysql -p -u nmd_frontend_staging` Entries above the line `# Single database restores` are meant for use by Nine employees and can't be executed by unprivileged users. ### Restoration into the Source Database To restore a backup, you can use the following command. Using this command will restore all tables for the database `nmd_frontend_staging` into the same, already existing database: ``` zstdcat /home/database-backup/mysql/2021-12-06-0134/customer/nmd_frontend_staging/*.zst | mysql -p -u nmd_frontend_staging "nmd_frontend_staging" ``` The parameter `-u` defines the username for the `mysql` command, `-p` the password. You can also provide the password directly to perform a restore without being asked for the password (`mysql -pSecurePass -u nmd_frontend_staging "nmd_frontend_staging"`). PLease mind that you need to provide the username and password for the database, not your `www-data` credentials. You will find the correct username and password within your application configuration if you don't have them at hand. ### Restoration into Another Database If you need or want to restore the backup into another database, you need to change the name of the target database. We recommend that you restore the structure of the database first: ``` zstdcat /home/database-backup/mysql/2021-12-06-0134/structure/nmd_frontend_staging.zst | mysql -p -u nmd_frontend_staging_restore "nmd_frontend_staging_restore" zstdcat /home/database-backup/mysql/2021-12-06-0134/customer/nmd_frontend_staging/*.zst | mysql -p -u nmd_frontend_staging_restore "nmd_frontend_staging_restore" ``` If you do want to restore a backup into another database, this database must exist beforehand [or be created with](./nine-manage-databases) `nine-manage-databases` . ### Restoration of Single Tables Instead of restoring all tables from the backup (using the "\*"), you can restore a single table instead: ``` zstdcat /home/database-backup/mysql/2021-12-06-0134/customer/nmd_frontend_staging/entities.zst | mysql -p -u nmd_frontend_staging_restore "nmd_frontend_staging_restore" ``` ## Additional information ### Backup Time Creating a database backup usually causes a bit higher load on your system. We've chosen a time during the night that is generally less busy. We're happy to find a better time suiting your needs if the time between 02:00 and 03:00 isn't fitting your applications well. ### Compression of Backups We're using the [Zstandard](https://facebook.github.io/zstd/) algorithm to compress the backups. This algorithm offers a great balance between speed, compression rate and resource usage and is superior to the common algorithms you might know or use. If you need the backup for usage on another systems where you don't have access to zst binaries, we recommend to decompress the existing dumps on the system and re-compress them afterwards. ``` www-data@nine01:~ $ mkdir dump_recompress www-data@nine01:~ $ cp /home/database-backup/mysql/latest/customer/nmd_frontend_staging/* dump_recompress/ ; cd dump_recompress/ # The option "--rm" will delete the zst archives after decompressing them www-data@nine01:~/dump_recompress $ unzstd --rm *.zst nmd_frontend_staging.zst: 3732915490 bytes www-data@nine01:~/dump_recompress $ bzip2 * www-data@nine01:~/dump_recompress $ ls -l *.bz2 -rw-r----- 1 www-data www-data 432683536 Dec 6 13:24 nmd_frontend_staging.bz2 ``` Please be cautious to not place database dumps publicly available within a public directory served by the webserver. ### Type of Backups By default, we're creating backups from each table within your databases. This comes with some advantages in the day to day handling of these backups, as single tables are usually smaller and allow for a single table restore. If you instead wish for a full backup of all tables into a single output file, we can change the behavior of the script to do this. The advantages in the handling will go, however, the dump itself will be more consistent as a whole. Based on our long time positive experience with dumping single tables, we only recommend to change this for edge cases or for explicit need. --- ## nine-manage-databases Nine provides a command line tool for managing databases and database users. `nine-manage-databases` can be used via SSH. Currently, MySQL / MariaDB and PostgreSQL are supported. :::info[PostgreSQL legacy password encryption] In recent PostgreSQL versions, the default password encryption method switched from `md5` to `scram-sha-256`. While both methods are still supported, all new passwords use `scram-sha-256` encryption. To maintain a high level of security, consider updating passwords whenever possible, such as during a major software deployment. ::: ## Help The following command lists all available options: ```shell-session $ sudo nine-manage-databases -h ``` ## Database Management `nine-manage-databases` is capable of listing, creating, and removing databases. ## Choose the Database Type (DBMS) If multiple database types are installed, you need to provide the DBMS with the option `--database-type` (or `-t`). Available DBMS types: - mysql (MySQL / MariaDB) - postgresql (PostgreSQL) ```shell-session $ sudo nine-manage-databases --database-type=postgresql database list ``` This command lists all the PostgreSQL databases created by `nine-manage-databases`. Usually, there is only one DBMS installed, which makes the usage of this option obsolete. ## List All Databases The following command lists all created databases: ```shell-session $ sudo nine-manage-databases database list ``` ## Create a Database ```shell-session $ sudo nine-manage-databases database create --user=nmd_user1 nmd_database1 ``` This command creates a database named **nmd_database1**. Notice the **nmd\_** prefix. The prefix separates the databases created by Nine from databases managed by `nine-manage-databases`. The command also creates the user **nmd_user1** which will automatically be granted read-write access to the database. A secure password for the user will be generated and displayed in the command output. You can provide your own password interactively with the `-p` option. If you want to create an additional database for an existing user, you can use the `--database-only` option for this purpose. ## Delete a Database ```shell-session $ sudo nine-manage-databases database drop nmd_database1 ``` This command drops the database **nmd_database1**. Before deletion, a confirmation is required. To skip the confirmation, use the option `--force`. This will drop the database immediately. Note that this command will also delete all the users which were created for this database. ## User Management ## List Users ```shell-session $ sudo nine-manage-databases user list --database=nmd_database1 ``` This command lists all users of the database **nmd_database1**. Without the `--database` option, all users are listed. ## Create a User ```shell-session $ sudo nine-manage-databases user create --database=nmd_database1 --read-only -p nmd_user2 ``` You can create as many users for a database as you want. You can also omit the database (`--database` option), because the permissions can be granted separately. The above command creates the user **nmd_user2**, the password will be asked for interactively (option `-p`. You can omit the password option `-p` to create a random one. This user will have read-only access to the **nmd_database1** database (option `--read-only`). By default, users have read-write access. ## Grant Database Access to User ⚠️ Note that the `user grant_rights` and `user revoke_rights` commands are only available for DBMS type 'mysql'. To manage database access for users on DBMS type 'postgresql', please contact to obtain access to an admin user for your specific purpose. One user can be granted access to multiple databases: ```shell-session $ sudo nine-manage-databases user grant_rights nmd_user2 --database=nmd_database2 --read-only ``` `nmd_user2` user has now read-only access to the database `nmd_database2`. To revoke this permission, use the `user revoke_rights` command: ```shell-session $ sudo nine-manage-databases user revoke_rights nmd_user2 --database=nmd_database2 ``` ## Change a Database Users Password ```shell-session $ sudo nine-manage-databases user update nmd_user1 update -p --database=nmd_database1 ``` This command updates the password of the user **nmd_user1** and asks for a new password interactively (option `-p`). ## Delete a User ```shell-session $ sudo nine-manage-databases user drop nmd_user1 ``` This command drops the user **nmd_user1**. --- ## PostgreSQL Backup and Restore ## Backups Nine creates daily backups of the Postgres databases on your managed server between 02:00 and 03:00 o clock. These backups are kept locally for 10 days and are transferred to our backup systems with the daily backup of your server. With the user 'www-data' available to you, you can access the local backups in order to archive them if necessary, to use them locally on your system or to restore them directly on your system. The backups are stored in the directory `/home/database-backup/postgresql/`. All backups are versioned in directories with the following time scheme: `2021-12-06-0134` The symbolic link `/home/database-backup/postgresql/latest/` will take you to the latest backup. In the folder `customer` you will find the backups of all databases you have created. If you are only interested in the structure of a database, you will find it in the folder `structure`. ## Create Additional Backups To allow you to create additional backups, we have enabled the script we use to create the backups to be called via `sudo` by the user `www-data`: ```shell-session www-data@nine01:~ $ sudo nine-postgresql-backup 2021-12-06T09:54:19+01:00 Dumped and compressed database 'nmd_frontend_production' in 53 seconds 2021-12-06T09:55:04+01:00 Dumped and compressed database 'nmd_frontend_staging' in 45 seconds ``` ## Restore ### General Information Within each backup you will find the file `restore.sh`, e.g. under `/home/database-backup/postgresql/2021-12-06-0134/restore.sh`. All databases including the file paths of the backup are listed here. You can copy the contents and adapt them to your needs. The commented entries are intended for a restore in the original database. The entry of the user name `-U nmd_` must be completed or adapted to the desired user name. ``` ... # Single database restores # zstdcat /home/database-backup/postgresql/2021-12-06-0134/customer/nmd_frontend_production/nmd_frontend_production.zst | pg_restore -d nmd_frontend_production -O -x -c --if-exists -U nmd_ # zstdcat /home/database-backup/postgresql/2021-12-06-0134/customer/nmd_frontend_staging/nmd_frontend_staging.zst | pg_restore -d nmd_frontend_staging -O -x -c --if-exists -U nmd_ ``` Entries above the line `# Single database restores` are for use by Nine staff and cannot be executed by unprivileged users. During recovery, the following warnings may occur: ```shell-session pg_restore: WARNING: no privileges could be revoked for "public". pg_restore: WARNING: no privileges were granted for "public". ``` These warnings can be ignored as the import is executed with the parameter `-O`. ### Parameters The following parameters are used in the `restore.sh` script: **-U (`--username`):** The database user to use. **-d (`--dbname`):** Specifies the target database **-O (`--no-owner`):** This option suppresses all commands to set the ownership of objects. All imported objects are assigned to the database user passed by **-U (--usename)**. **-x (`--no-acl / --no-privileges`):** This option prevents restoration of access privileges, which have been already set. **-c (`--clean`):** This option deletes existing database objects. **`--if-exists`:** This option is required if `--clean` is used. If the option is omitted, error messages would occur if objects do not exist in the target database. ### Restore to Source Database To restore a backup, you can use the following command. This restores the backup of the database 'nmd_frontend_staging' to the database with the same name. At first, all existing data of the database is removed: ```shell-session zstdcat /home/database-backup/postgresql/2021-12-06-0134/customer/nmd_frontend_staging/*.zst | pg_restore -d nmd_frontend_staging -O -x -c --if-exists -U nmd_frontend_staging ``` The parameter `-U` passes the database user name to the `pg_restore` command. The password of the database user is requested interactively during execution. Please note that you must specify the user name and password of the database user and that the access data of the `www-data` user are not requested here. If you do not have this information to hand, it can be read out from the configuration of your application. ### Restore to a Different Database If necessary, the restore can also be made to a different database. In this case, the name of the target database must be adapted. We also recommend that you first import the structure into the new database: ```shell-session # Restore of the database structure zstdcat /home/database-backup/postgresql/2021-12-06-0134/structure/nmd_frontend_staging.zst | pg_restore -d nmd_frontend_staging_restore -O -x -c --if-exists -U nmd_frontend_staging_restore # Restore of the database objects zstdcat /home/database-backup/postgresql/2021-12-06-0134/customer/nmd_frontend_staging/nmd_frontend_staging.zst | pg_restore -d nmd_frontend_staging_restore -O -x -c --if-exists -U nmd_frontend_staging_restore ``` If you want to restore to another database, it must already exist or be [created via `nine-manage-databases`](./nine-manage-databases). ### Restoring Individual Tables Instead of restoring all tables, a single table can also be selected and imported by passing the parameter `-t` and the desired table name to the restore command. Important note: When `-t` is specified, pg_restore makes no attempt to restore any other database objects that the selected table(s) might depend upon. Therefore, there is no guarantee that a specific-table restore into a clean database will succeed. And, while pg_dump's `-t` flag will also dump subsidiary objects (such as indexes) of the selected table(s), pg_restore's `-t` flag does not include such subsidiary objects. ```shell-session zstdcat /home/database-backup/postgresql/2021-12-06-0134/customer/nmd_frontend_staging/nmd_frontend_staging.zst | pg_restore -d nmd_frontend_staging -O -x -c --if-exists -U nmd_frontend_staging -t tablename ``` ## Additional information ### Time of Backups Backups of databases usually create an increased load on your system. Therefore, we have chosen a time period during the night, which is usually less frequented. If the period between 02:00 and 03:00 should be unfavourable for your application(s), we are happy to adjust it. ### Compression of Backups We use the [Zstandard](https://facebook.github.io/zstd/) algorithm to compress the backups. This algorithm offers an excellent balance of speed, compression and resource requirements and in some cases significantly outperforms the established algorithms. If you need the backups for use on a system where the zst binary packages are not available, we advise you to unpack and recompress the backups on your server yourself: ```shell-session www-data@nine01:~ $ mkdir dump_recompress www-data@nine01:~ $ cp /home/database-backup/postgresql/latest/customer/nmd_frontend_staging/* dump_recompress/ ; cd dump_recompress/ # The option "--rm" deletes the zst archives after unpacking www-data@nine01:~/dump_recompress $ unzstd --rm *.zst nmd_frontend_staging.zst: 3732915490 bytes www-data@nine01:~/dump_recompress $ bzip2 * www-data@nine01:~/dump_recompress $ ls -l *.bz2 -rw-r----- 1 www-data www-data 432683536 Dec 6 13:24 nmd_frontend_staging.bz2 ``` Please take care not to store backups publicly accessible on the web server. --- ## Redis-Compatible In-Memory Databases (key-value store) On this page you will find information on Redis-compatible [in-memory databases](https://en.wikipedia.org/wiki/In-memory_database) as a managed service, which store data as [keys and values](https://en.wikipedia.org/wiki/Key%E2%80%93value_database) (key-value store) in non-relational form. Due to [licence changes](https://redis.io/blog/what-redis-license-change-means-for-our-managed-service-providers/) and the associated uncertainty about the future development of Redis, we offer Valkey as a Redis-compatible alternative starting with Ubuntu 26.04 (Resolute Raccoon). See [Which Software Versions Are Available on My Server?](/docs/managed-server-services/operating-system-and-software-versions/which-software-versions-are-available-on-my-server) for the versions available per Ubuntu release. Running a key-value store as a managed service is a good option if your applications are already running on managed servers. Of course, use is also possible for applications on [Nine Kubernetes Engine](/docs/managed-kubernetes/nke/), [Kubernetes vcluster](/docs/managed-kubernetes/nke/kubernetes-cluster-backed-by-vcluster) and [Deploio](/docs/deplo-io/getting-started-with-deploio). Key-Value Stores can also be obtained in self-service as [On-Demand Database](/docs/on-demand-services/) via the Cockpit; configuration options and limitations are described in the linked product overview. ## IP/Port By default, the instance is reachable on port `6379` and is bound to the address `127.0.0.1` and therefore cannot be called from an external system. ## Persistence To ensure that previously written keys are not lost when the service or managed server is restarted, snapshots are regularly written to the hard drive. ## Memory Management ### What Happens If No More Memory Can Be Allocated? A key-value store instance is allocated a quarter of the memory available on the managed server by default. [Predefined strategies](https://redis.com/blog/cache-eviction-strategies/) set by the `maxmemory_policy` define what should happen when the available memory is exhausted. By default, the policy `allkeys-lru` is being used, which behaves as follows: > **allkeys-lru**: Keeps most recently used keys; removes least recently used (LRU) keys This ensures that no data is lost during writing due to a lack of available memory. To free up the required space, keys that have not been accessed for the longest time are automatically deleted. This form of balancing is designed to ensure the highest possible availability of the application in the most variety of scenarios. If you are familiar with your application's access patterns, are dependent on longevity of the keys or are working with TTLs and would like to use e.g. `volatile-lru`, please [contact our support](/docs/general/contact) to adjust the default configuration. An overview of all policies can be found in the [Redis documentation](https://redis.io/docs/reference/eviction/#eviction-policies). --- ## Price & Product Overview Managed Server ProductPrice, ProductSetupPrice, FormatPrice, PriceCalculator, } from "@site/src/components/ProductPrice" Prices in CHF excl. VAT [Price Calculator](https://calculator.nine.ch/?category=Managed%20Server) [Managed Server Product Page](https://nine.ch/products/managedserver/) ## Managed Virtual Server | | Managed nV4 | Managed nV8 | Managed nV16 | Managed nV32 | Managed nV64 | Managed nV-X | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | --------------------------------------------- | --------------------------------------------- | --------------------------------------------- | -------------------------------------------- | | **Monthly fees** | or \* | | | | | On request | | **Setup fees** | | | | | | | | **Virtual CPU (VCPU)** | 4 | 6 | 8 | 12 | 18 | max. 24 | | **RAM** | 4 GB | 8 GB | 16 GB | 32 GB | 64 GB | max. 64 GB | | **Storage space** | 100 GB | 100 GB | 100 GB | 100 GB | 100 GB | 100 GB | \* _: introductory price for Managed nV4 basic version (without additional resources). As soon as additional RAM and vCPUs are added, the monthly fee is _ ### Additional Resources Managed Virtual Server | Additional Resources | Price per Month | | ------------------------- | ------------------------------------------------------ | | Per vCPU | | | Per GB RAM | | | Per 50 GB storage space\* | | _\* Note_: Added storage space can't be reduced. ## Managed Dedicated Server | | Managed Dedicated Server | Managed Dedicated Server Performance | Managed Dedicated Server GPU | Managed Dedicated Server GPU Advanced | | -------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------- | | **Monthly fees** | | | | | | **Setup fees** | | | | | | **Processor** | AMD EPYC 7313P | AMD EPYC 7443P | AMD EPYC 7763P | AMD EPYC 9554 | | **CPU Cores** | 16 | 24 | 64 | 64 | | **RAM** | 32 GB (max. 1TB) | 64 GB (max. 1TB) | 64 GB (max. 2TB) | 64 GB DDR5 (max. 2TB) | | **Hard Drives (Max. number of hard drives)** | 2 x 960 GB SSD (max. 2x NVMe, 2x NVMe/SATA hybrid, 6x SATA SSD drives) | 2 x 960 GB SSD (max. 2x NVMe, 2x NVMe/SATA hybrid, 6x SATA SSD drives) | 2 x 960 GB NVMe SSD (max. 4x NVMe and 8x SATA drives) | No disks included (up to 4x NVMe and 4x SATA or 8x SATA) | | **RAID Configuration** | Software RAID-1 | Hardware RAID-1 | Software RAID-1 | Software RAID-1 | | **GPU Slots** | - | - | 2 | 4 | ### Additional Resources Managed Dedicated Server | Additional Resources | Price per Month | One-time Fee | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | --------------------------------------------------------------------- | | Per additional 32 GB RAM | | | | Per additional 32 GB RAM DDR5 | | | | 2x 960 GB SSD | | | | 2x 1.92 TB SSD | | | | 2x 3.84 TB SSD | | | | 2x 960 GB NVMe | | | | 2x 1.92 TB NVMe | | | | 2x 3.84 TB NVMe | | | | **Nvidia H100 NVL** VRAM 94GB, Memory Bandwidth 3'938 GB/s, Tensor Cores 456, CUDA Cores 14'592, TFLOPS (FP32) 60. Only for Managed Dedicated Server GPU | \* | \* | | **Nvidia L40s** VRAM 48 GB, Memory Bandwidth 864 GB/s, Tensor Cores 568, CUDA Cores 18'176, TFLOPS (FP32) 91.6. Only for Managed Dedicated Server GPU | \* | \* | | **Nvidia RTX 4500 ADA** VRAM 24 GB, Memory Bandwidth 432 GB/s, Tensor Cores 240, CUDA Cores 7'680, TFLOPS (FP32) 39.6. Only for Managed Dedicated Server GPU | \* | \* | | **Nvidia RTX Pro 6000 Blackwell Server Edition** VRAM 96 GB, Memory Bandwidth 1'792 GB/s, Tensor Cores 752, CUDA Cores 24'064, TFLOPS (FP32) 125. Only for Managed Dedicated Server GPU Advanced | \* | \* | _Note_: Added SSD / NVMe storage media can't be removed. \*) For longer commitments (6 months, 1 year, 2 years or more), please [contact us](/docs/general/contact) for an individual quote. Upgrade to a new card anytime! Your existing commitment will continue and be extended by a new term starting from the upgrade date — for the same or a longer duration (e.g. 1–3 years) and at the same or a higher price. ## Features | Features | | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Virtualization (only for Virtual Server) | Full virtualization (All virtual servers are isolated from each other and provided with the booked amount of RAM and CPU cores) | | Connection / Bandwidth | 1 Gbps flat (Fair-Usage) | | Server Management | Monitoring, 7-Day Backup, Firewall, 24/7-Stand-by duty, Support | | Service Level Agreement (SLA) | Basic (included) | | Operating System | Ubuntu LTS | ## Managed Services Two Managed Services are included in the basic price of all Managed Server products. | Managed Service | Price per month | | ------------------------------------- | -------------------------------------------------------------------------------------------- | | Apache | | | Apache with PHP | | | Nginx | | | Nginx with PHP | | | DB-Replication (MySQL/MariaDB) | per instance: | | Docker / Container Runtime (Podman) | | | FTP (Requires Apache or Nginx) | | | IPsec Tunnel | | | Key-Value Store (Redis compatible) | | | Loadbalancing | | | MariaDB | | | Memcached | | | MySQL | | | NFS | | | New Relic Infrastructure | | | OpenSearch (Elasticsearch compatible) | | | OpenVPN | | | PostgreSQL | | | RabbitMQ | | | Shibboleth | | | Solr | | | WireGuard | | ## Additional Options | Additional Options | Price per Month | One-time Fee | | ---------------------------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- | | per additional IPv4 address | | | | Backup retention time 7 days | included | | | Backup retention time 14 days | | | | Backup retention time 14 days, 2 monthly revisions | | | | Backup retention time 30 days, 6 monthly revisions | | | | Backup retention time 30 days, 12 monthly revisions | | | | Custom backup retention | On request | | | TLS certificate (automatically renewed) | | | | Wildcard certificate (automatically renewed) | | | | EV TLS certificate (1 year) | - | | | Multi-domain certificate incl. 2 SAN (1 year) | - | | | Additional SAN for multi-domain certificate (1 year) | - | | | Installing your own certificate | - | | | Let's Encrypt | included | | | ClimateCare (Comprehensive compensation for CO2 emissions) | | | --- ## Managed Service FTPAdmin FTPAdmin is a browser based tool for creating and managing FTP user accounts. The FTPAdmin web interface is accessible over the servers fully qualified hostname and the '/ftpadm' path, f.e. https://myserver.nine.ch/ftpadm . The TLS certificate used for the servers fully qualified hostname is self signed. ## User Administration User accounts share the following properties: - The login is restricted to the SFTP / FTP protocols - No interactive login possible (SSH) - The access can be restricted to a certain sub-directory inside the home directory ### Create When you create the user, you specify a home directory. By default, the home directory is located at /home/www-\*. For easy file sharing between multiple users, you can create multiple users that share a home directory. ### Delete If a user account is no longer required, the account can be deleted via "Edit" -> "Delete this account". The users home directory will not be deleted automatically. ### Edit User accounts can be edited via the "Edit" function in the user overview panel. The users password, login name and home directory can be changed here. ### Change the Admin Password The admin password can be changed with the "change password" function in the title bar. ## Customize Configuration with `.ftpaccess` Similar to Apache httpd and `.htaccess`, the software used in the background allows the configuration to be customized via [`.ftpaccess`](http://www.proftpd.org/docs/howto/ftpaccess.html) files, directly in the directory structure. The configuration is being applied recursively, which means that it applies to the respective directory as well as to all subordinate directories. Please note that no `` sections are required when using `.ftpaccess`. ### Restrict Access Rights Access to the directory structure can be restricted by using [Limits](http://www.proftpd.org/docs/howto/Limit.html). #### Example: Restrict Access to Certain IP Addresses ```apacheconf title=".ftpaccess" Allow from 5.4.5.6 Allow from 5.4.5.7 Deny from all ``` #### Example: Read-Only Access for Specific User ```apacheconf title=".ftpaccess" AllowUser testuser01 DenyUser testuser01 ``` ## Access via SFTP / FTP(S) All users set up through FTPAdmin can connect to the server both via FTP(S) and `scp`/SFTP. For `scp`/SFTP , port `1122` must be used to connect to the server. If you're `scp` on the command line, the port can be specified with `-P` in the command line as follows: ```bash $ scp -P 1122 source.file user@server.nine.ch:target ``` SSH public key authentication is available alongside password authentication. The SSH public key must be stored in RFC4716 format in the file `~/.sftp/authorized_keys` in the home directory of the specific user. The following command can be used to convert an SSH public key from OpenSSH format to RFC4716: ```bash $ ssh-keygen -e -f .ssh/id_rsa | grep -v Comment ``` **Important**: The comment must be removed from the public key. ### Windows Clients We recommend the following clients for Windows: - [Fillezilla](https://filezilla-project.org/) - [Cyberduck](https://cyberduck.io/) - [WinSCP](http://winscp.net/eng/docs/lang:de) ### macOS We recommend the following clients for macOS: - [Cyberduck](https://cyberduck.io/) - [Transmit](https://panic.com/transmit/) - [Fillezilla](https://filezilla-project.org/) --- ## Managed Service Podman Podman is a daemonless container engine for developing, managing, and running OCI containers on Linux. On our managed servers, you can run containers in rootless mode. Because the syntax is similar to Docker in many cases, you can add the following alias for convenience: ```shell-session alias docker=podman ``` Keep in mind that Podman is not a drop-in replacement for Docker in every scenario. Command syntax and behavior can differ, especially across Podman versions. For full upstream documentation, see: https://podman.readthedocs.io/en/latest/index.html ## Cleanup Jobs On our managed Podman service, we create two cleanup jobs by default: - One runs during the night from Sunday to Monday and removes unused images. - One runs during the night before the first day of each month and removes unused volumes. These jobs help keep Podman's disk usage under control and reduce the risk of running out of disk space. If you want to change these cleanup jobs, open a support ticket and we can adjust them to your needs. ## Search, Pull, and List Images Search remote registries for images: ```shell-session podman search ``` Filter the search results: ```shell-session podman search ghost --filter=is-official ``` Pull an image locally: ```shell-session podman pull docker.io/library/ghost ``` List local images: ```shell-session podman images ``` :::note Podman can search multiple registries. We recommend using the fully qualified image name, for example `docker.io/library/ghost` instead of `ghost`, to make sure you pull the expected image. ::: ## Run a Container The following example starts a Ghost container. It uses development mode (`-e NODE_ENV=development`), so Ghost uses its built-in SQLite database instead of requiring an external MySQL database. ```shell-session podman run --detach=true --tty --name ghost-cms -p 8080:2368/tcp -e NODE_ENV=development docker.io/library/ghost ``` :::note The `-d` or `--detach=true` option starts the container in detached mode and prints the container ID after startup. The `-t` or `--tty` option allocates a pseudo-TTY, which is useful for interactive use cases. The `-p 8080:2368/tcp` option publishes Ghost's web server on TCP port 2368 inside the container as TCP port 8080 on the host. Use `--detach=true` and `--tty` instead of `-d` and `-t` if you want to generate a Quadlet later with Podlet. Podlet parses the stored `podman run` command more strictly than Podman itself. ::: ## Show Running Containers Use `podman ps -a` to list created and running containers: ```shell-session $ podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5728ad900bc4 docker.io/library/ghost:latest node current/inde... 4 hours ago Up 4 hours ago 0.0.0.0:8080->2368/tcp gifted_edison ``` ## Attach to a Running Container Use the `CONTAINER ID` from `podman ps` to attach to a running container: ```shell-session $ podman attach b3376ff455a0 [2020-06-10 09:17:15] INFO "GET /" 200 512ms ``` ## Test the Running Container The container is now reachable on port `8080` on the host. You can verify that Ghost is responding correctly with `curl`: ```shell-session $ curl -s localhost:8080 | grep "og:site" ``` ## Publish the Port with nine-manage-vhosts If you already use managed Nginx or Apache2, you can publish your containerized application with `nine-manage-vhosts` and terminate TLS with Let's Encrypt. Create the virtual host: ```shell-session sudo nine-manage-vhosts virtual-host create testdomain.org --template=proxy --template-variable=PROXYPORT=8080 ``` Register the Let's Encrypt client: ```shell-session sudo nine-manage-vhosts certificate register-client ``` Create the certificate and switch the vhost to the HTTPS proxy template: ```shell-session sudo nine-manage-vhosts certificate create --virtual-host=testdomain.org sudo nine-manage-vhosts virtual-host update testdomain.org --template=proxy_letsencrypt_https --template-variable=PROXYPORT=8080 ``` If you use Apache and want automatic HTTP-to-HTTPS redirects, use the `proxy_letsencrypt_https_redirect` template. ## Compose Files Podman can run Compose files. The command you use depends on the Ubuntu version. On Ubuntu Noble and Resolute, use `podman compose` as the default command for Compose files. If you still have legacy workflows that rely on Docker-compatible commands, `docker compose` is available as a fallback when `podman-docker` is installed. ```shell-session podman compose -f docker-compose.yaml up ``` If `podman-docker` is installed on your server, the following fallback command also works: ```shell-session docker compose -f docker-compose.yaml up ``` Ubuntu 22.04 ships [`docker-compose` v1](https://packages.ubuntu.com/search?keywords=docker-compose&searchon=names), which uses a hyphen: ```shell-session docker-compose -f docker-compose.yaml up ```
Install the latest [v2](https://github.com/docker/compose) version of `docker-compose` ```shell-session mkdir -p ~/bin curl -L $(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.assets[] | select(.name == "docker-compose-linux-x86_64") | .browser_download_url') -o ~/bin/docker-compose chmod +x ~/bin/docker-compose ```
## Example Compose File The following example runs Ghost with a Compose file. It uses the default rootless network backend, `pasta` on Resolute and `slirp4netns` on Jammy and Noble, so no explicit network mode is required in the Compose file. This example connects Ghost to a MySQL database that already runs on the host and is named `nmd_ghost`. To reach a host database from a rootless container, connect to `host.containers.internal` instead of `localhost`. If the database listens only on the host loopback, make sure the network backend is configured accordingly as described in [Connecting to a Database on the Host](#connecting-to-a-database-on-the-host). Create your `docker-compose.yaml` file: ```yaml title="docker-compose.yaml" # This example configures Ghost to use a local MySQL database running on the host. services: ghost: image: docker.io/library/ghost restart: always ports: - 8080:2368 environment: # see https://docs.ghost.org/docs/config#section-running-ghost-with-config-env-variables database__client: mysql database__connection__host: host.containers.internal database__connection__user: nmd_ghost database__connection__password: EeNae5xaoapoh5RoDah1muwu database__connection__database: nmd_ghost # this url value is only an example and is likely wrong for your environment url: http://testdomain.org ``` Start the Compose file as follows: ```shell-session podman compose -f docker-compose.yaml up ``` ```shell-session docker-compose -f docker-compose.yaml up ``` Add `-d` if you want to run the Compose stack in the background. ## Start Containers Automatically with systemd We recommend creating a systemd user service so the container starts automatically after a reboot. The exact workflow depends on the OS version. Although newer Podman versions still support `podman generate systemd`, we recommend Quadlets on Ubuntu Noble and Resolute because they produce clearer and easier-to-control systemd units. Quadlet is a format for Podman-managed systemd units that is built into Podman 4.4 and later. A Quadlet file ends with `.container`, `.image`, `.volume`, `.network`, or `.kube` and is typically stored in `~/.config/containers/systemd/` for rootless services. Quadlet files use the same general structure as regular systemd unit files. Standard sections such as `[Service]` and `[Install]` are passed through to systemd, while Podman-specific sections such as `[Container]` describe how the final `podman run` command should be generated. For upstream details, see the [Podman Quadlet documentation](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html). If Podlet is available on your server, you can use it to generate a ready-to-use Quadlet file from an existing container. Create the container with a stable name and all the environment variables, volume mounts, and published ports that you want Podlet to capture: ```shell-session podman run --detach=true --tty --name ghost-cms \ -e NODE_ENV=development \ -e database__connection__filename=/var/lib/ghost/content/data/ghost.db \ -v some-ghost-data:/home/www-data/ghost \ -p 8080:2368/tcp \ docker.io/library/ghost ``` Then generate the Quadlet file: ```shell-session podlet --unit-directory --install --overwrite generate container ghost-cms Wrote to file: /home/www-data/.config/containers/systemd/ghost-cms.container ``` :::note Use `podman ps` and `podman pod ps` to list container and pod names. ::: The Quadlet file is written to `~/.config/containers/systemd/`: ```shell-session cat ~/.config/containers/systemd/ghost-cms.container [Container] Environment=NODE_ENV=development database__connection__filename=/var/lib/ghost/content/data/ghost.db Image=docker.io/library/ghost PodmanArgs=--tty PublishPort=8080:2368/tcp Volume=some-ghost-data:/home/www-data/ghost [Install] WantedBy=default.target ``` Edit the file as needed. Then reload the user systemd instance: ```shell-session systemctl --user daemon-reload ``` :::note `systemctl --user daemon-reload` does not create a separate static unit file. Quadlet files are converted into generated units by a systemd generator. Because these generated services are transient, they cannot be enabled with `systemctl --user enable`. Instead, systemd applies the supported keys from the Quadlet file's `[Install]` section during generation, which is why `WantedBy=default.target` still enables autostart on reboot. ::: Start the service: ```shell-session systemctl --user start ghost-cms.service systemctl --user status ghost-cms.service ● ghost-cms.service Loaded: loaded (/home/www-data/.config/containers/systemd/ghost-cms.container; generated) Active: active (running) since Wed 2024-12-04 13:39:10 CET; 1s ago Main PID: 540254 (conmon) Tasks: 25 (limit: 3337) Memory: 80.8M (peak: 80.8M) CPU: 1.100s CGroup: /user.slice/user-33.slice/user@33.service/app.slice/ghost-cms.service ├─libpod-payload-805ff2ecbcaa737f9b969dd21323b0115fb3c46c7800ab88563663a1e41341b8 │ └─540256 node current/index.js └─runtime └─540254 /usr/bin/conmon --api-version 1 -c 805ff2ecbcaa737f9b969dd21323b0115fb3c46c7800ab88563663a1e413> ``` :::note Podlet can also generate Quadlets from Podman commands or Compose files. For full documentation, see the [Podlet README](https://github.com/containers/podlet/blob/main/README.md). ::: Generate a systemd user unit for the pod named `app`: ```shell-session mkdir -p ~/.config/systemd/user/ podman generate systemd --new --name app > ~/.config/systemd/user/app.service ``` If the container or pod already exists, for example because it was created with Compose, omit `--new`. :::note Use `podman ps` and `podman pod ps` to list container and pod names. ::: Then reload systemd and enable the service: ```shell-session systemctl --user daemon-reload systemctl --user enable app.service ``` ### Set Resource Limits You can limit container resources either with Podman options such as `--cpus` or `--memory` or with systemd settings such as `CPUWeight` or `MemoryMax`. For more details, see our [resource limits](./applications/manage-daemons-as-a-user-with-systemd#resource-limits) documentation. ## Networking ### Modes (`pasta`, `slirp4netns`, `host`) The default rootless network backend depends on the Podman version. Since Podman 5.0, the default is `pasta`; before that, it is `slirp4netns`. On our managed servers, Ubuntu Resolute (Podman 5.7) therefore defaults to `pasta`, while Ubuntu Jammy (Podman 3.4.4) and Ubuntu Noble (Podman 4.9.3) default to `slirp4netns`. Containers in the same Podman pod share the same network namespace, so they share the same IP address, MAC address, and published ports and can always reach each other through `localhost`. You can also use `--network=host`. In host mode, the container shares the host's network namespace and does not get its own IP address. This is the simplest way to reach services on the host, but it removes network isolation. ### Connecting to a Database on the Host Inside a rootless container, `localhost` and `127.0.0.1` refer to the container itself, not to the host. An error such as `connect ECONNREFUSED 127.0.0.1:3306` usually means that the application is trying to reach a database on its own loopback interface. To reach a database on the host, connect to `host.containers.internal`. Podman usually adds that hostname automatically to the container's `/etc/hosts`. The details depend on the network backend: Resolute defaults to `pasta`. In the default Podman configuration, `host.containers.internal` resolves to `169.254.1.2` because Podman passes `--map-guest-addr 169.254.1.2` to `pasta`. In most cases, connecting to `host.containers.internal` is enough: ```shell-session podman run -dt -p 8080:2368/tcp \ -e database__connection__host=host.containers.internal \ docker.io/library/ghost ``` If you override `pasta` defaults in `~/.config/containers/containers.conf`, make sure the configuration still provides `--map-guest-addr 169.254.1.2`, for example: ```ini title="~/.config/containers/containers.conf" [network] pasta_options = ["--map-guest-addr", "169.254.1.2"] ``` Jammy and Noble default to `slirp4netns`. If the database listens only on the host loopback, enable `allow_host_loopback` and connect to `host.containers.internal` or `10.0.2.2`: ```shell-session podman run -dt -p 8080:2368/tcp \ --network slirp4netns:allow_host_loopback=true \ -e database__connection__host=host.containers.internal \ docker.io/library/ghost ``` :::note Alternatively, use `--network=host` so that `localhost:3306` inside the container points directly to the host database. This is simpler, but it removes network isolation. ::: ### Rootless Mode Our containers run rootless. Podman creates the rootless networking automatically. ### Port Publishing Rootless containers can publish only unprivileged ports on the host. Ports below 1024 are privileged and therefore cannot be published directly by a rootless container. The application can still listen on any port inside the container. What matters is that you publish an unprivileged host port. In this example, Ghost listens on port 2368 inside the container and is published on port 8080 on the host: ```shell-session podman run -dt -p 127.0.0.1:8080:2368/tcp -e NODE_ENV=development docker.io/library/ghost ``` Use `-P` if you want Podman to assign a random free host port automatically. Check the published ports: ```shell-session $ podman port -a c0194f22266c 2368/tcp -> 127.0.0.1:8080 ``` ### Container-to-Container Communication There are several ways for rootless containers to communicate. If both containers run in the same Podman pod, they can reach each other directly through `localhost`. If they run separately, the simplest approach is often to use published ports on the host: ```shell-session podman ps ``` Show the published ports and the host IP address: ```shell-session podman port ip a ``` Start another container and connect to the published port through the host IP: ```shell-session podman run --rm docker.io/curlimages/curl -s :8080 ``` ## Volumes: Mount Volumes and Directories into a Container To persist data, either write it to an external service such as a database or mount a volume or directory into the container with the `-v` [volume](https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options) flag. The two main persistence options are: - Bind mounts - Named volumes Both survive container recreation. As a rule of thumb, use a named volume when Podman should manage the data location for you, and use a bind mount when the container needs a specific file or directory from the host. :::note Podman also supports non-persistent mounts such as in-memory `tmpfs` mounts and overlay mounts with the `:O` option, whose changes are discarded when the container stops. ::: ### Bind Mounts A bind mount maps an existing host path into the container, for example `-v /home/user/config:/etc/app/config`. This is useful for configuration files or content that you also want to edit directly on the host. Because the containers run rootless, container UIDs and GIDs are mapped to subordinate IDs on the host. Files written through a bind mount may therefore be owned by a mapped UID that your normal user cannot remove directly. Remove them with: ```shell-session podman unshare rm -r ${bind_mount_dir} ``` ### Named Volumes Named volumes are managed by Podman and can be listed, inspected, and removed with the Podman CLI, for example `podman volume ls`, `podman volume inspect`, and `podman volume rm`. They are stored under `~/.local/share/containers/storage/volumes/`. Because Podman manages their location and lifecycle, named volumes are the recommended default for application data such as databases. ### Mount Options Append options after a colon, for example `-v some-data:/data:ro`. The most useful options on a managed server are: - `:ro` mounts the volume read-only, which is useful for configuration or shared reference data. - `:U` recursively changes the ownership of the source to the UID and GID used by the container process. This can fix permission errors, but it walks the full directory tree and can therefore be slow on large volumes. :::note Volume contents are not guaranteed to be in a consistent state, for example when they belong to a running database. See [Backup](#backup) for recommendations. ::: ## Backup All volume data is included in the [managed backup](./data-backups-and-restores) of your server. Even so, we recommend creating application-level backups as well for two reasons: - You cannot restore individual files or directories from a volume backup, only the whole volume. - Depending on the workload, for example a running database, the volume contents may not be in a consistent state at backup time. We therefore recommend creating regular backups of important application data, for example database dumps, and writing them to a mounted volume. Those files are then backed up by the regular backup routine for your server. --- ## Managed Service Solr Apache Solr is an open source search server based on Java. We offer you the possibility to let us run Apache Solr as a managed service. Nine is responsible for configuration, patching and monitoring of the service, while you can create Solr cores yourself, fill them with data and use them within your applications. The official documentation for Solr can be found here: https://solr.apache.org/guide/solr/latest/ ## IP/Port By default, Solr is accessible on port 8983. The service is bound to the localhost by default, so it cannot be accessed externally. If you use a separate server for Solr, we can adjust the configuration accordingly so that the service can be accessed from your web server, for example. If further external access is required (e.g. to tTo increase the security of the service, Solr can be operated with user authentication. In this case, authentication with username and password must be performed for each API request. This also applies to other applications that want to connect to Solr. ## Authentication To increase the security of the service, Solr can be operated with user authentication. In this case, authentication with username and password must be performed for each API request. This also applies to other applications that want to connect to Solr. When operating with user authentication, we will provide you with a user account. In this case you will find the authentication data in your user directory (usually `/home/www-data`) in the file `.solr_client_credentials`: ```bash www-data@host:~ $ cat .solr_client_credentials httpBasicAuthUser=client httpBasicAuthPassword=LRaXifKBn3LNfT3g ``` ## Cores Management By default, we configure a default core based on the default configset provided with Solr. However, this is only intended for test operation. For production operation, a separate core should be created. Since Solr runs under a separate system user `solr` and the cores must be created in the `/var/lib/solr/$VERSION` directory, you cannot use the script provided by Solr to create cores because of the permissions on the server. Instead, we provide the nine-manage-solr script for managing Solr cores. ## nine-manage-solr Solr provides a script to create and delete cores. Other actions like a backup or restore of a core can be triggered via API. However, direct use of Solr's own script is not possible due to system permissions of your user. Instead, with `nine-manage-solr`, Nine provides all the necessary functions for managing Solr cores. The script must be called by you via `sudo`. With the `--help` option you can display the available commands: ``` Usage: ------- nine-manage-solr core list - List cores nine-manage-solr core details - Show details of a given core nine-manage-solr core create [--configdir=] - Create a new core nine-manage-solr core delete - Delete a core nine-manage-solr backup list - List all core backups nine-manage-solr backup create [--backupdir=] - Backup a core nine-manage-solr backup status - Status of last backup nine-manage-solr backup restore [--backupdir=] - Restore a core nine-manage-solr backup restore status - Show restore status nine-manage-solr backup delete - Delete a core nine-manage-solr -h | --help - Show this help Parameters: ----------- -d / --configdir Specify core config directory. Only available for create option. (default: $SOLR_INSTALL_DIR/server/solr/configsets/_default) -d / --backupdir Specify custom backup directory to create and restore core backups from. (default: /var/lib/solr/9.0.0//data/) ``` ### Create Core You can create a Solr Core using ```shell-session sudo nine-manage-solr core create ``` The core is created in `$SOLR_HOME`, for Solr 9 for example under `/var/lib/solr/9.0.0`. This uses a default configuration (default configset), which is not recommended for production use. Usually you will therefore want to use your own configuration, which you can copy to your home folder and pass to the script with the `--configdir` parameter: ```shell-session sudo nine-manage-solr core create --configdir ./configdirectory ``` How a configset must be structured can be found in the [Solr documentation](https://solr.apache.org/guide/solr/latest/configuration-guide/config-sets.html). ### List Cores You can list the available cores with the `list` command: ```shell-session sudo nine-manage-solr core list ``` ### Delete Core You can remove an existing Solr Core with the `delete` command: ```shell-session sudo nine-manage-solr core delete ``` ### Create and Restore Backups With the `backup create` command you can create a consistent backup of a running core: ```shell-session sudo nine-manage-solr backup create ``` The backup is created in the `/data` folder of the corresponding core. Optionally, `--backupdir` can be used to make a backup to a specific directory. With the `backup status` command you can display the creation time as well as the status of a backup for a core: ```shell-session sudo nine-manage-solr backup status ``` The output contains, among other things, the name of the last backup and the creation time. With the `backup restore` command you can restore a core from a backup: ```shell-session sudo nine-manage-solr backup restore ``` The last created backup of the core is always restored. In a future iteration of the integration it will be possible to select the backup to be restored. --- ## Ubuntu Focal Upgrades Ubuntu Focal has [entered the ESM period](./ubuntu-support-periods-and-esm-information) in April 2025. The successor Ubuntu Jammy will be actively supported until April 2027, when it will enter the 5-year ESM period. Nine is therefore not offering Ubuntu Focal upgrades anymore, as the target operating system Ubuntu Jammy will enter the ESM period in less than a year. Nine will instead help you migrate your system(s) to Ubuntu Resolute (26.04). --- ## Ubuntu Support Periods and ESM Information Canonical, the organisation that publishes Ubuntu Linux, supports each version release for a pre-determined period of time. During the term of support, the packages included in the release are updated, thus ensuring a secure environment. Support is divided into two time windows. The regular support period is followed by the "ESM" time window. ## Recommendation for Upgrades Nine advises to upgrade systems regularly. For you as a customer, smaller but more frequent innovation steps result in a more predictable effort than it would be the case with a larger amount of changes. Available software versions Nine provides for the different Ubuntu release are [listed here](./which-software-versions-are-available-on-my-server). Please [reach out to us](/docs/general/contact) to discuss the details of an upgrade or replacement of your current environment. ## What Does ESM Mean? ESM stands for "Expanded Security Maintenance" and describes the time between the end of the regular support period and the end of support for an Ubuntu version. During the ESM period, security-relevant patches are backported by Canonical. This ensures that security updates are also available for software that has already reached the end of its support period. ### Operational Impact of the ESM Period Continued operation during the ESM period will be possible without restrictions. No adjustments from you are required on the start of the ESM period. Nine will make all necessary adjustments to ensure the reliable operation of your servers. ## Additional Costs During the ESM Period ### Costs of the ESM Licence In order to be able to use the Expanded Security Maintenance, Canonical requires the purchase of a license. Nine purchases the ESM license and then passes that license, along with the expense, to the specific server and customer. The license cost is different for virtual and dedicated servers. Nine will inform customers of the additional costs before the start of the ESM period. Subject to price adjustments by Canonical and exchange rate fluctuations, the license costs charged by Nine (as of writing September 2024) are: - per month for virtual servers - per month for dedicated servers The additional costs will be charged for each server operated at the start of the ESM period. ### Additional Costs for PHP Environments Canonical provides only one PHP version per release. This is not reflective of the needs of our customers. Nine therefore provides a different PHP environment on its managed servers. This allows our customers to choose between multiple PHP versions without having to run multiple servers. To ensure that the PHP environment can be kept up-to-date even after the start of the ESM period, Nine includes paid package sources. The charge is applied to all customers who use a PHP environment on their server. As the number of ESM environments decreases, the cost per installation increases. Nine publishes the fee for the upcoming quarterly billing period on this site. For Q1 2026, the following fees will be charged: - per month for Ubuntu Focal servers - per month for Ubuntu Bionic servers _Fees updated: October 1, 2025_ ## Support Period for Ubuntu Versions Ubuntu versions are actively supported for 5 years. This is followed by a 5-year ESM period. | Version | Release | ESM Period | End of Support by Nine | | ------------------- | :--------: | :---------------------: | :--------------------: | | Ubuntu Noble 24.04 | April 2024 | April 2029 - April 2034 | Not yet scheduled | | Ubuntu Jammy 22.04 | April 2022 | April 2027 - April 2032 | Not yet scheduled | | Ubuntu Focal 20.04 | April 2020 | April 2025 - April 2030 | Not yet scheduled | | Ubuntu Bionic 18.04 | April 2018 | June 2023 - April 2028 | expected Q3 2026 | | Ubuntu Xenial 16.04 | April 2016 | April 2021 - April 2026 | Q3 2025 | | Ubuntu Trusty 14.04 | April 2014 | April 2019 - April 2024 | September 2022 | ## Support Period by Nine Nine actively supports customers in upgrading to the next newer Ubuntu version. By means of "in-place" upgrades for virtual servers, or hardware renewal in the case of dedicated servers, there are always suitable opportunities to align system environments with new requirements. Nine aims to replace systems before the end of the ESM period and therefore does not offer operation until the end of the ESM period. ## Detailed Clarifications Every server environment and application is unique. We will be happy to support you with your ESM and system renewal questions by email at or via our [Service Desk Portal](https://portal.nine.ch). --- ## Which Software Versions Are Available on My Server? Our managed servers are usually delivered with the latest Ubuntu LTS release. ## Supported Ubuntu Versions The following overview shows Ubuntu versions that are available for new orders: | Service | Ubuntu 26.04(Resolute Raccoon) | Ubuntu 24.04(Noble Numbat) | | ---------- | :---------------------------------: | :-----------------------------: | | Apache | 2.4.66 | 2.4.58 | | Nginx | 1.28.3 | 1.24.0 | | PHP | 8.4, 8.5 _(\*1)_ | 8.2 - 8.5 _(\*2)_ | | MySQL | 8.4 | 8.0 | | PostgreSQL | 18 | 16 | | Ruby | 3.3 | 3.2 | | Redis | / | 7.0 | | Valkey | 9.0 | / | | Podman | 5.7.0 | 4.9.3 | _\*1_: The provision of two additional versions is planned _\*2_: No further versions will be provided PHP is the most widely used scripting language for delivering websites. Nine provides 4 PHP versions for each Ubuntu version. Nine will provide the two latest PHP versions from an earlier Ubuntu version in the next newer version. Please also refer to the overview of the support period of the respective PHP version: [PHP Release Cycle](https://www.php.net/supported-versions.php) If your software or the service you require is not listed, please contact us by email at or via our [Service Desk Portal](https://portal.nine.ch). ## Older Ubuntu Versions The following table shows older Ubuntu versions that are no longer used for new orders: | Service | Ubuntu 22.04(Jammy Jellyfish) | Ubuntu 20.04(Focal Fossa) | Ubuntu 18.04(Bionic Beaver) | | ---------- | :--------------------------------: | :----------------------------: | :------------------------------: | | Apache | 2.4.52 | 2.4.41 | 2.4.18 | | Nginx | 1.18.0 | 1.18.0 | 1.14.0 | | PHP | 8.0 - 8.3 | 7.4, 8.0 - 8.2 | 7.0 - 7.4, 8.0 | | MySQL | 8.0 | 8.0 | 5.7 | | PostgreSQL | 14 | 12 | 10 | | Ruby | 3.0 | 2.7 | 2.3 | | Redis | 6.0 | 5.0 | 4.0 | | Podman | 3.4.4 | 3.4.2 | 3.4.2 | --- ## Can I obtain root access? nine does not allow privileged access by customers on their "Managed" servers. Access for customers is limited to unprivileged users only. --- ## Connecting to Your Server: SFTP, FTP, and SSH This article will guide you through the process of connecting to your server using the three common protocols SFTP, FTP, and SSH. The different protocols - **SFTP:** Use SFTP when you need a secure and encrypted way to transfer files, especially for sensitive data. - **FTP:** FTP is for file transfers and does not use encryption. It should only be used if no other option is available. - **FTPS** FTPS is FTP with added security through TLS/SSL encryption. It is suitable for secure file transfers. - **SSH:** To remotely connect to your server ## Preparation Before we begin, ensure you have the following information ready: - Server IP address or hostname - Username and password for the user you'd like to connect with - A Client, if needed - Port number: | Protocol | Port Number | | --------- | ----------- | | SSH | 22 | | FTP, FTPS | 21 | | SFTP | 1122 | ## Connecting to Your Server ### Using FTP (File Transfer Protocol) or FTPS (FTP over TLS/SSL) - **Port:** Port 21 is used for both FTP and FTPS. - **User:** Users created via FTPAdmin, nine-manage-vhosts, Linux OS users, and www-data users can use FTP and FTPS. To connect to your server, you can use the `ftp` command: ```shell-session ftp userName@Server-IP_Address ``` ### Using SSH (Secure Shell) - **Port:** Port 22 is used for access via the SSH protocol - **User:** Only users created with nine-manage-vhosts and the www-data user You can simply use the command `ssh`: ```shell-session ssh userName@Server-IP_Address ``` ### Using SFTP (Secure File Transfer Protocol) - **Port:** Port 1122 and 22 are used for access via the SFTP protocol depending on the kind of user - **User:** the users that are created using FTPAdmin, the users created with nine-manage-vhosts and the www-data user can access To connect to your FTP server, you have to use the `sftp` command: FTPAdmin users: ```shell-session sftp -P 1122 userName@Server-IP_Address ``` www-data and nine-manage-vhost users: ```shell-session sftp -P 22 userName@Server-IP_Address ``` ### Access via Client - **Client:** A client should be installed, for example, FileZilla. - **Protocol:** Protocols should be given, for example, if you want to connect via SFTP: sftp:// ### Connect to Your Server with FileZilla FileZilla is a client application that makes connecting to servers via FTP or SFTP a simple task. Follow these steps to connect to your server with FileZilla. **Step 1**: Start FileZilla First, open FileZilla on your local computer (if not installed, you will have to install it). The FileZilla user interface should look similar to the screenshot below. ![Image 1](../../../static/img/Image1.png) **Step 2**: Configure the connection details In the top section of FileZilla, enter the required connection details and click the "Connect" button. **Server:** enter the IP or host name of your server. For this example, "saad-test01". **Important:** you will to enter the protocol you want to use before the host name or ip: - FTP Connection: `ftp://saad-test01` - SFTP Connection: `sftp://saad-test01` - SSH Connection: `saad-test01` **Username:** Enter the username. In this case, we use "saad". **Password:** Enter the password of the user **Port:** choose the port that you want to connect to: - SSH: 22 - FTP: 21 - SFTP: 1122 ![Image 2](../../../static/img/Image2.png) **Step 3**: Access Server Files As soon as the connection to your web hosting is established, FileZilla displays the server-side directory structure on the right side of the user interface. ![Image 3](../../../static/img/Image3.png) ## Access via SCP/SFTP - **Port:** To connect to the server via SCP or SFTP, you must use port 1122. - **Users:** All users set up through FTPAdmin can connect to the server using both FTP(S) and SFTP. When using `scp`, the port can be specified with the command line option `-P` as follows: ```shell-session scp -P 1122 your_file userName@Server-IP_Address:/path/to/destination/ ``` --- ## Cron Jobs The `cron` command-line utility is a job scheduler. You can use cron jobs, to run periodically at fixed times, dates, or intervals. ## Create or Edit Jobs Login to your server using an ssh client. Type `crontab -e` to create or edit your cron jobs. ## Limitations Your crontab file has to end with a new line, otherwise the last cron job will not be executed. ## Run Times The run times in the crontab file are defined as followed this: ```bash ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── weekday (0 - 6) │ │ │ │ │ * * * * * /home/www-data/scripts/job.sh ``` `*` A wildcard for every minute / hour and so on. This cron job runs every minute, in every hour, on every day of the month, every month, on every weekday. More examples can be found on [crontab.guru/examples.html](https://crontab.guru/examples.html). ## Shell, PATH and Mailings ```bash SHELL=/bin/sh PATH=/home/www-data/scripts:/usr/local/bin:/bin:/usr/sbin:/usr/bin MAILTO=user@example.org * * * * * job.sh ``` The cron daemon runs with a basic PATH variable: `/usr/bin:/bin`. If you want to use commands with relative paths, you can extend your `PATH` at the beginning of the crontab file. Alternatively, you can use absolute paths. The default shell of the cron jobs can be defined with the variable `SHELL`. Without changes, the default shell is `dash`. By setting the variable `MAILTO=user@domain.org`, the output of your cron jobs will be sent as an email to the given address. ## Lock Functionality Cron jobs should have a locking functionality to prevent processing the same command multiple times. It is common to execute cron jobs frequently, for example, every minute. The execution time, in tendency, increases with a growing data set. If the execution time of a cron job is greater than the frequency, job overlaps will occur. This will frequently lead to an exponential increase in execution time as well as system resource usage (CPU, RAM). We have made available the `run-one` wrapper which will ensure cron jobs will only run one instance of a job at a time. The wrapper can be used as a prefix for cron jobs like this: ```bash * * * * * /usr/bin/run-one /home/www-data/scripts/job.sh &> /dev/null ``` `run-one` is a wrapper from the likewise named Ubuntu package. More information about "run-one" and similar helpful wrappers can be found in the [Ubuntu manpages](https://manpages.ubuntu.com/manpages/focal/man1/run-one.1.html): ## Optional Output Control - The optional parameter `&> /dev/null` "throws away" the output of the command in the cron job (SYSOUT) - The optional parameter `>/dev/null 2>&1` "throws away" both the output and the error output of the command in the cron job (SYSOUT+SYSERR) ## Examples ### Every 5th Minute at Every Hour Request an URL using `curl`: ```bash 5 * * * * /usr/bin/run-one /usr/bin/curl https://domain.org/cron/run &> /dev/null ``` ### Every 4th Hour Information: The script has to be executable: `chmod u+x /home/www-data/scripts/refresh_cache.sh` ```bash 0 */4 * * * /usr/bin/run-one /home/www-data/scripts/refresh_cache.sh &> /dev/null ``` ### Every Wednesday at 00:30 AM Information: The script has to be executable: `chmod u+x /home/www-data/scripts/weekly_report.sh` ```bash 30 5 * * 3 /usr/bin/run-one /home/www-data/scripts/weekly_report.sh &> /dev/null ``` More information regarding the syntax of the crontab file and more examples can be found in the following Wikipedia article: https://en.wikipedia.org/wiki/Cron --- ## How do I work out a good String Check / Action Plan for my SLA? As part of our Service Level Agreement(SLA), we need two things from you for its implementation: a string check and an action plan. In the following article we would like to explain to you in more detail what these are and what you should pay attention to when creating them. ## What Is the Purpose of a String Check? The purpose of a string check is to verify that a web page behaves as expected and can be delivered properly. Since elements are often generated across multiple application and infrastructure components (e.g. web server, PHP execution, database connection, key-value stores), such a test ensures that your application works correctly and the interaction between services is successful. ## What Can a String Check Look Like? A simple example of such a test is checking for the output of the sales tax number from the imprint. The imprint is usually generated dynamically by your application and draws data from several infrastructure components. Moreover, the sales tax number is not subject to fluctuations and is therefore a reliable indicator. Based on this example, core criteria of a check can already be seen: - The return should be reproducible and consistent - The check should include as many components of the application and infrastructure as possible The string check can also be generated by you in a more complex procedure. It is conceivable that you test individual components of your application in the background and then issue a success message. This success message is checked by us. In case of a deviation, we trigger an alert, which is sent once to your technical contacts, as well as transmitted to our alerting. The following checks can be performed, for example: - Can a database connection be established? - Is write access to the database possible? - Is the creation and access of sessions possible? ## What Must Not Be Checked by the String Check? - Third party dependencies such as systems that are not hosted by Nine or external media dependencies - Dependencies on any other Nine service not covered under the systems SLA - System metrics such as storage space or CPU utilization. These are monitored by Nine, independently of the SLA. ## Adjustments to Your Application It must be possible to check the string check at any time. This includes times when you release a new version of your application. If this cannot be guaranteed, you are obliged to notify Nine Internet Solutions AG in advance of the time of the planned downtime. The monitoring of the string check will then be suspended for this period. You can notify us via https://cockpit.nine.ch under the menu item "Account" - "Events" or via email to . ## Action Plan ## Why Is an Action Plan Needed? An action plan defines steps to be taken in case of deviations from the expected return value of the string check. Without an action plan, we have limited ability to help you keep your application available. When considering the string check or action plan, you should always assume that your application is completely unknown to us. It is therefore essential for us that you document the measures to be taken in the application environment as precisely as possible for us. ## What Do You Have to Consider When Working Out a Plan of Measures? - Assume that we do not know your application. Explain step by step which action we should take under which circumstances. - We execute the action plan "blindly". If a step should only be executed under certain conditions (because it is destructive, for example), these conditions should be clearly formulated. - It is _only_ about your application. Assume that the system environment is working correctly. - For each step, describe the following: - What is it supposed to do? - How will it be executed (exact "step by step" instructions)? - What is the expected result/output? - An action plan may include "if, then, else" conditions. - Document the location where your application creates log files. This information helps in analyzing a problem at hand. ## What Does Not Belong in the Action Plan? - Restart of services / servers that are managed by Nine (e.g. Apache / MySQL). These are monitored by us independently of the SLA, a restart of services in case of failures is entirely at our discretion. - Restart of services in "User Space". If you want to run applications in "user space", you can find more information about this in our article [Manage daemons as users with systemd](../applications/manage-daemons-as-a-user-with-systemd). - Trivial instructions. If your string check can detect error "X" and the solution to this is to flush the cache, you should automate this action. - Escalation. At the moment when we receive the information about an error, you have also already been informed automatically via the stored contact option (SMS or email). We will inform again when the problem is solved or when the actions defined in the action plan cannot solve the problem. From this point on, until the customer actively asks for support, the issue is considered closed for us. ## Acceptance / Adjustments The string check and action plan must be accepted and confirmed by us. We must be notified of any changes to the application that affect the string check or action plan, and the changes to the string check and/or action plan must also be accepted by us. New responsibilities, changed telephone or email contacts or adjustments to existing contacts must be communicated to us immediately so that we can inform you of problems at any time. --- ## How to mount S3 bucket as a file system In this article, we describe the setup and mounting of a S3 bucket in your user space environment with Rclone for Ubuntu 16.04 and newer. This allows you to use the Nine S3 storage as a local file system. Rclone offers a wide range of features and could prevail in our comparison between s3fs and Goofys as the best option in terms of performance and compatibility. ## Requirements - Access Key and Secret for access to S3 bucket (can be accessed in your [Nine cockpit](https://cockpit.nine.ch/de/flightdeck/storage/buckets)). - In order for an unprivileged user to create his own filesystem in the user space, the kernel module FUSE (Filesystem in Userspace) must be installed. The module is available on our managed servers. - Root Server: Fuse is available in the Ubuntu repository and can be installed via `apt`: `apt-get install fuse` ## Installation Download the latest Linux binary (Intel/AMD - 64 bit) (https://rclone.org/downloads/) and place it in your user space: ```bash :~ $ wget https://downloads.rclone.org/rclone-current-linux-amd64.zip :~ $ unzip rclone-current-linux-amd64.zip ; rm rclone-current-linux-amd64.zip :~ $ mkdir ~/bin && mv rclone-v*-linux-amd64/rclone ~/bin/rclone && chmod u+x ~/bin/rclone :~ $ ~/bin/rclone version rclone v1.56.1 - os/version: ubuntu 20.04 (64 bit) - os/kernel: 5.4.0-80-generic (x86_64) - os/type: linux - os/arch: amd64 - go/version: go1.16.8 - go/linking: static - go/tags: none ``` ## Configuration To be automatically recognized by Rclone, the configuration must be placed in the file `~/.config/rclone/rclone.conf`. Adjust the values according to the information for your user and bucket. Multiple endpoints and users can be configured separated by `[section]`. ```bash :~ $ mkdir -p ~/.config/rclone/ ``` ```systemd title="~/.config/rclone/rclone.conf" [s3-nine] type = s3 provider = Other access_key_id = 6aaf50b18357446ab1a25a6c93361569 secret_access_key = fcf2c9c6bc5c4384a4e1dbff99d2cc52 region = nine-cz42 endpoint = https://cz42.objectstorage.nineapis.ch ``` Afterwards, the bucket can be setup as a mount point. The mount point can be an existing directory within your current directory structure or a new directory. A new directory can be created using `mkdir ~/path`. The following command mounts the S3 bucket: > **Note:** > Older versions of rclone use `--vfs-cache-mode write` (without "s"). ```bash :~ $ ~/bin/rclone mount s3-nine: ~/ --vfs-cache-mode writes --use-server-modtime ``` > **Note:** > The Rclone process runs in the foreground in your current shell. You can now open a second shell and check the status. If everything works as desired, you can proceed to the next step and create a systemd service that runs in the background. In our tests `--vfs-cache-mode writes` has been proven the most sensible option between compatibility and disk usage. To improve performance, read operations can also be cached with some implications. For cache modes `minimal` and `full`, disk usage can be higher or certain operations on the file system won't work. With those options, it may also makes sense to adjust the buffer and cache sizes with the parameters `--buffer-size` and `--vfs-cache-max-size`. We suggest looking up the official rclone documentation if you're considering using these modes: https://rclone.org/commands/rclone_mount/#vfs-file-caching ## Auto Start / Monitoring We use a Systemd Service Unit to automatically start Rclone and mount the buckert after a reboot. We create the following service unit configuration in `~/.config/systemd/user/rclone.service`. Adjust the values marked with `< >` according to the information for your user, bucket and mount point: ```systemd title="~/.config/systemd/user/rclone.service" [Unit] Description=rclone mount Documentation=http://rclone.org/docs/ Wants=network-online.target After=network-online.target StartLimitInterval=500 StartLimitBurst=5 [Service] Type=notify Environment=MOUNTPOINT= Environment=REMOTE_NAME=s3-nine Environment=BUCKETNAME= Restart=on-failure RestartSec=5 ExecStartPre=/bin/bash -c "/usr/bin/fusermount -uzq ${MOUNTPOINT} || true" ExecStart=/usr/bin/env "${HOME}/bin/rclone" mount \ --vfs-cache-mode writes \ --use-server-modtime \ ${REMOTE_NAME}:${BUCKETNAME} ${MOUNTPOINT} ExecStop=/bin/fusermount -uzq ${MOUNTPOINT} [Install] WantedBy=multi-user.target ``` The new Systemd configuration must then be loaded with the command `systemctl --user daemon-reload`. In order for the service to start automatically after a system reboot, the following command must be executed: `systemctl --user enable rclone.service`. The newly created service unit can now be started or the status of the unit can be retrieved with the following commands: ```bash :~ $ systemctl --user start rclone.service :~ $ systemctl --user status rclone.service ● rclone.service - rclone mount Loaded: loaded (/home/www-data/.config/systemd/user/rclone.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-09-30 15:45:56 CEST; 1min 2s ago Docs: http://rclone.org/docs/ Main PID: 12582 (rclone) Status: "[15:46] vfs cache: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)" CGroup: /user.slice/user-33.slice/user@33.service/rclone.service └─12582 /home/www-data/bin/rclone mount --vfs-cache-mode writes --use-server-modtime s3-nine:bucket1 /home/www-data/testmountpoint Sep 30 15:45:56 server systemd[805]: rclone.service: Service hold-off time over, scheduling restart. Sep 30 15:45:56 server systemd[805]: Stopped rclone mount. Sep 30 15:45:56 server systemd[805]: Starting rclone mount... Sep 30 15:45:56 server systemd[805]: Started rclone mount. ``` ## Update Updates of Rclone can be done using the built-in "selfupdate" function. This will download the latest version marked as "stable" and replace the binary that was used before: `~/bin/rclone selfupdate` ## Troubleshooting Error: ```bash :~/s3mount $ ls ls: cannot open directory '.': Transport endpoint is not connected ``` If Rclone mounted the bucket to the path you had your current shell session pointing to, you must change to the directory again (`cd; cd -`). Error: ```bash :~ $ ~/bin/rclone mount s3-nine:test-bucket ~/s3mount --vfs-cache-mode writes --use-server-modtime 2021/09/23 14:27:56 Fatal error: Can not open: /home/www-data/s3mount: open /home/www-data/s3mount: transport endpoint is not connected ``` If the Rclone process terminates unexpectedly, the mount point must be removed with the command `fusermount -u `. After a restart of the systemd unit the mountpoint should be available again: `systemctl --user restart rclone.service`. --- ## inodes An inode (index node) is a fundamental data structure in Linux file systems that stores essential metadata about files and directories, including: - File type - Ownership (user and group) - Permissions - File size - Data location - Timestamps Every file and directory in `ext` file systems (e.g. `ext2`, `ext3`, `ext4`) is represented by a unique inode. ## Limitations 1. The total number of inodes is fixed when creating the file system. 2. Once set, this number cannot be changed. 3. Each ext4 inode occupies 256 bytes of storage space. ## Allocation `ext4` uses a ratio of 1 inode per 16 KiB of disk space. This means that a 100 GB partition will have about 6 million inodes available. ## Notifications Customers with a managed server will receive an automatic notification by email at 80 respectively at 90% inode usage. If you have received such a notification, you can use the following command to find out where in the file system these inodes are being used: ```shell-session www-data@myserver:~ find /home /tmp -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -nr | head -n 20 ``` An overview of the current inode usage can be displayed with the following command: ```shell-session df -ih ``` ### Cron Job Typically, increased inode usage involves many small files, such as temporary files like sessions or cache. In such cases, we recommend using a cron job to automatically delete files older than 14 days: :::tip For testing purposes, before setting up the cron jobs, please use the command without the `-delete` option, as the command will delete any files found within the specified path without asking. ::: ```cron 0 3 * * * /usr/bin/find /path/to/files* -type f -mtime +14 -delete > /dev/null 2>&1 ``` For more information on setting up a cron job, see the related [support article](./cron-jobs). --- ## Migrate website files/databases to new server This article describes how you can migrate a website from one server to another with as little interruption as possible. Please replace 'YOURDOMAIN' with the domain / directory you wish to migrate. This guide assumes that the virtual host configuration already exists on the target system. ## Preparation To ensure that the migration can be carried out as quickly and with as little downtime as possible, we recommend to prepare the following configurations: **.htaccess** If a `.htaccess` file already exists in the directory of the domain to be migrated, we recommend creating a copy of it. _Source system_ ```bash cp /home/www-data/YOURDOMAIN/.htaccess /home/www-data/YOURDOMAIN/.htaccess_backup ``` **DNS** We recommend setting the TTL ("Time To Live", validity period of the entry) of your DNS entries to a low value, e.g. 300 seconds, ideally two days in advance. This speeds up the distribution of the DNS entries that need to be adjusted later. **Maintenance page** During data migration, it should be ensured that no data is changing. We recommend delivering a maintenance page for the duration of the data migration. If the software you are using does not support a maintenance function, please create a file called `maintenance.html` in the 'DocumentRoot' of the domain to be migrated. The following content can be stored as an example: ``` (EN) We are currently improving the user experience of our website and look forward to welcoming you again soon. (DE) Wir verbessern derzeit das Benutzererlebnis unserer Webseite und freuen uns, dich in Kürze wieder begrüssen zu dürfen. ``` _HTML version_ ```html title="maintenance.html" Under Maintenance

We’re back soon

(EN) We are currently improving the user experience of our website and look forward to welcoming you again soon.

(DE) Wir verbessern derzeit das Benutzererlebnis unserer Webseite und freuen uns, dich in Kürze wieder begrüssen zu dürfen.

``` To activate the maintenance page at a later time, prepare the necessary `.htaccess` file with the following content: _Source system_ `/home/www-data/.htaccess_maintenance` : ```apacheconf title="/home/www-data/.htaccess_maintenance" RewriteEngine on RewriteCond %{REQUEST_URI} !/maintenance.html$ [NC] RewriteRule .* /maintenance.html [R=302,L] ``` **Proxying to the new server** Despite careful preparation and a reduction in the TTL of a domain's DNS entries, it may take some time for end users to receive the updated the DNS entries. To avoid accesses on the source system, prepare another `.htaccess` file that allows the source system to act as a reverse proxy. This ensures that all accesses are processed on the target system, while the DNS changes are propagated on the Internet. The server subdomain of the target system is used as the target of the reverse proxy configuration. To do this, please create an `.htaccess` file with the name `/home/www-data/.htaccess_proxy` and the following content: _Source system_ ```apacheconf title="/home/www-data/.htaccess_proxy" RewriteEngine on ProxyPreserveHost on RewriteBase / RewriteRule ^(.*)$ http(s)://YOURDOMAIN.NEWSERVERNAME.nine.ch/$1 [P] ProxyPassReverse / http(s)://YOURDOMAIN.NEWSERVERNAME.nine.ch/ ``` ## Migration Before you start the migration, first activate the prepared maintenance page. This ensures that no more changes can be made to the data on the source system. _Source system_ ```bash cp /home/www-data/.htaccess_maintenance /home/www-data/YOURDOMAIN/.htaccess nine-flush-fpm ``` `nine-flush-fpm` empties the PHP OpCode cache. This ensures that the modified `.htaccess` is read immediately. If the maintenance page appears as intended, you can start the data synchronization to the target system: > Note: We recommend carrying out an initial synchronization a few days before the planned migration if you have a > large volume of data. This drastically reduces the time required for the data synchronization. The `--delete` option deletes data from the `/home/www-data/YOURDOMAIN/` directory on the target system that does not (or no longer) exist on the source system. This ensures that the data deleted on the source system after a previous data synchronization is also removed on the target system. ```bash rsync -avzH --delete /home/www-data/YOURDOMAIN/ \ NEWSERVER:/home/www-data/YOURDOMAIN ``` During the copying process, you can take care of the database migration. If an up-to-date version of the database is required, please proceed as follows. Please replace the placeholder values `YOURDATABASEUSER`, `DATABASENAME`, `SSHUSER` und `NEWSERVER` with the corresponding values. _Source system_ ```bash mysqldump --opt --quote-names --single-transaction --user=YOURDATABASEUSER --password DATABASENAME | \ ssh SSHUSER@NEWSERVER mysql --user=YOURDATABASEUSER --password DATABASENAME ``` The password prompt requires you to enter the password of the SSH user with whom you want to connect to the target system. Nine creates nightly backups of your databases. If the previous night's backup is sufficient, you will find the necessary information in [this article](../databases/mysql-backup-and-restoration). Once the data synchronization and the import of the database have been completed, you can replace the file `/home/www-data/YOURDOMAIN/.htaccess` with your saved `.htaccess`. The `.htaccess` used to activate the maintenance page was also copied by the data synchronization. _Target system_ ```bash cp /home/www-data/YOURDOMAIN/.htaccess_backup /home/www-data/YOURDOMAIN/.htaccess ``` Under the URL "http(s)://YOURDOMAIN.NEWSERVERNAME.nine.ch/" you can access and test your website on the target system. Please note that many CMS redirect to the domain used. It may therefore be necessary to adapt the CMS before the migration or change the `hosts` entries on your PC for the test phase. A guide for adapting the `hosts` file, see [How to Edit Your Hosts File](https://www.howtogeek.com/27350/beginner-geek-how-to-edit-your-hosts-file/). After you have checked the correct functioning on the target system, you can activate the prepared proxy configuration on the source system. _Source system_ ```bash cp /home/www-data/.htaccess_proxy /home/www-data/YOURDOMAIN/.htaccess ``` All accesses are now processed by the target system. To complete the migration, the DNS entries must be adjusted. Adjust all DNS A entries that point to the source system so that they point to the target system. Depending on the TTL of the entries and the geographical location of the accesses, this adjustment may take effect with a delay of up to 24 hours. A few days after the DNS adjustment, make sure that it has been carried out successfully. The easiest way to achieve this is to deactivate the `.htaccess` file that provides the reverse proxy configuration: _Source system_ ```bash mv /home/www-data/YOURDOMAIN/.htaccess /home/www-data/YOURDOMAIN/.htaccess_disabled ``` If the DNS adjustment has been made correctly, all access to the website should continue to function without error. --- ## Out-of-Memory (OOM) Event The Linux kernel constantly monitors a system's memory usage. If usage reaches a critical threshold, the kernel will issue an "out of memory" routine. In order to avoid instability of the entire system, this process will typically terminate the most memory-intensive processes at this point in time. The routine does not stop processes in a coordinated manner. This may have a negative effect on data integrity, for example, in database services. If these "out of memory" events happen regularly, we strongly recommend to upgrade the system memory. ## Notifications We will notify you of all "out of memory" events that occur on your system. You will receive a separate information for each process (e.g., Java, MySQL, PHP) that caused an "out of memory" event in the past six hours. ## Often Affected Processes Since the Linux kernel prefers to terminate processes that use a lot of memory, some processes are affected more than others: - MySQL - Java - User space processes, for example Atlassian software - OpenSearch / Elasticsearch - PHP-FPM in the webserver context - PHP CLI processes, for example PHP executed by cron jobs Please take into consideration that the resource usage of these processes depends on the amount of accesses to your web page or the amount of data processed. In the vast majority the root cause of these "out of memory" events will be found within the application environment. ## Linux Memory Usage and Memory Usage Display When using tools like `(h)top` or `free` to check the memory usage, there is a fair chance of being misled by the metrics shown. These are: - `total`: Total memory available - `used`: Memory used by services "directly" - `free`: Unused memory, excluding `buff/cache` - `shared`: Mostly irrelevant for systems managed by Nine - `buff/cache`: Memory used by kernel buffers, file and page caches and shared memory segments - `available`: Total minus used memory One often neglected metric is `buff/cache`. Some services don't directly allocate memory in a way the kernel shows it in the `used` category. Notable examples include PostgreSQL and NFS. These services almost only allocate file and page caches. This might be misleading as it could indicate that a system is oversized. As an example, one of our customer servers with 128 GB memory running PostgreSQL shows these metrics: ```bash root@redacted:~ # free -m total used free shared buff/cache available Mem: 128752 9823 4959 19814 113970 98170 Swap: 7811 75 7736 ``` The fact that less than 10 GB of memory is used, yet there are over 110 GB in the buffers and cache, demonstrates how deceptive it can be to only look at one of these metrics. Another system with 512 GB memory running MySQL shows these metrics: ```bash root@redacted:~ # free -m total used free shared buff/cache available Mem: 515773 392181 7565 3 120592 123591 Swap: 8191 1714 6477 ``` In a stark contrast to the PostgreSQL system, the majority of memory usage is found in the `used` area, while still making intense use of the `buff/cache` area. While the kernel can potentially free up some memory in the `buff/cache` area, it's important to understand that this comes with side effects, such as higher latencies caused by more hard disk reads and writes. A reasonably sized environment should always have some spare room in the form of used `buff/cache`. Very low `buff/cache` usage and a `used` memory amount close to the `total` memory often indicate a shortage without room for growth and unforeseen events. In fact, these are the systems on which we see "out of memory events" most regularly. High `Swap` usage also indicates an issue, especially for smaller sized environments. This usually becomes an issue if the `Swap` area usage exceeds ~50%, the `used` memory is close to the `total` memory and there is little to no `buff/cache` used. Using `Swap` heavily will cause high latencies as `Swap` content needs to be written to and read from disk, which generally should be avoided. A slight usage of `Swap` in the area of a few mega byte isn't concerning though. The kernel might decide to reallocate used memory to the `Swap` area, for example if the algorithm sees very little use for a part of the `used` memory. We know it's challenging to interpret and understand metrics within each system's context, especially when a system serves more than one purpose. We are happy to guide you through this process and will recommend a sizing that's right for your application and system. ## Database Service Memory Usage Database services are a critical part of application performance. Regardless of the chosen database engine, database services aim to minimize hard disk reads for requested data by using internal caches. Database developers recommend allocating up to 70% of a system's memory for caching. Ideally, the caches are large enough to contain **all** database contents. With a database growing in size, this might not be possible or not feasible within a given budget. This recommendation applies to systems that **only** run a database service. In most cases, a system will run multiple services alongside the database service, such as a web server, a PHP environment and a key value store. Nine therefore automatically adjusts the database cache configuration to fit the chosen system size. It is common to see database services use between 40% and 60% of a system's memory. High memory utilization by a database service does not indicate a performance issue. In fact, it's the desired state, as performance for every application depending on the database service could otherwise significantly degrade. ## Show Current Memory Usage You can get an overview of the memory usage with the following shell command: ``` www-data@server:~ # ps -eo pid,cmd,%cpu,%mem --sort=-%mem | head -n 11 PID CMD %CPU %MEM 986 /usr/sbin/mysqld --daemoniz 0.1 7.3 125872 ruby2.5 /usr/lib/hello-worl 0.0 2.0 234301 ruby2.5 /usr/lib/find-file- 0.0 1.4 208475 ruby2.5 /usr/lib/find-dir-a 0.0 1.2 310 ruby2.5 /usr/lib/find-dir-b 0.0 1.0 1325 ruby2.5 /usr/lib/find-dir-c 0.0 0.9 125826 ruby2.5 /usr/lib/find-dir-d 0.0 0.9 126039 ruby2.5 /usr/lib/find-dir-e 0.1 0.9 2089 ruby2.5 /usr/lib/find-file- 0.1 0.8 166352 ruby2.5 /usr/lib/exec-comma 0.4 0.7 ``` This will show the 10 processes using the most memory. Please mind that this will be a snapshot of this very moment and depending on the application or service, the memory usage might vary by a significant margin in a short time frame. ## Order Additional Memory If you need additional memory or have questions about this message, please feel free to contact us: --- ## Rotate Log Files Sometimes log files can become quite large. To prevent this, they should be "rotated" frequently. `logrotate` is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large. We cover the configuration of `logrotate` for the log files of a Tomcat 8 instance in this article, which are located in `/home/www-data/example.ch/logs/catalina.out`. ## Configure Logrotate You will need to add a configuration file (e.g. as a hidden file in the home directory): `~/.logrotate.conf`. A simple configuration might look like this: ```logrotate title="~/.logrotate.conf" ~/*/logs/catalina.out { daily rotate 14 notifempty compress copytruncate compresscmd /usr/bin/zstdmt compressoptions -18 -T0 --rm -qq compressext .zst uncompresscmd /usr/bin/unzstd } ``` With this configuration, `logrotate`: - rotates all files matching `~/*/logs/catalina.out` once per day - keeps 14 log files - does not rotate if the file is empty - compresses the log files with [ZSTD](./zstd) - copies the log file and empties the log file afterwards (`copytruncate`) This will end up in daily log files like this: - `.../catalina.out-20161212.zst` - `.../catalina.out-20161213.zst` - `.../catalina.out-20161214.zst` More configuration options can be found in the `logrotate` [man page](http://www.linuxcommand.org/man_pages/logrotate8.html). ## Execute Logrotate Daily Logrotate can be executed like this: ```bash /usr/sbin/logrotate -s ~/.logrotate.status.tmp ~/.logrotate.conf ``` We have to provide a custom state file with the `-s` option, since the global state file is only writable by the `root` user. To execute this command daily, we should add it to the crontab by executing `crontab -e`. Simply add following line to the end of the file: ```crontab 22 0 * * * /usr/sbin/logrotate -s ~/.logrotate.status.tmp ~/.logrotate.conf >/dev/null 2>&1 ``` Now, the `logrotate` is executed once a day at 00:22. Be aware that with `>/dev/null 2>&1` at the and of the command all output of the command is suppressed. For more information on configuring cron, see the dedicated [Cron Jobs](./cron-jobs) documentation page. --- ## Security recommendations for your Managed Server Ensuring data integrity and confidentiality is crucial for any application or environment. Therefore, we would like to provide you with an overview of recommendations and best practices that are easy to implement within your usual workflows. With these practices in place, you can have a significant impact on the security of your environment with very little effort. ## Passwordless Authentication A best practice for `ssh` and `sftp` access is to use a [public / private key](https://en.wikipedia.org/wiki/Public-key_cryptography) based authentication. After setting your `ssh` and `sftp` up for the key based authentication, Nine will happily disable the possibility to use a password based authentication. This will also negate all `ssh` and `sftp` brute force attacks to your system, which is an improvement over the [brute force protection](/docs/general/why-cant-i-access-my-server-using-sshftp) that is already in place for all Managed systems. ## Use Different SSH Users for Each Environment The user `www-data` is provided for administrative purposes. It is acceptable to use this user in a single project environment. However, if you intend to run multiple projects or versions, or grant access to contractors, it is advisable to create additional users for each project or environment, such as a `staging` or `test` environment. The creation of additional user accounts is documented [in this support article](../webserver/nine-manage-vhosts/nine-manage-vhosts-with-multi-user#user-management). ## Use Different Databases and Database Users for Each Environment The application frontend is the most vulnerable target for attacks. To minimize the risk of a successful attack causing harm to other environments, it is strongly advised to use separate databases and database users for each environment. The same principles that apply to the previously described SSH users also apply here. The main objective is to grant the minimum necessary privileges. ## Restrict Access to Specific IP Addresses If you're using a dedicated IP address range or VPN, we gladly restrict access to specific services or your whole environment to your dedicated IP address(es). ## Avoid Using Insecure Connections and Protocols Data confidentiality can only be achieved if the data transmission channel is properly secured. Therefore, TLS secured connections should always be preferred. FTP is one of the most widely spread insecure protocols. Without further extensions, FTP transfers credentials and data in plain text, making it easy for third parties to intercept this information. We strongly recommend using the FTPS or FTPES extensions, which use TLS encryption, instead of the plain text protocol FTP. If you're using our Managed Service [FTPAdmin2](../ftpadmin), we gladly disable plain text FTP for you. ## Use TLS Certificates for All Your Web Projects We recommend to use TLS connections for all web projects, including test environments. We cover the vast majority of use cases with the free [Let's Encrypt](../webserver/nine-manage-vhosts/nine-manage-vhosts-with-lets-encrypt) integration and offer a variety of [TLS certificates](../tls-ssl-information/how-do-i-order-a-tls-certificate) for more specific use cases. ## HTTP Header for Your Application **Origin** / **Access-Control-Allow-Origin** The [Origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Origin) and [Access-Control-Allow-Origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin) go hand in hand and allow to control what sources can include and access your web application. **Content-Security-Policy** The [Content-Security-Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) controls the resources the user agent is allowed to load for a given page, which can be used as a mitigation against Cross-Site-Scripting (XSS) attacks. **Strict-Transport-Security (HSTS)** The [Strict-Transport-Security](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) causes browsers to access websites only with the https protocol. Yelp released an interesting [blog post](https://engineeringblog.yelp.com/2017/09/the-road-to-hsts.html) from their engineering point of view, we recommend reading the article for larger deployments and especially if you plan to include subdomains in the policy. ## Cloudflare - Attack Detection and Prevention, Performance Improvements The web frontend and web application, which are the most prominent attack targets, can be protected by a CDN. A world-wide distributed CDN not only provides automatic attack detection and mitigation against various attack vectors but also improves the delivery times of your content globally. The CDN uses caching mechanisms to deliver content directly without hitting the backends, effectively reducing load times and system load. Cloudflare utilises various attack vector detections and implements mitigations for popular CMS/CRM platforms such as Wordpress, Drupal, Joomla, Typo3, and Magento. As a Cloudflare partner, Nine is available to provide guidance and assistance in configuring the world's most widely used CDN. --- ## Interpreting security scan reports for your server Application security scans are a great way to review and improve the security of customer-facing applications. Often, these security scans also perform basic infrastructure checks that routinely generate large amounts of false positives or misleading results. This article is a summary of the most common "security issues" reported to us, and will provide you with the information you need to analyze the report you've received. Prefacing the summary, we want to make you aware of our article about [Security recommendations for your Managed Server](./security-recommendations). ## Ubuntu Software Versions and Patching Philosophy Nine uses Ubuntu LTS versions (Long Term Support), which are released every two years. With the release of each version, a major version for each particular software is chosen by Canonical. Over the lifespan of each LTS version, these major versions won't change. To ensure the LTS releases remain safe from critical security issues, Canonical is actively backporting security patches for critical parts of the operating system and software stack. Therefore, while versions may appear to be outdated and potentially vulnerable, they're perfectly safe to use and security issues have been addressed by backported patches from newer releases. A prime example of this is OpenSSH. On Ubuntu Focal, the OpenSSH version appears to be 8.2p1, when connecting to the server: ``` Remote protocol version 2.0, remote software version OpenSSH_8.2p1 ``` However, at the time of writing this article (July 2024), the detailed package information is as follows: ``` apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.11 ``` The installed version `8.2p1-4ubuntu0.11` reflects the patch level (`0.11`) of the OpenSSH server package and illustrates the philosophy with which Canonical backports security updates. The patch level is deliberately not shown in the connection information above. ## SSH OpenSSH is the standard for establishing remote shell connections to servers. This makes OpenSSH a potential target for attacks, requiring a strict security approach in its development. When it comes to this aspect of security, the OpenSSH developers have done exemplary work. > OpenSSH is one of the most secure software in the world. Its defense-in-depth design and code are a model and an > inspiration, and we thank OpenSSH's developers for their exemplary work. These are the words of experts of [Qualys](https://www.qualys.com/), a company well established in the IT security scene with a record of hundreds of security issues found in various software projects over the years. However, no software is free of bugs or issues. Critical security vulnerabilities in OpenSSH have been extremely rare. In addition, the developers have been excellent at triaging and responding to issues that were found. It is important to note that security reports play a "guessing game" when it comes to OpenSSH. By design, OpenSSH presents a version number during the connection handshake to allow negotiation of connection parameters between client and server. > The version number shown during the connection handshake **does not indicate** the patch level of the service. Following is a summary of fixed or disputed issues that we regularly see in security reports: | CVE | Affected OpenSSH Version | Affected Ubuntu Versions | Note | | ------------------------------------------------------------ | :----------------------: | :----------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [CVE-2016-20012](https://ubuntu.com/security/CVE-2016-20012) | < 8.7 | All before Jammy | Disputed issue, ignored on upstream. For more information, please refer to the [Github discussion](https://github.com/openssh/openssh-portable/pull/270). | | [CVE-2020-14145](https://ubuntu.com/security/CVE-2020-14145) | < 8.4 | All before Jammy | Only affects the client integration. Fix possibly breaking RFC's. Ignored on upstream. | | [CVE-2020-15778](https://ubuntu.com/security/CVE-2020-15778) | < 8.3 | All before Jammy | Only affects _authenticated_ users, see [detailed information](https://github.com/cpandya2909/CVE-2020-15778). | | [CVE-2021-28041](https://ubuntu.com/security/CVE-2021-28041) | < 8.2 | Focal | False positive, fixed in `8.2p1-4ubuntu0.2` | | [CVE-2021-36368](https://ubuntu.com/security/CVE-2021-36368) | < 8.9 | All before Jammy | Disputed issue, ignored on upstream. Only applicable to FIDO authentication mechanism (not used at Nine). | | [CVE-2023-38408](https://ubuntu.com/security/CVE-2023-38408) | < 9.3p2 | All | False positive, fixed in:Xenial: `7.2p2-4ubuntu2.10+esm3`Bionic: `7.6p1-4ubuntu0.7+esm1`Focal: `8.2p1-4ubuntu0.8`Jammy: `8.9p1-3ubuntu0.3`Noble: `9.3p1-1ubuntu2` | | [CVE-2023-48795](https://ubuntu.com/security/CVE-2023-48795) | < 9.6 | All | False positive, fixed in:Xenial: `7.2p2-4ubuntu2.10+esm`Bionic: `7.6p1-4ubuntu0.7+esm3`Focal: `8.2p1-4ubuntu0.10`Jammy: `8.9p1-3ubuntu0.5`Noble: `9.6p1-3ubuntu1` | | [CVE-2023-51384](https://ubuntu.com/security/CVE-2023-51384) | 8.9 - 9.6 | Jammy, Noble | Only affects the `ssh-agent` | | [CVE-2023-51385](https://ubuntu.com/security/CVE-2023-51385) | < 9.6 | All | False positive, fixed in:Xenial: `7.2p2-4ubuntu2.10+esm5`Bionic: `7.6p1-4ubuntu0.7+esm3`Focal: `8.2p1-4ubuntu0.11`Jammy: `8.9p1-3ubuntu0.6`Noble: `9.6p1-3ubuntu1` | | [CVE-2024-6387](https://ubuntu.com/security/CVE-2024-6387) | 8.5 - 9.6 | Jammy, Noble | False positive, fixed in:Jammy: `8.9p1-3ubuntu0.10`Noble: `9.6p1-3ubuntu13.3` | | [CVE-2024-39894](https://ubuntu.com/security/CVE-2024-39894) | 9.5 - 9.7 | Noble | False positive, fixed in:Noble: `9.6p1-3ubuntu13.4` | | [CVE-2025-26465](https://ubuntu.com/security/CVE-2025-26465) | 6.8 - 9.9 | Xenial - Noble | False positive, fixed in:Xenial: `7.2p2-4ubuntu2.10+esm7`Bionic: `7.6p1-4ubuntu0.7+esm4`Focal: `8.2p1-4ubuntu0.12`Jammy: `8.9p1-3ubuntu0.11 `Noble: `9.6p1-3ubuntu13.8` | | [CVE-2025-26466](https://ubuntu.com/security/CVE-2025-26466) | 9.5 - 9.9 | Noble | False positive, fixed in:Noble: `9.6p1-3ubuntu13.8` | ## TLS ### Ciphers TLS ciphers define the cryptographic algorithm that is used to establish a TLS connection. Client and server negotiate a cipher during the connection's handshake. When client and server don't support a mutual cipher combination, connections can't be established. To support a broad range of clients without sacrificing security, Nine supports a few ciphers for backwards compatibility that are recommended by the Mozilla Foundation. These ciphers are a compromise between security and compatibility, with the goal of achieving high overall ratings on the well known [Qualys SSL check](https://globalsign.ssllabs.com/analyze.html). If you only want to support the most modern ciphers and client compatibility isn't a concern for you, we're happy to adjust the ciphers offered by the server to your demand. ### Default Vhost and Server Name Indication (SNI) All Nine managed environments have a so called "default vhost" configured. This vhost is used by the webserver if an access can't be assigned to a specific vhost, for example when the IP address of the server is requested directly in the browser. In this case, a simple "No website configured" message is shown. This ensures that search engines do not inadvertently crawl and index customer content under a Nine controlled URL. The default vhost is also served if a client does not support [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication). Otherwise, the first customer vhost (in alphabetical order) would be delivered instead, likely serving unwanted content to the end user. ### Self Signed TLS Certificates The default vhost is also accessible via https. Nine creates a self-signed certificate for this purpose, with the same goals in mind mentioned above. If a customer vhost does not already use a TLS certificate, the self-signed certificate will be used when accessing that vhost using the https protocol. This can be easily avoided by creating a Let's Encrypt certificate for that vhost with our [nine-manage-vhosts](../webserver/nine-manage-vhosts/nine-manage-vhosts-with-lets-encrypt) integration. In case you're using our Managed Service [FTPAdmin](../ftpadmin), you might see a reference in a security report that the FTP service is using a self signed certificate as well. This is intended to support FTP(E)S connections. If required, a [certificate](../tls-ssl-information/how-do-i-order-a-tls-certificate) from a recognized certificate authority can be integrated for the FTP service. ## FTP The existence of an FTP server is often cited as a problem in security reports. We couldn't agree more! The FTP protocol dates back to 1972, with a more "modern" RFC replacing the original in 1985. Unless your application or use case is strictly dependent on FTP, we strongly recommend that you use only SFTP, which can easily replace most use cases for FTP. We're more than happy to deactivate the FTP protocol. This does not have any implication on the SFTP integration of FTPAdmin. ## HTTP Header From time to time, security scans report missing HTTP headers. We agree that certain HTTP headers can increase the security and recommend setting these headers within the scope of the application, either via `.htaccess` or within the code / application itself. Our article about [security recommendations](./security-recommendations#http-header-for-your-application) mentions a few headers, that might be worth looking at. --- ## Zstd Zstd (Zstandard) is a modern compression algorithm developed by Facebook that Nine uses for compressing database dumps and log files. ## Advantages Compared to gzip's deflate algorithm, zstd offers: - Better compression ratios in most cases - Significantly faster compression speeds - Lower CPU usage (20-30% vs gzip's 100%) ## Usage ### On Your Managed Server `zstd`, `zstdcat`, `zstdgrep` and `zstdless` are the most commonly used command line tools for managing zstd archives. The most notable options for `zstd` are `-d` to decompress an archive and `--rm` to delete a compressed source file after it has been decompressed. To learn more about the options, the `--help` command line parameter will show all options for the tools. ### On Your Local Desktop We recommend [7zip](https://www.7-zip.org/) if you want to decompress zstd archives on your local desktop. 7zip is available for all platforms. --- ## Create a Certificate Signing Request (CSR) A CSR (Certificate Signing Request) is a digital request for issuing a TLS certificate. TLS certificates ordered through Nine are not limited to managed services. They can also be used with root servers and third-party infrastructure. :::info[Not required for single-domain and wildcard certificates] If you order a single-domain or wildcard certificate through Nine, you don't need to create a CSR. Nine handles this automatically. ::: Creating a CSR is required for: - EV and multi-domain certificates - Certificates from third-party providers ## Where to Create the CSR Generate the CSR and private key on your Managed Server. This ensures the private key never leaves the secure server environment. ## Preparation This guide uses OpenSSL. On managed environments, OpenSSL is pre-installed. For root environments, install it first if needed. The following steps use default directories for Nine managed environments. Adjust paths for root environments as needed. 1. Create a directory that is not publicly accessible via a web server. Private keys must remain secret: ```shell-session mkdir -p ~/.ssl/ cd ~/.ssl/ ``` 2. Create an OpenSSL config: ```ini title="~/.ssl/openssl.conf" [req] distinguished_name = req_distinguished_name req_extensions = v3_req [req_distinguished_name] [ v3_req ] subjectAltName = ${ENV::SAN} # keyUsage = keyEncipherment, dataEncipherment # extendedKeyUsage = serverAuth ``` ## Create the CSR Run the following command to create both the CSR and private key for your domain. This uses an [ECDSA key with the P-256 curve](https://www.ssl.com/article/comparing-ecdsa-vs-rsa/), the current industry standard for fast and secure TLS handshakes: ```shell-session SAN=DNS:example.com,DNS:www.example.com openssl req -new -subj "/C=CH/ST=Zuerich/L=Zuerich/O=Example AG/CN=example.com/" -sha256 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -keyout SSL_example.com.key -out SSL_example.com.csr -config openssl.conf ``` **Avoid umlauts, special characters (such as French accents), and any abbreviations in the certificate fields.** ### Country Name 2-digit country code per [ISO 3166](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) (CH = Switzerland). ### State or Province Name The canton or province where the person or company is registered. ### Organization Name Your company or association name. For certificates issued to individuals, enter the full name. ### Common Name The domain name to protect with the certificate, or `*.example.com` for wildcard certificates. **Wildcard certificates cover one subdomain level.** For example, `*.example.com` covers `www.example.com` and `staging.example.com`, but not `www.staging.example.com`. To cover `www.staging.example.com`, use `*.staging.example.com`. **The TLS certificate is only valid for the domain specified here.** --- ## Order a TLS Certificate A TLS certificate makes your domain accessible via HTTPS, establishing a secure channel that encrypts data transmission and verifies the server's identity. :::note[TLS vs SSL] TLS (Transport Layer Security) is the successor to SSL (Secure Sockets Layer). While the term "SSL" is still widely used, all modern certificates use the TLS protocol. In this documentation, we use "TLS" to refer to both. ::: :::info[Outlook] The maximum validity of TLS certificates [will be reduced to 47 days by 2029](https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/). We are working on further automating certificate issuance and renewal to keep up with these changes. ::: ## Certificate Types The following certificate types are available through Nine: | Type | Price | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Single domain certificate ¹ ² | / month | | Wildcard certificate ² ³ | / month | | EV certificate | / year | | Multi-domain certificate ⁴ | / year (includes 2 SANs), / year per additional SAN | | Own certificate (installation only) | / certificate | ¹ Covers the `www` subdomain.\ ² Usage-based billing. See note below.\ ³ Covers one subdomain level. For example, `*.example.com` is valid for `docs.example.com`, but not for `dev.docs.example.com`.\ ⁴ Does **not** cover `www` subdomains. Order `www` as an additional SAN. :::info[Usage-based billing] Single-domain and wildcard certificates are billed based on usage. You only pay for the days the certificate is active, invoiced monthly together with your other managed services. ::: ## Place an Order Ordering a TLS certificate requires you to prove control over the domain. You can validate domain ownership using one of these methods: - Create a DNS record - Place a file on the web server - Receive an email at one of these addresses: - `admin@example.com` - `administrator@example.com` - `hostmaster@example.com` - `postmaster@example.com` - `webmaster@example.com` The validation email contains a link and verification code to enter at the certification authority. When ordering, specify the certificate type and your preferred validation method. Send your order to . ## EV Certificate Requirements EV (Extended Validation) certificates require additional information: 1. Administrative contact: - First name - Last name - Email address - Phone number 2. Company data (from phone directory, trade register, or similar): - Company name - Department - Main phone number - Address, state/canton, city, postal code The certification authority may request additional information. EV certificate issuance typically takes several days. --- ## TLS Compatibility All Nine managed services support TLS 1.2 and TLS 1.3 by default. If you require a TLS 1.3-only configuration, contact to disable TLS 1.2. ## Test Your Configuration Use the [SSL Labs Server Test](https://www.ssllabs.com/ssltest/) to analyze your domain's TLS configuration. The scan may flag certain ciphers as weak. Our base configuration is regularly checked for vulnerabilities. It deliberately includes certain weaker ciphers to ensure compatibility with as many clients as possible. For details, see the [security scans](../operations/security-scans.md#tls) article. Alternatively, test from the command line: ```shell-session openssl s_client -connect example.com:443 -tls1_3 ``` ## Client Compatibility TLS 1.2 and 1.3 are supported by all modern browsers and operating systems. For detailed compatibility information, see the [TLS browser support tables](https://caniuse.com/?search=tls) on caniuse. --- ## Managed Service PHP The PHP versions available depend on the Ubuntu version in use. We have summarized an overview of the available PHP versions [in this support article](../../operating-system-and-software-versions/which-software-versions-are-available-on-my-server). PHP versions can be set individually per Virtual Host (when using PHP-FPM). ## PHP-FPM FPM stands for "FastCGI Process Manager". With this integration, the PHP code is interpreted by pre-started PHP processes. Apache passes the requests to the FastCGI Process Manager and does not interpret the code itself. A big advantage of the FPM integration is the increased security when using multiple environments on one server. Each environment is isolated and runs under its own user ID. In addition, thanks to process separation, different PHP versions can be used for different vhosts at the same time. ## Select the PHP Version with [nine-manage-vhosts](../nine-manage-vhosts/manage-virtualhosts-with-nine-manage-vhosts) If PHP-FPM is installed, the desired version can be selected for each virtual host. Create virtual host with PHP 8.0: ```bash www-data@nine01:~ $ sudo nine-manage-vhosts virtual-host create example.org \ --template-variable=PHP_VERSION=8.0 ``` Change to PHP 8.1 for above example: ```bash www-data@nine01:~ $ sudo nine-manage-vhosts virtual-host update example.org \ --template-variable=PHP_VERSION=8.1 ``` ## Empty the PHP OPcache To speed up the loading and parsing of scripts, the PHP OPcache is enabled by default. This reduces the execution time of PHP scripts by storing precompiled bytecode in memory. You can flush this cache, for example after deployments, by calling our tool `nine-flush-fpm`; no admin rights are required for this. `nine-flush-fpm` flushes the cache for all PHP-FPM environments of all users. You can also flush the cache for a specific PHP environment of a user. For this purpose we provide the tool `nine-php-cachetool`: Query status for PHP 8.1: ```bash www-data@nine01:~ $ nine-php-cachetool -v 8.1 status ``` Clear cache for PHP 8.1: ```bash www-data@nine01:~ $ nine-php-cachetool -v 8.1 reset ``` ## Customize PHP CLI Version _Note: This feature is available on Ubuntu 20.04 Focal and newer. Please contact our support for an update via support request in or by email to ._ The available PHP CLI versions can be displayed with the following command: ```bash www-data@nine01:~ $ sudo nine-manage-vhosts php-cli list php7.4 php8.0 php8.1 ``` The available PHP versions can be run directly. ```bash www-data@nine01:~ $ php8.0 --version ``` The currently active version can be displayed as follows: ```bash www-data@nine01:~ $ sudo nine-manage-vhosts php-cli show php8.0 ``` The default version can be customized as follows: ```bash www-data@nine01:~ $ sudo nine-manage-vhosts php-cli update php8.1 PHP version successfully updated to php8.1 ``` --- ## PHP Composer [Composer](https://getcomposer.org/) is a tool for dependency management in PHP. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. ## Installation We recommend installing composer in userspace, as this allows you to use a newer version than the one [provided by Ubuntu](https://packages.ubuntu.com/search?keywords=composer&searchon=names&suite=all§ion=all). 1. Download the setup file: ```shell-session www-data@server:~$ php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" ``` 2. Install composer: ```shell-session www-data@server:~$ php composer-setup.php --install-dir=bin --filename=composer All settings correct for using Composer Downloading...Composer (version 2.1.3) successfully installed to: /home/www-data/bin/composer www-data@server:~$ composer --help Usage: help [options] [--] []Arguments: command The command to execute command_name The command name [default: "help"] ``` ## Update Run the following command to update composer: ```shell-session www-data@server:~$ composer self-update You are already using the latest available Composer version 2.1.3 (stable channel). ``` --- ## PHP Settings With .user.ini A [`.user.ini`](https://www.php.net/manual/en/configuration.file.per-user.php) file allows you to easily customize PHP settings. Starting with Ubuntu Bionic, only `.user.ini` settings are supported. All PHP settings defined in this file will be applied recursively to the folder where the file is located. :::warning Ensure that you're only configuring [changeable settings](#changeable-configuration-options) and to [clear the OPcache](#apply-settings) after making changes. ::: ## Example ```ini title="/home/www-data/example.com/current/.user.ini" register_globals=on upload_max_filesize="5M" ``` ## Apply Settings To ensure all settings come into effect immediately, you'll need to [clear the PHP OPcache](./managed-service-php#empty-the-php-opcache) by running `nine-flush-fpm` after making changes to a `.user.ini`. ## Changeable Configuration Options Only INI settings with the modes [INI_PERDIR](https://www.php.net/manual/en/info.constants.php#constant.ini-perdir), [INI_USER](https://www.php.net/manual/en/info.constants.php#constant.ini-user) and [INI_ALL](https://www.php.net/manual/en/info.constants.php#constant.ini-all) will be recognized in `.user.ini`-style INI files. Check the [list of php.ini directives](https://www.php.net/manual/en/ini.list.php) to determine if a setting is changeable by a `.user.ini`. ## Migration From .htaccess If you have been using our old PHP stack, you will need to migrate your PHP settings from `.htaccess` to `.user.ini`. Simply remove the first keyword (`php_value`, `php_flag`) and replace it with `[KEY]=[VALUE]`. Please surround the PHP settings in the `.htaccess` file with `IfModule` conditions, otherwise they will cause problems after migration: ### Find Affected .htaccess Files To find affected files, use the following command: ```shell-session find /home/www-*/ -type f -name .htaccess -exec grep -nH "php_" {} + ``` ### .htaccess ```apacheconf title=".htaccess" php_value include_path ".:/usr/local/lib/php" php_flag display_errors Off php_value upload_max_filesize 500M ``` ### .user.ini ```ini title=".user.ini" include_path=".:/usr/local/lib/php" display_errors=Off upload_max_filesize=500M ``` --- ## Blocking Bot Requests Websites are frequently made inaccessible due to an overwhelming surge in traffic, which is often caused by aggressive bot activity. While traditional Distributed Denial of Service (DDoS) attacks require specialized protection, such as [Cloudflare](./does-nine-ch-provide-d-dos-protection-options), bot traffic can also be a significant issue. Fortunately, we can mitigate this issue by using an .htaccess file to block requests. This is only possible when using the Apache web server; Nginx does not support .htaccess files. ## Checking the Log Files If your site is not available, take a closer look at your web server's logs first. You can find them in the /logs subdirectory of your home directory (use ls $HOME/logs). In the access log file, look at the last column to find the User-Agent header. The following example illustrates a common problematic script [FacebookExternalHit](https://www.facebook.com/externalhit_uatext.php), a crawler that often causes issues. In the following example, the bot makes multiple requests per second: ``` 173.252.83.40 - - [15/Aug/2024:01:19:00 +0200] "GET /*** HTTP/2.0" 200 434490 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 69.171.249.116 - - [15/Aug/2024:01:19:00 +0200] "GET /***/***/*** HTTP/2.0" 200 248774 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 69.171.249.15 - - [15/Aug/2024:01:19:01 +0200] "GET /***/*** HTTP/2.0" 200 94504 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 173.252.87.11 - - [15/Aug/2024:01:19:02 +0200] "GET /***/*** HTTP/2.0" 200 262990 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 66.220.149.115 - - [15/Aug/2024:01:19:02 +0200] "GET /***/***/*** HTTP/2.0" 200 258529 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 69.171.249.112 - - [15/Aug/2024:01:19:03 +0200] "GET /***/***/*** HTTP/2.0" 200 356111 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" ``` This could lead to problems with the availability of the website over a longer period of time. ## Creating the Rewrite Rule To prevent the `FacebookExternalHit` crawler from accessing your site, you can create an `.htaccess` file in your document root with the following configuration: ```apacheconf title=".htaccess" RewriteEngine On RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC] RewriteRule .* - [F,L] ``` If an .htaccess file already contains other rewrite rules, you can simply append the above rule to it. To ensure the new rule takes precedence, place it at the very top of the .htaccess file. To block multiple bots within the same rewrite condition, you can use the following syntax: ```apacheconf title=".htaccess" RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (Amazonbot|Bytespider|facebookexternalhit) [NC] RewriteRule .* - [F,L] ``` By implementing this simple .htaccess rule, you can effectively block malicious bot traffic and protect your website from downtime. Remember to regularly review your access logs to ensure the rule is working as intended. With a little effort, you can safeguard your online presence and keep your website available to users around the world. --- ## Does Nine provide (D)DoS protection options? ## Official Cloudflare Partner Nine is a Cloudflare partner and offers the services of Cloudflare with the needed consulting and configuration for your environment. This is available as option for our products (including "Unmanaged" products). ## Cloudflare Protection To secure your infrastructure from (D)DoS attacks, web access (HTTP & HTTPS) will be routed via the systems of Cloudflare. Cloudflare is checking the accesses and will block these if there is an attack detected. This way, Cloudflare disguises the target infrastructure and protects it from an overload. Furthermore, Cloudflare offers plentiful options to protect your applications from attacks. This includes Cross-Site-Scripting and SQL injections attacks. Cloudflare also offers a base rule set for the common CMS / CRM. ## Performance Cloudflare is not only a great protection layer, but also excels at speeding up the delivery of content. The global network makes sure that requests are always served from an exchange that is geographically near the user. As these exchange knots are also able to cache content, even users outside of Europe will profit from the enhancements of the Cloudflare network. With the direct links to all major internet exchanges, not only far away will profit, but also companies specialized to the swiss market will profit from the various options and improvements Cloudflare offers. ## First Steps The services and options of Cloudflare are plentiful, a support article will never cover your exact needs. If you're interested in how Cloudflare can add value to your environment, please get in touch with us via email at or phone . --- ## How do I redirect from HTTP to HTTPS? Redirection from HTTP to HTTPS can be performed in various places: in the application, in the web server or in an upstream proxy/CDN such as [Cloudflare](./does-nine-ch-provide-d-dos-protection-options). If all requests to your application are made using HTTPS, the use of [Strict Transport Security (HSTS)](../operations/security-recommendations#http-header-for-your-application) can provide additional security. ## Redirection Behind a Load Balancer If secure connections are being terminated with a load balaner by Nine, the internal connection to the backend is done without encryption. The check above does not work and results in a redirection loop. But there is an alternate check which is based on an HTTP header: ``` RewriteEngine on RewriteCond %{HTTP:X-Forwarded-Proto} !https RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L,QSA] ``` --- ## Let's Encrypt on Load Balancers :::note[For Load Balancer Setups only] This article describes how to set up Let's Encrypt certificates on a load balancer. For any other type of web server, please consult the article [nine-manage-vhosts with Let's Encrypt](./nine-manage-vhosts-with-lets-encrypt). ::: `nine-manage-letsencrypt` enables you to request and setup Let's Encrypt certificates automatically on your load balancer. Let's Encrypt supports up to 100 (sub)domains per certificate. There is no limitation regarding the amount of certificates. Wildcard certificates from Let's Encrypt are not supported. ## Prerequisites Let's Encrypt will verify the (sub)domain you want to create a certificate for, therefore the A- or CNAME- record for the (sub)domain is required to point to the `failover address` of the load balancer. If you are unsure about this, please contact . ## Usage and Options `nine-manage-letsencrypt` can only be used on the primary load balancer. There is an automated sync to the secondary (standby) load balancer. The usage on the secondary load balancer is not possible. The help can be shown by executing `nine-manage-letsencrypt --help`. The following options are available: ``` nine-manage-letsencrypt register nine-manage-letsencrypt certificate list nine-manage-letsencrypt certificate create nine-manage-letsencrypt certificate remove nine-manage-letsencrypt certificate renew-expiring nine-manage-letsencrypt alias add nine-manage-letsencrypt alias remove ``` ## Registration To request Let's Encrypt certificates, you need to register at the Let's Encrypt API. You have to provide a valid email address. This email address will be used to send notifications when there is an issue with renewing a certificate. You therefore should use an email address that is checked regularly. The registration can be done via command line: ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt register devops_AT_domain.ch ``` ## Manage Certificates ### List Certificates ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt certificate list ``` ### Create Certificates ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt certificate create lb.nine.ch ``` ### Delete Certificates The deletion of a certificate removes the vhost and revokes the certificate. ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt certificate remove lb.nine.ch ``` ### Expand Certificates An existing certificate can be expanded by up to 100 (sub)domains or aliases. After adding an alias, a new validation cycle will be triggered and a new certificate will be issued. ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt alias add www.nine.ch lb.nine.ch ``` ### Remove Alias from Certificate You can also remove a formerly created alias. After removal of an alias, there will be a new certificate issued that no longer contains the deleted alias. ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt alias remove www.nine.ch lb.nine.ch ``` ## Renew Certificates Let's Encrypt certificates are valid for 90 days and are **automatically** renewed 30 days before expiration. If there are errors while renewing expiring certificates, Let's Encrypt will send you a notification to the registered email address. The following are the most common reasons that lead to a failing renewal: - The (sub)domain does not point to the `failover address` or there is no A- or CNAME-record. - The request is processed by a CDN (for example Cloudflare or Akamai) or an external load balancer is not forwarding the plain request. Please make sure that requests to `/.well-known/acme-challenge/` are forwarded unmodified. The automatic renewal happens once a day. If it is necessary to renew certificates immediately, you can force a renewal: ``` www-data@nine-lb01:~ $ sudo nine-manage-letsencrypt certificate renew-expiring ``` Usually, it is **not** necessary to take care of the renewal by yourself. --- ## Manage VirtualHosts with nine-manage-vhosts ## What Is a Virtual Host? Virtual Hosting is the simultaneous operation of multiple websites (domains) on a common host (server). Each of these environments is called a Virtual Host and can be individually configured for hosting. Thus, multiple domains or IP addresses can be used or user-specific ports can be configured. ```mermaid graph LR A1[www\.example.com] -->|https| B1((Webserver)) A3[www\.example2.org] -->|https| B1 A4[www\.example3.org] -->|https| B1 B1 --> D1["/home/www-data/example.org/"] B1 --> D2["/home/www-data/example2.org/"] B1 --> D3["/home/www-data/example3.org/"] ``` ## What Is nine-manage-vhost? On Nine Managed (V)Servers, `nine-manage-vhosts` is used by default to manage websites. This makes it easier and faster to manage the Virtual Hosts. This VirtualHost configuration tells the web server where the data is located and under which address the page can be called up. `nine-manage-vhosts` supports the web servers Apache2 as well as Nginx and is compatible with the FTP administration tool "FTPAdmin". Basically, `nine-manage-vhosts -h` displays the help text with all options. ``` www-data@server:~ $ # sudo nine-manage-vhosts -h Manage your virtual hosts, users and certificates. Usage: nine-manage-vhosts user create [--no-password | --ask-password | --password=] nine-manage-vhosts user update (--no-password | --ask-password | --password=) nine-manage-vhosts user remove nine-manage-vhosts user list [--json] nine-manage-vhosts webserver reload nine-manage-vhosts virtual-host create [--user=] [--webroot=] [--template=