Skip to content
Book a call
Menu

Networking8 min read

Cloud-Init Basics

The industry standard for instance initialization

What Is Cloud-Init?

Cloud-init is the de facto standard for configuring Linux instances at first boot. It handles everything from setting hostnames and creating users to writing files, installing packages, and running arbitrary scripts — all before anyone ever SSHes in.

Where Cloud-Init Runs

  • AWS EC2 — every Amazon Linux, Ubuntu, Debian, RHEL AMI ships with cloud-init
  • GCP Compute Engine — all standard images
  • Azure VMs — standard on all Linux marketplace images
  • OpenStack — cloud-init was originally built for it (Canonical)
  • Proxmox VE — cloud-init drive support for VM templates
  • libvirt / QEMU / KVM — via NoCloud datasource (seed ISO or directory)
  • LXD / Incus — native cloud-init integration for containers and VMs
  • Vagrant, Multipass, DigitalOcean, Oracle Cloud, Hetzner... — the list goes on

Note

Key insight: Cloud-init is cloud-agnostic. You write one cloud-config YAML and it works across providers. The abstraction layer between "what you want configured" and "which cloud you're on" is the datasource — cloud-init auto-detects where it's running and fetches metadata accordingly.

Boot Stages

Cloud-init runs across five distinct stages during the systemd boot process. Each stage is a separate systemd unit with explicit ordering dependencies. Understanding the sequence is critical for debugging "why didn't my config apply" issues.

  1. Generator

    systemd generator — decide if cloud-init should run at all

  2. init-local

    cloud-init-local.service — before networking

  3. init-network

    cloud-init.service — networking is up

  4. config

    cloud-config.service — config modules

  5. final

    cloud-final.service — final modules

  1. Generator (systemd generator)
    Runs very early in systemd boot. Checks if cloud-init should be enabled at all. Looks for /etc/cloud/cloud-init.disabled, kernel command-line flags (cloud-init=disabled), or a missing datasource. If disabled, no cloud-init units start. This is how you permanently disable cloud-init on a golden image.
  2. init-local (cloud-init-local.service)
    Runs before networking is up. Can only access local datasources: attached config drives, seed directories, or SMBIOS/DMI data. This stage determines the datasource, applies network configuration (so the next stage can bring up networking), and sets up the instance identity. Critically, this is where network config is rendered (e.g., writing Netplan YAML).
  3. init-network (cloud-init.service)
    Networking is now available. Cloud-init fetches remote datasources: the EC2 metadata service at 169.254.169.254, GCE metadata at metadata.google.internal, etc. Retrieves user-data, vendor-data, and full instance metadata. Runs disk_setup, mounts, and bootcmd.
  4. config (cloud-config.service)
    Runs config modules only. This is where the bulk of declarative configuration happens: creating users/groups, installing packages, writing files, configuring SSH, setting timezone, NTP, CA certs, apt/yum repos, etc. Modules run in a defined order set by /etc/cloud/cloud.cfg.
  5. final (cloud-final.service)
    Runs final modules: runcmd, user scripts (/var/lib/cloud/scripts/), phone_home, package install completion, and final message. This is the "do whatever you want" stage — arbitrary shell commands run here, after everything else is configured.

Tip

Debugging tip: If your runcmd script needs a package installed by packages:, it will work — packages install in the config stage (4), and runcmd runs in the final stage (5). But if your bootcmd needs a package, it will fail — bootcmd runs in stage 3, before package installation.

Datasources

A datasource is the mechanism cloud-init uses to discover where it's running and retrieve configuration. Each cloud provider exposes instance metadata and user-data through a provider-specific interface. Cloud-init abstracts these behind a common API.

Datasource Provider How It Works Endpoint / Path
Ec2 AWS Link-local HTTP metadata service (IMDSv1/v2) http://169.254.169.254/latest/
GCE Google Cloud HTTP metadata service with Metadata-Flavor: Google header http://metadata.google.internal/
Azure Azure IMDS + wireserver, Metadata: true header http://169.254.169.254/metadata/
OpenStack OpenStack HTTP metadata or config drive (attached disk) http://169.254.169.254/openstack/
NoCloud Local VMs / libvirt / Proxmox Attached ISO or seed directory with meta-data + user-data files /var/lib/cloud/seed/nocloud/
ConfigDrive OpenStack (alt) Attached disk (FAT/ISO9660) with JSON metadata Mounted at /config-2/
VMware vSphere GuestInfo properties or OVF environment vmtoolsd --cmd "info-get guestinfo.userdata"

Datasource Detection Order

Cloud-init doesn't blindly try every datasource. Detection follows a priority list defined in /etc/cloud/cloud.cfg.d/ and built-in defaults:

  • First, check DMI/SMBIOS data — hypervisors embed vendor identifiers (e.g., Amazon EC2, Google Compute Engine) in system product name fields
  • Then check for config drives — attached disks with known labels (cidata, config-2)
  • Then attempt HTTP metadata endpoints — probe link-local addresses with timeouts
  • Finally, fall back to None datasource if nothing matches

You can override detection with datasource_list in /etc/cloud/cloud.cfg to force a specific datasource or change priority order.

Warning

Security note: AWS IMDSv1 is vulnerable to SSRF attacks (any process can curl the metadata endpoint). IMDSv2 requires a PUT-based token exchange, which mitigates this. Always enforce IMDSv2 on production instances. Cloud-init supports both, but your launch template should mandate v2.

Data Types

Cloud-init distinguishes between three categories of data it receives from datasources. Understanding the distinction matters for both security and configuration layering.

Instance Metadata

Source: Cloud provider

Content:

  • Instance ID, region, availability zone
  • Instance type / machine type
  • Network configuration (IPs, MACs, VPC)
  • SSH host keys (provider-generated)
  • Block device mappings
  • Placement info (host, rack, etc.)

Who controls it: The cloud provider, based on the instance's properties.

User-Data

Source: You (the operator)

Content:

  • Cloud-config YAML (#cloud-config)
  • Shell scripts (#!/bin/bash)
  • Multi-part MIME bundles
  • Jinja2-templated configs
  • Gzip-compressed data
  • Include files (URLs to fetch)

Who controls it: Whoever launches the instance. This is your primary configuration mechanism.

Vendor-Data

Source: Cloud provider (supplemental)

Content:

  • Provider-specific agents (e.g., AWS SSM agent)
  • Monitoring hooks
  • Default package repos / mirrors
  • License activation scripts
  • Provider NTP configuration

Who controls it: The cloud vendor. Runs after user-data, so user config takes precedence where there's overlap.

Note

Merge order matters: When cloud-init processes configuration, the merge order is: base defaults → vendor-data → user-data. User-data wins. This means your cloud-config overrides whatever the vendor sets. You can also use merge_how directives for fine-grained control over list/dict merging behavior.

Instance Identity & Re-Run Behavior

Cloud-init uses the instance-id from metadata to decide whether to re-run. This is fundamental to understanding cloud-init's idempotency model.

Same Instance ID

Result: Cloud-init skips initialization

  • Compares instance-id against /var/lib/cloud/data/instance-id
  • If they match, the instance was already initialized
  • Only bootcmd and per-boot scripts run again
  • No user creation, package install, runcmd, etc.

Use case: Normal reboots of an existing instance.

Different Instance ID

Result: Cloud-init runs full initialization

  • New instance-id detected, treats as new instance
  • All modules run: users, packages, write_files, runcmd, etc.
  • Previous instance data archived to /var/lib/cloud/data/previous-*

Use case: Cloning a VM template, re-deploying with new metadata, or manually running cloud-init clean.

Tip

Operational tip: To force cloud-init to re-run on an existing instance (e.g., testing your user-data), use cloud-init clean --logs then reboot. This wipes /var/lib/cloud/ and resets the instance-id check. Alternatively, cloud-init clean --logs --seed also removes seed data for NoCloud setups.

Filesystem Paths

Cloud-init uses two primary directory trees. Knowing what lives where is essential for debugging.

Runtime: /run/cloud-init/

Ephemeral — cleared on every boot (tmpfs).

text
bash

          /run/cloud-init/

            cloud-init-generator.log

            enabled — marker that cloud-init is active

            ds-identify.log — datasource detection log

            instance-data.json — resolved metadata (sanitized)

            instance-data-sensitive.json — full metadata (root-only)

            status.json — current run status per stage

            result.json — final result after all stages
Persistent: /var/lib/cloud/

Persistent — survives reboots. This is cloud-init's state store.

text
bash

          /var/lib/cloud/

            data/ — instance-id, datasource name, result

            instance/ — symlink to current instance

            instances/ — per-instance data dirs

              i-0abc123/

                cloud-config.txt — merged cloud-config

                user-data.txt — raw user-data

                vendor-data.txt — raw vendor-data

                sem/ — semaphore files (per-once tracking)

                scripts/ — cached scripts

            scripts/ — per-boot, per-instance, per-once scripts

            seed/ — NoCloud seed directory

Script Frequency Directories

Scripts placed in /var/lib/cloud/scripts/ subdirectories run at different frequencies:

Directory Frequency Use Case
per-boot/ Every boot Dynamic registration, health check hooks
per-instance/ Once per instance-id Initial setup, one-time provisioning
per-once/ Once ever (tracked by semaphore) Migrations, one-off fixes

Warning

Common pitfall: /run/cloud-init/instance-data-sensitive.json contains all metadata including potential secrets (e.g., user-data with embedded passwords). It's root-only (mode 0600), but be aware it exists — don't expose it via a web endpoint or logging pipeline.

Configuration Hierarchy

Cloud-init's own behavior is configured through a layered system. Understanding the precedence is critical when debugging why a setting isn't taking effect.

  1. /etc/cloud/cloud.cfg — Base configuration file. Defines default user, module run order, datasource list, and system-level defaults. Managed by the distro's cloud-init package. Avoid editing directly — use drop-ins instead.
  2. /etc/cloud/cloud.cfg.d/*.cfg — Drop-in configuration files. Merged in lexicographic order (so 99-custom.cfg overrides 50-defaults.cfg). This is where you customize cloud-init on a base image: force a datasource, change the default user, disable modules, etc.
  3. User-data — Provided at launch time. Highest precedence for module configuration. Your #cloud-config YAML overrides everything above where keys overlap.

Note

Kernel command line: You can also pass cc:{"key":"value"} on the kernel cmdline to inject cloud-config at the earliest possible stage. Useful for PXE boot scenarios where you need to set the datasource before cloud-init even runs.

Disabling Cloud-Init

On images that ship with cloud-init but don't need it (e.g., converted VM templates running on bare metal), there are several ways to disable it:

Method Scope How
Marker file Persistent touch /etc/cloud/cloud-init.disabled
Kernel cmdline Per boot Add cloud-init=disabled to GRUB
Environment variable Per invocation CLOUD_INIT_DISABLED=1
Uninstall Permanent apt remove cloud-init / dnf remove cloud-init

Tip

Best practice: For Packer-built images that will run on a cloud, don't disable cloud-init. Instead, run cloud-init clean --logs at the end of your Packer provisioner so the next boot treats the instance as new. This is standard practice for AMI/image baking pipelines.

Solidnines — solidnines.com