In the past, if your Docker build failed or something didn’t act like you thought it would in your Dockerfile, your only real options were to add a bunch of RUN echo or RUN ls commands. You would then need to rebuild repeatedly, and hope you eventually found the issue. Now, Docker’s BuildKit debugger can actually open up an interactive debugging session right inside VS Code, letting you step through each instruction in your Dockerfile like real code.

You start by adding the new --debug flag to your Docker build command, like this:

docker build --debug .

Then VS Code will recognize the BuildKit debug session. It will then allow you to connect to it. Once you connect, you can see the build stages, inspect environment variables, and even poke around the container’s filesystem at each step. It’s a much more visual way to troubleshoot. This will allow you to much more easily see what's happening inside your image during the build process.

Below showing adding a breakpoint to your code and you can see that it has stopped where the breakpoint was added.

File explorer with the new docker build kit tool.

Inspecting variables with the new buildkit.

This new integration is really great for developers who work with multi-stage builds. You can pause inside any stage, check dependencies, and verify installed binaries. You can then make sure your COPY or RUN instructions are doing what you think they are before moving on.

To make it work, you’ll need to have the latest Docker extension for VS Code and make sure you’re using BuildKit. You need this since the debugger relies on BuildKit’s capabilities. Once you’ve got that, you can either attach to a running build or launch one straight from VS Code using the “Docker: Build with Debug” command from the Command Palette.

If you are like me, this feels like a big step forward if you have ever spent too long trying to fix broken Docker builds, which I have definitely sunk many hours into this in the home lab and production environments. Instead of guessing what went wrong, you can now literally step through your build and inspect the state of things in real time.

I’ve tried it out on a small self-hosted app I have built, and it works really well. You can drop into the environment mid-build and even run commands manually to test things out. I think the work between Docker and Microsoft and their collaboration are really tightening the loop between development and building containers.

If you want to dig deeper, the full write-up from Docker has step-by-step details and screenshots showing how it all works: Debug Docker Builds with Visual Studio Code

Definitely worth checking out if you’re building images regularly or tired of “debugging by rebuild.” This should save a ton of time for anyone who’s serious about Docker development.

- Kubernetes v1.34: Of Wind & Will (O' WaW) | Kubernetes

Here are a few of the highlights:

• Dynamic Resource Allocation (now Stable) – it now has first class handling of GPUs, FPGAs, and accelerators. It also has better visibility, smarter scheduling, and the ability to share slices. This I think will be a big deal for ML in production, but even more exciting for home lab GPU configurations

• Per-container restart policy (Alpha) – restart a single container instead of rescheduling the whole pod. Less disruption, more efficiency. Great in prod, and handy in labs when you’re testing multi-service pods.

• Short-lived ServiceAccount tokens (Beta/Default) – replaces long-lived imagePullSecrets. More secure and less secret management overhead.

• Built-In Pod mTLS (Alpha) – native short-lived certs for workload-to-workload encryption, no sidecars needed. Moves Kubernetes toward zero-trust by default.

• KYAML (Alpha) – stricter, safer YAML that avoids those frustrating “why did this deploy break on whitespace” problems.

• OCI artifact volumes (Beta) – mount configs, models, or binaries from a registry without bloating images. Clean and simple.

• Graceful windows node shutdown (Beta) – Windows nodes finally respect termination grace periods.

From an SRE’s perspective, this is a release that is all about stability, observability, and security. From a home labber’s perspective, I think it helps with some of the friction with running Kubernetes in a home lab.

I’m personally excited about per-container restarts and the DRA graduation. Those are two that solve real pain points that I’ve seen in both production and lab environments. I am curious what you all think. Will you be testing 1.34 right away? Or will you be waiting until these features harden a bit?

🔄 Volume Populators Go GA

Volume Populators are now stable. This means you can now pre-populate your PVCs with data from sources other than just volume snapshots or clones.
This is enabled using the dataSourceRef field. It also requires a CRD and the volume-data-source-validator controller in your cluster. Super helpful for scenarios like injecting test data or restoring from backups using custom controllers.

Below is an example of how to use the volume populators:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
spec:
  ...
  dataSourceRef:
    apiGroup: provider.example.com
    kind: Provider
    name: provider1

🧹 PersistentVolume Reclaim Policy Fixes

There’s now a new protection in place for persistent volumes (PVs) when using the “Delete” reclaim policy. Before, if you deleted a PV before the PVC, the deletion logic might get skipped. Kubernetes 1.33 introduces new finalizers to make sure the reclaim policy is carried out no matter the order in which things are deleted.

⚠️ Endpoints API Deprecation

The old Endpoints API is now officially deprecated. If you're still using it, you'll start seeing warnings. The Kubernetes team recommends switching to EndpointSlices. These have been stable since the release of 1.21 and have better scalability.

📈 Horizontal Pod Autoscaler Tolerance (Alpha)

K8s 1.33 adds another new feature in alpha that allows you to tweak the tolerance level for HPA metrics. This will mean fewer unnecessary scale-up/scale-down events. Even when your resource usage fluctuates just slightly. This gives more control for production environments.

🔥 nftables Mode in Kube-Proxy (GA)

The long-awaited nftables mode for kube-proxy is coming to GA in 1.33. It replaces iptables for better performance and scalability. This is true especially on modern Linux kernels. The iptables module is the default for now, but nftables is the path forward.

These changes bring exciting new capabilities and performance boosts, especially for home labbers and platform engineers. If you are working with persistent storage and autoscaling in Kubernetes this release contains a lot of great new functionality.

Check out the full release notes and blog here: https://kubernetes.io/blog/2025/05/08/kubernetes-v1-33-volume-populators-ga/

Anyone updated to 1.33 already and testing things out?

flux check --pre

See here for documentation on the initial installation and package managers you can use: Flux CLI | Flux.

After you have updated your Flux CLI, you can see this with the command:

flux --version

You can check and see what version your controllers are sitting at with the command:

flux check

You can see below I am at an older version than my newly updated Flux CLI at 2.5.1. Below we are at v2.4.0.

If you have used the Flux CLI bootstrap command to install Flux in your Kubernetes cluster, you can simply rerun the bootstrap command using the same parameters you used initially to bootstrap. The command will look something like this:

flux bootstrap git --url=https://<git server URL>/devops/k8s-gitops.git --branch=main --path=clusters/clkube --token-auth

After rerunning the bootstrap command, you will see the update process work through and the controllers should be redeployed with the latest version. Then, you can run the flux check command again, and you should see your distribution has been updated.

Starting April 1, Docker will enforce the following pull rate limits according to the GitLab KB here: Prepare now: Docker Hub rate limits will impact GitLab CI/CD

To avoid this, you need to add authentication to your CI/CD pipelines, no matter which solution you are using. However, how do we do this with GitLab?

We can add the authentication using a special DOCKER_AUTH_CONFIG parameter in your config.toml. This is the configuration file for your runner and the contents get created when you pair a runner with your GitLab instance using the gitlab-runner register command.

Generating the Docker Hub authentication token

First, we need to sign up for Docker Hub and generate a personal token. You can do this after you have a free Docker account, navigate to Account Settings > Personal access tokens.

After you click to generate new token, you will see this screen. You can add a description, choose or just leave None for the expiration date, and set the permissions for the token. For what we are trying to accomplish pulling images from the online public repo, you can just leave it set to Public Repo Read-only.

Once we have the token file, we need to create a BASE64 version of our authentication information that combines our username and the token together. On a Linux machine, you can do this using the command, note, below is not literal, you will replace dockeruser and the tokentexttokentexttokentext with your real token that you create in the Personal access tokens area above.

echo -n "dockeruser:tokentexttokentexttokentext" | base64

When you run this command from a Linux terminal, you will get the BASE64 encoded token that we can use in the next step.

Updating the config.toml file

in the special config.toml file, we can use the environment parameter to configure the authentication like in the below example. Note the following:

• environment parameter
• URL is https://index.docker.io/v1
• Then enter your generated token that you get from using the process above where you see REDACTED_DOCKER_AUTH in the environment string below

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0


  session_timeout = 1800

[]
  name = "my-runner"
  url = "https://gitlab.example.com"
  id = 0
  token = "REDACTED_TOKEN"
  token_obtained_at = 2024-10-21T00:31:49Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  environment = 
  clone_url = "https://gitlab.example.com"
  
  
    MaxUploadedArchiveSize = 0
    
    
    
  
    tls_verify = false
    image = "rocker/verse:latest"
    dns = 
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = 
    extra_hosts = 
    shm_size = 0
    network_mtu = 0

Confirming your pipeline is authenticating to Docker Hub

Now that we have everything in place, we just need to run the pipeline once again and make sure it shows that it is now authenticating using the special $DOCKER_AUTH_CONFIG parameter.

Check the version of netdata that is installed. If you want to see what version is currently installed, run this command. Replace "netdata" with the namespace you have netdata installed in.

helm list -n netdata

Update your helm repos to the latest versions:

helm repo update

To upgrade netdata to the latest after updating your helm repo, run the command:

helm upgrade <release-name> netdata/netdata -n <namespace> --reuse-values

Below I have replaced with the appropriate values for netdata in my environment. Also, in a microk8s cluster, so need to add the "microk8s" in front of helm.

Building Docker images isn't a difficult process and is a great skill for DevOps. At its most simple definition, it has you run the docker build command to create a new image. It then reads instructions from your Dockerfile. These instructions typically have the directive to pull a base image, like Ubuntu, and then execute commands and other instructions to create the final Docker image.

Creating a Custom Dockerfile

The Dockerfile is a special file that is a simple text file with the Docker image instructions. A Dockerfile is a text file containing the instructions needed to build a Docker image. The instructions are executed in order they appear in the text file. These can include commands to copy files, set environment variables, run commands, and other important directives to make your application work as intended.

Here is a simple example of a Dockerfile that runs a Python application. We can see the different directives in the file that tells Docker how to build the resulting image. The below is a simple docker image defined in the Dockerfile.

# Use an official Python runtime as a parent image
FROM python:3.8-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Make port 80 available to the world outside this container
EXPOSE 80

# Define environment variable
ENV NAME World

# Run app.py when the container launches
CMD 

Dissecting the commands in the Docker file

• FROM: Specifies the base image to use.
• WORKDIR: Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY, and ADD instructions.
• COPY: This copies new files or directories to the container's filesystem.
• RUN instruction: The run command directive executes any commands in a new layer on top of the current image and then commits the results to the resulting image
• CMD instruction: This tells the image which command to run within the container.

Command to Build Docker Image

Use the docker build command to build a Docker image from a custom Dockerfile. This command takes several options and arguments to define how the build process should be carried out.

Using the Docker Build Command

Below is the command to build Docker image from custom created Dockerfile using the Docker build command from the command line and specifying the root directory (current working directory) for the Dockerfile, etc.

docker build -t mycustomimage:latest .

• -t - Tags the image with a name and optionally a tag in the nameformat.
• The "." at the end - This specifies the build context. It tells Docker to look for files located in the specified directory. The period tells it to look in the current directory for local files. Here we start to see the build output that will result in the built image.

Build Context

The build context is the set of files located in the specified directory and its subdirectories. Docker sends these files to the Docker daemon when building the image. One important consideration is to keep the build context as small as possible. This helps to speed up the build process and reduce resource usage.

Managing Build Context

Use a .dockerignore file to exclude files and directories from the build context. This file works similarly to a .gitignore file, listing patterns for files and directories to ignore. As an example, we are ignoring node modules and log files below:

# Ignore the node_modules directory
node_modules

# Ignore all files with the .log extension
*.log

Multiple Build Stages

One of the advanced build features with Docker is performing a multi-stage build. These are an advanced build feature of Docker. When doing a multi-stage build, you use multiple FROM statements in a single Dockerfile.

One reason you might do this is to optimize your build and reduce the size of the resulting container. When you separate the build environment from the runtime environment, you can include all necessary dependencies for building your application in one stage. Then, only the final artifacts are copied to the runtime stage.

Example of Multi-Stage Build

Here’s an example of a multiple build stage in a Dockerfile:

# Stage 1: Build
FROM maven:3.6.3-jdk-8 as builder
WORKDIR /app
COPY . .
RUN mvn clean install

# Stage 2: Run
FROM openjdk:8-jre-alpine
WORKDIR /app
COPY --from=builder /app/target/app.jar .
CMD 

In this example:

1. The first stage uses a Maven image to compile the Java application.
2. The second stage uses a lightweight OpenJDK runtime image to run the compiled application.
3. This separation makes sure that the final image only contains the runtime dependencies that are necessary. It makes it result in a smaller image.

Benefits of Multi-Stage Builds

There are many benefits to note for a multiple build stage Dockerfile. Note the following:

• Smaller Image Size: By copying only the necessary artifacts to the final image, you can significantly reduce the size of the image.
• Better Security: Smaller images with fewer components reduces the attack surface and vulnerabilities
• Separation: By separating the build environment from the runtime environment it helps to make sure that the final image is clean. Also, it helps to know that it only contains what is needed to run the app.

Environment Variables in Docker

Environment variables are a good way to configure applications and services. You can set the build time variables in a Dockerfile using the ENV instruction.

Setting Environment Variables

Note the following environment variables where we are setting the app environment and debug settings:

ENV APP_ENV=production
ENV APP_DEBUG=false

Advanced Docker Build Options

Build Arguments

Build arguments (ARG) are used to pass variables at build time.

ARG VERSION=1.0
RUN echo "Version: $VERSION"

Using a Build Context

You can specify a different destination path directory as the build context rather than the local directory by the following syntax:

docker build -t my-image:latest /path/to/build/context

Using Docker Hub

The Docker Hub service is Docker's cloud repository. It is where you can find Docker images and share your newly created image. To push your custom Docker images to Docker Hub, you need to log in and use the docker push command.

Note the following syntax for a Docker login tagging the image, and finally pushing it to the Docker Hub repo.

docker login
docker tag my-image:latest mydockerhubusername/my-image:latest
docker push mydockerhubusername/my-image:latest

Dockerfile and Docker Build Example

Let's go through an example of building a Docker image for a Python application. First, create a Dockerfile:

FROM python:3.9-slim
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
CMD 

Now, we can build the Docker Image:

docker build -t my-python-app:latest .

Finally, run the Docker Container:

docker run -d -p 5000:5000 my-python-app:latest

The Docker container starts up and runs from the custom image.

Load Build Definition

The build definition is an important part of Docker images. As you would expect, the Dockerfile is the build definition for the resulting Docker image. It details the steps needed to create the image. The definition must be correct and well-structured. By giving attention to this, it helps make sure the build process runs smoothly and produces the desired container image.

Best Practices for Building Docker Images

There are definitely best practices for building Docker image files that you want to keep in mind. Note the following

Minimize Image Size

• Use a minimal base image - In other words, why use a bloated base image if you don't need one?
• Clean up temporary files and package managers - also use a docker ignore file to prevent unnecessary files from creeping in
• Use multi-stage builds to separate the build environment from the runtime environment which results in a smaller image size as unnecessary files and resources are not inadvertently copied in.

Security Considerations

• Regularly update your base images to include the newest security updates
• Use official images from Docker Hub as these are properly vetted for security concerns
• Scan images for vulnerabilities using third-party tools or native Docker tools like Docker Scout

Looking at Docker Scout:

Viewing vulnerabilities in Docker Desktop for a Docker image using Docker Scout:

It is a great exercise to build Docker container images and looking at the command to build Docker image from custom created Dockerfile. By going through the process to build your own custom image with a custom Dockerfile, it helps you understand the components of a Docker image and the commands needed to bring everything together.

Practice in a lab environment building custom images so you understand the process. It will help you become proficient with building custom images. I used this process to build a custom arpwatch container for my home lab environment and learned a ton doing this.

Let me know what you guys think about building your own custom Docker images. Have you done this before? What projects and solutions make sense to build your own custom container image?

What Does Docker Expose Port Mean?

When you “expose” a port or multiple ports, it means that you are making that network port on a Docker container available to be connected to from either the Docker network or the outside world. By default, Docker containers are isolated in the way their networking is designed and are not accessible unless you configure this.

Isolation helps with security as if a port doesn’t need to be exposed to the outside, it shouldn’t be. And, if you can limit the number of network ports you expose, the better off you will be from an attack surface. However, the downside is that you will need to think about which ports need to be open for necessary or expected communication with your containers.

How to Expose Ports in Docker

Let’s take a look at how to expose ports in Docker and see what commands are needed for opening a Docker port.

To expose ports in Docker, you use the docker run command with the -p or –publish flag for publish ports functionality. This flag specifies which ports are available for connection. Let’s look at a simple example of exposing port 80 on an Nginx Docker image web server using port 8080 on the outside.

docker run -d -p 8080:80 nginx

In this command, 8080 is the host port, and 80 is the container port. The -d flag runs the container in detached mode which is how you want to run containers to keep them running without having to have a console connection to your Docker container host.

The port expose configuration via a port number means that you are exposing the container on the same network internally to the Docker network. By default when you spin up a new container and don’t specify the container network, they are connected to the default bridge network on the Docker host.

Configuring container ports to listen

When creating a Docker container from a docker container image, configuring container ports that you want the container to listen on is an important step. To do this, we use a special instruction called EXPOSE in the Dockerfile. For example let’s look at what expose instructions for ports 80 and 443 would look like. Pretty simple:

EXPOSE 80 EXPOSE 443

This EXPOSE instruction tells Docker that the container should expose multiple ports, 80 and 443.

Do you have to do this necessarily? No, if the container image already is exposing the ports, then the configuration will have this by default and you won’t have to “re-expose” the port in your Docker directives.

Case in point, if you see the below, you can see the port 81/tcp is being exposed on the container, but you don’t see a host side port mapping.

In addition, the Docker Compose code for the container above (Nginx Proxy Manager) looks like the following, so you don’t see any directives on the ports side or an expose directive for port 81. So, it means the container image is already exposing this port and the container listens on this exposed port by default.

Docker Container Port Mapping

Let’s look at the Docker container port mapping construct. This is the configuration that allows mapping internal container ports to external host ports. When you do this, it allows traffic that is destined for the host IP network interface to be forwarded inward to the container and vice versa.

docker run -d -p 5000:5000 testapp

This command maps port 5000 on the host to port 5000 on the container. This configures communication to the container’s ports via the host IP address and makes the container accessible by the host system. Otherwise, you wouldn’t be able to communicate with it.

Configuring the exposed Ports with Docker Compose

Docker Compose is the means by which you can spin up Docker container “stacks” that allow multiple containers to be provisioned at once. This is a great way to work with Docker containers if you have an application that requires multiple containers as part of the overall application architecture.

Docker Compose uses YAML code to describe the container configuration. The ports directive configures published ports available for connection from the host into the container.

version: '3'
services:
  web:
    image: nginx
    ports:
      - "8080:80"

In this configuration, the web service exposes port 80 of the container on port 8080 of the host machine.

Verifying Exposed Ports with Docker PS and Docker-Compose PS

When you provision a container, you can verify the exposed port mappings using the docker ps command.

docker ps

Look for the PORTS column in the output to see the mappings

When you use docker-compose ps command, it shows the same information, just for the application stack configured via the Docker Compose file:

Troubleshooting Docker expose port mappings

There are a few issues that could come up when you are trying to expose a specified port for a container. Note the following:

• Make sure the port is not in use by another container
• Make sure the container is attached to the Docker network that you assume it is connected to
• Is the port mapped to the host port you expect?
• If you are just exposing a port to use with something like Nginx Proxy Manager, make sure the port proxy is configured corrected to forward traffic to the internally exposed ports of the Docker container.

Best Practices for Exposing Ports

1. Use Non-Standard Ports: Common ports like 80 and 443 are going to be widely used, so you can use non-standard ports to avoid conflicts
2. Limit Exposed Ports: Only expose Docker container ports as this will help to avoid security vulnerabilities by only exposing what is absolutely needed
3. Document Exposed Ports: Maintain clear documentation of which ports are exposed and their purposes.

sudo vi /var/snap/microk8s/current/certs/csr.conf.template

Add the IP address as IP.99="a.b.c.d" to the file. Then refresh your microk8s certificates:

sudo microk8s refresh-certs -e ca.crt

Create a values.yaml file. You can find an example here:

https://github.com/kube-vip/helm-charts/blob/main/charts/kube-vip/values.yaml

It will look something like this below. Replace the config: address with your address that will be the VIP and change any other settings here you would like, but most can use defaults.

# Default values for kube-vip.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
image:
  repository: ghcr.io/kube-vip/kube-vip
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  # tag: "v0.7.0"

config:
  address: ""

# Check https://kube-vip.io/docs/installation/flags/
env:
  vip_interface: ""
  vip_arp: "true"
  lb_enable: "true"
  lb_port: "6443"
  vip_cidr: "32"
  cp_enable: "false"
  svc_enable: "true"
  svc_election: "false"
  vip_leaderelection: "false"

extraArgs: {}
  # Specify additional arguments to kube-vip
  # For example, to change the Prometheus HTTP server port, use the following:
  # prometheusHTTPServer: "0.0.0.0:2112"

envValueFrom: {}
  # Specify environment variables using valueFrom references (EnvVarSource)
  # For example we can use the IP address of the pod itself as a unique value for the routerID
  # bgp_routerid:
  #  fieldRef:
  #    fieldPath: status.podIP

envFrom: []
  # Specify an externally created Secret(s) or ConfigMap(s) to inject environment variables
  # For example an externally provisioned secret could contain the password for your upstream BGP router, such as
  #
  # apiVersion: v1
  # data:
  #   bgp_peers: "<address:AS:password:multihop>"
  # kind: Secret
  #   name: kube-vip
  #   namespace: kube-system
  # type: Opaque
  #
  # - secretKeyRef:
  #    name: kube-vip

extraLabels: {}
  # Specify extra labels to be added to DaemonSet (and therefore to Pods)

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
# Custom namespace to override the namespace for the deployed resources.
namespaceOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {}

podSecurityContext: {}
# fsGroup: 2000

securityContext:
  capabilities:
    add:
      - NET_ADMIN
      - NET_RAW
    drop:
      - ALL

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

volumes: []
  # Specify additional volumes
  #   - hostPath:
  #       path: /etc/rancher/k3s/k3s.yaml
  #       type: File
  #     name: kubeconfig

volumeMounts: []
  # Specify additional volume mounts
  # - mountPath: /etc/kubernetes/admin.conf
  #   name: kubeconfig

hostAliases: []
  # Specify additional host aliases
  # - hostnames:
  #     - kubernetes
  #   ip: 127.0.0.1

nodeSelector: {}

tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
affinity: {}
  # nodeAffinity:
  #   requiredDuringSchedulingIgnoredDuringExecution:
  #     nodeSelectorTerms:
  #     - matchExpressions:
  #       - key: node-role.kubernetes.io/master
  #         operator: Exists
  #     - matchExpressions:
  #       - key: node-role.kubernetes.io/control-plane
  #         operator: Exists

podMonitor:
  enabled: false
  labels: {}
  annotations: {}

priorityClassName: ""

Finally, we need to add the helm repo for kube-vip, update, and then install:

microk8s helm3 repo add kube-vip https://kube-vip.io/helm-charts
microk8s helm3 repo update
microk8s helm3 install kube-vip kube-vip/kube-vip --namespace kube-system -f values.yaml

Hopefully, this will help anyone trying to get this up and running.

By default, the pod CIDR is 10.1.0.0/16. This may be fine in your environment. However, if your production LAN is running in a subnet that is contained or overlapped by this /16 subnet, you need to change it since it will cause issues in getting traffic in and out of your cluster.

First, edit your file:

/var/snap/microk8s/current/args/cni-network/cni.yaml

You will want to search for the value "10.1.0.0/16" and change it to the subnet you want to use instead. For instance:

Change "10.1.0.0/16" to "10.100.0.0/16"

Next, edit the file:

 /var/snap/microk8s/current/args/kube-proxy

Change the line:

Change "10.1.0.0/16" to "10.100.0.0/16"

Apply your YAML file with the command:

microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml

Next, restart your Microk8s service on all nodes:

sudo snap restart microk8s

Note, if your cluster has already created the default ippools with the other subnet that gets created by default, you will need to delete this configuration. Why? If you don't, the cluster will continue to serve out IPs from this default pool and you won't see your new configuration take effect on new pods spun up. Here are the steps in addition I took to get that done. Note, the commands just copied and pasted from the official documentation didn't work for me in this step. So, here are the correct commands:

microk8s kubectl get ippools.crd.projectcalico.org

microk8s kubectl delete ippool default-ipv4-ippool

microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml

After this, restart your deployments, daemon sets, etc. The new pods that are created should be provisioned with the new IP addresses. Hopefully, these steps will help ones struggling with this.