CKS Notes - Audit log

This is a quick notes and summary from office doc.

1. this is a quick look at log audit.

2. more detail about the log audit, pls refer to the official doc about audit

3. and also can check my previous article: kubernets related logs & configurations which shows clear log file paths and the config related files and their locations.

Audit workflow

edit the audit-policy.yaml:

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

add flags to /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=8
    - --audit-log-maxsize=9
    - --audit-log-maxbackup=3

• --audit-policy-file=/etc/kubernetes/audit-policy.yaml

• --audit-log-path=/var/log/kubernetes/audit/audit.log

• --audit-log-maxage=8 : max number of days

• --audit-log-maxsize=9 : max file size in megabytes before rotate

• --audit-log-maxbackup=3 : copy of audit log files

edit the volumeMounts

volumeMounts:
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    readOnly: true
  - mountPath: /var/log/kubernetes/audit/
    name: audit-log
    readOnly: false

edit the hostPath corresponding with --audit-policy-file and --audit-log-path

volumes:
- name: audit
  hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: File

- name: audit-log
  hostPath:
    path: /var/log/kubernetes/audit/
    type: DirectoryOrCreate

and then wait for some time and we can check the logs on path /var/log/kubernetes/audit/.

Notice:

in above Volume and VolumeMount, we see some difference settings for audit policy and audit logs.

Audit policy

• Path is a single YAML file: /etc/kubernetes/audit-policy.yaml

• readOnly: API server only needs to read it

• type: File: The file already exists, Kubelet should fail to start the apiserver if the file is missing

Audit logs

• Path is a directory /var/log/kubernetes/audit/, API server writes out multiple log files (with rotation) audit.log, audit.log.1, etc

• readOnly: false

• type: DirectoryOrCreate: It may not exist yet on a fresh system, Kubelet will automatically create it so the API server can write logs

Policy

A Kubernetes audit policy has three important layers:

1. Resources you target

resources:
  - group: ""    # core API group
    resources: ["pods", "configmaps", "secrets", "persistentvolumeclaims", ...]

group: "" means Core API group, like pods, configmaps, secrets, services, pv, pvc.

but there are also non-core-api

so the resource eg.

resources:
  - group: "apps"
    resources: ["deployments"]

2. Which stages you log

stages: ["Request", "Response", "Metadata"]

• Request: log the incoming request before it hits handlers

• Response: log the full response (including responseBody if enabled)

• Metadata: only logs the request metadata (verb, user, resource, namespace)

Normally not used, but we often define omitStages to reduce noise.

omitStages: ["RequestReceived"] is used to: Reduce extremely noisy logs with no loss of useful information.

3. Log level fields

level: None | Metadata | Request | RequestResponse

3.1. level: None

No audit event produced, which is used to ignore noisy resources. eg.

- level: None
  resources:
    - group: ""
      resources: ["events"]

level: Metadata

Logs:

• user (e.g., system:serviceaccount:default:sa)

• verb (get, list, update, patch…)

• resource (configmaps, pods…)

• namespace

• request URI

• response code

No request/response bodies. This is lightweight and safe for Secrets.

3.2. level: Request

Logs everything from “Metadata” plus the request body. eg.

• the new version of a ConfigMap

• the PVC spec submitted by kubectl

• the PATCH operations

Does NOT log the response body.

3.3. level: RequestResponse

Logs:

• metadata

• request body

• response body

eg. when we run command kubectl apply -f my-pv.yaml

the audit logs include:

• the PV YAML sent

• the PV YAML that apiserver stored after admission

This is the most expensive but also the most complete.

3.4 Summary