Kubernetes

• Kubernetes Eviction
  • Out-of-memory (OOM) in Kubernetes – Part 4: Pod evictions, OOM scenarios and flows leading to them
    • Pod evictions
      • Allocatable
      • --kube-reserved
      • Eviction mechanism at a glance
      • Node Allocatable, illustrated
      • Metric watched for during pod evictions
      • OOM Scenario #2: Pods’ memory usage exceeds node’s “allocatable” value
      • --eviction-hard
      • OOM Scenario #3: Node available memory drops below the --eviction-hard flag value
      • Changing the –eviction-hard memory threshold
      • Interactions between Kubelet’s pod eviction mechanism and the kernel’s OOM killer
      • Is Kubelet killing containers due to OOM?
      • Conclusions around pod evictions
    • Is it a problem that kubectl top pod shows a memory usage >100%?
    • Signals and exit codes
    • Metrics Testing
      • Grafana
      • kubectl top pod
      • kubectl top node
      • Resource Metrics API endpoint
      • Summary API endpoint
      • cAdvisor
      • cgroups pseudo-filesystem
      • htop
      • Why is there a difference between what htop shows for the container process and kubectl top pod?
    • Flows leading to out-of-memory situations
    • OOM Scenarios
      • OOM1: Container is OOMKilled when it exceeds its memory limit
      • OOM2: Pods’ memory usage exceeds node’s “allocatable” value
      • OOM3: Node available memory drops below the hard eviction threshold
      • OOM4: Pods’ memory usage exceeds node’s “allocatable” value (fast allocation)
      • OOM5: Container has a limit set, app inside allocates memory but the app’s runtime eventually fails the allocations way before the limit
    • Q&A
      • Q&A: Pod evictions
      • Q&A: Signals and exit codes
      • Q&A: Metrics Testing
      • Q&A: Flows and OOM Scenarios
    • Utils
      • How to see the Kubelet logs
      • How to modify the Kubelet log level
      • How to modify Kubelet startup parameters --eviction-hard and --kube-reserved on an AKS node
      • How to SSH into an AKS Linux node and run apps directly on it
        • Step 1: Open an SSH session to the AKS Linux node
        • Step 2: Copy and run the app
        • Notes
      • How to obtain data about who sends kill signals in Linux
• Kubernetes Network
  • Kube Proxy
    • Cracking Kubernetes Node Proxy (aka kube-proxy)
    • 1. Background knowledge
      • 1.1 Netfilter
      • 1.2 eBPF
      • 1.3 VIP and load balancing (LB)
      • 1.4 Cross-host networking model
    • 2. Kubernetes node proxy model
      • 2.1 Service types
      • 2.2 Service/proxy model
      • 2.3 Role of the node proxy: reverse proxy
      • 2.4 Performance issues
    • 3. Test environment
      • 3.1 Cluster topology and test environment
      • 3.2 Create a Service
      • 3.3 Reachability test
    • 4. Implementation 1: proxy via userspace socket
      • 4.1 The middleman model
      • 4.2 Implementation
        • 4.2.1 Code
        • 4.2.2 Some explanations
          • 1. Traffic interception
          • 2. Create proxy
        • 4.3 Reachability test
      • 4.3 Clean up
      • 4.4 Improvements
      • 4.5 Pros and Cons
    • 5. Implementation 2: proxy via iptables
      • 5.1 Host -> ClusterIP (single backend)
        • 5.1.1 Where to perform the DNAT
        • 5.1.2 Check current NAT rules
        • 5.1.3 Add DNAT rules
        • 5.1.4 Test reachability
        • 5.1.5 Cleanup
      • 5.2 Host -> ClusterIP (multiple backends)
        • 5.2.1 Scale up webapp
        • 5.2.2 Add DNAT rules with load balancing
        • 5.2.3 Verify load balancing
        • 5.2.4 Cleanup
      • 5.3 Pod (app A) -> ClusterIP (app B)
      • 5.4 Re-structure the iptables rules
      • 5.5 Further Re-structure the iptables rules
    • 6. Implementation 3: proxy via IPVS
      • 6.1 Install IPVS
      • 6.2 Add virtual/real servers
      • 6.3 Verify
      • 6.4 Clean up
    • 7. Implementation 4: proxy via tc-level ebpf
      • 7.1 Prerequisites
      • 7.2 Implementation
      • 7.3 Compile and load into kernel
      • 7.4 Verify
      • 7.5 Cleanup
      • 7.6 Explanations
      • 7.7 Improvements
    • 8. Implementation 5: proxy via socket-level ebpf
      • 8.1 Hook earlier
      • 8.2 Implementation
      • 8.3 Compile, load and attach BPF code
      • 8.3 Verify
      • 8.4 Clean up
      • 8.5 Performance comparison
      • 8.6 Other explanations
    • 9. Summary
    • References
    • Appendix
      • A1. webapp.yaml
      • A2. Service struct definition in Kubernetes
• Destructive test
  • Worker Node
    • What happens when one of your Kubernetes nodes fails?
• Monitoring
  • Resource Monitoring
    • memory monitoring
      • How to alert for Pod Restart & OOMKilled in Kubernetes
        • Context
        • Data source
        • Container Restart Metric
        • OOMKilled Metric
        • Alerting
          • low-capacity alerts
          • Pod container restart rate too high
          • OOMEvents
        • Conclusion
• Pod
  • pod phase
    • Container states
      • Waiting
      • Running
      • Terminated
    • Pod conditions
      • Garbage collection of terminated Pods
• Persistent Volumes
  • Lifecycle of a volume and claim
    • Provisioning(供应 Volume 的方法)
      • Static
      • Dynamic
    • Binding(PVC 绑定 PV)
    • Using
    • Storage Object in Use Protection
    • Reclaiming(回收)
      • Retain
      • Delete
      • Recycle
    • Reserving a PersistentVolume(静态绑定 PVC 与 PV)
    • Expanding Persistent Volumes Claims(扩容)
      • CSI Volume expansion
      • Resizing a volume containing a file system
      • Resizing an in-use PersistentVolumeClaim
  • Persistent Volumes(PV)
    • Capacity
    • Volume Mode
    • Access Modes
    • Class
    • Reclaim Policy
    • Phase
  • PersistentVolumeClaims(PVC)
    • Access Modes
    • Volume Modes
    • Resources
    • Selector
    • Class
  • Claims As Volumes - Pod 挂载
    • A Note on Namespaces
    • PersistentVolumes typed hostPath
• CSI
  • Ceph-CSI Architecture
    • K8s CSI & ceph 构架图解，还有一些闲话
      • 引
        • K8s 几大难点
        • 如何开始
          • 如何学习抽象
        • 本文的阅读方法
          • 互动图片
          • 读者对象
      • 基本的依赖关系
      • Ceph Cluster
      • CSI
      • Ceph-CSI Driver
      • 小结
      • 闲话
        • AI / ChatGPT
          • “不可控” 天然的抗拒
          • 只要 how，还是要 why
  • CSI Spec
    • CSI Spec
      • Terminology
      • Solution Overview
        • Architecture
          • Centralized - 中心化架构
          • Headless - 去中心架构
      • Volume Lifecycle
        • 动态创建 Volume
        • 预先提供 Volume
      • API
        • Concurrency - 并发访问同一 Volume
        • Controller Service RPC
          • CreateVolume - 创建 Volume
          • ControllerPublishVolume - 挂载 Volume
            • ControllerPublishVolume Errors
        • Node Service RPC
          • NodePublishVolume
• HPA
  • design-proposals
  • HPA Architecture
  • Responsiveness
  • Illustrated Case
    • Target Utilization
  • Concerns
• Container
  • container shim stack
    • Implementing Container Runtime Shim: runc
      • What is a shim?
      • Playing with runc
      • A naive attempt to use runc from code
      • The full scope of the shim
      • Shim implementation overview
      • Stay tuned
  • containerd
    • containerd CRI plugin e.g
    • Container Lifecycle
    • Container Lifecycle
      • containerd
      • shim
        • containerd-shim-runc-v2
          • Flow
  • Introducing Container Runtime Interface (CRI) in Kubernetes
    • Overview of CRI
    • design-docs-and-proposals
    • Container Runtime Interface (CRI) Networking Specifications
      • Requirements
  • containerd CRI plugin e.g
  • Kubernetes uses different container runtimes via CRI API.
  • Ref
  • What Is a Standard Container: Diving Into the OCI Runtime Spec Containers Aren’t Linux Processes
    • What is Open Container Initiative (OCI)
    • What is an OCI Container
    • Linux Container vs. Other Containers
    • Virtual Machines vs. Containers
    • MicroVMs vs. Containers
    • Instead of conclusion
      • OCI Runtime Specification( reference implmentation: runc )
  • hot swap restart container process
    • k8s 中不停容器热替换/重启主进程 - gdb exec 法
      • setup target e.g pod
      • inspect process
      • gdb
        • close fd
        • exec
        • check
        • detach
        • check after restarted
    • depends
    • exec hot swap gdb exec simple way
      • setup target e.g pod
      • inspect process
      • gdb
        • close fd
        • exec
        • check
        • detach
        • check after restarted
    • depends
• K8s metrics
  • Container Metrics
    • Memory
      • 常被误解的 K8s 指标
        • container_memory_usage_bytes
        • container_memory_working_set_bytes
        • kubectl top
        • container_memory_cache 与 container_memory_mapped_file 的关系
        • 什么 metric 才是 OOM Kill 相关
      • 参考