docker源码1-命令的调用流程

谢绝转载

序言

之前研究了一段时间的docker源码的调用机制，主要是想学习一下go，并了解一下docker volume plugin的加载机制，最近有点忘记了，就写下来加深记忆。

docker 版本：docker-ce(18.09)

本文会列出一些docker源码中的函数,因为篇幅原因会用...来省略一些内容,只留下我认为在调用流程中重要的一些内容. 在部分函数中会用注释标注关键点.

注意事项:
1.本文共有四篇,每篇都有编号,编号类似1.2.1这种,其中1是文章编号,因为后面的调用关系需要去前面篇幅中找,所以我标注了这个方便寻找.

2.我是按调用过程列出代码,如果当前函数有多个地方需要讲解,比如函数1.2中有两个地方需要讲解,那么要展开的地方便是1.2.1,1.2.2这样排列.

3.链接:
第一篇:https://www.jianshu.com/p/9900ec52f2c1 (命令的调用流程)
第二篇:https://www.jianshu.com/p/db08b7d57721 (卷服务初始化)
第三篇:https://www.jianshu.com/p/bbc73f5687a2 (plugin的管理)
第四篇:https://www.jianshu.com/p/a92b1b11c8dd (卷相关命令的执行)

以下开始正文

docker命令的调用流程

以下是docker-ce代码的主要结构,主要关注前两个文件夹,即客户端和服务端.

markdown-img-paste-2018102921143250.png

下面通过docker命令的执行过程,来探究一下docker的代码调用流程.

1.1 客户端部分

客户端入口是docker.go中的main函数:

markdown-img-paste-20181029212322244.png

func main() {
    // Set terminal emulation based on platform as required.
    stdin, stdout, stderr := term.StdStreams()
    logrus.SetOutput(stderr)

    # 新建dockerCli
    dockerCli := command.NewDockerCli(stdin, stdout, stderr, contentTrustEnabled(), containerizedengine.NewClient)
    # 绑定命令
    cmd := newDockerCommand(dockerCli)

    if err := cmd.Execute(); err != nil {
        ...
    }
}

main函数先创建了dockerCli客户端,然后调用了newDockerCommand函数来初始化docker命令.

1.1.1 DockerCli结构体

DockerCli是docker命令行客户端:

// DockerCli is an instance the docker command line client.
// Instances of the client can be returned from NewDockerCli.
type DockerCli struct {
    configFile            *configfile.ConfigFile
    in                    *InStream
    out                   *OutStream
    err                   io.Writer
    client                client.APIClient
    serverInfo            ServerInfo
    clientInfo            ClientInfo
    contentTrust          bool
    newContainerizeClient func(string) (clitypes.ContainerizedClient, error)
}

1.1.2 newDockerCommand函数

newDockerCommand函数来初始化docker命令, 用到的是golang的命令行库Cobra.

func newDockerCommand(dockerCli *command.DockerCli) *cobra.Command {
    ...
    cmd := &cobra.Command{
        Use:              "docker [OPTIONS] COMMAND [ARG...]",
        Short:            "A self-sufficient runtime for containers",
        SilenceUsage:     true,
        SilenceErrors:    true,
        TraverseChildren: true,
        Args:             noArgs,
        ...
    }
    # 设置默认信息
    cli.SetupRootCommand(cmd)
  ...
    # 加载子命令
    commands.AddCommands(cmd, dockerCli)

    ...
}

1.1.3 AddCommands函数

函数commands.AddCommands定义如下,可知加载了所有的子命令:

func AddCommands(cmd *cobra.Command, dockerCli *command.DockerCli) {
    cmd.AddCommand(
        ...
        // container
        container.NewContainerCommand(dockerCli),
        container.NewRunCommand(dockerCli),

        // image
        image.NewImageCommand(dockerCli),
        image.NewBuildCommand(dockerCli),

        ...

        // network
        network.NewNetworkCommand(dockerCli),

        ...

        // volume
        volume.NewVolumeCommand(dockerCli),
        ...
    )
}

看到这里我们已经知道了客户端命令是如何绑定的,也可以说是知道了docker命令执行的起点,再通过具体命令的执行过程来看一下客户端命令是如何和服务端的代码对应起来的.以docker run命令为例.

1.2.docker run命令执行流程

根据上面的分析,我们找到docker run命令对应的函数NewRunCommand

1.2.1 NewRunCommand函数

func NewRunCommand(dockerCli command.Cli) *cobra.Command {
  ...
    cmd := &cobra.Command{
        Use:   "run [OPTIONS] IMAGE [COMMAND] [ARG...]",
        Short: "Run a command in a new container",
        Args:  cli.RequiresMinArgs(1),
        RunE: func(cmd *cobra.Command, args []string) error {
            copts.Image = args[0]
            if len(args) > 1 {
                copts.Args = args[1:]
            }
            return runRun(dockerCli, cmd.Flags(), &opts, copts)
        },
    }
    ...
    return cmd
}

命令运行则会运行同路径下的runRun函数,runRun函数继续调用到同路径下runContainer函数,runContainer函数最终调用createContainer函数,可以发现docker run与docker create命令调用的是同一个函数:

1.2.2 createContainer函数

func createContainer(ctx context.Context, dockerCli command.Cli, containerConfig *containerConfig, name string, platform string) (*container.ContainerCreateCreatedBody, error) {
    ...

    //create the container
    response, err := dockerCli.Client().ContainerCreate(ctx, config, hostConfig, networkingConfig, name)
    ...
}

这里可以看到调用的是dockerCli.Client()的ContainerCreate方法,我们追踪溯源,看一下dockerCli.Client()是什么. 这里的dockerCli是command.Cli接口:

ps:以下两个小章节都是根据上面的函数进行展开,所以就用这种排版了,以后出现类似的排版同理.

• 1.2.2.1 Cli接口

// Cli represents the docker command line client.
type Cli interface {
    Client() client.APIClient
    ...
}

通过Cli的接口定义可知Client()返回的是client.APIClient, 再看一下DockerCli[1.1章节]中对接口方法Client()的具体实现:

• 1.2.2.2 Client方法

// Client returns the APIClient
func (cli *DockerCli) Client() client.APIClient {
    return cli.client
}

通过接口的实现可知返回值就是cli.client,而cli.client是在newDockerCommand函数中调用dockerCli.Initialize初始化的:

1.2.3 Initialize方法

func (cli *DockerCli) Initialize(opts *cliflags.ClientOptions) error {
    ...
    cli.client, err = NewAPIClientFromFlags(opts.Common, cli.configFile)
    ...
}

在NewAPIClientFromFlags中最终调用到了服务端的NewClientWithOpts函数,注意,连接客户端代码和服务端代码的关键来了:

1.2.4 NewClientWithOpts函数

func NewClientWithOpts(ops ...func(*Client) error) (*Client, error) {
    client, err := defaultHTTPClient(DefaultDockerHost)
    if err != nil {
        return nil, err
    }
    c := &Client{
        host:    DefaultDockerHost,
        version: api.DefaultVersion,
        scheme:  "http",
        client:  client,
        proto:   defaultProto,
        addr:    defaultAddr,
    }

    ...

    return c, nil
}

首先,可知返回的c是一个Client类型的结构体指针,再看成员变量Client.client则是docker服务端的http客户端. Client的具体定义如下:

1.2.5 Client结构体

// Client is the API client that performs all operations
// against a docker server.
type Client struct {
    // scheme sets the scheme for the client
    scheme string
    // host holds the server address to connect to
    host string
    // proto holds the client protocol i.e. unix.
    proto string
    // addr holds the client address.
    addr string
    // basePath holds the path to prepend to the requests.
    basePath string
    // client used to send and receive http requests.
    client *http.Client
    // version of the server to talk to.
    version string
    // custom http headers configured by users.
    customHTTPHeaders map[string]string
    // manualOverride is set to true when the version was set by users.
    manualOverride bool
}

在client.go 的相同路径下找到ContainerCreate方法,可知最后发出的是一个post请求:

1.2.6 ContainerCreate方法

func (cli *Client) ContainerCreate(ctx context.Context, config *container.Config, hostConfig *container.HostConfig, networkingConfig *network.NetworkingConfig, containerName string) (container.ContainerCreateCreatedBody, error) {
    var response container.ContainerCreateCreatedBody

    ...

    serverResp, err := cli.post(ctx, "/containers/create", query, body, nil)
    if err != nil {
        if serverResp.statusCode == 404 && strings.Contains(err.Error(), "No such image") {
            return response, objectNotFoundError{object: "image", id: config.Image}
        }
        return response, err
    }

    err = json.NewDecoder(serverResp.body).Decode(&response)
    ensureReaderClosed(serverResp)
    return response, err
}

这里我们注意的是url的路径 "/containers/create".
那另一个疑问来了,该请求最终调用到什么地方去了?让我们话分两头,再去看看服务端的进程是如何启动的.

To be continued...

1.3 服务端程序

同样,我们找到入口函数,即服务端代码的main函数:

func main() {
    ...
    cmd := newDaemonCommand()
    cmd.SetOutput(stdout)
    if err := cmd.Execute(); err != nil {
        fmt.Fprintf(stderr, "%s\n", err)
        os.Exit(1)
    }
}

main函数中只调用一个命令,那就是后台进程启动命令,没有子命令,newDaemonCommand函数如下:

1.3.1 newDaemonCommand函数

func newDaemonCommand() *cobra.Command {
    opts := newDaemonOptions(config.New())

    cmd := &cobra.Command{
        Use:           "dockerd [OPTIONS]",
        Short:         "A self-sufficient runtime for containers.",
        SilenceUsage:  true,
        SilenceErrors: true,
        Args:          cli.NoArgs,
        RunE: func(cmd *cobra.Command, args []string) error {
            opts.flags = cmd.Flags()
            return runDaemon(opts)
        },
        ...
    }
    ...

    return cmd
}

类似客户端命令,这个命令执行后会调用到runDaemon函数,看一下进程启动都会做些什么:

1.3.2 runDaemon函数

func runDaemon(opts *daemonOptions) error {
    daemonCli := NewDaemonCli()
    return daemonCli.start(opts)
}

NewDaemonCli()返回值是DaemonCli的结构体指针,先看下DaemonCli 的定义:

• 1.3.2.1 DaemonCli结构体

type DaemonCli struct {
    *config.Config
    configFile *string
    flags      *pflag.FlagSet

    api             *apiserver.Server
    d               *daemon.Daemon
    authzMiddleware *authorization.Middleware // authzMiddleware enables to dynamically reload the authorization plugins
}

再看DaemonCli的start方法,重点来了,这个方法包含整个docker进程的启动参数,我们只找到我们要找的关键部分:

• 1.3.2.2 start方法

func (cli *DaemonCli) start(opts *daemonOptions) (err error) {
    ...
    # 创建apiserver
    cli.api = apiserver.New(serverConfig)

    # 绑定监听端口
    hosts, err := loadListeners(cli, serverConfig)
    if err != nil {
        return fmt.Errorf("Failed to load listeners: %v", err)
    }

    ctx, cancel := context.WithCancel(context.Background())
    if cli.Config.ContainerdAddr == "" && runtime.GOOS != "windows" {
            ...
            # 启动containerd
            r, err := supervisor.Start(ctx, filepath.Join(cli.Config.Root, "containerd"), filepath.Join(cli.Config.ExecRoot, "containerd"), opts...)
            ...
            cli.Config.ContainerdAddr = r.Address()
            ...
    }
    ...

    # 创建pluginstore,用来保存plugin的相关信息
    pluginStore := plugin.NewStore()
    ...
    # 创建进程,并为守护进程设置一切服务
    d, err := daemon.NewDaemon(ctx, cli.Config, pluginStore)
    if err != nil {
        return fmt.Errorf("Error starting daemon: %v", err)
    }

    ...

    cli.d = d

    routerOptions, err := newRouterOptions(cli.Config, d)
    if err != nil {
        return err
    }
    routerOptions.api = cli.api
    routerOptions.cluster = c

    # 初始化路由
    initRouter(routerOptions)

    ...

    return nil
}

我们只关注调用流程,所以重点是初始化路由,这个函数的主要目的是绑定http请求到对应方法:

• 1.3.2.3 initRouter函数

func initRouter(opts routerOptions) {
    decoder := runconfig.ContainerDecoder{}

    routers := []router.Router{
        // we need to add the checkpoint router before the container router or the DELETE gets masked
        checkpointrouter.NewRouter(opts.daemon, decoder),
        container.NewRouter(opts.daemon, decoder),
        image.NewRouter(opts.daemon.ImageService()),
        systemrouter.NewRouter(opts.daemon, opts.cluster, opts.buildCache, opts.buildkit, opts.features),
        volume.NewRouter(opts.daemon.VolumesService()),
        build.NewRouter(opts.buildBackend, opts.daemon, opts.features),
        sessionrouter.NewRouter(opts.sessionManager),
        swarmrouter.NewRouter(opts.cluster),
        pluginrouter.NewRouter(opts.daemon.PluginManager()),
        distributionrouter.NewRouter(opts.daemon.ImageService()),
    }

    ...

    opts.api.InitRouter(routers...)
}

1.4 容器相关的命令执行

我们以容器为例具体看一下container.NewRouter():

1.4.1 NewRouter函数

func NewRouter(b Backend, decoder httputils.ContainerDecoder) router.Router {
    r := &containerRouter{
        backend: b,
        decoder: decoder,
    }
    r.initRoutes()
    return r
}

具体的绑定过程在initRoutes中执行:

1.4.2 NewRouter函数

func (r *containerRouter) initRoutes() {
    r.routes = []router.Route{
        // HEAD
        router.NewHeadRoute("/containers/{name:.*}/archive", r.headContainersArchive),
        // GET
        router.NewGetRoute("/containers/json", r.getContainersJSON),
        router.NewGetRoute("/containers/{name:.*}/export", r.getContainersExport),
        router.NewGetRoute("/containers/{name:.*}/changes", r.getContainersChanges),
        router.NewGetRoute("/containers/{name:.*}/json", r.getContainersByName),
        router.NewGetRoute("/containers/{name:.*}/top", r.getContainersTop),
        router.NewGetRoute("/containers/{name:.*}/logs", r.getContainersLogs, router.WithCancel),
        router.NewGetRoute("/containers/{name:.*}/stats", r.getContainersStats, router.WithCancel),
        router.NewGetRoute("/containers/{name:.*}/attach/ws", r.wsContainersAttach),
        router.NewGetRoute("/exec/{id:.*}/json", r.getExecByID),
        router.NewGetRoute("/containers/{name:.*}/archive", r.getContainersArchive),
        // POST
        # 找到重点了,"/containers/create"
        router.NewPostRoute("/containers/create", r.postContainersCreate),
        router.NewPostRoute("/containers/{name:.*}/kill", r.postContainersKill),
        router.NewPostRoute("/containers/{name:.*}/pause", r.postContainersPause),
        router.NewPostRoute("/containers/{name:.*}/unpause", r.postContainersUnpause),
        router.NewPostRoute("/containers/{name:.*}/restart", r.postContainersRestart),
        router.NewPostRoute("/containers/{name:.*}/start", r.postContainersStart),
        router.NewPostRoute("/containers/{name:.*}/stop", r.postContainersStop),
        router.NewPostRoute("/containers/{name:.*}/wait", r.postContainersWait, router.WithCancel),
        router.NewPostRoute("/containers/{name:.*}/resize", r.postContainersResize),
        router.NewPostRoute("/containers/{name:.*}/attach", r.postContainersAttach),
        router.NewPostRoute("/containers/{name:.*}/copy", r.postContainersCopy), // Deprecated since 1.8, Errors out since 1.12
        router.NewPostRoute("/containers/{name:.*}/exec", r.postContainerExecCreate),
        router.NewPostRoute("/exec/{name:.*}/start", r.postContainerExecStart),
        router.NewPostRoute("/exec/{name:.*}/resize", r.postContainerExecResize),
        router.NewPostRoute("/containers/{name:.*}/rename", r.postContainerRename),
        router.NewPostRoute("/containers/{name:.*}/update", r.postContainerUpdate),
        router.NewPostRoute("/containers/prune", r.postContainersPrune, router.WithCancel),
        router.NewPostRoute("/commit", r.postCommit),
        // PUT
        router.NewPutRoute("/containers/{name:.*}/archive", r.putContainersArchive),
        // DELETE
        router.NewDeleteRoute("/containers/{name:.*}", r.deleteContainers),
    }
}

通过以上的执行过程,我们找到了docker run最终调用的命令,接上文,继续docker run命令的执行.

1.4.3 postContainersCreate方法

通过路由绑定的http路径,我们找到了docker run调用的方法r.postContainersCreate:

func (s *containerRouter) postContainersCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    ...

    ccr, err := s.backend.ContainerCreate(types.ContainerCreateConfig{
        Name:             name,
        Config:           config,
        HostConfig:       hostConfig,
        NetworkingConfig: networkingConfig,
        AdjustCPUShares:  adjustCPUShares,
    })
    ...
}

s.backend也就是前面DaemonCli的start方法中创建的deamon:

d, err := daemon.NewDaemon(ctx, cli.Config, pluginStore)

我们看一下deamon的定义:

1.4.4 Daemon结构体

// Daemon holds information about the Docker daemon.
type Daemon struct {
    ID                string
    repository        string
    containers        container.Store
    containersReplica container.ViewDB
    execCommands      *exec.Store
    imageService      *images.ImageService
    idIndex           *truncindex.TruncIndex
    configStore       *config.Config
    statsCollector    *stats.Collector
    defaultLogConfig  containertypes.LogConfig
    RegistryService   registry.Service
    EventsService     *events.Events
    netController     libnetwork.NetworkController
    volumes           *volumesservice.VolumesService
    discoveryWatcher  discovery.Reloader
    root              string
    seccompEnabled    bool
    apparmorEnabled   bool
    shutdown          bool
    idMapping         *idtools.IdentityMapping
    // TODO: move graphDrivers field to an InfoService
    graphDrivers map[string]string // By operating system

    PluginStore           *plugin.Store // todo: remove
    pluginManager         *plugin.Manager
    linkIndex             *linkIndex
    containerdCli         *containerd.Client
    containerd            libcontainerd.Client
    defaultIsolation      containertypes.Isolation // Default isolation mode on Windows
    clusterProvider       cluster.Provider
    cluster               Cluster
    genericResources      []swarm.GenericResource
    metricsPluginListener net.Listener

    machineMemory uint64

    seccompProfile     []byte
    seccompProfilePath string

    diskUsageRunning int32
    pruneRunning     int32
    hosts            map[string]bool // hosts stores the addresses the daemon is listening on
    startupDone      chan struct{}

    attachmentStore       network.AttachmentStore
    attachableNetworkLock *locker.Locker
}

在相同路径下,找到Daemon的ContainerCreate方法:

1.4.5 ContainerCreate方法

// ContainerCreate creates a regular container
func (daemon *Daemon) ContainerCreate(params types.ContainerCreateConfig) (containertypes.ContainerCreateCreatedBody, error) {
    return daemon.containerCreate(params, false)
}

对外可见方法ContainerCreate调用了私有方法containerCreate:

1.4.6 containerCreate方法

func (daemon *Daemon) containerCreate(params types.ContainerCreateConfig, managed bool) (containertypes.ContainerCreateCreatedBody, error) {
    start := time.Now()
    if params.Config == nil {
        return containertypes.ContainerCreateCreatedBody{}, errdefs.InvalidParameter(errors.New("Config cannot be empty in order to create a container"))
    }

    os := runtime.GOOS
    if params.Config.Image != "" {
        img, err := daemon.imageService.GetImage(params.Config.Image)
        if err == nil {
            os = img.OS
        }
    } else {
        // This mean scratch. On Windows, we can safely assume that this is a linux
        // container. On other platforms, it's the host OS (which it already is)
        if runtime.GOOS == "windows" && system.LCOWSupported() {
            os = "linux"
        }
    }

    warnings, err := daemon.verifyContainerSettings(os, params.HostConfig, params.Config, false)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    err = verifyNetworkingConfig(params.NetworkingConfig)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    if params.HostConfig == nil {
        params.HostConfig = &containertypes.HostConfig{}
    }
    err = daemon.adaptContainerSettings(params.HostConfig, params.AdjustCPUShares)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    container, err := daemon.create(params, managed)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, err
    }
    containerActions.WithValues("create").UpdateSince(start)

    return containertypes.ContainerCreateCreatedBody{ID: container.ID, Warnings: warnings}, nil
}

由此,我们找到了客户端docker run命令执行后的整个代码流程.

本篇的主要目的是通过命令的代码流程, 把客户端的代码与服务端的代码连接起来,为以后探究其他命令的执行过程打下基础,简而言之,就是找到最终执行命令的服务端代码.

总结

1.起点

找到该命令调用的dockerCli.Client()下的函数名

2.然后跳转到对应的服务端的客户端代码

3.然后跳转到对应的api server路由找到调用的函数

4.上一步找到的函数一般在daemon中定义具体执行