The magic behind a simple kubectl command! -- Part1



Hey folks, as promised this is the first article in the k8s code's collection.
We gonna talk about kubectl its code implementation and how it works during the runtime.
First, we need to clone the k8s repository, of course am assuming that you have go installed and all the variables related to it like (GOPATH and GOROOT) are alraedy set.
go get -d k8s.io/kubernetes
Once this done, you will see the structure of k8s's repository

Godeps/ 
api/ 
build/ 
cluster/ 
cmd/ 
docs/ 
hack/ 
logo/ 
pkg/ 
plugin/ 
staging/ 
test/ 
third_party/ 
translations/ 
vendor/ 

For now, while we are talking about kubectl, we will be interested by pkg/ folder. There we can find some sub-directories and files.

pi   auth   capabilities  cloudprovider  credentialprovider  fieldpath  kubeapiserver  kubelet   master  printers  proxy  registry  scheduler  securitycontext  ssh   version  watch 
apis  BUILD  client        controller     features            generated  kubectl        kubemark  OWNERS  probe     quota  routes    security   serviceaccount   util  volume   windows

As you can see, we have the folder kubectl. The folder in there which is cmd is the entry point for all kubectl commands.
Every kubectl command has an initial entry point, it can be directly a file or if the command have several usage which will contain many go files, it will be under a folder.
Let's pick an example, you have your k8s cluster up and running and you want to create a new namespace. In this case, your command will be kubectl create namespace
Sounds easy right? Okay :) let's trace that kubectl create command and see where the implementation of the code.
There is always more than one way to do that. You can dump or display information from the kubectl binary itself using objdump for example
or just go and read the documentation and have a quick look on the sub-directories in pkg/kubectl/cmd/ and you will notice that there is a folder with create as a name.
Let's open the go file pkg/kubectl/cmd/create/create.go which is associated to the kubectl create command including every possible argument.

 81 func NewCreateOptions(ioStreams genericclioptions.IOStreams) *CreateOptions { 
 82         return &CreateOptions{ 
 83                 PrintFlags:  genericclioptions.NewPrintFlags("created").WithTypeSetter(scheme.Scheme),
 84                 RecordFlags: genericclioptions.NewRecordFlags(),
 85
 86                 Recorder: genericclioptions.NoopRecorder{},
 87
 88                 IOStreams: ioStreams,
 89         }
 90 }
 91
 92 func NewCmdCreate(f cmdutil.Factory, ioStreams genericclioptions.IOStreams) *cobra.Command {
 93         o := NewCreateOptions(ioStreams)
 94
 95         cmd := &cobra.Command{
 96                 Use: "create -f FILENAME",
 97                 DisableFlagsInUseLine: true,
 98                 Short:   i18n.T("Create a resource from a file or from stdin."),
 99                 Long:    createLong,
100                 Example: createExample,
101                 Run: func(cmd *cobra.Command, args []string) {
102                         if cmdutil.IsFilenameSliceEmpty(o.FilenameOptions.Filenames) {
103                                 defaultRunFunc := cmdutil.DefaultSubCommandRun(ioStreams.ErrOut)
104                                 defaultRunFunc(cmd, args)
105                                 return
106                         }
107                         cmdutil.CheckErr(o.Complete(f, cmd))
108                         cmdutil.CheckErr(o.ValidateArgs(cmd, args))
109                         cmdutil.CheckErr(o.RunCreate(f, cmd))
110                 },
111         }
112
113         // bind flag structs
114         o.RecordFlags.AddFlags(cmd)
115
116         usage := "to use to create the resource"
117         cmdutil.AddFilenameOptionFlags(cmd, &o.FilenameOptions, usage)
118         cmd.MarkFlagRequired("filename")
119         cmdutil.AddValidateFlags(cmd)
120         cmd.Flags().BoolVar(&o.EditBeforeCreate, "edit", o.EditBeforeCreate, "Edit the API resource before creating")
121         cmd.Flags().Bool("windows-line-endings", runtime.GOOS == "windows",
122                 "Only relevant if --edit=true. Defaults to the line ending native to your platform.")
123         cmdutil.AddApplyAnnotationFlags(cmd)
124         cmdutil.AddDryRunFlag(cmd)
125         cmd.Flags().StringVarP(&o.Selector, "selector", "l", o.Selector, "Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2)")
126         cmd.Flags().StringVar(&o.Raw, "raw", o.Raw, "Raw URI to POST to the server.  Uses the transport specified by the kubeconfig file.")
127
128         o.PrintFlags.AddFlags(cmd)
129
130         // create subcommands
131         cmd.AddCommand(NewCmdCreateNamespace(f, ioStreams))
132         cmd.AddCommand(NewCmdCreateQuota(f, ioStreams))
133         cmd.AddCommand(NewCmdCreateSecret(f, ioStreams))
134         cmd.AddCommand(NewCmdCreateConfigMap(f, ioStreams))
135         cmd.AddCommand(NewCmdCreateServiceAccount(f, ioStreams))
136         cmd.AddCommand(NewCmdCreateService(f, ioStreams))
137         cmd.AddCommand(NewCmdCreateDeployment(f, ioStreams))
138         cmd.AddCommand(NewCmdCreateClusterRole(f, ioStreams))
139         cmd.AddCommand(NewCmdCreateClusterRoleBinding(f, ioStreams))
140         cmd.AddCommand(NewCmdCreateRole(f, ioStreams))
141         cmd.AddCommand(NewCmdCreateRoleBinding(f, ioStreams))
142         cmd.AddCommand(NewCmdCreatePodDisruptionBudget(f, ioStreams))

I took this piece of code from create.go file to demonstrate the implementation of the command kubectl create -f file at the runtime.
The function associated to this starts from the line 92 and ends at the line 111
You can notice there that kubernetes commands are implemented using the Cobra command framework. Cobra provides a lot of great features for building CLI.
As you can see here

        
         cmd := &cobra.Command{
                 Use: "create -f FILENAME",
                 DisableFlagsInUseLine: true,
                 Short:   i18n.T("Create a resource from a file or from stdin."),
                 Long:    createLong,
                 Example: createExample,
                 Run: func(cmd *cobra.Command, args []string) {
                         if cmdutil.IsFilenameSliceEmpty(o.FilenameOptions.Filenames) {
                                 defaultRunFunc := cmdutil.DefaultSubCommandRun(ioStreams.ErrOut)
                                 defaultRunFunc(cmd, args)
                                 return
                         }
                         cmdutil.CheckErr(o.Complete(f, cmd))
                         cmdutil.CheckErr(o.ValidateArgs(cmd, args))
                         cmdutil.CheckErr(o.RunCreate(f, cmd))
                 },
         }

Cobra makes it very easy to locate which file implements each command line option. It gives also the ability to put the command usage alongside a brief description of what the command can do.
While this is implemented in every piece of kuberenetes. So, you can walk through all kubectl commands and you can read their descriptions which will make you jump to the piece of code you need.
As shown in lines 96-101 in the snippet of code, the strings Use, Short, Long, and Example all hold information describing the command and Run points to a function that actually runs the command.
We can also see that there are three functions in order to run the command kubectl create -f FILENAME. These are the following funcs:
The RunCreate function is the most important one, it is invoked on line 109 where the bulk of the kubectl create command is implemented. The implementation of this function can be found in the same file create.go

205 func (o *CreateOptions) RunCreate(f cmdutil.Factory, cmd *cobra.Command) error {
206         // raw only makes sense for a single file resource multiple objects aren't likely to do what you want.
207         // the validator enforces this, so
208         if len(o.Raw) > 0 {
209                 return o.raw(f)
210         }
211
212         if o.EditBeforeCreate {
213                 return RunEditOnCreate(f, o.PrintFlags, o.RecordFlags, o.IOStreams, cmd, &o.FilenameOptions)
214         }
215         schema, err := f.Validator(cmdutil.GetFlagBool(cmd, "validate"))
216         if err != nil {
217                 return err
218         }
219
220         cmdNamespace, enforceNamespace, err := f.ToRawKubeConfigLoader().Namespace()
221         if err != nil {
222                 return err
223         }
224
225         r := f.NewBuilder().
226                 Unstructured().
227                 Schema(schema).
228                 ContinueOnError().
229                 NamespaceParam(cmdNamespace).DefaultNamespace().
230                 FilenameParam(enforceNamespace, &o.FilenameOptions).
231                 LabelSelectorParam(o.Selector).
232                 Flatten().
233                 Do()
234         err = r.Err()
235         if err != nil {
236                 return err
237         }
238
239         count := 0
240         err = r.Visit(func(info *resource.Info, err error) error {
241                 if err != nil {
242                         return err
243                 }
244                 if err := kubectl.CreateOrUpdateAnnotation(cmdutil.GetFlagBool(cmd, cmdutil.ApplyAnnotationsFlag), info.Object, cmdutil.InternalVersionJSONEncoder()); err != nil {
245                         return cmdutil.AddSourceToErr("creating", info.Source, err)
246                 }
247
248                 if err := o.Recorder.Record(info.Object); err != nil {
249                         glog.V(4).Infof("error recording current command: %v", err)
250                 }
251
252                 if !o.DryRun {
253                         if err := createAndRefresh(info); err != nil {
254                                 return cmdutil.AddSourceToErr("creating", info.Source, err)
255                         }
256                 }
257
258                 count++
259
260                 return o.PrintObj(info.Object)
261         })
262         if err != nil {
263                 return err
264         }
265         if count == 0 {
266                 return fmt.Errorf("no objects passed to create")
267         }
268         return nil
269 }

As you can see into the RunCreate func, there is the builder mechanism which is implemented in many piece of kubernetes. It is a little bit tricky for any newcomer to the kubernetes code or to golang in general.
So, f.NewBuilder will take arguments and parameters from the command line and converts them into a list of resources. That's in a brief description of what NewBuilder do.
Besides, it's also responsible for creating a visitor construct that can be used to iterate across all the resources.
The code is complex because it uses a variant of the builder pattern where individual functions are each doing a separate portion of the data initialization.
The functions Unstructed, Schema, ContinueOnError, NamespaceParam, DefaultNamespace, FilenameParam, SelectorParam and Flatten all take in a pointer to a Builder struct, perform some form of modification
on the Builder struct, and then return the pointer to the Builder struct for the next method in the chain to use when it performs its modifications.
All of these methods are basically implemented in another go file. You can find it under staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/builder.go

func (b *Builder) Schema(schema ContentValidator) *Builder {
       b.schema = schema
       return b
}
func (b *Builder) ContinueOnError() *Builder { b.continueOnError = true return b }
func (b *Builder) NamespaceParam(namespace string) *Builder { b.namespace = namespace return b }
func (b *Builder) FilenameParam(enforceNamespace bool, filenameOptions *FilenameOptions) *Builder { recursive := filenameOptions.Recursive paths := filenameOptions.Filenames for _, s := range paths { switch { case s == "-": b.Stdin() case strings.Index(s, "http://") == 0 || strings.Index(s, "https://") == 0: url, err := url.Parse(s) if err != nil { b.errs = append(b.errs, fmt.Errorf("the URL passed to filename %q is not valid: %v", s, err)) continue } b.URL(defaultHttpGetAttempts, url) default: if !recursive { b.singleItemImplied = true } b.Path(recursive, s) } } if enforceNamespace { b.RequireNamespace() } return b }
func (b *Builder) LabelSelectorParam(s string) *Builder { selector := strings.TrimSpace(s) if len(selector) == 0 { return b } if b.selectAll { b.errs = append(b.errs, fmt.Errorf("found non-empty label selector %q with previously set 'all' parameter. ", s)) return b } return b.LabelSelector(selector) }
func (b *Builder) Flatten() *Builder { b.flatten = true return b }
Once all the initializers have completed the f.NewBuilder func calls the Do func.
Which will returs a Result object that will be used to drive and begin the creaton of our resource.
Here where it comes the concept of Visitor which is also an object will make it through the list of resources that were associated with this invocation of f.NewBuilder.
This is the implementation of the Do func:

func (b *Builder) Do() *Result {
       r := b.visitorResult()
       r.mapper = b.Mapper()
       if r.err != nil {
               return r
       }
       if b.flatten {
               r.visitor = NewFlattenListVisitor(r.visitor, b.objectTyper, b.mapper)
       }
       helpers := []VisitorFunc{}
       if b.defaultNamespace {
               helpers = append(helpers, SetNamespace(b.namespace))
       }
       if b.requireNamespace {
               helpers = append(helpers, RequireNamespace(b.namespace))
       }
       helpers = append(helpers, FilterNamespace)
       if b.requireObject {
               helpers = append(helpers, RetrieveLazy)
       }
       r.visitor = NewDecoratedVisitor(r.visitor, helpers...)
       if b.continueOnError {
               r.visitor = ContinueOnErrorVisitor{r.visitor}
       }
       return r
}

the NewDecoratedVisitor is created and stored as part of the Result object that is returned by the Builder Do func.
The DecoratedVisitor has a Vist func that will call the Visitor func which is passed already into it.
As usual, this is the implementation of the NewDecoratedVisitor func:

// NewDecoratedVisitor will create a visitor that invokes the provided visitor functions before
// the user supplied visitor function is invoked, giving them the opportunity to mutate the Info
// object or terminate early with an error.
func NewDecoratedVisitor(v Visitor, fn ...VisitorFunc) Visitor {
        if len(fn) == 0 {
                return v
        }
        return DecoratedVisitor{v, fn}
}

// Visit implements Visitor
func (v DecoratedVisitor) Visit(fn VisitorFunc) error {
        return v.visitor.Visit(func(info *Info, err error) error {
                if err != nil {
                        return err
                }
                for i := range v.decorators {
                        if err := v.decorators[i](info, nil); err != nil {
                                return err
                        }
                }
                return fn(info, nil)
        })
}


Well, I think this article is getting too long, I will try to devide it into parts. I think it will be 4 parts related to the implementation of kubectl.

I hope you learned something in this article! Please let me know in e-mail or twitter if I missed anything or if there is any typo.
Well, that's it for today.

Cheers o/