Restore API group version by priority (#3133)

* Restore API group version by priority

Signed-off-by: F. Gold <fgold@vmware.com>

* Add changelog

Signed-off-by: F. Gold <fgold@vmware.com>

* Correct spelling

Signed-off-by: F. Gold <fgold@vmware.com>

* Refactor userResourceGroupVersionPriorities(...) to accept config map, adjust unit test

Signed-off-by: F. Gold <fgold@vmware.com>

* Move some unit tests into e2e

Signed-off-by: F. Gold <fgold@vmware.com>

* Add three e2e tests using Testify Suites

Summary of changes

Makefile - add testify e2e test target
go.sum - changed with go mod tidy
pkg/install/install.go - increased polling timeout
test/e2e/restore_priority_group_test.go - deleted
test/e2e/restore_test.go - deleted
test/e2e/velero_utils.go - made restic optional in velero install
test/e2e_testify/Makefile - makefile for testify e2e tests
test/e2e_testify/README.md - example command for running tests
test/e2e_testify/common_test.go - helper functions
test/e2e_testify/e2e_suite_test.go - prepare for tests and run
test/e2e_testify/restore_priority_apigv_test.go - test cases

Signed-off-by: F. Gold <fgold@vmware.com>

* Make changes per @nrb code review

Signed-off-by: F. Gold <fgold@vmware.com>

* Wait for pods in e2e tests

Signed-off-by: F. Gold <fgold@vmware.com>

* Remove testify suites e2e scaffolding moved to PR #3354

Signed-off-by: F. Gold <fgold@vmware.com>

* Make changes per @brito-rafa and Velero maintainers code reviews

- Made changes suggested by @brito-rafa in GitHub.
- We had a code review meeting with @carlisia, @dsu-igeek, @zubron, and @nrb
- and changes were made based on their suggetions:
  - pull in logic from 'meetsAPIGVResotreReqs()' to restore.go.
  - add TODO to remove APIGroupVersionFeatureFlag check
  - have feature flag and backup version format checks in separate `if` statements.
  - rename variables to be sourceGVs, targetGVs, and userGVs.

Signed-off-by: F. Gold <fgold@vmware.com>

* Convert Testify Suites e2e tests to existing Ginkgo framework

Signed-off-by: F. Gold <fgold@vmware.com>

* Made changes per @zubron PR review

Signed-off-by: F. Gold <fgold@vmware.com>

* Run go mod tidy after resolving go.sum merge conflict

Signed-off-by: F. Gold <fgold@vmware.com>

* Add feature documentation to velero.io site

Signed-off-by: F. Gold <fgold@vmware.com>

* Add config map e2e test; rename e2e test file and name

Signed-off-by: F. Gold <fgold@vmware.com>

* Update go.{mod,sum} files

Signed-off-by: F. Gold <fgold@vmware.com>

* Move CRDs and CRs to testdata folder

Signed-off-by: F. Gold <fgold@vmware.com>

* Fix typos in cert-manager to pass codespell CICD check

Signed-off-by: F. Gold <fgold@vmware.com>

* Make changes per @nrb code review round 2

- make checkAndReadDir function private
- add info level messages when priorties 1-3 API group versions can not be used

Signed-off-by: F. Gold <fgold@vmware.com>

* Make user config map rules less strict

Signed-off-by: F. Gold <fgold@vmware.com>

* Update e2e test image version in example

Signed-off-by: F. Gold <fgold@vmware.com>

* Update case A music-system controller code

Signed-off-by: F. Gold <fgold@vmware.com>

* Documentation updates

Signed-off-by: F. Gold <fgold@vmware.com>

* Update migration case documentation

Signed-off-by: F. Gold <fgold@vmware.com>
pull/3286/head
codegold79 2021-02-16 09:36:17 -08:00 committed by GitHub
parent 2c6adab903
commit 6bdd4ac192
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 33696 additions and 30 deletions

View File

@ -0,0 +1,2 @@
Restore API group version by priority
Increase timeout to 3 minutes in DeploymentIsReady(...) function in the install package

2
go.mod
View File

@ -20,7 +20,7 @@ require (
github.com/hashicorp/go-plugin v0.0.0-20190610192547-a1bc61569a26
github.com/joho/godotenv v1.3.0
github.com/kubernetes-csi/external-snapshotter/client/v4 v4.0.0
github.com/onsi/ginkgo v1.14.2
github.com/onsi/ginkgo v1.15.0
github.com/onsi/gomega v1.10.2
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.7.1

14
go.sum
View File

@ -429,8 +429,8 @@ github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.1/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/ginkgo v1.14.2 h1:8mVmC9kjFFmA8H4pKMUhcblgifdkOIXPvbhN1T36q1M=
github.com/onsi/ginkgo v1.14.2/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/ginkgo v1.15.0 h1:1V1NfVQR87RtWAgp1lv9JZJ5Jap+XFGKPi00andXGi4=
github.com/onsi/ginkgo v1.15.0/go.mod h1:hF8qUzuuC8DJGygJH3726JnCZX4MYbRB8yFfISqnKUg=
github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
@ -532,6 +532,7 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1/go.mod h1:QcJo0QPSfTONNIgpN5RA8prR7fF8nkF6cTWTcNerRO8=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
@ -627,6 +628,7 @@ golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b h1:uwuIcX0g4Yl1NC5XAz37xsr2lTtcqevgzYNVt49waME=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
@ -641,6 +643,7 @@ golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -682,6 +685,8 @@ golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201112073958-5cba982894dd h1:5CtCZbICpIOFdgO940moixOPjc0178IU44m4EjOO5IY=
golang.org/x/sys v0.0.0-20201112073958-5cba982894dd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210112080510-489259a85091 h1:DMyOG0U+gKfu8JZzg2UQe9MeaC1X+xQWlAKcRnjxjCw=
golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@ -727,10 +732,14 @@ golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtn
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200616133436-c1934b75d054 h1:HHeAlu5H9b71C+Fx0K+1dGgVFN1DM1/wz4aoGOA5qS8=
golang.org/x/tools v0.0.0-20200616133436-c1934b75d054/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e h1:4nW4NLDYnU28ojHaHO8OVxFHk/aQ33U01a9cjED+pzE=
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gomodules.xyz/jsonpatch/v2 v2.1.0 h1:Phva6wqu+xR//Njw6iorylFFgn/z547tw5Ne3HZPQ+k=
gomodules.xyz/jsonpatch/v2 v2.1.0/go.mod h1:IhYNNY4jnS53ZnfE4PAmpKtDpTCj1JFXc+3mwe7XcUU=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
@ -808,6 +817,7 @@ gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

View File

@ -1,5 +1,5 @@
/*
Copyright 2019 the Velero contributors.
Copyright The Velero Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@ -17,11 +17,13 @@ limitations under the License.
package archive
import (
"os"
"path/filepath"
"strings"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/util/filesystem"
@ -165,3 +167,102 @@ func (p *Parser) getResourceItemsForScope(dir, archiveRootDir string) ([]string,
return items, nil
}
// checkAndReadDir is a wrapper around fs.DirExists and fs.ReadDir that does checks
// and returns errors if directory cannot be read.
func (p *Parser) checkAndReadDir(dir string) ([]os.FileInfo, error) {
exists, err := p.fs.DirExists(dir)
if err != nil {
return []os.FileInfo{}, errors.Wrapf(err, "finding %q", dir)
}
if !exists {
return []os.FileInfo{}, errors.Errorf("%q not found", dir)
}
contents, err := p.fs.ReadDir(dir)
if err != nil {
return []os.FileInfo{}, errors.Wrapf(err, "reading contents of %q", dir)
}
return contents, nil
}
// ParseGroupVersions extracts the versions for each API Group from the backup
// directory names and stores them in a metav1 APIGroup object.
func (p *Parser) ParseGroupVersions(dir string) (map[string]metav1.APIGroup, error) {
resourcesDir := filepath.Join(dir, velerov1api.ResourcesDir)
// Get the subdirectories inside the "resources" directory. The subdirectories
// will have resource.group names like "horizontalpodautoscalers.autoscaling".
rgDirs, err := p.checkAndReadDir(resourcesDir)
if err != nil {
return nil, err
}
resourceAGs := make(map[string]metav1.APIGroup)
// Loop through the resource.group directory names.
for _, rgd := range rgDirs {
group := metav1.APIGroup{
Name: extractGroupName(rgd.Name()),
}
rgdPath := filepath.Join(resourcesDir, rgd.Name())
// Inside each of the resource.group directories are directories whose
// names are API Group versions like "v1" or "v1-preferredversion"
gvDirs, err := p.checkAndReadDir(rgdPath)
if err != nil {
return nil, err
}
var supportedVersions []metav1.GroupVersionForDiscovery
for _, gvd := range gvDirs {
gvdName := gvd.Name()
// Don't save the namespaces or clusters directories in list of
// supported API Group Versions.
if gvdName == "namespaces" || gvdName == "cluster" {
continue
}
version := metav1.GroupVersionForDiscovery{
GroupVersion: strings.TrimPrefix(group.Name+"/"+gvdName, "/"),
Version: gvdName,
}
if strings.Contains(gvdName, velerov1api.PreferredVersionDir) {
gvdName = strings.TrimSuffix(gvdName, velerov1api.PreferredVersionDir)
// Update version and group version to be without suffix.
version.Version = gvdName
version.GroupVersion = strings.TrimPrefix(group.Name+"/"+gvdName, "/")
group.PreferredVersion = version
}
supportedVersions = append(supportedVersions, version)
}
group.Versions = supportedVersions
resourceAGs[rgd.Name()] = group
}
return resourceAGs, nil
}
// extractGroupName will take a concatenated resource.group and extract the group,
// if there is one. Resources like "pods" which has no group and will return an
// empty string.
func extractGroupName(resourceGroupDir string) string {
parts := strings.SplitN(resourceGroupDir, ".", 2)
var group string
if len(parts) == 2 {
group = parts[1]
}
return group
}

View File

@ -1,5 +1,5 @@
/*
Copyright 2019 the Velero contributors.
Copyright The Velero Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@ -17,28 +17,28 @@ limitations under the License.
package archive
import (
"errors"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"github.com/vmware-tanzu/velero/pkg/test"
)
func TestParse(t *testing.T) {
tests := []struct {
name string
files []string
dir string
wantErr error
want map[string]*ResourceItems
name string
files []string
dir string
wantErrMsg string
want map[string]*ResourceItems
}{
{
name: "when there is no top-level resources directory, an error is returned",
dir: "root-dir",
wantErr: errors.New("directory \"resources\" does not exist"),
name: "when there is no top-level resources directory, an error is returned",
dir: "root-dir",
wantErrMsg: "directory \"resources\" does not exist",
},
{
name: "when there are no directories under the resources directory, an empty map is returned",
@ -109,8 +109,8 @@ func TestParse(t *testing.T) {
}
res, err := p.Parse(tc.dir)
if tc.wantErr != nil {
assert.Equal(t, err.Error(), tc.wantErr.Error())
if tc.wantErrMsg != "" {
assert.EqualError(t, err, tc.wantErrMsg)
} else {
assert.Nil(t, err)
assert.Equal(t, tc.want, res)
@ -118,3 +118,154 @@ func TestParse(t *testing.T) {
})
}
}
func TestParseGroupVersions(t *testing.T) {
tests := []struct {
name string
files []string
backupDir string
wantErrMsg string
want map[string]metav1.APIGroup
}{
{
name: "when there is no top-level resources directory, an error is returned",
backupDir: "/var/folders",
wantErrMsg: "\"/var/folders/resources\" not found",
},
{
name: "when there are no directories under the resources directory, an empty map is returned",
backupDir: "/var/folders",
files: []string{"/var/folders/resources/"},
want: map[string]metav1.APIGroup{},
},
{
name: "when there is a mix of cluster-scoped and namespaced items for resources with preferred or multiple API groups, all group versions are correctly returned",
backupDir: "/var/folders",
files: []string{
"/var/folders/resources/clusterroles.rbac.authorization.k8s.io/v1-preferredversion/cluster/system/controller/attachdetach-controller.json",
"/var/folders/resources/clusterroles.rbac.authorization.k8s.io/cluster/system/controller/attachdetach-controller.json",
"/var/folders/resources/horizontalpodautoscalers.autoscaling/namespaces/myexample/php-apache-autoscaler.json",
"/var/folders/resources/horizontalpodautoscalers.autoscaling/v1-preferredversion/namespaces/myexample/php-apache-autoscaler.json",
"/var/folders/resources/horizontalpodautoscalers.autoscaling/v2beta1/namespaces/myexample/php-apache-autoscaler.json",
"/var/folders/resources/horizontalpodautoscalers.autoscaling/v2beta2/namespaces/myexample/php-apache-autoscaler.json",
"/var/folders/resources/pods/namespaces/nginx-example/nginx-deployment-57d5dcb68-wrqsc.json",
"/var/folders/resources/pods/v1-preferredversion/namespaces/nginx-example/nginx-deployment-57d5dcb68-wrqsc.json",
},
want: map[string]metav1.APIGroup{
"clusterroles.rbac.authorization.k8s.io": {
Name: "rbac.authorization.k8s.io",
Versions: []metav1.GroupVersionForDiscovery{
{
GroupVersion: "rbac.authorization.k8s.io/v1",
Version: "v1",
},
},
PreferredVersion: metav1.GroupVersionForDiscovery{
GroupVersion: "rbac.authorization.k8s.io/v1",
Version: "v1",
},
},
"horizontalpodautoscalers.autoscaling": {
Name: "autoscaling",
Versions: []metav1.GroupVersionForDiscovery{
{
GroupVersion: "autoscaling/v1",
Version: "v1",
},
{
GroupVersion: "autoscaling/v2beta1",
Version: "v2beta1",
},
{
GroupVersion: "autoscaling/v2beta2",
Version: "v2beta2",
},
},
PreferredVersion: metav1.GroupVersionForDiscovery{
GroupVersion: "autoscaling/v1",
Version: "v1",
},
},
"pods": {
Name: "",
Versions: []metav1.GroupVersionForDiscovery{
{
GroupVersion: "v1",
Version: "v1",
},
},
PreferredVersion: metav1.GroupVersionForDiscovery{
GroupVersion: "v1",
Version: "v1",
},
},
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
p := &Parser{
log: test.NewLogger(),
fs: test.NewFakeFileSystem(),
}
for _, file := range tc.files {
require.NoError(t, p.fs.MkdirAll(file, 0755))
if !strings.HasSuffix(file, "/") {
res, err := p.fs.Create(file)
require.NoError(t, err)
require.NoError(t, res.Close())
}
}
res, err := p.ParseGroupVersions(tc.backupDir)
if tc.wantErrMsg != "" {
assert.EqualError(t, err, tc.wantErrMsg)
} else {
assert.Nil(t, err)
assert.Equal(t, tc.want, res)
}
})
}
}
func TestExtractGroupName(t *testing.T) {
tests := []struct {
name string
rgDir string
want string
}{
{
name: "Directory has no dots (only a group name)",
rgDir: "pods",
want: "",
},
{
name: "Directory has one concatenation dot (has both resource and group name which have 0 dots",
rgDir: "cronjobs.batch",
want: "batch",
},
{
name: "Directory has 3 dots in name (group has 2 dot)",
rgDir: "leases.coordination.k8s.io",
want: "coordination.k8s.io",
},
{
name: "Directory has 4 dots in name (group has 3 dots)",
rgDir: "roles.rbac.authorization.k8s.io",
want: "rbac.authorization.k8s.io",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
grp := extractGroupName(tc.rgDir)
assert.Equal(t, tc.want, grp)
})
}
}

View File

@ -132,7 +132,7 @@ func DeploymentIsReady(factory client.DynamicFactory, namespace string) (bool, e
// declare this variable out of scope so we can return it
var isReady bool
var readyObservations int32
err = wait.PollImmediate(time.Second, time.Minute, func() (bool, error) {
err = wait.PollImmediate(time.Second, 3*time.Minute, func() (bool, error) {
unstructuredDeployment, err := c.Get("velero", metav1.GetOptions{})
if apierrors.IsNotFound(err) {
return false, nil

View File

@ -0,0 +1,351 @@
/*
Copyright The Velero Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package restore
import (
"context"
"sort"
"strings"
"github.com/pkg/errors"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/version"
velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
"github.com/vmware-tanzu/velero/pkg/archive"
"github.com/vmware-tanzu/velero/pkg/client"
)
// ChosenGroupVersion is the API Group version that was selected to restore
// from potentially multiple backed up version enabled by the feature flag
// APIGroupVersionsFeatureFlag
type ChosenGroupVersion struct {
Group string
Version string
Dir string
}
// chooseAPIVersionsToRestore will choose a version to restore based on a user-
// provided config map prioritization or our version prioritization.
func (ctx *restoreContext) chooseAPIVersionsToRestore() error {
sourceGVs, targetGVs, userGVs, err := ctx.gatherSourceTargetUserGroupVersions()
if err != nil {
return err
}
OUTER:
for rg, sg := range sourceGVs {
// Default to the source preferred version if no other common version
// can be found.
cgv := ChosenGroupVersion{
Group: sg.Name,
Version: sg.PreferredVersion.Version,
Dir: sg.PreferredVersion.Version + velerov1api.PreferredVersionDir,
}
tg := findAPIGroup(targetGVs, sg.Name)
if len(tg.Versions) == 0 {
ctx.chosenGrpVersToRestore[rg] = cgv
ctx.log.Debugf("Chose %s/%s API group version to restore", cgv.Group, cgv.Version)
continue
}
// Priority 0: User Priority Version
if userGVs != nil {
uv := findSupportedUserVersion(userGVs[rg].Versions, tg.Versions, sg.Versions)
if uv != "" {
cgv.Version = uv
cgv.Dir = uv
if uv == sg.PreferredVersion.Version {
cgv.Dir += velerov1api.PreferredVersionDir
}
ctx.chosenGrpVersToRestore[rg] = cgv
ctx.log.Debugf("APIGroupVersionsFeatureFlag Priority 0: User defined API group version %s chosen for %s", uv, rg)
continue
}
ctx.log.Infof("Cannot find user defined version in both the cluster and backup cluster. Ignoring version %s for %s", uv, rg)
}
// Priority 1: Target Cluster Preferred Version
if versionsContain(sg.Versions, tg.PreferredVersion.Version) {
cgv.Version = tg.PreferredVersion.Version
cgv.Dir = tg.PreferredVersion.Version
if tg.PreferredVersion.Version == sg.PreferredVersion.Version {
cgv.Dir += velerov1api.PreferredVersionDir
}
ctx.chosenGrpVersToRestore[rg] = cgv
ctx.log.Debugf(
"APIGroupVersionsFeatureFlag Priority 1: Cluster preferred API group version %s found in backup for %s",
tg.PreferredVersion.Version,
rg,
)
continue
}
ctx.log.Infof("Cannot find cluster preferred API group version in backup. Ignoring version %s for %s", tg.PreferredVersion.Version, rg)
// Priority 2: Source Cluster Preferred Version
if versionsContain(tg.Versions, sg.PreferredVersion.Version) {
cgv.Version = sg.PreferredVersion.Version
cgv.Dir = cgv.Version + velerov1api.PreferredVersionDir
ctx.chosenGrpVersToRestore[rg] = cgv
ctx.log.Debugf(
"APIGroupVersionsFeatureFlag Priority 2: Cluster preferred API group version not found in backup. Using backup preferred version %s for %s",
sg.PreferredVersion.Version,
rg,
)
continue
}
ctx.log.Infof("Cannot find backup preferred API group version in cluster. Ignoring version %s for %s", sg.PreferredVersion.Version, rg)
// Priority 3: The Common Supported Version with the Highest Kubernetes Version Priority
for _, tv := range tg.Versions[1:] {
if versionsContain(sg.Versions[1:], tv.Version) {
cgv.Version = tv.Version
cgv.Dir = tv.Version
ctx.chosenGrpVersToRestore[rg] = cgv
ctx.log.Debugf(
"APIGroupVersionsFeatureFlag Priority 3: Common supported but not preferred API group version %s chosen for %s",
tv.Version,
rg,
)
continue OUTER
}
}
ctx.log.Infof("Cannot find non-preferred a common supported API group version. Using %s (default behavior without feature flag) for %s", sg.PreferredVersion.Version, rg)
// Use default group version.
ctx.chosenGrpVersToRestore[rg] = cgv
ctx.log.Debugf(
"APIGroupVersionsFeatureFlag: Unable to find supported priority API group version. Using backup preferred version %s for %s (default behavior without feature flag).",
tg.PreferredVersion.Version,
rg,
)
}
return nil
}
// gatherSourceTargetUserGroupVersions collects the source, target, and user priority versions.
func (ctx *restoreContext) gatherSourceTargetUserGroupVersions() (
map[string]metav1.APIGroup,
[]metav1.APIGroup,
map[string]metav1.APIGroup,
error,
) {
sourceRGVersions, err := archive.NewParser(ctx.log, ctx.fileSystem).ParseGroupVersions(ctx.restoreDir)
if err != nil {
return nil, nil, nil, errors.Wrap(err, "parsing versions from directory names")
}
// Sort the versions in the APIGroups in sourceRGVersions map values.
for _, src := range sourceRGVersions {
k8sPrioritySort(src.Versions)
}
targetGroupVersions := ctx.discoveryHelper.APIGroups()
// Sort the versions in the APIGroups slice in targetGroupVersions.
for _, target := range targetGroupVersions {
k8sPrioritySort(target.Versions)
}
// Get the user-provided enableapigroupversion config map.
cm, err := userPriorityConfigMap()
if err != nil {
return nil, nil, nil, errors.Wrap(err, "retrieving enableapigroupversion config map")
}
// Read user-defined version priorities from config map.
userRGVPriorities := userResourceGroupVersionPriorities(ctx, cm)
return sourceRGVersions, targetGroupVersions, userRGVPriorities, nil
}
// k8sPrioritySort sorts slices using Kubernetes' version prioritization.
func k8sPrioritySort(gvs []metav1.GroupVersionForDiscovery) {
sort.SliceStable(gvs, func(i, j int) bool {
return version.CompareKubeAwareVersionStrings(gvs[i].Version, gvs[j].Version) > 0
})
}
// userResourceGroupVersionPriorities retrieves a user-provided config map and
// extracts the user priority versions for each resource.
func userResourceGroupVersionPriorities(ctx *restoreContext, cm *corev1.ConfigMap) map[string]metav1.APIGroup {
if cm == nil {
ctx.log.Debugf("No enableapigroupversion config map found in velero namespace. Using pre-defined priorities.")
return nil
}
priorities := parseUserPriorities(ctx, cm.Data["restoreResourcesVersionPriority"])
if len(priorities) == 0 {
ctx.log.Debugf("No valid user version priorities found in enableapigroupversion config map. Using pre-defined priorities.")
return nil
}
return priorities
}
func userPriorityConfigMap() (*corev1.ConfigMap, error) {
cfg, err := client.LoadConfig()
if err != nil {
return nil, errors.Wrap(err, "reading client config file")
}
fc := client.NewFactory("APIGroupVersionsRestore", cfg)
kc, err := fc.KubeClient()
if err != nil {
return nil, errors.Wrap(err, "getting Kube client")
}
cm, err := kc.CoreV1().ConfigMaps("velero").Get(
context.Background(),
"enableapigroupversions",
metav1.GetOptions{},
)
if err != nil {
if apierrors.IsNotFound(err) {
return nil, nil
}
return nil, errors.Wrap(err, "getting enableapigroupversions config map from velero namespace")
}
return cm, nil
}
func parseUserPriorities(ctx *restoreContext, prioritiesData string) map[string]metav1.APIGroup {
userPriorities := make(map[string]metav1.APIGroup)
// The user priorities will be in a string of the form
// rockbands.music.example.io=v2beta1,v2beta2\n
// orchestras.music.example.io=v2,v3alpha1\n
// subscriptions.operators.coreos.com=v2,v1
lines := strings.Split(prioritiesData, "\n")
lines = formatUserPriorities(lines)
for _, line := range lines {
err := validateUserPriority(line)
if err == nil {
rgvs := strings.SplitN(line, "=", 2)
rg := rgvs[0] // rockbands.music.example.io
versions := rgvs[1] // v2beta1,v2beta2
vers := strings.Split(versions, ",")
userPriorities[rg] = metav1.APIGroup{
Versions: versionsToGroupVersionForDiscovery(vers),
}
} else {
ctx.log.Debugf("Unable to validate user priority versions %q due to %v", line, err)
}
}
return userPriorities
}
// formatUserPriorities removes extra white spaces that cause validation to fail.
func formatUserPriorities(lines []string) []string {
trimmed := []string{}
for _, line := range lines {
temp := strings.ReplaceAll(line, " ", "")
if len(temp) > 0 {
trimmed = append(trimmed, temp)
}
}
return trimmed
}
func validateUserPriority(line string) error {
if strings.Count(line, "=") != 1 {
return errors.New("line must have one and only one equal sign")
}
pair := strings.Split(line, "=")
if len(pair[0]) < 1 || len(pair[1]) < 1 {
return errors.New("line must contain at least one character before and after equal sign")
}
// Line must not contain any spaces
if strings.Count(line, " ") > 0 {
return errors.New("line must not contain any spaces")
}
return nil
}
// versionsToGroupVersionForDiscovery converts version strings into a Kubernetes format
// for group versions.
func versionsToGroupVersionForDiscovery(vs []string) []metav1.GroupVersionForDiscovery {
gvs := make([]metav1.GroupVersionForDiscovery, len(vs))
for i, v := range vs {
gvs[i] = metav1.GroupVersionForDiscovery{
Version: v,
}
}
return gvs
}
// findAPIGroup looks for an API Group by a group name.
func findAPIGroup(groups []metav1.APIGroup, name string) metav1.APIGroup {
for _, g := range groups {
if g.Name == name {
return g
}
}
return metav1.APIGroup{}
}
// findSupportedUserVersion finds the first user priority version that both source
// and target support.
func findSupportedUserVersion(userGVs, targetGVs, sourceGVs []metav1.GroupVersionForDiscovery) string {
for _, ug := range userGVs {
if versionsContain(targetGVs, ug.Version) && versionsContain(sourceGVs, ug.Version) {
return ug.Version
}
}
return ""
}
// versionsContain will check if a version can be found in a a slice of versions.
func versionsContain(list []metav1.GroupVersionForDiscovery, version string) bool {
for _, v := range list {
if v.Version == version {
return true
}
}
return false
}

View File

@ -0,0 +1,374 @@
/*
Copyright The Velero Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package restore
import (
"testing"
"github.com/stretchr/testify/assert"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"github.com/vmware-tanzu/velero/pkg/builder"
"github.com/vmware-tanzu/velero/pkg/test"
)
func TestK8sPrioritySort(t *testing.T) {
tests := []struct {
name string
orig []metav1.GroupVersionForDiscovery
want []metav1.GroupVersionForDiscovery
}{
{
name: "sorts Kubernetes API group versions per k8s priority",
orig: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v11alpha2"},
{Version: "foo10"},
{Version: "v10"},
{Version: "v12alpha1"},
{Version: "v3beta1"},
{Version: "foo1"},
{Version: "v1"},
{Version: "v10beta3"},
{Version: "v11beta2"},
},
want: []metav1.GroupVersionForDiscovery{
{Version: "v10"},
{Version: "v2"},
{Version: "v1"},
{Version: "v11beta2"},
{Version: "v10beta3"},
{Version: "v3beta1"},
{Version: "v12alpha1"},
{Version: "v11alpha2"},
{Version: "foo1"},
{Version: "foo10"},
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
k8sPrioritySort(tc.orig)
assert.Equal(t, tc.want, tc.orig)
})
}
}
func TestUserResourceGroupVersionPriorities(t *testing.T) {
tests := []struct {
name string
cm *corev1.ConfigMap
want map[string]metav1.APIGroup
wantErrMsg string
}{
{
name: "retrieve version priority data from config map",
cm: builder.
ForConfigMap("velero", "enableapigroupversions").
Data(
"restoreResourcesVersionPriority",
`rockbands.music.example.io=v2beta1,v2beta2
orchestras.music.example.io=v2,v3alpha1
subscriptions.operators.coreos.com=v2,v1`,
).
Result(),
want: map[string]metav1.APIGroup{
"rockbands.music.example.io": {Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2beta1"},
{Version: "v2beta2"},
}},
"orchestras.music.example.io": {Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v3alpha1"},
}},
"subscriptions.operators.coreos.com": {Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v1"},
}},
},
},
{
name: "incorrect data format returns an error",
cm: builder.
ForConfigMap("velero", "enableapigroupversions").
Data(
"restoreResourcesVersionPriority",
`rockbands.music.example.io=v2beta1,v2beta2\n orchestras.music.example.io=v2,v3alpha1`,
).
Result(),
want: nil,
wantErrMsg: "parsing user priorities: validating user priority: line must have one and only one equal sign",
},
{
name: "spaces and empty lines are removed before storing user version priorities",
cm: builder.
ForConfigMap("velero", "enableapigroupversions").
Data(
"restoreResourcesVersionPriority",
` pods=v2,v1beta2
horizontalpodautoscalers.autoscaling = v2beta2
jobs.batch=v3
`,
).
Result(),
want: map[string]metav1.APIGroup{
"pods": {Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v1beta2"},
}},
"horizontalpodautoscalers.autoscaling": {Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2beta2"},
}},
"jobs.batch": {Versions: []metav1.GroupVersionForDiscovery{
{Version: "v3"},
}},
},
},
}
fakeCtx := &restoreContext{
log: test.NewLogger(),
}
for _, tc := range tests {
t.Log(tc.name)
priorities := userResourceGroupVersionPriorities(fakeCtx, tc.cm)
assert.Equal(t, tc.want, priorities)
}
}
func TestFindAPIGroup(t *testing.T) {
tests := []struct {
name string
targetGrps []metav1.APIGroup
grpName string
want metav1.APIGroup
}{
{
name: "return the API Group in target list matching group string",
targetGrps: []metav1.APIGroup{
{
Name: "rbac.authorization.k8s.io",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v2"},
},
{
Name: "",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v1"},
},
{
Name: "velero.io",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2beta1"},
{Version: "v2beta2"},
{Version: "v2"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v2"},
},
},
grpName: "velero.io",
want: metav1.APIGroup{
Name: "velero.io",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2beta1"},
{Version: "v2beta2"},
{Version: "v2"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v2"},
},
},
{
name: "return empty API Group if no match in target list",
targetGrps: []metav1.APIGroup{
{
Name: "rbac.authorization.k8s.io",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v2"},
},
{
Name: "",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v1"},
},
{
Name: "velero.io",
Versions: []metav1.GroupVersionForDiscovery{
{Version: "v2beta1"},
{Version: "v2beta2"},
{Version: "v2"},
},
PreferredVersion: metav1.GroupVersionForDiscovery{Version: "v2"},
},
},
grpName: "autoscaling",
want: metav1.APIGroup{},
},
}
for _, tc := range tests {
grp := findAPIGroup(tc.targetGrps, tc.grpName)
assert.Equal(t, tc.want, grp)
}
}
func TestFindSupportedUserVersion(t *testing.T) {
tests := []struct {
name string
userGVs []metav1.GroupVersionForDiscovery
targetGVs []metav1.GroupVersionForDiscovery
sourceGVs []metav1.GroupVersionForDiscovery
want string
}{
{
name: "return the single user group version that has a match in both source and target clusters",
userGVs: []metav1.GroupVersionForDiscovery{
{Version: "foo"},
{Version: "v10alpha2"},
{Version: "v3"},
},
targetGVs: []metav1.GroupVersionForDiscovery{
{Version: "v9"},
{Version: "v10beta1"},
{Version: "v10alpha2"},
{Version: "v10alpha3"},
},
sourceGVs: []metav1.GroupVersionForDiscovery{
{Version: "v10alpha2"},
{Version: "v9beta1"},
},
want: "v10alpha2",
},
{
name: "return the first user group version that has a match in both source and target clusters",
userGVs: []metav1.GroupVersionForDiscovery{
{Version: "v2beta1"},
{Version: "v2beta2"},
},
targetGVs: []metav1.GroupVersionForDiscovery{
{Version: "v2beta2"},
{Version: "v2beta1"},
},
sourceGVs: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
{Version: "v2beta2"},
{Version: "v2beta1"},
},
want: "v2beta1",
},
{
name: "return empty string if there's only matches in the source cluster, but not target",
userGVs: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
},
targetGVs: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
},
sourceGVs: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
},
want: "",
},
{
name: "return empty string if there's only matches in the target cluster, but not source",
userGVs: []metav1.GroupVersionForDiscovery{
{Version: "v3"},
{Version: "v1"},
},
targetGVs: []metav1.GroupVersionForDiscovery{
{Version: "v3"},
{Version: "v3beta2"},
},
sourceGVs: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v2beta1"},
},
want: "",
},
{
name: "return empty string if there is no match with either target and source clusters",
userGVs: []metav1.GroupVersionForDiscovery{
{Version: "v2beta2"},
{Version: "v2beta1"},
{Version: "v2beta3"},
},
targetGVs: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v1"},
{Version: "v2alpha1"},
},
sourceGVs: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
{Version: "v2alpha1"},
},
want: "",
},
}
for _, tc := range tests {
uv := findSupportedUserVersion(tc.userGVs, tc.targetGVs, tc.sourceGVs)
assert.Equal(t, tc.want, uv)
}
}
func TestVersionsContain(t *testing.T) {
tests := []struct {
name string
GVs []metav1.GroupVersionForDiscovery
ver string
want bool
}{
{
name: "version is not in list",
GVs: []metav1.GroupVersionForDiscovery{
{Version: "v1"},
{Version: "v2alpha1"},
{Version: "v2beta1"},
},
ver: "v2",
want: false,
},
{
name: "version is in list",
GVs: []metav1.GroupVersionForDiscovery{
{Version: "v2"},
{Version: "v2alpha1"},
{Version: "v2beta1"},
},
ver: "v2",
want: true,
},
}
for _, tc := range tests {
assert.Equal(t, tc.want, versionsContain(tc.GVs, tc.ver))
}
}

View File

@ -1,5 +1,5 @@
/*
Copyright 2020 the Velero contributors.
Copyright The Velero Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@ -22,6 +22,7 @@ import (
"fmt"
"io"
"io/ioutil"
"path/filepath"
"sort"
"strings"
"sync"
@ -49,6 +50,7 @@ import (
"github.com/vmware-tanzu/velero/pkg/archive"
"github.com/vmware-tanzu/velero/pkg/client"
"github.com/vmware-tanzu/velero/pkg/discovery"
"github.com/vmware-tanzu/velero/pkg/features"
listers "github.com/vmware-tanzu/velero/pkg/generated/listers/velero/v1"
"github.com/vmware-tanzu/velero/pkg/kuberesource"
"github.com/vmware-tanzu/velero/pkg/label"
@ -231,6 +233,7 @@ func (kr *kubernetesRestorer) Restore(
restore: req.Restore,
resourceIncludesExcludes: resourceIncludesExcludes,
namespaceIncludesExcludes: namespaceIncludesExcludes,
chosenGrpVersToRestore: make(map[string]ChosenGroupVersion),
selector: selector,
log: req.Log,
dynamicFactory: kr.dynamicFactory,
@ -308,6 +311,7 @@ type restoreContext struct {
restoreDir string
resourceIncludesExcludes *collections.IncludesExcludes
namespaceIncludesExcludes *collections.IncludesExcludes
chosenGrpVersToRestore map[string]ChosenGroupVersion
selector labels.Selector
log logrus.FieldLogger
dynamicFactory client.DynamicFactory
@ -384,6 +388,16 @@ func (ctx *restoreContext) execute() (Result, Result) {
return warnings, errs
}
// TODO: Remove outer feature flag check to make this feature a default in Velero.
if features.IsEnabled(velerov1api.APIGroupVersionsFeatureFlag) {
if ctx.backup.Status.FormatVersion >= "1.1.0" {
if err := ctx.chooseAPIVersionsToRestore(); err != nil {
errs.AddVeleroError(errors.Wrap(err, "choosing API version to restore"))
return warnings, errs
}
}
}
// Iterate through an ordered list of resources to restore, checking each one to see if it should be restored.
// Note that resources *may* be in this list twice, i.e. once due to being a prioritized resource, and once due
// to being in the backup tarball. We can't de-dupe this upfront, because it's possible that items in the prioritized
@ -732,6 +746,17 @@ func (ctx *restoreContext) restoreResource(resource, targetNamespace, originalNa
groupResource := schema.ParseGroupResource(resource)
// Modify `resource` so that it has the priority version to restore in its
// path. For example, for group resource "horizontalpodautoscalers.autoscaling",
// "/v2beta1" will be appended to the end. Different versions would only
// have been stored if the APIGroupVersionsFeatureFlag was enabled during backup.
// The chosenGrpVersToRestore map would only be populated if APIGroupVersionsFeatureFlag
// was enabled for restore and the minimum required backup format version has been met.
cgv, ok := ctx.chosenGrpVersToRestore[groupResource.String()]
if ok {
resource = filepath.Join(groupResource.String(), cgv.Dir)
}
for _, item := range items {
itemPath := archive.GetItemFilePath(ctx.restoreDir, resource, originalNamespace, item)

View File

@ -43,6 +43,8 @@ Features on the Velero server can be enabled using the `--features` flag to the
velero install --features=EnableCSI
```
Another example is enabling the support of multiple API group versions, as documented at [- -features=EnableAPIGroupVersions](enable-api-group-versions-feature.md).
Feature flags, passed to `velero install` will be passed to the Velero deployment and also to the `restic` daemon set, if `--use-restic` flag is used.
Similarly, features may be disabled by removing the corresponding feature flags from the `--features` flag.

View File

@ -0,0 +1,113 @@
---
title: "Enable API Group Versions Feature"
layout: docs
---
## Background
Velero serves to both restore and migrate Kubernetes applications. Typically, backup and restore does not involve upgrading Kubernetes API group versions. However, when migrating from a source cluster to a destination cluster, it is not unusual to see the API group versions differing between clusters.
> &#9432; **API Group Version** | Kubernetes applications are made up of various resources. Common resources are pods, jobs, and deployments. Custom resources are created via custom resource definitions (CRDs). Every resource, whether custom or not, is part of a group, and each group has a version called the API group version.
Kubernetes by default allows changing API group versions between clusters as long as the upgrade is a single version, for example, v1 -> v2beta1. Jumping multiple versions, for example, v1 -> v3, is not supported out of the box. This is where the Enable API Group Version feature comes in.
Currently, the Enable API Group Version feature is in beta and can be enabled by installing Velero with a [feature flag](https://velero.io/docs/v1.5/customize-installation/#enable-server-side-features), `--features=EnableAPIGroupVersions`.
## How the Enable API Group Versions Feature Works
When the Enable API Group Versions feature is enabled on the source cluster, Velero will not only back up Kubernetes preferred API group versions, but it will also back up all supported versions on the cluster. As an example, consider the resource `horizontalpodautoscalers` which falls under the `autoscaling` group. Without the feature flag enabled, only the preferred API group version for autoscaling, `v1` will be backed up. With the feature enabled, the remaining supported versions, `v2beta1` and `v2beta2` will also be backed up. Once the versions are stored in the backup tarball file, they will be available to be restored on the destination cluster.
When the Enable API Group Versions feature is enabled on the destination cluster, Velero restore will choose the version to restore based on an API group version priority order.
The version priorities are listed from highest to lowest priority below:
- Priority 1: destination cluster preferred version
- Priority 2: source cluster preferred version
- Priority 3: non-preferred common supported version with the highest [Kubernetes version priority](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-priority)
The highest priority (Priority 1) will be the destination cluster's preferred API group version. If the destination preferred version is found in the backup tarball, it will be the API group version chosen for restoration for that resource. However, if the destination preferred version is not found in the backup tarball, the next version in the list will be selected: the source cluster preferred version (Priority 2).
If the source cluster preferred version is found to be supported by the destination cluster, it will be chosen as the API group version to restore. However, if the source preferred version is not supported by the destination cluster, then the next version in the list will be considered: a non-preferred common supported version (Priority 3).
In the case that there are more than one non-preferred common supported version, which version will be chosen? The answer requires understanding the [Kubernetes version priority order](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-priority). Kubernetes prioritizes group versions by making the latest, most stable version the highest priority. The highest priority version is the Kubernetes preferred version. Here is a sorted version list example from the Kubernetes.io documentation:
- v10
- v2
- v1
- v11beta2
- v10beta3
- v3beta1
- v12alpha1
- v11alpha2
- foo1
- foo10
Of the non-preferred common versions, the version that has the highest Kubernetes version priority will be chosen. See the example for Priority 3 below.
To better understand which API group version will be chosen, the following provides some concrete examples. The examples use the term "target cluster" which is synonymous to "destination cluster".
![Priority 1 Case A example](/docs/main/img/gv_priority1-caseA.png)
![Priority 1 Case B example](/docs/main/img/gv_priority1-caseB.png)
![Priority 2 Case C example](/docs/main/img/gv_priority2-caseC.png)
![Priority 3 Case D example](/docs/main/img/gv_priority3-caseD.png)
## Procedure for Using the Enable API Group Versions Feature
1. [Install Velero](https://velero.io/docs/v1.5/basic-install/) on source cluster with the [feature flag enabled](https://velero.io/docs/v1.5/customize-installation/#enable-server-side-features). The flag is `--features=EnableAPIGroupVersions`. For the enable API group versions feature to work, the feature flag needs to be used for Velero installations on both the source and destination clusters.
2. Back up and restore following the [migration case instructions](https://velero.io/docs/v1.5/migration-case/). Note that "Cluster 1" in the instructions refers to the source cluster, and "Cluster 2" refers to the destination cluster.
## Advanced Procedure for Customizing the Version Prioritization
Optionally, users can create a config map to override the default API group prioritization for some or all of the resources being migrated. For each resource that is specified by the user, Velero will search for the version in both the backup tarball and the destination cluster. If there is a match, the user-specified API group version will be restored. If the backup tarball and the destination cluster does not have or support any of the user-specified versions, then the default version prioritization will be used.
Here are the steps for creating a config map that allows users to override the default version prioritization. These steps must happen on the destination cluster before a Velero restore is initiated.
1. Create a file called `restoreResourcesVersionPriority`. The file name will become a key in the `data` field of the config map.
- In the file, write a line for each resource group you'd like to override. Make sure each line follows the format `<resource>.<group>=<highest user priority version>,<next highest>`
- Note that the resource group and versions are separated by a single equal (=) sign. Each version is listed in order of user's priority separated by commas.
- Here is an example of the contents of a config map file:
```cm
rockbands.music.example.io=v2beta1,v2beta2
orchestras.music.example.io=v2,v3alpha1
subscriptions.operators.coreos.com=v2,v1
```
2. Apply config map with
```bash
kubectl create configmap enableapigroupversions --from-file=<absolute path>/restoreResourcesVersionPriority -n velero
```
3. See the config map with
```bash
kubectl describe configmap enableapigroupversions -n velero
```
The config map should look something like
```bash
Name: enableapigroupversions
Namespace: velero
Labels: <none>
Annotations: <none>
Data
====
restoreResourcesVersionPriority:
----
rockbands.music.example.io=v2beta1,v2beta2
orchestras.music.example.io=v2,v3alpha1
subscriptions.operators.coreos.com=v2,v1
Events: <none>
```
## Troubleshooting
1. Refer to the [troubleshooting section](https://velero.io/docs/v1.5/troubleshooting/) of the docs as the techniques generally apply here as well.
2. The [debug logs](https://velero.io/docs/v1.5/troubleshooting/#getting-velero-debug-logs) will contain information on which version was chosen to restore.
3. If no API group version could be found that both exists in the backup tarball file and is supported by the destination cluster, then the following error will be recorded (no need to activate debug level logging): `"error restoring rockbands.music.example.io/rockstars/beatles: the server could not find the requested resource"`.

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

View File

@ -3,7 +3,7 @@ title: "Cluster migration"
layout: docs
---
*Using Backups and Restores*
## Using Backups and Restores
Velero can help you port your resources from one cluster to another, as long as you point each Velero instance to the same cloud object storage location. This scenario assumes that your clusters are hosted by the same cloud provider. **Note that Velero does not natively support the migration of persistent volumes snapshots across cloud providers.** If you would like to migrate volume data between cloud platforms, please enable [restic][2], which will backup volume contents at the filesystem level.
@ -32,7 +32,7 @@ Velero can help you port your resources from one cluster to another, as long as
velero restore create --from-backup <BACKUP-NAME>
```
## Verify both clusters
## Verify Both Clusters
Check that the second cluster is behaving as expected:
@ -50,5 +50,9 @@ Check that the second cluster is behaving as expected:
If you encounter issues, make sure that Velero is running in the same namespace in both clusters.
## Migrating Workloads Across Different Kubernetes Versions
Migration across clusters that are not running the same version of Kubernetes might be possible, but some factors need to be considered: compatibility of API groups between clusters for each custom resource, and if a Kubernetes version upgrade breaks the compatibility of core/native API groups. For more information about API group versions, please see [EnableAPIGroupVersions](enable-api-group-versions-feature.md).
[1]: how-velero-works.md#set-a-backup-to-expire
[2]: restic.md

View File

@ -78,10 +78,9 @@ However, a major version release of Velero does not necessarily mean that the ba
### File Format Version: 1.1 (Current)
Version 1.1 added support of API groups versions as part of the backup (previously, only the preferred version of each API Groups was backed up). Each resource has one or more sub-directories, one sub-directory for each supported version of the API group. The preferred version API Group of each resource has the suffix "-preferredversion" as part of the sub-directory name. For backward compatibility, we kept the classic directory structure without the API Group version, which sits on the same level as the API Group sub-directory versions.
By default, only the preferred API group of each resource is backed up.
To take a backup of all API group versions, you need to run the Velero server with `--features=EnableAPIGroupVersions` feature flag. This is an experimental flag and the restore logic to handle multiple API Group Versions will be added in the future.
Version 1.1 added support of API groups versions as part of the backup. Previously, only the preferred version of each API groups was backed up. Each resource has one or more sub-directories: one sub-directory for each supported version of the API group. The preferred version API Group of each resource has the suffix "-preferredversion" as part of the sub-directory name. For backward compatibility, we kept the classic directory structure without the API group version, which sits on the same level as the API group sub-directory versions.
By default, only the preferred API group of each resource is backed up. To take a backup of all API group versions, you need to run the Velero server with the `--features=EnableAPIGroupVersions` feature flag. This is an experimental flag and the restore logic to handle multiple API group versions is documented at [EnableAPIGroupVersions](enable-api-group-versions-feature.md).
When unzipped, a typical backup directory (`backup1234.tar.gz`) taken with this file format version looks like the following (with the feature flag):

View File

@ -31,6 +31,8 @@ toc:
url: /disaster-case
- page: Cluster migration
url: /migration-case
- page: Enable API group versions
url: /enable-api-group-versions-feature
- page: Resource filtering
url: /resource-filtering
- page: Backup reference

View File

@ -67,6 +67,10 @@ For example, E2E tests can be run from Velero repository roots using the below c
BSL_CONFIG="resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID" BSL_BUCKET=velero CREDS_FILE=~/bin/velero-dev/aks-creds PLUGIN_PROVIDER=azure make test-e2e
```
Please refer to `velero-plugin-for-microsoft-azure` documentation for instruction to [set up permissions for Velero](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#set-permissions-for-velero) and to [set up azure storage account and blob container](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#setup-azure-storage-account-and-blob-container)
1. Run Ginko-focused Restore Multi-API Groups tests using an image built for PR #3133 and Minio as the backup storage location:
```bash
BSL_CONFIG="region=minio,s3ForcePathStyle=\"true\",s3Url=http://192.168.1.124:9000" BSL_PREFIX=veldat BSL_BUCKET=velero CREDS_FILE=~/go/src/github.com/vmware-tanzu/velero/frankie-secrets/credentials-minio PLUGIN_PROVIDER=aws VELERO_IMAGE=projects.registry.vmware.com/tanzu_migrator/velero-pr3133:0.0.5 GINKGO_FOCUS="API group versions" make test-e2e
```
## Filtering tests

View File

@ -0,0 +1,580 @@
package e2e
import (
"context"
"encoding/json"
"fmt"
"os/exec"
"regexp"
"strconv"
"strings"
"time"
"github.com/google/uuid"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"github.com/pkg/errors"
corev1api "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/kubernetes"
"github.com/vmware-tanzu/velero/pkg/builder"
veleroexec "github.com/vmware-tanzu/velero/pkg/util/exec"
)
var _ = Describe("[KinD] Velero tests on KinD clusters with various CRD API group versions", func() {
var (
resource, group string
certMgrCRD map[string]string
client *kubernetes.Clientset
err error
ctx = context.Background()
)
BeforeEach(func() {
resource = "rockbands"
group = "music.example.io"
certMgrCRD = map[string]string{
"url": "testdata/enable_api_group_versions/cert-manager.yaml",
"namespace": "cert-manager",
}
client, err = GetClusterClient()
Expect(err).NotTo(HaveOccurred())
err = InstallCRD(ctx, certMgrCRD["url"], certMgrCRD["namespace"])
Expect(err).NotTo(HaveOccurred())
uuidgen, err = uuid.NewRandom()
Expect(err).NotTo(HaveOccurred())
})
AfterEach(func() {
cmd := exec.CommandContext(ctx, "kubectl", "delete", "namespace", "music-system")
_, _, _ = veleroexec.RunCommand(cmd)
cmd = exec.CommandContext(ctx, "kubectl", "delete", "crd", "rockbands.music.example.io")
_, _, _ = veleroexec.RunCommand(cmd)
_ = DeleteCRD(ctx, certMgrCRD["url"], certMgrCRD["namespace"])
// Uninstall Velero.
if client != nil {
_ = client.CoreV1().Namespaces().Delete(
context.Background(),
"velero",
metav1.DeleteOptions{},
)
}
})
Context("When EnableAPIGroupVersions flag is set", func() {
It("Should back up API group version and restore by version priority", func() {
Expect(RunEnableAPIGroupVersionsTests(
ctx,
resource,
group,
client,
)).To(Succeed(), "Failed to successfully backup and restore multiple API Groups")
})
})
})
func RunEnableAPIGroupVersionsTests(ctx context.Context, resource, group string, client *kubernetes.Clientset) error {
tests := []struct {
name string
namespaces []string
srcCRD map[string]string
srcCRs map[string]string
tgtCRD map[string]string
tgtVer string
cm *corev1api.ConfigMap
gvs map[string][]string
want map[string]map[string]string
}{
{
name: "Target and source cluster preferred versions match; Preferred version v1 is restored (Priority 1, Case A).",
srcCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-a-source.yaml",
"namespace": "music-system",
},
srcCRs: map[string]string{
"v1": "testdata/enable_api_group_versions/music_v1_rockband.yaml",
"v1alpha1": "testdata/enable_api_group_versions/music_v1alpha1_rockband.yaml",
},
tgtCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-a-target.yaml",
"namespace": "music-system",
},
tgtVer: "v1",
cm: nil,
want: map[string]map[string]string{
"annotations": {
"rockbands.music.example.io/originalVersion": "v1",
},
"specs": {
"leadSinger": "John Lennon",
},
},
},
{
name: "Latest common non-preferred supported version v2beta2 is restored (Priority 3, Case D).",
srcCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-b-source-manually-added-mutations.yaml",
"namespace": "music-system",
},
srcCRs: map[string]string{
"v2beta2": "testdata/enable_api_group_versions/music_v2beta2_rockband.yaml",
"v2beta1": "testdata/enable_api_group_versions/music_v2beta1_rockband.yaml",
"v1": "testdata/enable_api_group_versions/music_v1_rockband.yaml",
},
tgtCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-d-target-manually-added-mutations.yaml",
"namespace": "music-system",
},
tgtVer: "v2beta2",
cm: nil,
want: map[string]map[string]string{
"annotations": {
"rockbands.music.example.io/originalVersion": "v2beta2",
},
"specs": {
"leadSinger": "John Lennon",
"leadGuitar": "George Harrison",
"drummer": "Ringo Starr",
},
},
},
{
name: "No common supported versions means no rockbands custom resource is restored.",
srcCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-a-source.yaml",
"namespace": "music-system",
},
srcCRs: map[string]string{
"v1": "testdata/enable_api_group_versions/music_v1_rockband.yaml",
"v1alpha1": "testdata/enable_api_group_versions/music_v1alpha1_rockband.yaml",
},
tgtCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-b-target-manually-added-mutations.yaml",
"namespace": "music-system",
},
tgtVer: "",
cm: nil,
want: nil,
},
{
name: "User config map overrides Priority 3, Case D and restores v2beta1",
srcCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-b-source-manually-added-mutations.yaml",
"namespace": "music-system",
},
srcCRs: map[string]string{
"v2beta2": "testdata/enable_api_group_versions/music_v2beta2_rockband.yaml",
"v2beta1": "testdata/enable_api_group_versions/music_v2beta1_rockband.yaml",
"v1": "testdata/enable_api_group_versions/music_v1_rockband.yaml",
},
tgtCRD: map[string]string{
"url": "testdata/enable_api_group_versions/case-d-target-manually-added-mutations.yaml",
"namespace": "music-system",
},
tgtVer: "v2beta1",
cm: builder.ForConfigMap("velero", "enableapigroupversions").Data(
"restoreResourcesVersionPriority",
`rockbands.music.example.io=v2beta1,v2beta2,v2`,
).Result(),
want: map[string]map[string]string{
"annotations": {
"rockbands.music.example.io/originalVersion": "v2beta1",
},
"specs": {
"leadSinger": "John Lennon",
"leadGuitar": "George Harrison",
"genre": "60s rock",
},
},
},
}
for i, tc := range tests {
fmt.Printf("\n====== Test Case %d ======\n", i)
err := InstallCRD(ctx, tc.srcCRD["url"], tc.srcCRD["namespace"])
if err != nil {
return errors.Wrap(err, "installing music-system CRD for source cluster")
}
for version, cr := range tc.srcCRs {
ns := resource + "-src-" + version
if err := CreateNamespace(ctx, client, ns); err != nil {
return errors.Wrapf(err, "creating %s namespace", ns)
}
if err := InstallCR(ctx, cr, ns); err != nil {
return errors.Wrapf(err, "installing %s custom resource on source cluster namespace %s", cr, ns)
}
tc.namespaces = append(tc.namespaces, ns)
}
if err := installVeleroForAPIGroups(ctx); err != nil {
return errors.Wrap(err, "install velero")
}
fmt.Println("Sleep 20s to wait for Velero to stabilize after install.")
time.Sleep(time.Second * 20)
backup := "backup-rockbands-" + uuidgen.String() + "-" + strconv.Itoa(i)
namespacesStr := strings.Join(tc.namespaces, ",")
err = VeleroBackupNamespace(ctx, veleroCLI, backup, namespacesStr)
if err != nil {
return errors.Wrapf(err, "backing up %s namespaces on source cluster", namespacesStr)
}
// Delete music-system CRD and controllers installed on source cluster.
if err := DeleteCRD(ctx, tc.srcCRD["url"], tc.srcCRD["namespace"]); err != nil {
return errors.Wrapf(err, "deleting music-system CRD from source cluster")
}
for _, ns := range tc.namespaces {
if err := client.CoreV1().Namespaces().Delete(ctx, ns, metav1.DeleteOptions{}); err != nil {
return errors.Wrapf(err, "deleting %s namespace from source cluster", ns)
}
if err := WaitNamespaceDelete(ctx, ns); err != nil {
return errors.Wrapf(err, "deleting %s namespace from source cluster", ns)
}
}
// Install music-system CRD for target cluster.
if err := InstallCRD(ctx, tc.tgtCRD["url"], tc.tgtCRD["namespace"]); err != nil {
return errors.Wrapf(err, "installing music-system CRD for target cluster")
}
// Apply config map if there is one.
if tc.cm != nil {
_, err := client.CoreV1().ConfigMaps("velero").Create(ctx, tc.cm, metav1.CreateOptions{})
if err != nil {
return errors.Wrap(err, "creating config map with user version priorities")
}
}
// Reset Velero to recognize music-system CRD.
if err := RestartPods(ctx, "velero"); err != nil {
return errors.Wrapf(err, "restarting Velero pods")
}
fmt.Println("Sleep 20s to wait for Velero to stabilize after restart.")
time.Sleep(time.Second * 20)
// Restore rockbands namespace.
restore := "restore-rockbands-" + uuidgen.String() + "-" + strconv.Itoa(i)
if tc.want != nil {
if err := VeleroRestore(ctx, veleroCLI, restore, backup); err != nil {
return errors.Wrapf(err, "restoring %s namespaces on target cluster", namespacesStr)
}
annoSpec, err := resourceInfo(ctx, group, tc.tgtVer, resource)
if err != nil {
return errors.Wrapf(
err,
"get annotation and spec from %s.%s/%s object",
resource,
group,
tc.tgtVer,
)
}
// Assertion
if containsAll(annoSpec["annotations"], tc.want["annotations"]) != true {
msg := fmt.Sprintf(
"actual annotations: %v, expected annotations: %v",
annoSpec["annotations"],
tc.want["annotations"],
)
return errors.New(msg)
}
// Assertion
if containsAll(annoSpec["specs"], tc.want["specs"]) != true {
msg := fmt.Sprintf(
"actual specs: %v, expected specs: %v",
annoSpec["specs"],
tc.want["specs"],
)
return errors.New(msg)
}
} else {
// No custom resource should have been restored. Expect "no resource found"
// error during restore.
err := VeleroRestore(ctx, veleroCLI, restore, backup)
if err.Error() != "Unexpected restore phase got PartiallyFailed, expecting Completed" {
return errors.New("expected error but not none")
}
}
// Delete namespaces created for CRs
for _, ns := range tc.namespaces {
fmt.Println("Delete namespace", ns)
_ = client.CoreV1().Namespaces().Delete(ctx, ns, metav1.DeleteOptions{})
_ = WaitNamespaceDelete(ctx, ns)
}
// Delete source cluster music-system CRD
_ = DeleteCRD(
ctx,
tc.srcCRD["url"],
tc.srcCRD["namespace"],
)
// Delete target cluster music-system CRD
_ = DeleteCRD(
ctx,
tc.tgtCRD["url"],
tc.srcCRD["namespace"],
)
// Delete Velero namespace
_ = client.CoreV1().Namespaces().Delete(ctx, "velero", metav1.DeleteOptions{})
_ = WaitNamespaceDelete(ctx, "velero")
}
return nil
}
func installVeleroForAPIGroups(ctx context.Context) error {
if err := EnsureClusterExists(ctx); err != nil {
return errors.Wrap(err, "check cluster exists")
}
// Pass global variables to option parameters.
options, err := GetProviderVeleroInstallOptions(
pluginProvider,
cloudCredentialsFile,
bslBucket,
bslPrefix,
bslConfig,
vslConfig,
getProviderPlugins(pluginProvider),
)
if err != nil {
return errors.Wrap(err, "get velero install options")
}
options.UseRestic = false
options.Features = "EnableAPIGroupVersions"
options.Image = veleroImage
if err := InstallVeleroServer(options); err != nil {
return errors.Wrap(err, "install velero server")
}
return nil
}
func InstallCRD(ctx context.Context, crdFile, ns string) error {
fmt.Printf("Install CRD %s.\n", crdFile)
cmd := exec.CommandContext(ctx, "kubectl", "apply", "-f", crdFile)
_, stderr, err := veleroexec.RunCommand(cmd)
if err != nil {
return errors.Wrap(err, stderr)
}
fmt.Println("Wait for CRD to be ready.")
if err := WaitForPodContainers(ctx, ns); err != nil {
return err
}
return err
}
// WaitForPodContainers will get the pods and container status in a namespace.
// If the ratio of the number of containers running to total in a pod is not 1,
// it is not ready. Otherwise, if all container ratios are 1, the pod is running.
func WaitForPodContainers(ctx context.Context, ns string) error {
err := wait.Poll(3*time.Second, 4*time.Minute, func() (bool, error) {
cmd := exec.CommandContext(ctx, "kubectl", "get", "pods", "-n", ns)
stdout, stderr, err := veleroexec.RunCommand(cmd)
if err != nil {
return false, errors.Wrap(err, stderr)
}
re := regexp.MustCompile(`(\d)/(\d)\s+Running`)
// Default allRunning needs to be false for when no match is found.
var allRunning bool
for i, v := range re.FindAllStringSubmatch(stdout, -1) {
if i == 0 {
allRunning = true
}
allRunning = v[1] == v[2] && allRunning
}
return allRunning, nil
})
if err == nil {
fmt.Println("Sleep for 20s for cluster to stabilize.")
time.Sleep(time.Second * 20)
}
return err
}
func DeleteCRD(ctx context.Context, crdFile, ns string) error {
fmt.Println("Delete CRD", crdFile)
cmd := exec.CommandContext(ctx, "kubectl", "delete", "-f", crdFile)
_, stderr, err := veleroexec.RunCommand(cmd)
if strings.Contains(stderr, "not found") {
return nil
}
if err != nil {
return errors.Wrap(err, stderr)
}
err = wait.Poll(1*time.Second, 3*time.Minute, func() (bool, error) {
cmd := exec.CommandContext(ctx, "kubectl", "get", "namespace", ns)
stdout, stderr, err := veleroexec.RunCommand(cmd)
if strings.Contains(stderr, "not found") {
return true, nil
}
if err != nil {
return false, errors.Wrap(err, stderr)
}
re := regexp.MustCompile(ns)
return re.MatchString(stdout), nil
})
return err
}
func RestartPods(ctx context.Context, ns string) error {
fmt.Printf("Restart pods in %s namespace.\n", ns)
cmd := exec.CommandContext(ctx, "kubectl", "delete", "pod", "--all", "-n", ns)
_, _, err := veleroexec.RunCommand(cmd)
if err == nil {
fmt.Println("Wait for pods to be ready.")
if err := WaitForPodContainers(ctx, ns); err != nil {
return err
}
}
return err
}
func InstallCR(ctx context.Context, crFile, ns string) error {
retries := 5
var stderr string
var err error
for i := 0; i < retries; i++ {
fmt.Printf("Attempt %d: Install custom resource %s\n", i+1, crFile)
cmd := exec.CommandContext(ctx, "kubectl", "apply", "-n", ns, "-f", crFile)
_, stderr, err = veleroexec.RunCommand(cmd)
if err == nil {
fmt.Printf("Successfully installed CR on %s.\n", ns)
return nil
}
fmt.Printf("Sleep for %ds before next attempt.\n", 20*i)
time.Sleep(time.Second * time.Duration(i) * 20)
}
return errors.Wrap(err, stderr)
}
func WaitNamespaceDelete(ctx context.Context, ns string) error {
err := wait.Poll(1*time.Second, 3*time.Minute, func() (bool, error) {
cmd := exec.CommandContext(ctx, "kubectl", "get", "namespace", ns)
stdout, stderr, err := veleroexec.RunCommand(cmd)
if err != nil {
return false, errors.Wrap(err, stderr)
}
re := regexp.MustCompile(ns)
return re.MatchString(stdout), nil
})
return err
}
func resourceInfo(ctx context.Context, g, v, r string) (map[string]map[string]string, error) {
rvg := r + "." + v + "." + g
ns := r + "-src-" + v
cmd := exec.CommandContext(ctx, "kubectl", "get", rvg, "-n", ns, "-o", "json")
stdout, errMsg, err := veleroexec.RunCommand(cmd)
if err != nil {
return nil, errors.Wrap(err, errMsg)
}
var info map[string]interface{}
if err := json.Unmarshal([]byte(stdout), &info); err != nil {
return nil, errors.Wrap(err, "unmarshal resource info JSON")
}
items := info["items"].([]interface{})
if len(items) < 1 {
return nil, errors.New("resource info is empty")
}
item := items[0].(map[string]interface{})
metadata := item["metadata"].(map[string]interface{})
annotations := metadata["annotations"].(map[string]interface{})
specs := item["spec"].(map[string]interface{})
annoSpec := make(map[string]map[string]string)
for k, v := range annotations {
if annoSpec["annotations"] == nil {
annoSpec["annotations"] = map[string]string{
k: v.(string),
}
} else {
annoSpec["annotations"][k] = v.(string)
}
}
for k, v := range specs {
if val, ok := v.(string); ok {
if annoSpec["specs"] == nil {
annoSpec["specs"] = map[string]string{
k: val,
}
} else {
annoSpec["specs"][k] = val
}
}
}
return annoSpec, nil
}
// containsAll returns true if all the map values in the needles argument
// are found in the haystack argument values.
func containsAll(haystack, needles map[string]string) bool {
for nkey, nval := range needles {
hval, ok := haystack[nkey]
if !ok {
return false
}
if hval != nval {
return false
}
}
return true
}

View File

@ -0,0 +1,62 @@
# Enable API Group Versions Test Data
This directory contains Kubernetes manifests that are used for the enable API group versions e2e tests.
## Documentation
Read more about cert-manager in the [Jet Stack repo](https://github.com/jetstack/cert-manager/blob/master/README.md).
Read more about the music-system custom resource definitions and rockbands custom resources created for Velero tests at [@brito-rafa's repo](https://github.com/brito-rafa/k8s-webhooks/blob/master/examples-for-projectvelero/README.md).
## Reference
These manifests, listed below, come from two different sources: github.com/jetstack/cert-manager and github.com/brito-rafa/k8s-webhooks:
cert-manager.yaml
- source: https://github.com/jetstack/cert-manager/releases/download/v1.0.3/cert-manager.yaml
- license: https://github.com/jetstack/cert-manager/blob/master/LICENSE
case-a-source.yaml
- source: https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-a/source/case-a-source.yaml
case-a-target.yaml
- source: https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-a/target/case-a-target.yaml
case-b-source-manually-added-mutations.yaml
- source: https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-b/source/case-b-source-manually-added-mutations.yaml
case-b-target-manually-added-mutations.yaml
- source: https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-b/target/case-b-target-manually-added-mutations.yaml
case-c-target-manually-added-mutations.yaml
- source: https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-a/source/case-a-source.yaml
case-c-target-manually-added-mutations.yaml
- source: https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-c/target/case-c-target-manually-added-mutations.yaml
music_v1_rockband.yaml
- source: https://github.com/brito-rafa/k8s-webhooks/blob/master/examples-for-projectvelero/case-a/source/music/config/samples/music_v1_rockband.yaml
music_v1alpha1_rockband.yaml
- source: https://github.com/brito-rafa/k8s-webhooks/blob/master/examples-for-projectvelero/case-a/source/music/config/samples/music_v1alpha1_rockband.yaml
music_v2_rockband.yaml
- source: https://github.com/brito-rafa/k8s-webhooks/blob/master/examples-for-projectvelero/case-c/target/music/config/samples/music_v2_rockband.yaml
music_v2beta1_rockband.yaml
- source: https://github.com/brito-rafa/k8s-webhooks/blob/master/examples-for-projectvelero/case-b/source/music/config/samples/music_v2beta1_rockband.yaml
music_v2beta2_rockband.yaml
- source: https://github.com/brito-rafa/k8s-webhooks/blob/master/examples-for-projectvelero/case-b/source/music/config/samples/music_v2beta2_rockband.yaml

View File

@ -0,0 +1,395 @@
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: music-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
controller-gen.kubebuilder.io/version: v0.2.5
name: rockbands.music.example.io
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /convert
conversionReviewVersions:
- v1
- v1alpha1
group: music.example.io
names:
kind: RockBand
listKind: RockBandList
plural: rockbands
singular: rockband
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
- name: v1alpha1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: music-leader-election-role
namespace: music-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps/status
verbs:
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: music-manager-role
rules:
- apiGroups:
- music.example.io
resources:
- rockbands
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- music.example.io
resources:
- rockbands/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: music-proxy-role
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: music-metrics-reader
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: music-leader-election-rolebinding
namespace: music-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: music-leader-election-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-proxy-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-proxy-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: v1
kind: Service
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager-metrics-service
namespace: music-system
spec:
ports:
- name: https
port: 8443
targetPort: https
selector:
control-plane: controller-manager
---
apiVersion: v1
kind: Service
metadata:
name: music-webhook-service
namespace: music-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
control-plane: controller-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager
namespace: music-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: controller-manager
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=10
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
command:
- /manager
image: quay.io/brito_rafa/music-controller:case-a-source-v0.1
name: manager
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: music-serving-cert
namespace: music-system
spec:
dnsNames:
- music-webhook-service.music-system.svc
- music-webhook-service.music-system.svc.cluster.local
issuerRef:
kind: Issuer
name: music-selfsigned-issuer
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: music-selfsigned-issuer
namespace: music-system
spec:
selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-mutating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- rockbands
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-validating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /validate-music-example-io-v1-rockband
failurePolicy: Fail
name: vrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- rockbands

View File

@ -0,0 +1,399 @@
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: music-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
controller-gen.kubebuilder.io/version: v0.2.5
name: rockbands.music.example.io
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /convert
conversionReviewVersions:
- v1
- v1alpha1
group: music.example.io
names:
kind: RockBand
listKind: RockBandList
plural: rockbands
singular: rockband
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
- name: v2beta1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: music-leader-election-role
namespace: music-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps/status
verbs:
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: music-manager-role
rules:
- apiGroups:
- music.example.io
resources:
- rockbands
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- music.example.io
resources:
- rockbands/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: music-proxy-role
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: music-metrics-reader
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: music-leader-election-rolebinding
namespace: music-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: music-leader-election-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-proxy-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-proxy-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: v1
kind: Service
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager-metrics-service
namespace: music-system
spec:
ports:
- name: https
port: 8443
targetPort: https
selector:
control-plane: controller-manager
---
apiVersion: v1
kind: Service
metadata:
name: music-webhook-service
namespace: music-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
control-plane: controller-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager
namespace: music-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: controller-manager
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=10
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
command:
- /manager
image: quay.io/brito_rafa/music-controller:case-a-target-v0.2
name: manager
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: music-serving-cert
namespace: music-system
spec:
dnsNames:
- music-webhook-service.music-system.svc
- music-webhook-service.music-system.svc.cluster.local
issuerRef:
kind: Issuer
name: music-selfsigned-issuer
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: music-selfsigned-issuer
namespace: music-system
spec:
selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-mutating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta1
operations:
- CREATE
- UPDATE
resources:
- rockbands
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-validating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /validate-music-example-io-v1-rockband
failurePolicy: Fail
name: vrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- rockbands

View File

@ -0,0 +1,475 @@
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: music-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
controller-gen.kubebuilder.io/version: v0.2.5
name: rockbands.music.example.io
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /convert
conversionReviewVersions:
- v1
- v2beta1
- v2beta2
group: music.example.io
names:
kind: RockBand
listKind: RockBandList
plural: rockbands
singular: rockband
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
- name: v2beta1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
- name: v2beta2
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
drummer:
type: string
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: music-leader-election-role
namespace: music-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps/status
verbs:
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: music-manager-role
rules:
- apiGroups:
- music.example.io
resources:
- rockbands
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- music.example.io
resources:
- rockbands/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: music-proxy-role
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: music-metrics-reader
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: music-leader-election-rolebinding
namespace: music-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: music-leader-election-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-proxy-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-proxy-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: v1
kind: Service
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager-metrics-service
namespace: music-system
spec:
ports:
- name: https
port: 8443
targetPort: https
selector:
control-plane: controller-manager
---
apiVersion: v1
kind: Service
metadata:
name: music-webhook-service
namespace: music-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
control-plane: controller-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager
namespace: music-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: controller-manager
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=10
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
command:
- /manager
image: quay.io/brito_rafa/music-controller:case-b-source-v0.1
name: manager
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: music-serving-cert
namespace: music-system
spec:
dnsNames:
- music-webhook-service.music-system.svc
- music-webhook-service.music-system.svc.cluster.local
issuerRef:
kind: Issuer
name: music-selfsigned-issuer
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: music-selfsigned-issuer
namespace: music-system
spec:
selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-mutating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta2-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta2
operations:
- CREATE
- UPDATE
resources:
- rockbands
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta1
operations:
- CREATE
- UPDATE
resources:
- rockbands
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- rockbands
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-validating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /validate-music-example-io-v2beta2-rockband
failurePolicy: Fail
name: vrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta2
operations:
- CREATE
- UPDATE
resources:
- rockbands

View File

@ -0,0 +1,422 @@
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: music-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
controller-gen.kubebuilder.io/version: v0.2.5
name: rockbands.music.example.io
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /convert
conversionReviewVersions:
- v2beta2
- v2beta1
- v1
group: music.example.io
names:
kind: RockBand
listKind: RockBandList
plural: rockbands
singular: rockband
scope: Namespaced
versions:
- name: v2beta1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
- name: v2beta2
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
drummer:
type: string
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: music-leader-election-role
namespace: music-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps/status
verbs:
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: music-manager-role
rules:
- apiGroups:
- music.example.io
resources:
- rockbands
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- music.example.io
resources:
- rockbands/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: music-proxy-role
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: music-metrics-reader
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: music-leader-election-rolebinding
namespace: music-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: music-leader-election-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-proxy-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-proxy-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: v1
kind: Service
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager-metrics-service
namespace: music-system
spec:
ports:
- name: https
port: 8443
targetPort: https
selector:
control-plane: controller-manager
---
apiVersion: v1
kind: Service
metadata:
name: music-webhook-service
namespace: music-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
control-plane: controller-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager
namespace: music-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: controller-manager
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=10
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
command:
- /manager
image: quay.io/brito_rafa/music-controller:case-b-target-v0.1
name: manager
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: music-serving-cert
namespace: music-system
spec:
dnsNames:
- music-webhook-service.music-system.svc
- music-webhook-service.music-system.svc.cluster.local
issuerRef:
kind: Issuer
name: music-selfsigned-issuer
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: music-selfsigned-issuer
namespace: music-system
spec:
selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-mutating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta1
operations:
- CREATE
- UPDATE
resources:
- rockbands
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta2-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta2
operations:
- CREATE
- UPDATE
resources:
- rockbands
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-validating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /validate-music-example-io-v2beta2-rockband
failurePolicy: Fail
name: vrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta2
operations:
- CREATE
- UPDATE
resources:
- rockbands

View File

@ -0,0 +1,421 @@
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: music-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
controller-gen.kubebuilder.io/version: v0.2.5
name: rockbands.music.example.io
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /convert
conversionReviewVersions:
- v2
- v1
group: music.example.io
names:
kind: RockBand
listKind: RockBandList
plural: rockbands
singular: rockband
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
- name: v2
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
bass:
type: string
drummer:
type: string
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: music-leader-election-role
namespace: music-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps/status
verbs:
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: music-manager-role
rules:
- apiGroups:
- music.example.io
resources:
- rockbands
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- music.example.io
resources:
- rockbands/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: music-proxy-role
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: music-metrics-reader
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: music-leader-election-rolebinding
namespace: music-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: music-leader-election-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-proxy-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-proxy-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: v1
kind: Service
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager-metrics-service
namespace: music-system
spec:
ports:
- name: https
port: 8443
targetPort: https
selector:
control-plane: controller-manager
---
apiVersion: v1
kind: Service
metadata:
name: music-webhook-service
namespace: music-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
control-plane: controller-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager
namespace: music-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: controller-manager
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=10
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
command:
- /manager
image: quay.io/brito_rafa/music-controller:case-c-target-v0.1
name: manager
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: music-serving-cert
namespace: music-system
spec:
dnsNames:
- music-webhook-service.music-system.svc
- music-webhook-service.music-system.svc.cluster.local
issuerRef:
kind: Issuer
name: music-selfsigned-issuer
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: music-selfsigned-issuer
namespace: music-system
spec:
selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-mutating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- rockbands
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2
operations:
- CREATE
- UPDATE
resources:
- rockbands
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-validating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /validate-music-example-io-v1-rockband
failurePolicy: Fail
name: vrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- rockbands

View File

@ -0,0 +1,482 @@
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: music-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
controller-gen.kubebuilder.io/version: v0.2.5
name: rockbands.music.example.io
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /convert
conversionReviewVersions:
- v2
- v2beta2
- v2beta1
- v1
group: music.example.io
names:
kind: RockBand
listKind: RockBandList
plural: rockbands
singular: rockband
scope: Namespaced
versions:
- name: v2
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
bass:
type: string
drummer:
type: string
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
- name: v2beta1
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
- name: v2beta2
schema:
openAPIV3Schema:
description: RockBand is the Schema for the rockbands API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RockBandSpec defines the desired state of RockBand
properties:
drummer:
type: string
genre:
type: string
leadGuitar:
type: string
leadSinger:
type: string
numberComponents:
format: int32
type: integer
type: object
status:
description: RockBandStatus defines the observed state of RockBand
properties:
lastPlayed:
type: string
required:
- lastPlayed
type: object
type: object
served: true
storage: false
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: music-leader-election-role
namespace: music-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps/status
verbs:
- get
- update
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: music-manager-role
rules:
- apiGroups:
- music.example.io
resources:
- rockbands
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- music.example.io
resources:
- rockbands/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: music-proxy-role
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: music-metrics-reader
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: music-leader-election-rolebinding
namespace: music-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: music-leader-election-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: music-proxy-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: music-proxy-role
subjects:
- kind: ServiceAccount
name: default
namespace: music-system
---
apiVersion: v1
kind: Service
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager-metrics-service
namespace: music-system
spec:
ports:
- name: https
port: 8443
targetPort: https
selector:
control-plane: controller-manager
---
apiVersion: v1
kind: Service
metadata:
name: music-webhook-service
namespace: music-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
control-plane: controller-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
name: music-controller-manager
namespace: music-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: controller-manager
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=10
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
command:
- /manager
image: quay.io/brito_rafa/music-controller:case-d-target-v0.1
name: manager
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: music-serving-cert
namespace: music-system
spec:
dnsNames:
- music-webhook-service.music-system.svc
- music-webhook-service.music-system.svc.cluster.local
issuerRef:
kind: Issuer
name: music-selfsigned-issuer
secretName: webhook-server-cert
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: music-selfsigned-issuer
namespace: music-system
spec:
selfSigned: {}
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-mutating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta2-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta2
operations:
- CREATE
- UPDATE
resources:
- rockbands
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2beta1-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta1
operations:
- CREATE
- UPDATE
resources:
- rockbands
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /mutate-music-example-io-v2-rockband
failurePolicy: Fail
name: mrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2
operations:
- CREATE
- UPDATE
resources:
- rockbands
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from: music-system/music-serving-cert
name: music-validating-webhook-configuration
webhooks:
- clientConfig:
caBundle: Cg==
service:
name: music-webhook-service
namespace: music-system
path: /validate-music-example-io-v2beta2-rockband
failurePolicy: Fail
name: vrockband.kb.io
rules:
- apiGroups:
- music.example.io
apiVersions:
- v2beta2
operations:
- CREATE
- UPDATE
resources:
- rockbands

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,11 @@
apiVersion: music.example.io/v1
kind: RockBand
metadata:
name: beatles
annotations:
rockbands.music.example.io/originalVersion: v1
spec:
# Add fields here
genre: '60s rock'
numberComponents: 4
leadSinger: John

View File

@ -0,0 +1,11 @@
apiVersion: music.example.io/v1alpha1
kind: RockBand
metadata:
name: beatles
annotations:
rockbands.music.example.io/originalVersion: v1alpha1
spec:
# Add fields here
genre: '60s rock'
numberComponents: 4

View File

@ -0,0 +1,12 @@
apiVersion: music.example.io/v2
kind: RockBand
metadata:
name: beatles
spec:
# Add fields here
genre: '60s rock'
numberComponents: 4
leadSinger: John
leadGuitar: George
drummer: Ringo
bass: Paul

View File

@ -0,0 +1,13 @@
apiVersion: music.example.io/v2beta1
kind: RockBand
metadata:
name: beatles
annotations:
rockbands.music.example.io/originalVersion: v2beta1
spec:
# Add fields here
genre: '60s rock'
numberComponents: 4
leadSinger: John
leadGuitar: George

View File

@ -0,0 +1,13 @@
apiVersion: music.example.io/v2beta2
kind: RockBand
metadata:
name: beatles
annotations:
rockbands.music.example.io/originalVersion: v2beta2
spec:
# Add fields here
genre: '60s rock'
numberComponents: 4
leadSinger: John
leadGuitar: George
drummer: Ringo

View File

@ -33,8 +33,13 @@ func getProviderPlugins(providerName string) []string {
}
// GetProviderVeleroInstallOptions returns Velero InstallOptions for the provider.
func GetProviderVeleroInstallOptions(providerName, credentialsFile, objectStoreBucket, objectStorePrefix string,
bslConfig, vslConfig string,
func GetProviderVeleroInstallOptions(
providerName,
credentialsFile,
objectStoreBucket,
objectStorePrefix string,
bslConfig,
vslConfig string,
plugins []string,
) (*cliinstall.InstallOptions, error) {
@ -100,10 +105,11 @@ func InstallVeleroServer(io *cliinstall.InstallOptions) error {
return errors.Wrap(err, errorMsg)
}
// restic enabled by default
fmt.Println("Waiting for Velero restic daemonset to be ready.")
if _, err = install.DaemonSetIsReady(factory, "velero"); err != nil {
return errors.Wrap(err, errorMsg)
if io.UseRestic {
fmt.Println("Waiting for Velero restic daemonset to be ready.")
if _, err = install.DaemonSetIsReady(factory, "velero"); err != nil {
return errors.Wrap(err, errorMsg)
}
}
return nil
@ -195,7 +201,6 @@ func CheckRestorePhase(ctx context.Context, veleroCLI string, restoreName string
func VeleroBackupNamespace(ctx context.Context, veleroCLI string, backupName string, namespace string) error {
backupCmd := exec.CommandContext(ctx, veleroCLI, "create", "backup", backupName, "--include-namespaces", namespace,
"--default-volumes-to-restic", "--wait")
fmt.Printf("backup cmd =%v\n", backupCmd)
err := backupCmd.Run()
if err != nil {
return err